On Friday 09 November 2018 20:46:03 Gene Heskett wrote:
> On Friday 09 November 2018 18:00:41 Nathan Stratton Treadway wrote:
> > On Fri, Nov 02, 2018 at 13:00:38 -0400, Nathan Stratton Treadway
wrote:
> > > On Fri, Nov 02, 2018 at 12:35:06 -0400, Gene Heskett wrote:
> > > > Fri Nov 02 03:42:06.085962877 2018: pid 13139: thd-0x9824e00:
> > > > driver: not updating because origsize or dumpsize is 0 Fri Nov
> > > > 02 03:42:06.086158866 2018: pid 13139: thd-0x9824e00: driver:
> > > > Building type FILE header of 32768-32768 bytes with
> > > > name='coyote' disk='/usr/games' dumplevel=1 and blocksize=0
> > > > Fri Nov 02 03:42:06.086317001 2018: pid 13139: thd-0x9824e00:
> > > > driver: Building type FILE header of 32768-32768 bytes with
> > > > name='coyote' disk='/usr/games' dumplevel=1 and blocksize=0
> > > > Fri Nov 02 03:42:06.092807051 2018: pid 13139: thd-0x9824e00:
> > > > driver: driver: send-cmd time 2461.017 to taper0: FILE-WRITE
> > > > worker0-0 00-00130 /usr/dumps/20181102030105/coyote._usr_games.1
> > > > coyote /usr/games 1 20181102030105 "" "" "" 1 "" "" "" "" 10
> > >
> > > Ah, interesting, looks like this applies to level 1 dumps as well.
> > > What does "amadmin CONFIG info coyote /usr/games shop
> > > /usr/lib/amanda" output?
> >
> > Okay, I believe I have tracked down this "info on small DLEs not
> > saved" bug...
> >
> >
> > Summary:
> >
> > The short-ish summary is that in Amanda 3.4 and 3.5, any dump which
> > ends but being shorter than 1024 bytes long (after compression) is
> > treated as not having happened at all, as far as Amanda's recording
> > of "info" statistics goes.
> >
> > This is most significant for full dumps (i.e. of very small
> > partitions), because it causes Amanda to think that DLE has never
> > been dumped (or, that the last time it was dumped was on the final
> > run made using Amanda 3.3 or older), and thus it schedules that DLE
> > for a full dump on every run.
> >
> > However, the bug actually also affects incremental dumps (as we
> > discussed in the above-quoted message) -- so DLEs that don't change
> > much on a particular day and thus end up with very tiny incrementals
> > end up recording those dumps as having taken place on 1969/12/31
> > rather than the day they actually occurred.
> >
> >
> > Neither of the above situations is "fatal" as far as preventing
> > Amanda from actually backup up your data, but for people (such as
> > Gene) who are effected, a workaround workaround is simply to make
> > sure that the dump on a particular day is at least 1024 bytes.
> >
> > For full dumps, you can do this just by creating a "dummy" file on
> > the otherwise-very-empty partition in question, using data that's
> > already compressed so that the dump file is still big enough after
> > Amanda compresses it. (In my tests, I just used the bytes off the
> > front of the amanda_3.4.2.orig.tar.gz file I happened to have
> > sitting around.)
> >
> > (For incrementals the trick is to make sure there is enough changing
> > on the partition that the incremental dump is over the cutoff size;
> > the best way to do that will depend on what data is on that
> > partition, etc.)
> >
> >
> >
> > Internal details and history:
> >
> > The bug happens because the messages that the chunker and driver
> > processes use to communicate with each other specify the size of the
> > files transferred in integral units of "kb" (=1024 bytes), and thus
> > the size given for very small files is "0" -- but the code in the
> > driver that handles updating the info database has a specific check
> > for zero-length dumps and refuses to update the database in that
> > case. (This check is found in driverio.c:update_info_dumper() .)
> >
> > It appears that the bug has existed since Amanda 3.4, when the old
> > chunker.c version of the chunker program was replaced with a new
> > Perl version.
> >
> > Both server-src/chunker.c as found in Amanda 3.3 and
> > server-src/dumper.c as it exists in 3.5 take special care to round
> > "kb" value passed for files which are short-but-not-empty files up
> > to "1" -- but that logic was not implemented in the Perl chunker
> > code when it was created..
> >
> > (Interestingly, if I am reading the old chunker.c code correctly, it
> > used to round up not just very-small-but-not-empty files to 1 kb,
> > but actually rounded the kb figure up to count any trailing partial
> > kilobytes at the end of the file... while the new program seems to
> > just ignore those trailing partial kilobytes. Presumably this
> > difference simply doesn't matter -- except for the when the size is
> > rounded down to zero.)
> >
> >
> > Proposed patch:
> >
> > This is where having an actual Amanda developer would be very
> > handy... but given that planner.c and old-chunker.c both have
> > special handling for small-but-not-empty files, I figured that
> > adding a similar check to the new Chunker implementation is probably
> > the best fix for this situation, and that hopefully doing so would
> > be safe from unwanted side-effects....
> >
> > So, I edited the perl/Amanda/Chunker/Controller.pm to implement such
> > a check, as shown in the attached patch. I've been running with
> > this patch in place for a couple days now, and so far it seems to
> > have resolved the issue for me....
> >
> >
> > Nathan
>
> I had to do some file renaming because you were saving the original,
> then it patched it withgout generating a new properly named file
> leaving it with the name you saved, but I finally got it to apply to
> 3.5.1, and its building now. We'll see how it runs later tonight.
> Thanks Nathan for playing bulldog. Its appreciated.
>
> And amcheck is happy. :)
Yup, patch is good, Nathan, all my 30+ day overdue's are gone.
I think we should republish it as 3.5.1-p1. Maybe.
But what is the correct gzip syntax to get a listing of the contents of
one of these smaller files?
All I can get out of last nights /usr/local file from the unpack by:
dd if=00043.coyote._usr_local.1 bs=32k skip=1 | /bin/gzip -dcl --name
is:
compressed uncompressed ratio uncompressed_name
-1 -1 0.0% stdout
as if there is nothing in the file, and it should have had in this case,
all the new sbin/am* files from the rebuilt amanda install, but that
file for /usr/local is just short of 47 megs. So its probably all
there, and I'm not holding my mouth right with the gzip invocation.
okteta says there is data starting at the 32k offset, another 46.9 or so
megs of it, but no names visible due to the compression. Call me
puzzled.
Copyright 2018 by Maurice E. Heskett
--
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>
