Thanks for the reply. The backups are not completing because the level 0 problem: planner thinks its running a backup that can complete, but ends up with 10x the data that a single run can handle, and it fills up the tape and the holding disk (which is a little more than a full tape), and then the run suspends until I kill it off with amcleankup -k. So that explains the new files.
The big question still remains is why is a level 1 or 2 backup (which by all accounts, amreport / amstatus show that it IS actually doing a level 1 or 2, it just happens to be 10000%+ of what it expected) is actually grabbing ALL files and not changed files like it should... I guess its possible the minor number is changing, but that too seems unlikely. The amclient is technically running in a FreeBSD "jail" (essentially a "ultra-lightweight VM") as required by FreeNAS, as anything installed in the main FreeNAS applaince is wiped next upgrade. Jails are how you install additional packages in a long-term way. The jail does have access to the underlying filesystem from FreeNAS. Maybe I do need that option to tell it not to pay attention to devices...but I thought tar would go by file path anyway, and any changes in underlying devices wouldn't have a difference... At this point, I also am not sure what is the best way to recover. I have a full holding disk (mostly of incomplete backups), and the .new gtar lists, etc. It seems like a waste of a tape (and 26 hours to fill said tape) to put a bunch of partial backups.... If it properly does the levels, then the backup run will fit on a single tape. If it does not honor the levels, then it would take like 5 or 7 tapes to do the run, and I do not have a tape changer (nor do I have the luxury to do unneeded fulls of everything anytime Amanda pukes at something). My data is too big (but not big enough to have a large library, multiple tape drives, and such...Big data on a budget backup system :( ) --Jim On Mon, Jun 17, 2019 at 11:35 AM Nathan Stratton Treadway < [email protected]> wrote: > On Mon, Jun 17, 2019 at 10:22:10 -0700, Jim Kusznir wrote: > > The client I'm trying to back up is a FreeNAS appliance, so its not > LVM. I > > You are running the Amanda client software directly on the FreeNAS > appliance, right? And also have shell access on the box to produce your > "ls" output, etc? > > > don't think the devices are changing boot to boot, but that's possible. > > Actually, now that I think about it, I don't think it is > possible....there > > is only one device, and I'm sub-dividing that with a bunch of > > I am not familiar with FreeNAS so I don't know how it is set up, but > note that the fact there is only one large filesystem doesn't actually > mean that the block device contianing that filesystem necessarily gets > the same (minor) device number each boot. However, this particular > issue should only cause problems for Amanda across reboots or other > occations when the block device would be re-detected, so at the very > least it would appear you ahve something else going on as well... and in > particular tracking down the .new files definitely seems the > higher-priority track of investigation at the moment... > > > -rw------- 1 amanda amanda 3695323 May 7 02:04 > amclient-tdriveCYS-General_0 > > -rw------- 1 amanda amanda 3767257 May 31 23:46 > amclient-tdriveCYS-General_1 > > -rw------- 1 amanda amanda 3695323 Jun 12 20:13 > amclient-tdriveCYS-General_1.new > > -rw------- 1 amanda amanda 9061561 May 8 04:43 > amclient-tdriveCreativeImage_0 > > -rw------- 1 amanda amanda 9061649 May 31 22:47 > amclient-tdriveCreativeImage_1 > > -rw------- 1 amanda amanda 9061561 Jun 13 20:51 > amclient-tdriveCreativeImage_1.new > > > > I'm bothered by the .new extensions, and I don't know why they are there. > > But those are in fact volumes that I've been getting level 0's of even > > though it claims its a level 1 or 2. > > Yes, that's not a good sign. GNU tar updates the snapshot file passed > in to it as it runs, so before invoking tar Amanda makes a copy of the > previous level's snapshot file (over to the file with the .new > extension), and when tar is finished Amanda renames the file to the > proper final name. So the fact that you have .new files still out there > means the dump is not runnning to completion somehow. > > (Am I correct you last ran a dump on Jun 12 or 13 or so, and the job was > no longer running at the point you took that directory listing?) > > So probably the next step is to look for the Amanda client debug files > (e.g. /var/log/amanda/client/<CONFIG>/ on my Ubuntu boxes) and look > through the sendbackup.201906*.debug and runtar.201906*.debug files to > see if you can find any indication of what's causing the jobs to fail. > > Also check the .../<CONFIG>/amdump.2019* file (on the *Amanda server* > side) to see if there are any messages in there regarding these DLEs. > > The Amanda Mail Report email doesn't list any error messages? > > Nathan > > > > ---------------------------------------------------------------------------- > Nathan Stratton Treadway - [email protected] - Mid-Atlantic region > Ray Ontko & Co. - Software consulting services - > http://www.ontko.com/ > GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 > Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239 >
