Re: GNU tar --one-file-system flag seems to have no effect
On Sun, May 20, 2012 at 00:20:10 +0300, Toomas Aas wrote: Jean-Louis wrote: It might be a feature/bug with level 1 backup, try a full of /storage. I did, and the dump of /storage did not include files under /storage/lists. So it *might* be a bug with level 1 backup, but in that case it's strange that it has never affected backup of /, as indicated by Nathan above. I did experimentation using a simple directory tree to emulate your move-to-new-partition situation, and confirmed that even in tar 1.26 (i.e. the current latest release) there's a bug that shows up in your particular situation. The conditions required to trigger the bug are pretty specific, though: you have to have a listed-incremental snapshot file that mentions files which were included in an earlier backup, and those *same archive member paths* have to be found in the current run, but now be found on separate partition. (So in other words the bug only triggers when you move files to a new partition but then mount that new partition back on top of the path where the files used to be found, and only when you do that between levels of an incremental backup. That's why normal backups of e.g. / are not affected, even though they do have other filesystems mounted under them -- and why your problem when away once you did a new level-0 backup of /storage.) Anyway, I posted a report of the bug to the bug-tar list: http://lists.gnu.org/archive/html/bug-tar/2012-05/msg00022.html , and will report back here to amanda-users if anything interesting comes out of the discussion there. Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: GNU tar --one-file-system flag seems to have no effect
Hello all, Just to keep the amount of messages down, I'll respond to several questions at once. Robert wrote: Is tar on the OP system actually GNUTar? On *Linux* systems it generally is, but I am not certain about *commercial* UNIX systems... It is true that 'standard' tar on FreeBSD is BSD tar not GNU tar, but GNU tar is always installed together with Amanda (as /usr/local/bin/gtar) and Amanda is configured to use that - the runtar.*.debug files show that /usr/local/bin/gtar is being launched. $ /usr/local/bin/gtar --version tar (GNU tar) 1.22 Copyright (C) 2009 Free Software Foundation, Inc. Does the disk spec in the disklist file contain wildcards? tar ... --one-file-system /storage/* will backup both disks (and no others mounted under them). Nope, there are no wildcards in the disklist Nathan wrote: $ stat -f %N %d /storage /storage/some-file-not-under-lists $ stat -f %N %d /storage/lists /storage/lists/some-filename Does that in fact show a different device number for the two different commands (but the same device number for the two files in each command)? Yes, it does: # stat -f %N %d /storage /storage/dumps/20120516/test.sql.gz /storage 111 /storage/dumps/20120516/test.sql.gz 111 # stat -f %N %d /storage/lists /storage/lists/bbb /storage/lists 125 /storage/lists/bbb 125 Do you also back up / on this system, by any chance? If so, that would seem to show that --one-file-system works some of the time. Good point that I completely missed! This machine has been in backup for years, and disklist includes / with comp-root-tar dumptype. I'm sure I would have noticed if it suddenly would have baced up the entire machine, not just the 500 MB that are actually in /. None of the relevant dumptypes have been changed from their defaults in amanda.conf. Jean-Louis wrote: It might be a feature/bug with level 1 backup, try a full of /storage. I did, and the dump of /storage did not include files under /storage/lists. So it *might* be a bug with level 1 backup, but in that case it's strange that it has never affected backup of /, as indicated by Nathan above. -- Toomas Aas
Re: GNU tar --one-file-system flag seems to have no effect
At Thu, 17 May 2012 12:37:24 -0400 amanda-users@amanda.org wrote: On Thu, May 17, 2012 at 03:17:32PM +0300, Toomas Aas wrote: Thu, 17 May 2012 kirjutas Christopher X. Candreva ch...@westnet.com: One file system means just that. Unless /storage/lists is a separate partition mounted at that point, /storage is one file system, as you say above. You need the explicit exclude. That was exactly my point. Until day before yesterday, /storage/lists was just a subdirectory on /storage filesystem. Then I created a new filesystem, moved the contents of /storage/lists to that filesystem and mounted it under /storage/lists. So it is a separate filesystem, but it was still backed up as part of /storage. $ df Filesystem 1K-blocks UsedAvail Capacity Mounted on /dev/da0s1a 507630 254546 21247455%/ devfs 110 100%/dev /dev/md0 31470 77642119027%/phpramdisk /dev/da0s1f 8119342 4804070 266572664%/usr /dev/da0s1e 4058062 2732312 100110673%/var /dev/da0s1d 126702220149455219%/var/tmp /dev/da0s2.journal 54576856 17684592 3252611635%/storage devfs 110 100%/var/named/dev /dev/da1s1a 34425972 9515844 2215605230%/storage/lists Grasping at straws, any chance you also left the original files in the directory lists after copying them to the new partition. After mounting the new partition on /storage/lists they would be masked and would not be accessible using file system semantics. But dump-like programs could still see, and back-up, the masked files. Tar should be using file system semantics, but like I said, grasping. There are a couple of other possibilities: Is tar on the OP system actually GNUTar? On *Linux* systems it generally is, but I am not certain about *commercial* UNIX systems... Does the disk spec in the disklist file contain wildcards? tar ... --one-file-system /storage/* will backup both disks (and no others mounted under them). (Yes, this is probably more straw grapsing..) jl -- Robert Heller -- 978-544-6933 / hel...@deepsoft.com Deepwoods Software-- http://www.deepsoft.com/ () ascii ribbon campaign -- against html e-mail /\ www.asciiribbon.org -- against proprietary attachments
Re: GNU tar --one-file-system flag seems to have no effect
On 05/18/2012 12:33 AM, Toomas Aas wrote: Hello Nathan! What exactly happened that convinced you the contents of /storage/lists was still getting included in the dump for /storage ? First I noticed in the e-mail report that level 1 backup of /storage was just a little bit bigger than level 0 backup of /storage/lists (9703 vs 9120 MB). Then I looked at the index file of /storage DLE on server for that day's amdump run, and noticed that all files in the /storage/lists directory were listed there. It might be a feature/bug with level 1 backup, try a full of /storage. Jean-Louis
Re: GNU tar --one-file-system flag seems to have no effect
On Fri, May 18, 2012 at 07:33:31 +0300, Toomas Aas wrote: First I noticed in the e-mail report that level 1 backup of /storage was just a little bit bigger than level 0 backup of /storage/lists (9703 vs 9120 MB). Then I looked at the index file of /storage DLE on server for that day's amdump run, and noticed that all files in the /storage/lists directory were listed there. Hmmm, interesting. I'm not all that familiar with FreeBSD, but one thing to double-check is to make sure that the filesystems are really showing up as being mounted on separate devices. From the FreeBSD stat man page, it looks like you can confirm this using a command something like $ stat -f %N %d /storage /storage/some-file-not-under-lists $ stat -f %N %d /storage/lists /storage/lists/some-filename Does that in fact show a different device number for the two different commands (but the same device number for the two files in each command)? On Thu, May 17, 2012 at 15:17:32 +0300, Toomas Aas wrote: $ df Filesystem 1K-blocks UsedAvail Capacity Mounted on /dev/da0s1a 507630 254546 21247455%/ devfs 110 100%/dev /dev/md0 31470 77642119027%/phpramdisk /dev/da0s1f 8119342 4804070 266572664%/usr /dev/da0s1e 4058062 2732312 100110673%/var /dev/da0s1d 126702220149455219%/var/tmp /dev/da0s2.journal 54576856 17684592 3252611635%/storage devfs 110 100%/var/named/dev /dev/da1s1a 34425972 9515844 2215605230%/storage/lists Do you also back up / on this system, by any chance? If so, that would seem to show that --one-file-system works some of the time. (I'm assuming you would have already noticed if the / DLE included all of /usr, /var and /storage...) (Or, if you back up /var, you could check to see if the files under /var/tmp/ are included in the index files for the /var DLE) Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: GNU tar --one-file-system flag seems to have no effect
On Thu, 17 May 2012, Toomas Aas wrote: I had one big filesystem mounted on /storage which was backed up by a DLE using comp-user-tar dumptype. Yesterday I split out one subdirectory, /storage/lists, into separate partition and added another DLE for /storage/lists, also using comp-user-tar dumptype. Tonight's amdump run seems to have backed up the contents of /storage/lists twice, once as /storage/lists DLE and once as part of /storage DLE. I thought that '--one-file-system' flag passed by Amanda to gtar is supposed to prevent that, but in my case it doesn't seem to have an effect. One file system means just that. Unless /storage/lists is a separate partition mounted at that point, /storage is one file system, as you say above. You need the explicit exclude. == Chris Candreva -- ch...@westnet.com -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/
Re: GNU tar --one-file-system flag seems to have no effect
Thu, 17 May 2012 kirjutas Christopher X. Candreva ch...@westnet.com: One file system means just that. Unless /storage/lists is a separate partition mounted at that point, /storage is one file system, as you say above. You need the explicit exclude. That was exactly my point. Until day before yesterday, /storage/lists was just a subdirectory on /storage filesystem. Then I created a new filesystem, moved the contents of /storage/lists to that filesystem and mounted it under /storage/lists. So it is a separate filesystem, but it was still backed up as part of /storage. $ df Filesystem 1K-blocks UsedAvail Capacity Mounted on /dev/da0s1a 507630 254546 21247455%/ devfs 110 100%/dev /dev/md0 31470 77642119027%/phpramdisk /dev/da0s1f 8119342 4804070 266572664%/usr /dev/da0s1e 4058062 2732312 100110673%/var /dev/da0s1d 126702220149455219%/var/tmp /dev/da0s2.journal 54576856 17684592 3252611635%/storage devfs 110 100%/var/named/dev /dev/da1s1a 34425972 9515844 2215605230%/storage/lists -- Toomas Aas
Re: GNU tar --one-file-system flag seems to have no effect
On Thu, 17 May 2012, Toomas Aas wrote: Thu, 17 May 2012 kirjutas Christopher X. Candreva ch...@westnet.com: One file system means just that. Unless /storage/lists is a separate partition mounted at that point, /storage is one file system, as you say above. You need the explicit exclude. That was exactly my point. Until day before yesterday, /storage/lists was just a subdirectory on /storage filesystem. Then I created a new filesystem, moved the contents of /storage/lists to that filesystem and mounted it under /storage/lists. So it is a separate filesystem, but it was still backed up as part of /storage. /dev/da0s2.journal 54576856 17684592 3252611635%/storage /dev/da1s1a 34425972 9515844 2215605230%/storage/lists Ah, then I think you are correct. == Chris Candreva -- ch...@westnet.com -- (914) 948-3162 WestNet Internet Services of Westchester http://www.westnet.com/
Re: GNU tar --one-file-system flag seems to have no effect
On Thu, May 17, 2012 at 07:02:51 +0300, Toomas Aas wrote: Tonight's amdump run seems to have backed up the contents of /storage/lists twice, once as /storage/lists DLE and once as part of /storage DLE. I thought that '--one-file-system' flag passed by Amanda to gtar is supposed to prevent that, but in my case it doesn't seem to have an effect. Yes, the --one-file-system is indeed supposed to prevent that... What exactly happened that convinced you the contents of /storage/lists was still getting included in the dump for /storage ? Nathan Nathan Stratton Treadway - natha...@ontko.com - Mid-Atlantic region Ray Ontko Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: GNU tar --one-file-system flag seems to have no effect
On Thu, May 17, 2012 at 03:17:32PM +0300, Toomas Aas wrote: Thu, 17 May 2012 kirjutas Christopher X. Candreva ch...@westnet.com: One file system means just that. Unless /storage/lists is a separate partition mounted at that point, /storage is one file system, as you say above. You need the explicit exclude. That was exactly my point. Until day before yesterday, /storage/lists was just a subdirectory on /storage filesystem. Then I created a new filesystem, moved the contents of /storage/lists to that filesystem and mounted it under /storage/lists. So it is a separate filesystem, but it was still backed up as part of /storage. $ df Filesystem 1K-blocks UsedAvail Capacity Mounted on /dev/da0s1a 507630 254546 21247455%/ devfs 110 100%/dev /dev/md0 31470 77642119027%/phpramdisk /dev/da0s1f 8119342 4804070 266572664%/usr /dev/da0s1e 4058062 2732312 100110673%/var /dev/da0s1d 126702220149455219%/var/tmp /dev/da0s2.journal 54576856 17684592 3252611635%/storage devfs 110 100%/var/named/dev /dev/da1s1a 34425972 9515844 2215605230%/storage/lists Grasping at straws, any chance you also left the original files in the directory lists after copying them to the new partition. After mounting the new partition on /storage/lists they would be masked and would not be accessible using file system semantics. But dump-like programs could still see, and back-up, the masked files. Tar should be using file system semantics, but like I said, grasping. jl -- Jon H. LaBadie j...@jgcomp.com 11226 South Shore Rd. (703) 787-0688 (H) Reston, VA 20190 (609) 477-8330 (C)
Re: GNU tar --one-file-system flag seems to have no effect
Hello Nathan! What exactly happened that convinced you the contents of /storage/lists was still getting included in the dump for /storage ? First I noticed in the e-mail report that level 1 backup of /storage was just a little bit bigger than level 0 backup of /storage/lists (9703 vs 9120 MB). Then I looked at the index file of /storage DLE on server for that day's amdump run, and noticed that all files in the /storage/lists directory were listed there. -- Toomas Aas
Re: GNU tar --one-file-system flag seems to have no effect
Hello Jon! Grasping at straws, any chance you also left the original files in the directory lists after copying them to the new partition. After mounting the new partition on /storage/lists they would be masked and would not be accessible using file system semantics. But dump-like programs could still see, and back-up, the masked files. Tar should be using file system semantics, but like I said, grasping. I just double-checked and I'm pretty sure I did delete the original files from the directory. When I umount /storage/lists, the directory appears empty. -- Toomas Aas
GNU tar --one-file-system flag seems to have no effect
I had one big filesystem mounted on /storage which was backed up by a DLE using comp-user-tar dumptype. Yesterday I split out one subdirectory, /storage/lists, into separate partition and added another DLE for /storage/lists, also using comp-user-tar dumptype. Tonight's amdump run seems to have backed up the contents of /storage/lists twice, once as /storage/lists DLE and once as part of /storage DLE. I thought that '--one-file-system' flag passed by Amanda to gtar is supposed to prevent that, but in my case it doesn't seem to have an effect. It's easy enough to just add an appropriate exclude statement to /storage DLE, but I'm curious whether someone else has seen such thing, and what might be the cause. It can't be very common, otherwise I would remember someone else reporting this. Relevant software versions: Client: amanda-client-2.6.1p2, gtar-1.22, FreeBSD 7.4 Server: amanda-server-3.2.2, FreeBSD 8.2 Contents of relevant runtar.debug file for DLE /storage: 1337197794.412932: runtar: pid 34403 ruid 1010 euid 0 version 2.6.1p2: start at Wed May 16 22:49:54 2012 1337197794.413066: runtar: version 2.6.1p2 1337197794.437903: runtar: /usr/local/bin/gtar version: tar (GNU tar) 1.22 1337197794.438480: runtar: config: BACKUP 1337197794.439032: runtar: pid 34403 ruid 0 euid 0 version 2.6.1p2: rename at Wed May 16 22:49:54 2012 1337197794.439334: runtar: running: /usr/local/bin/gtar --create --file - --directory /storage --one-file-system --listed-incremental /var/amanda/client/gnutar-lists/kuller.raad.tartu.ee_storage_1.new --sparse --ignore-failed-read --totals . 1337197794.439383: runtar: pid 34403 finish time Wed May 16 22:49:54 2012 -- Toomas Aas
