Re: again level 1 backup problems.

[Jean-Louis Martineau]

Charles,

It looks to be a bug in gnutar, I sent a bug report to bug-...@gnu.org but I 
got no reply.
http://lists.gnu.org/archive/html/bug-tar/2018-02/msg0.html

You could try an older version 1.12, 1.13, 1.14
Or the patch I posted to bug-...@gnu.org

Jean-Louis


From: owner-amanda-us...@amanda.org <owner-amanda-us...@amanda.org> on behalf 
of Charles Stroom <char...@stremen.xs4all.nl>
Sent: February 13, 2018 4:28:02 PM
To: amanda-users@amanda.org
Subject: Re: again level 1 backup problems.

To sum up the problems I have had:

I have now a properly working amanda working with my pictures
directory divided over 5 disks defined in the disklist, that all resides
on a Windows formatted separate disk.

I have done a number of tests with all kind of variations on my vfat
disk, but I couldn't get a handle on what or where the problem is.
Level 1 backups are totally wrong, because they are in all but 1 case
either full backups or very sizeable backups while they should be near
0. And then they were also different in size (that is unpredictable) in
a next level 1 run and all this on a directory tree in which no changes
were made between the test runs. The culprit seems to be the vfat disk
which holds the directory, because when I copy the pictures directory to
a linux ext4 partition, everything works just fine, including proper
level 1 dumps.

As variations I have done (all with a vtape test setup):

- running on a read-only mounted directory and actually checked
date/time settings with a find (for recently changed files) after a
level 1 amdump: failed although there was no change in file times.
- running with program "GNUTAR" with an empty gnutar-list-dir
definition or with one: failure
- tested each defined disk separately: failure for each with
consistently 1 exception: de Pictures_M10 behaved as should.
- level 1 for Pictures_M10 backups are ok, both for a no changed
directory or with some test files added. I could not find any reason
why this disk was different from the others.
- However when I amdumped another disk together with the Pictures_M10
disk, both failed.
- actual precise amdump results can vary for the same disks over time.
- tested with tar 1.27 and tar 1.30: all fail
- with or without the property "CHECK-DEVICE" "NO" in amgtar: no
difference
and then I have probably forgotten some variations.

It seems that amanda (or probably tar?) does not produce the correct
information to perform a level 1 backup. I have looked to find the
details about what type of info amanda (time info of files and/or
directories, size, etc?) is using to create the tar info, which I
suppose goes into the files in the gnutar-lists directory. These are
binary files, which makes investigation difficult (for me). So
I have given up on the vfat format. But as it is claimed that amanda
also runs on Windows, how does it works in Windows on vfat format?

Anyhow, I then decided to reformat my picture disk to ntfs and see what
that would deliver. In a first test, level 1 was a near 0 size dump, so
everything seemed ok. However when I added a few large files, level 1
remained 0. It seemed an inverse problem. But then I noticed that only
accessing a file did not only change atime, but also ctime equally. I
remounted the picture disk with a noatime option added to the mount and
this now solved all problems. In hindsight, I should have tested that
option as well with the vfat disk, but when I tested file times after
an amdump on vfat , I noticed no time changes at all. So noatime
on vfat would probably not have solved the problems.

The ntfs disk is mounted as:
/dev/sdc1 /home/charles/pictures ntfs-3g \
defaults,uid=1000,gid=100,umask=0022,nofail,noatime,rw 0 0

Cheers, Charles















--
Charles Stroom
email: charles at no-spam.stremen.xs4all.nl<http://no-spam.stremen.xs4all.nl> 
(remove the "no-spam.")
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail

Re: again level 1 backup problems.

[Charles Stroom]

To sum up the problems I have had:

I have now a properly working amanda working with my pictures
directory divided over 5 disks defined in the disklist, that all resides
on a Windows formatted separate disk.

I have done a number of tests with all kind of variations on my vfat
disk, but I couldn't get a handle on what or where the problem is.
Level 1 backups are totally wrong, because they are in all but 1 case
either full backups or very sizeable backups while they should be near
0.  And then they were also different in size (that is unpredictable) in
a next level 1 run and all this on a directory tree in which no changes
were made between the test runs. The culprit seems to be the vfat disk
which holds the directory, because when I copy the pictures directory to
a linux ext4 partition, everything works just fine, including proper
level 1 dumps.

As variations I have done (all with a vtape test setup):

- running on a read-only mounted directory and actually checked
  date/time settings with a find (for recently changed files) after a
  level 1 amdump: failed although there was no change in file times.
- running with program "GNUTAR" with an empty gnutar-list-dir
  definition or with one: failure
- tested each defined disk separately: failure for each with
  consistently 1 exception: de Pictures_M10 behaved as should.
- level 1 for Pictures_M10 backups are ok, both for a no changed
  directory or with some test files added.  I could not find any reason
  why this disk was different from the others.
- However when I amdumped another disk together with the Pictures_M10
  disk, both failed.
- actual precise amdump results can vary for the same disks over time.
- tested with tar 1.27 and tar 1.30: all fail
- with or without the property "CHECK-DEVICE" "NO" in amgtar: no
  difference
and then I have probably forgotten some variations.

It seems that amanda (or probably tar?) does not produce the correct
information to perform a level 1 backup. I have looked to find the
details about what type of info amanda (time info of files and/or
directories, size, etc?) is using to create the tar info, which I
suppose goes into the files in the gnutar-lists directory. These are
binary files, which makes investigation difficult (for me).  So
I have given up on the vfat format. But as it is claimed that amanda
also runs on Windows, how does it works in Windows on vfat format?

Anyhow, I then decided to reformat my picture disk to ntfs and see what
that would deliver.  In a first test, level 1 was a near 0 size dump, so
everything seemed ok.  However when I added a few large files, level 1
remained 0.  It seemed an inverse problem.  But then I noticed that only
accessing a file did not only change atime, but also ctime equally.  I
remounted the picture disk with a noatime option added to the mount and
this now solved all problems.  In hindsight, I should have tested that
option as well with the vfat disk, but when I tested file times after
an amdump on vfat , I noticed no time changes at all. So noatime
on vfat would probably not have solved the problems.

The ntfs disk is mounted as:
/dev/sdc1  /home/charles/pictures   ntfs-3g  \
defaults,uid=1000,gid=100,umask=0022,nofail,noatime,rw  0 0

Cheers, Charles















-- 
Charles Stroom
email: charles at no-spam.stremen.xs4all.nl (remove the "no-spam.")

Re: again level 1 backup problems.

[Jean-Louis Martineau]

You didn't say how you do the backup, you are using the GNUTAR program or the 
amgtar application? or ... ?

Read 
https://marc.info/?l=amanda-users=2=1=GNU+tar+estimates+for+vfat+filesystems=b

Try to set gnutar-list-dir to the empty string in 
/etc/amanda/$CONF/amanda-client.conf
eg.   gnutar-list-dir ""
and use the program"GNUTAR" to backup the vfat filesystem.

That will not be good for other unix filesystem, so for them, use the 
application amgtar and set the GNUTAR-LISTDIR property where you want it to 
store the files.

Let me know if it works for your vfat, and  I will add that feature to amgtar.

Jean-Louis


From: owner-amanda-us...@amanda.org  on behalf 
of Charles Stroom 
Sent: February 6, 2018 5:58:23 PM
To: amanda-users@amanda.org
Subject: again level 1 backup problems.

I have had level 1 problems about 4 years ago. In that case, the level
1 behaved as a level 0, so level 1 in fact did a real full backup.
I have similar problems now again, similar, but not the same.

I have amanda 3.5.5 running which on a daily base makes cycled backup
on DDS-4 tapes of my Linux PC. This included a directory ~/pictures,
which was, because of its size, split over 5 different amanda disks (say
A-E). Disks A-D each defined 1 sub-directory (via an include), while
disk E did all remaining stuff by excluding the sub-directories defined
in A-D. This went all without any hitch and no problems.

However, recently I installed a new physical disk and transferred the
whole ~/picture tree to this extra disk, which was formatted vfat, as I
wanted to have access from Windows as well. I also thought it
better to have a different amanda.conf to enable a different
tapecycle just for ~/pictures. So a new 2nd amanda.conf was made in
its own directory and the corresponding disklist only contained the A-E
disks from the other amanda setup, where they were deleted. After I did
so, the level 1 problem started in the new setup. Looking at the
solutions of 4 years ago, I could not see the reason. As tape backups
take a long time, I made a test setup with virtual tapes (with the
wiki.zmanda as starting point) and started a number of tests. After
tweeting the various parameters I got a running amanda but with very
strange results.

My initial test disklist (with only the A-E ~/picture disks) was limited
to only 1 (A) to speed up testing, it takes a couple of minutes only.
No compression is used, because a better size estimate can be
done and pictures do not compress much anyhow. The 1st backup creates a
level 0 and the subsequent test a level 1 which is done a few minutes
later. Nothing is touching the ~/picture disk in the mean time.

For disk A (results are KB):
A: level 0 6,849,090 - level 1 40

So I was quite happy, however too early. The tests with all other
disks, which I tested one-by-one and with resetting the previous
indexfile and curinfo info in between are strange.

B: level 0 8,439,510 - level 1 3,717,090
C: level 0 6,870,230 - level 1 5,837,490
D: level 0 6,093,580 - level 1 4,058,520
E: level 0 10,240,370 - level 1 10,175,430

I repeated the tests for E, D, C & A. All level 0 backups
were identical. Level 1 for E was identical, for D & C level 1 were
different in size (5% & 20&), and for A it was correct again: only
40 KB.

The ~/picture disk is mounted as:

/dev/sdc1 /home/charles/pictures vfat
defaults,uid=1000,gid=100,umask=0022,nofail 0 0

And the disklist definitions for A-D are simply as:

fiume5.localnet Pictures_A /home/charles/pictures {
#
# pictures in the M10 directory
#
nocomp-generic
include "./A"
} 2

or B, C etc and "nocomp-generic" defined in amanda.conf.

After a couple of days looking at options, I am lost. So any suggestion
where or how to look for solutions are welcome. As far as I can see,
the new disklist and amanda.conf do not differ in essence from the
previous instalment when they were part of a single backup setup which
worked flawless (but with ~/pictures on a different linux formatted
disk). Strange.

Charles







--
Charles Stroom
email: charles at no-spam.stremen.xs4all.nl 
(remove the "no-spam.")
This message is the property of CARBONITE, INC. and may contain confidential or 
privileged information.
If this message has been delivered to you by mistake, then do not copy or 
deliver this message to anyone.  Instead, destroy it and notify me by reply 
e-mail