Re: Problems with Incremental Backups
On Thu, Aug 11, 2005 at 11:53:07AM -0400, Jon LaBadie wrote: > > You mean, if I edit a file but the size doesn't change > it need not be backed up in an incremental? Hardly. > Size is the determining factor, nor should it be. ^^^ insert "not" -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Problems with Incremental Backups
On Thu, Aug 11, 2005 at 10:41:23AM +1000, Haroon Anwar wrote: > Hi Matt, > > Thanks for you reply. Just got a question, what if I > mount filesystems with the "noatime" options so that > when it accesses a file it does not update the atime > on the inode? What will amanda do then? Work on file > size? > Is there any way, I can force amanda to work on file > size rather than on the basis of "atime" for the > incremental backups. You mean, if I edit a file but the size doesn't change it need not be backed up in an incremental? Hardly. Size is the determining factor, nor should it be. atime is not the basis for determing the need for a backup either. The "a" in atime stands for "access", i.e. when were the data read. Looking at the data in a file should not cause it to be a candidate for incremental backup. Your system may vary, but on Solaris the noatime mount option is not absolute. I.e. under some conditions atime is updated. Basically if atime needs updating when ctime or mtime are also being updated the system will update atime even with the option turned on. What I think Matt was suggesting is that some process on your system "might" be forcing the files atime to be updated. One example of a user level command that does this is touch(1). When a command like touch is used to force a time stamp change, the file's ctime is automatically changed as well. A ctime change will stimulate and incremental backup. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Problems with Incremental Backups
On Monday 08 August 2005 22:33, Haroon Anwar wrote: >Hi, > >I am having problem with incremental backups. Every >thing is working >fine. I force full backup of all my servers on >Saturday. From Sunday >to Friday I am taking incremental backup. > >It is working fine. But, what my understanding of >incremental backup >is, starting from level 0 backup, amanda will backup >only those files >which change. It should backup only the difference or >the changes. But >this does not seem to be happening. E.g, One of my >server disk is 18G. >On Saturday I backup 18G. But on Sunday, I expect >amanda to backup >only the difference. But, its backing up around 10G >still and I dont >think that the data on the server disk change that >much. All the other >servers are behaving in the same manner. > >Can someone help me how can I achieve only to backup >the new files or >the files that change when the incremental backup >kicks off. > >I force incremental backups by putting no-full line in >global secation >of the dumptype and inhereting global settings in the >other dumptypes. Bad. Very bad. Awfull even.. You aren't understanding the theory behind amandas scheduler at all. Amanda must have a full backup in her database within dumpcycle days history in order to be able to discern whats changed when doing an incremental. As someone else has said, you are forcing amanda to do something thats against the way amanda is designed and expected to work. Amanda, left to do her own thing, will do a very good job of backing up your data by useing her own regimen & scheduleing. This will result in an overall schedule that uses about the same amount of tape every night, and you can often set runtapes to 1, depending on the size of the drive. If you insist on bending your backups to your fixed schedule view of how things should be done, then maybe you should look for backup solutions other than amanda. But I don't believe there is a competing solution that will do incrementals only. >Thanks > >Regards > >Haroon I have to ask since you didn't state, is this a samba share by any chance? Samba, like most windows file systems, does not support ctime and returns garbage in that field when queried. So amanda thinks its nearly all new and acts accordingly. The solution to that problem here was to install amandas client stuff on the targeted box and change the disklist entry from a /mnt/x/y path to a 'FQDN /path dumptypespindle_number leo' style of entry. Everything I do here is on this side of the firewall so thats not a problem. -- Cheers, Gene "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) 99.35% setiathome rank, not too shabby for a WV hillbilly Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
Re: Problems with Incremental Backups
--On Thursday, August 11, 2005 10:41:23 +1000 Haroon Anwar <[EMAIL PROTECTED]> wrote: > Hi Matt, > > Thanks for you reply. Just got a question, what if I > mount filesystems with the "noatime" options so that > when it accesses a file it does not update the atime > on the inode? What will amanda do then? Work on file > size? > Is there any way, I can force amanda to work on file > size rather than on the basis of "atime" for the > incremental backups. Amanda (or more accurately, dump or tar) doesn't care about atime. I have atime updates disabled on my NFS servers to improve performance (with the tradeoff of loss of information on how recently files were accessed) and incremental backups work as expected. atime is the last time a file was read mtime is the last time the data in a file was written (either originally or modified) ctime is the last time the file's inode was modified Things that change mtime also change ctime, but not the other way around, i.e. editing will update the mtime, which also causes a change in the ctime. Changing the ownership or permissions, however, only updates the ctime and not the mtime. dump uses ctime, tar can use ctime or mtime depending on the options used, but I believe for --listed-incremental (the way Amanda calls it) it uses ctime. As someone else pointed out, you don't want to use file size as a criteria, much can change and still have the same size. Since you said your full backup was 18 GB and your incremental was only 10GB, you must actually be doing incrementals (or it would have been 18GB also), so something is modifying nearly half of your data. It could be a process that chowns some directories, or even that you are archiving part of your data onto the same filesystem and have one huge file that is always 'new' even though its components are mostly the same data. Another possibility is something like a CVS tree that gets cleaned and checked out making the files appear new. Try looking at the indexes of what is on your incremental and see what is there, and then figure out what makes it appear 'new'. Frank > > Your help in this regard will be higly appareciated. > > Thanks > > Regards > > haroon > > > > > --- Matt Hyclak <[EMAIL PROTECTED]> wrote: > >> On Tue, Aug 09, 2005 at 12:33:26PM +1000, Haroon >> Anwar enlightened us: >> > I am having problem with incremental backups. >> Every >> > thing is working >> > fine. I force full backup of all my servers on >> > Saturday. From Sunday >> > to Friday I am taking incremental backup. >> > >> > It is working fine. But, what my understanding of >> > incremental backup >> > is, starting from level 0 backup, amanda will >> backup >> > only those files >> > which change. It should backup only the difference >> or >> > the changes. But >> > this does not seem to be happening. E.g, One of my >> > server disk is 18G. >> > On Saturday I backup 18G. But on Sunday, I expect >> > amanda to backup >> > only the difference. But, its backing up around >> 10G >> > still and I dont >> > think that the data on the server disk change that >> > much. All the other >> > servers are behaving in the same manner. >> > >> > Can someone help me how can I achieve only to >> backup >> > the new files or >> > the files that change when the incremental backup >> > kicks off. >> > >> > I force incremental backups by putting no-full >> line in >> > global secation >> > of the dumptype and inhereting global settings in >> the >> > other dumptypes. >> > >> >> First, you're really fighting what amanda was >> designed to do...you're trying >> to make "her" do what you want instead of letting >> her do her own thing. >> >> That being said, I would look at the 10GB of files >> and see what is causing >> them to be backed up. There are several things that >> could cause this, such >> as some script that accesses the files (changing >> their atime attribute, >> causing them to be considered new), etc. I would >> examine what processes are >> running on your servers that might be modifying the >> files in such a manner >> to get at the root of the problem. >> >> Matt >> >> -- >> Matt Hyclak >> Department of Mathematics >> Department of Social Work >> Ohio University >> (740) 593-1263 >> > > > > > > Do you Yahoo!? > Yahoo! Photos: Now with unlimited storage > http://au.photos.yahoo.com -- Frank Smith[EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: Problems with Incremental Backups
On Thu, Aug 11, 2005 at 10:41:23AM +1000, Haroon Anwar enlightened us: > Thanks for you reply. Just got a question, what if I > mount filesystems with the "noatime" options so that > when it accesses a file it does not update the atime > on the inode? What will amanda do then? Work on file > size? > Is there any way, I can force amanda to work on file > size rather than on the basis of "atime" for the > incremental backups. > [EMAIL PROTECTED] ~]$ echo 'hello' > foo.txt [EMAIL PROTECTED] ~]$ ls -l foo.txt -rw-r--r-- 1 hyclak faculty 6 Aug 10 21:28 foo.txt [EMAIL PROTECTED] ~]$ echo 'help!' > foo.txt [EMAIL PROTECTED] ~]$ ls -l foo.txt -rw-r--r-- 1 hyclak faculty 6 Aug 10 21:29 foo.txt Notice that the file size is still 6 for both versions of that file. You probably don't want to only back up based on size. Anyway, it may not be atime that's being modified, it could be the mtime or maybe even ctime. You're going to have to figure out what it is that is causing the files to change. Ignoring files that don't change size is a bad way to fix the problem. Perhaps someone else can jump in here with a better knowledge of filesystem internals. Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263 pgpx0mBac2dJ3.pgp Description: PGP signature
Re: Problems with Incremental Backups
Hi Matt, Thanks for you reply. Just got a question, what if I mount filesystems with the "noatime" options so that when it accesses a file it does not update the atime on the inode? What will amanda do then? Work on file size? Is there any way, I can force amanda to work on file size rather than on the basis of "atime" for the incremental backups. Your help in this regard will be higly appareciated. Thanks Regards haroon --- Matt Hyclak <[EMAIL PROTECTED]> wrote: > On Tue, Aug 09, 2005 at 12:33:26PM +1000, Haroon > Anwar enlightened us: > > I am having problem with incremental backups. > Every > > thing is working > > fine. I force full backup of all my servers on > > Saturday. From Sunday > > to Friday I am taking incremental backup. > > > > It is working fine. But, what my understanding of > > incremental backup > > is, starting from level 0 backup, amanda will > backup > > only those files > > which change. It should backup only the difference > or > > the changes. But > > this does not seem to be happening. E.g, One of my > > server disk is 18G. > > On Saturday I backup 18G. But on Sunday, I expect > > amanda to backup > > only the difference. But, its backing up around > 10G > > still and I dont > > think that the data on the server disk change that > > much. All the other > > servers are behaving in the same manner. > > > > Can someone help me how can I achieve only to > backup > > the new files or > > the files that change when the incremental backup > > kicks off. > > > > I force incremental backups by putting no-full > line in > > global secation > > of the dumptype and inhereting global settings in > the > > other dumptypes. > > > > First, you're really fighting what amanda was > designed to do...you're trying > to make "her" do what you want instead of letting > her do her own thing. > > That being said, I would look at the 10GB of files > and see what is causing > them to be backed up. There are several things that > could cause this, such > as some script that accesses the files (changing > their atime attribute, > causing them to be considered new), etc. I would > examine what processes are > running on your servers that might be modifying the > files in such a manner > to get at the root of the problem. > > Matt > > -- > Matt Hyclak > Department of Mathematics > Department of Social Work > Ohio University > (740) 593-1263 > Do you Yahoo!? Yahoo! Photos: Now with unlimited storage http://au.photos.yahoo.com
Re: Problems with Incremental Backups
On Tue, Aug 09, 2005 at 12:33:26PM +1000, Haroon Anwar enlightened us: > I am having problem with incremental backups. Every > thing is working > fine. I force full backup of all my servers on > Saturday. From Sunday > to Friday I am taking incremental backup. > > It is working fine. But, what my understanding of > incremental backup > is, starting from level 0 backup, amanda will backup > only those files > which change. It should backup only the difference or > the changes. But > this does not seem to be happening. E.g, One of my > server disk is 18G. > On Saturday I backup 18G. But on Sunday, I expect > amanda to backup > only the difference. But, its backing up around 10G > still and I dont > think that the data on the server disk change that > much. All the other > servers are behaving in the same manner. > > Can someone help me how can I achieve only to backup > the new files or > the files that change when the incremental backup > kicks off. > > I force incremental backups by putting no-full line in > global secation > of the dumptype and inhereting global settings in the > other dumptypes. > First, you're really fighting what amanda was designed to do...you're trying to make "her" do what you want instead of letting her do her own thing. That being said, I would look at the 10GB of files and see what is causing them to be backed up. There are several things that could cause this, such as some script that accesses the files (changing their atime attribute, causing them to be considered new), etc. I would examine what processes are running on your servers that might be modifying the files in such a manner to get at the root of the problem. Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263 pgpTyAqInqNHO.pgp Description: PGP signature
