Re: [Bacula-users] Bacula-users Digest, Vol 181, Issue 21
On Sunday 2021-05-23 11:53:25 Joseph Zatarski wrote: > In response to Bill, Phil, and Josip, I have checked the logs to be > sure and the job is not being upgraded to full. I can't say for > certain that the files haven't been touched by anything, but I'll be > checking some attributes and doing some comparisons between what bacula > has in the DB for those files, and what they are on disk. I believe > last time I did such a thing, I came to the conclusion that the only > difference I saw was in the device number field of the stat(2) > attribute blob. I'll follow up again once I've looked at it. So, if Bacula didn't upgrade incremental job to a full backup job and you only got suspiciously large backup, I would suspect that problem lays somewhere else. I had to check your original post to refresh my memory. You asked about the change of the device number of the storage device... I am pretty sure it doesn't affect Bacula and the way it determines which files differ from those already backed up. Bacula deals with files and directories that reside on the file system you have decided to backup and it doesn't check the device numbers of the underlaying device. But the fact that you have noticed unreasonably large amount of files retrieved by the incremental job might mean that something else is changing "attributes" of the files. Here are few things you might wan to consider: - Are you using normal file system (e.g. ext4, xfs) or some experimental file system? - Is the OS you are trying to backup hardware machine or virtual machine? - Is your system time correct and your ntpd/chronyd properly configured. Bacula is trying to take into account the time difference between server and client but still, it's worth checking the system time. You might also want to manually synchronize your hardware clock with your system clock using the command "hwclock -w" in case it is not regularly performed. - What kind of data are getting copied by the job? Are we talking about the files that belong to OS or some custom user data? - If the data that get copied belong to the OS than it would be a good thing to check if there were some unattended upgrades before the backup. - Also, prelinking (in case it's turned on and forced to run) could modify large number of system files. - Are you using selinux or some other subsystem or tools that could trigger a change of large number of files? - Have you noticed any warnings or errors in your logs and dmesg output lately? - Have you noticed any corrupted files or databases on that file system you are trying to backup? > It would also be useful to know what the default options are for the > "Accurate = " flags in the fileset. Unfortunately, I don't see any > listing of default options in the documentation, beyond when they > implemented the feature and it pops up in the change summary sections. > At least back then, it was mtime, ctime, and file size. I'll have to > work on the assumption that it has not changed since then. In the Bacula documentation of version 5..0.x https://www.bacula.org/5.0.x-manuals/en/main/main.pdf in section 4.5 it says: -BEGIN In previous versions, the accurate code used the file creation and modification times to determine if a file was modified or not. Now you can specify which attributes to use (time, size, checksum, permission, owner, group, . . . ), similar to the Verify options. -END- -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula-users Digest, Vol 181, Issue 21
On 5/22/21 10:57 PM, bacula-users-requ...@lists.sourceforge.net wrote: Message: 7 Date: Sat, 22 May 2021 21:14:20 + From: Bill Arlofski To:bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] noticing an occasional incremental backup act like full Message-ID:<92e818f8-173d-874d-188c-1637c5ab4...@protonmail.com> Content-Type: text/plain; charset=utf-8 Hello Joe, I can think of three things that can cause this to happen, and at least two of them will show some evidence in the job log Option 1: - You have edited the Fileset, and the "IgnoreFilesetChanges = yes" option was not set at the top of the Fileset. In this case Bacula will log as one of the first lines in the job's log something like this: 8< 05-May 11:05 bacula-dir JobId 38357: No prior or suitable Full backup found in catalog. Doing FULL backup. 8< Additionally, in the Summary, where it shows the "Backup Level:" it will also tell you the job was upgraded from an Inc or Diff 8< Backup Level: Full (upgraded from Differential) 8< One other thing in the summary to help diagnose if an edit of the Fileset was the cause is that the "Fileset:" line shows the last time the Fileset that was used in the job was edited: 8< FileSet:"HomeFileSet" 2015-07-03 12:35:32 8< Option 2: - Something or someone has 'touched' all of the files in the tree. I have seen this a few times where someone implements a script - perhaps to do some virus scan or something - and the script or program modifies the attributes of every single file, causing Bacula to (correctly) noticed a change and backup each file. To mitigate this type of case, you can set specific attributes to check in the Fileset with the "Accurate = " setting - where is a list of characters to enable the attributes you want to check. Using this feature you can tell Bacula to ignore attributes that may get modified in cases like I described. Option 3: - You have set the "MaxFullInterval" in a Job or JobDefs resource. Personally, I like and use this feature. It is a simple way to help "spread out" my Full backups from all kicking off at the same time each month. And, since I don't really care what day or time my Fulls run as long as they run once a month - this is a prefect feature for me. I set it for about 33 days, and then set all of my Jobs to have "Level = Incremental", and my Schedules are just simple "run every day at x time" with no mention of running a Full on the First Sunday of a month and Incrementals every other day. Then, when 33 days have passed, instead of doing the specified Incremental or Differential level configured in the Job/JobDefs, Bacula will log this in the job log: 8< 12-May 23:00 bacula-dir JobId 38614: Max Full Interval exceeded. Doing FULL backup. 8< and then this in the Summary: 8< Backup Level: Full (upgraded from Differential) 8< Also, if I kick of a Full for whatever reason, this starts the MaxFullInterval counter for this job again at this time, adding a little more entropy into the mix of when Fulls are kicked off. Hope this helps, Best regards, Bill -- Bill Arlofski w...@protonmail.com -- Message: 10 Date: Sat, 22 May 2021 23:36:50 -0400 From: Phil Stracchino To:bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] noticing an occasional incremental backup act like full Message-ID: Content-Type: text/plain; charset=utf-8 On 5/22/21 4:45 PM, Joseph Zatarski wrote: I notice that every once in a while, I'll have an incremental run that seems to back up the entire file set with no regard to what files have actually changed. Firstly, has anybody else noticed this happening? JUST TO BE CERTAIN here: Is this a case where the job RAN as an incremental, but backed up everything anyway, or a case where the job was promoted to Full? I'm pretty sure you're talking about the former, but I want to be sure. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 Message: 1 Date: Sun, 23 May 2021 11:50:29 +0200 From: Josip Deanovic To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] noticing an occasional incremental backup act like full Message-ID: <2308799.O7ZQRL7jbA@libria.borealis> Content-Type: text/plain; charset="us-ascii" Option 4: - If no valid full backup for the job could be found in the catalog, the backup job will be upgraded from incremental or differential to full backup. The message would be the same as the one described in option 1. This could happen if the job is prematurely pruned (before a new full backup has been performed). Regards! -- Josip Deanovic - In response to Bill, Phil, and Josip, I have checked the logs to be sure and the job
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Sunday 2021-05-23 14:31:51 Bill Arlofski via Bacula-users wrote: > On Sunday, May 23, 2021 8:25 AM, Bill Arlofski wrote: > > Check this out: > > 8< > > 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified > > on 2021-05-23 08:12:32, this is after the last successful backup on > > 2021-05-22 17:35:16. > > > > 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup > > found in catalog for the current FileSet. Doing FULL backup. > > 8< > > I just checked and I see that this was implemented in June 2019. :) I must have got you wrong. I thought you are referring to "No prior or suitable Full backup". -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Sunday 2021-05-23 14:25:37 Bill Arlofski via Bacula-users wrote: > Right, good point! And when a job (typically set to Level = Incremental) > is first run, this will always happen too. :) > > > Oh! And I just remembered that I had made an internal Bacula Enterprise > feature request some time ago. It was accepted and implemented, so > Option 1 is even better than I explained because now it tells you very > clearly why there are "no prior suitable Full backups". > > Check this out: > 8< > 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified on > 2021-05-23 08:12:32, this is after the last successful backup on > 2021-05-22 17:35:16. > > 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup found > in catalog for the current FileSet. Doing FULL backup. 8< > > > I am looking forward to Joe's feedback in this thread. :) This message is very helpful. I don't even remember when it first appeared. -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Sunday, May 23, 2021 8:25 AM, Bill Arlofski wrote: > > Check this out: > 8< > 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified on > 2021-05-23 08:12:32, this is after the last > successful backup on 2021-05-22 17:35:16. > > 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup found in > catalog for the current FileSet. Doing FULL backup. > 8< I just checked and I see that this was implemented in June 2019. :) Best regards, Bill ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On 5/23/21 3:50 AM, Josip Deanovic wrote: > > There is another option. > > Option 4: > - > If no valid full backup for the job could be found in the catalog, the > backup job will be upgraded from incremental or differential to full > backup. The message would be the same as the one described in option 1. > > This could happen if the job is prematurely pruned (before a new full > backup has been performed). > Right, good point! And when a job (typically set to Level = Incremental) is first run, this will always happen too. :) Oh! And I just remembered that I had made an internal Bacula Enterprise feature request some time ago. It was accepted and implemented, so Option 1 is even better than I explained because now it tells you very clearly why there are "no prior suitable Full backups". Check this out: 8< 23-May 08:12 speedy-dir JobId 6: The FileSet "Full Set" was modified on 2021-05-23 08:12:32, this is after the last successful backup on 2021-05-22 17:35:16. 23-May 08:12 speedy-dir JobId 6: No prior or suitable Full backup found in catalog for the current FileSet. Doing FULL backup. 8< I am looking forward to Joe's feedback in this thread. :) Best regards, Bill -- Bill Arlofski w...@protonmail.com ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] noticing an occasional incremental backup act like full
On Saturday 2021-05-22 21:14:20 Bill Arlofski via Bacula-users wrote: > Hello Joe, > > I can think of three things that can cause this to happen, and at least > two of them will show some evidence in the job log > > > Option 1: > - > You have edited the Fileset, and the "IgnoreFilesetChanges = yes" option > was not set at the top of the Fileset. In this case Bacula will log as > one of the first lines in the job's log something like this: 8< > 05-May 11:05 bacula-dir JobId 38357: No prior or suitable Full backup > found in catalog. Doing FULL backup. 8< > > Additionally, in the Summary, where it shows the "Backup Level:" it will > also tell you the job was upgraded from an Inc or Diff 8< > Backup Level: Full (upgraded from Differential) > 8< > > One other thing in the summary to help diagnose if an edit of the > Fileset was the cause is that the "Fileset:" line shows the last time > the Fileset that was used in the job was edited: > 8< > FileSet:"HomeFileSet" 2015-07-03 12:35:32 > 8< > > > Option 2: > - > Something or someone has 'touched' all of the files in the tree. I have > seen this a few times where someone implements a script - perhaps to do > some virus scan or something - and the script or program modifies the > attributes of every single file, causing Bacula to (correctly) noticed > a change and backup each file. > > To mitigate this type of case, you can set specific attributes to check > in the Fileset with the "Accurate = " setting - where is a > list of characters to enable the attributes you want to check. Using > this feature you can tell Bacula to ignore attributes that may get > modified in cases like I described. > > > Option 3: > - > You have set the "MaxFullInterval" in a Job or JobDefs resource. > Personally, I like and use this feature. It is a simple way to help > "spread out" my Full backups from all kicking off at the same time each > month. And, since I don't really care what day or time my Fulls run as > long as they run once a month - this is a prefect feature for me. > > I set it for about 33 days, and then set all of my Jobs to have "Level = > Incremental", and my Schedules are just simple "run every day at x > time" with no mention of running a Full on the First Sunday of a month > and Incrementals every other day. > > Then, when 33 days have passed, instead of doing the specified > Incremental or Differential level configured in the Job/JobDefs, Bacula > will log this in the job log: > 8< > 12-May 23:00 bacula-dir JobId 38614: Max Full Interval exceeded. Doing > FULL backup. 8< > > and then this in the Summary: > 8< > Backup Level: Full (upgraded from Differential) > 8< > > Also, if I kick of a Full for whatever reason, this starts the > MaxFullInterval counter for this job again at this time, adding a > little more entropy into the mix of when Fulls are kicked off. > > Hope this helps, There is another option. Option 4: - If no valid full backup for the job could be found in the catalog, the backup job will be upgraded from incremental or differential to full backup. The message would be the same as the one described in option 1. This could happen if the job is prematurely pruned (before a new full backup has been performed). Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
