Re: [Bacula-users] waiting on max Job jobs - not cancelled?
> On Thu, 28 May 2020 06:10:36 +0100, Bernie Elbourn said: > > On 01/04/2020 16:10, Bernie Elbourn wrote: > > Oddly, jobs run sequentially as follows rather than duplicates being > > cancelled: > > > > Running Jobs: > > Console connected at 29-Mar-20 12:36 > > JobId Type Level Files Bytes Name Status > > == > > 70172 Back Diff 120 36.70 M Backup-pc is running > > 70173 Back Incr 0 0 Backup-pc is waiting on max Job jobs > > > > > > Are there any pointers to trace why the duplicate job 70173 is not > > cancelled? > > So I have cloned the system (less backup volumes) to test system with a test > pc. That test system has exactly the same > database and same bacula sever and same pc setup. > > On the test system the second incremental job IS cancelled. The reason is > pretty clear > > 12-May 15:35 sv-dir JobId 71204: Fatal error: JobId 71202 already running. > Duplicate job not allowed. > > Is there any logging or tracing information available on an actual live > system that might reveal why the production > bacula system decides to wait on max job jobs? There is no useful logging for this, but the function that checks for duplicates is allow_duplicate_job in src/dird/job.c. You could either add some more logging to it or run the Director under gdb and step through it. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 01/04/2020 16:10, Bernie Elbourn wrote: Oddly, jobs run sequentially as follows rather than duplicates being cancelled: Running Jobs: Console connected at 29-Mar-20 12:36 JobId Type Level Files Bytes Name Status == 70172 Back Diff 120 36.70 M Backup-pc is running 70173 Back Incr 0 0 Backup-pc is waiting on max Job jobs Are there any pointers to trace why the duplicate job 70173 is not cancelled? So I have cloned the system (less backup volumes) to test system with a test pc. That test system has exactly the same database and same bacula sever and same pc setup. On the test system the second incremental job IS cancelled. The reason is pretty clear 12-May 15:35 sv-dir JobId 71204: Fatal error: JobId 71202 already running. Duplicate job not allowed. Is there any logging or tracing information available on an actual live system that might reveal why the production bacula system decides to wait on max job jobs? Thanks ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 19/04/2020 20:49, Pierre Bernhardt wrote: [SNIP] For running jobs only that the running job with same level or higher will be abortet the running. For queued job it is ok, but for running job it should not start a new at same level. Better want to wait or want to cancel. If there is a way that a higher job will wait for each running job has to been finished would be also welcome. So I can start a new full after a inc has to been canceled. But for queued jobs the behavior is exactly as expected and needed. So any idea what I must change to have the effect on running jobs without changing the queued behavior? Are your full/differential/incremental jobs running with different priorities? You haven't shown anything that sets them differently, there is no priority difference between full/differential/incremental jobs unless you set it. Cheers, GaryB-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
Am 03.04.20 um 04:58 schrieb Pierre Bernhardt: > I cannot see an error. By the way here my JobDefs configuration for my daily > jobs > which should work nearly as you expect: > > 34 JobDefs { > 35 Name = "RotateJob" > 36 Type = Backup > 37 Level = Incremental > 38 Schedule = "Cycle" > 39 Max Start Delay = 124 > 40 Max Wait Time = 360 > 41 Max Run Time = 360 > 42 Spool Data = yes > 43 Spool Attributes = yes > 44 Messages = Standard > 45 Pool = Daily > 46 Storage = "Disk2" > 47 Incremental Backup Pool = Daily > 48 Differential Backup Pool = Weekly > 49 Full Backup Pool = Monthly > 50 Rerun Failed Levels = yes > 51 Allow Duplicate Jobs = no > 52 Cancel lower level duplicates = yes > 53 Cancel Queued Duplicates = yes > 54 Cancel Running Duplicates = no > 55 Accurate = yes > 56 Priority = 9 > 57 Allow Mixed Priority = yes > 58 } > > By the way against my meaning I remember it cancel also lower level > running jobs (Inc if Diff is started) althoug I should first finish > the Inc and then start the diff job. Only if the Inc is waiting in > the queue it should be cancel. I must check it again later. I testet something: FULL job is running: Start of INC, DIFF and FULL will aborted immediately FULL job is queued: Start of INC will aborted immediately. Start of FULL will abort the waiting FULL and New job will be queued. DIFF job is running: Start of INC and DIFF will aborted immediately. Start of FULL will abort the DIFF and New FULL job will be queued. DIFF job is queued: Start of INC will aborted immediately. Start of DIFF and FULL will abort the DIFF and New job will be queued. INC job is running: Start of INC will aborted immediately. Start of DIFF and FULL will abort the INC and New FULL job will be queued INC job is queued: Start of INC, DIFF and FULL will abort the INC and New job will be queued. For running jobs only that the running job with same level or higher will be abortet the running. For queued job it is ok, but for running job it should not start a new at same level. Better want to wait or want to cancel. If there is a way that a higher job will wait for each running job has to been finished would be also welcome. So I can start a new full after a inc has to been canceled. But for queued jobs the behavior is exactly as expected and needed. So any idea what I must change to have the effect on running jobs without changing the queued behavior? Cheers, Pierre ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 03/04/2020 13:19, Gary R. Schmidt wrote: And the reason it doesn't get cancelled is because they are *not* identical jobs, one is "Differential" and the other is "Incremental." You need to run the Differential, Incremental, and Full jobs at different priorities - that reflect how you want things done - and set "Cancel Lower Level Duplicate = Yes" and "Allow Mixed Priority = Yes" in them all. Hi, Its an intriguing thought but the bacula 7 reference manual only mentions the name... "Allow Duplicate Jobs = A duplicate job in the sense we use it here means a second or sub- sequent job with the same name starts. This happens most frequently when the first job runs longer than expected because no tapes are available." ... it then goes on to explain options for lower or higher jobs without reference to priority... "Cancel Lower Level Duplicates = If Allow Duplicate Jobs is set to no and this directive is set to yes, Bacula will choose between duplicated jobs the one with the highest level. For example, it will cancel a previous Incremental to run a Full backup. It works only for Backup jobs. The default is no. If the levels of the duplicated jobs are the same, nothing is done and the other Cancel XXX Duplicate directives will be examined." I am pretty sure that the configuration files are ok. I have now copied the configuration to another system, and on that setup lower level duplicates are cancelled as expected rather than queued "max job jobs". Have to say debugging this peculiarity is baffling me... Rgs Bernie ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
Am 03.04.20 um 14:19 schrieb Gary R. Schmidt: > You need to run the Differential, Incremental, and Full jobs at different > priorities - that reflect how you want things done - and set "Cancel Lower > Level Duplicate = Yes" and "Allow Mixed Priority = Yes" in them all. Hi, that could not be really true because I did not use different priorities für different levels, but it is mostly working as expected and if an diff job is queued a new full will be cancel the diff, and the new inc will also be canceled. For the moment I'm migrate some tapes so next daysI have no time to test it again. Schedule { Name = "Cycle" Run = Level=Full 1st sun at 23:50 Run = Level=Differential 2nd-5th sun at 23:50 Run = Level=Incremental mon-sat at 23:50 } But I will recheck later. Cheers, Pierre ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 03/04/2020 19:38, Bernie Elbourn wrote: On 03/04/2020 03:58, Pierre Bernhardt wrote: It is exactly same job configured? A good thought but yes it is same simple job with no inherited defaults. Some extra detail this time using sed to reliably obfuscate :-) Running Jobs: Console connected at 29-Mar-20 12:36 JobId Type Level Files Bytes Name Status == 70172 Back Diff 120 36.70 M Backup-pc is running 70173 Back Incr 0 0 Backup-pc is waiting on max Job jobs And the reason it doesn't get cancelled is because they are *not* identical jobs, one is "Differential" and the other is "Incremental." You need to run the Differential, Incremental, and Full jobs at different priorities - that reflect how you want things done - and set "Cancel Lower Level Duplicate = Yes" and "Allow Mixed Priority = Yes" in them all. I have my schedule set that Full jobs run at 50, Differential Jobs run at 49, and Incremental Jobs run at 48, as some of my Full jobs last more than 24 hours. I have a several machines, and some of them cannot be fully backed-up in twenty-four hours. I run Differential and Full backups on Saturday mornings (at 00:01), but nor for *all* machines, and no backup on Sunday morning. All machines do an incremental backup from Monday to Friday. And I have a complicated schedule: Schedule { Name = "WeeklyOneNoSun" Run = Level=Full FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental 1st sat at 00:01 Run = Level=Differential FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental Priority=49 2nd-5th sat at 00:01 Run = Level=Incremental FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental Priority=48 mon-fri at 00:01 } Schedule { Name = "WeeklyTwoNoSun" Run = Level=Full FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental 2nd sat at 00:01 Run = Level=Differential FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental Priority=49 1st,3rd-5th sat at 00:01 Run = Level=Incremental FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental Priority=48 mon-fri at 00:01 } Schedule { Name = "WeeklyThreeNoSun" Run = Level=Full FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental 3rd sat at 00:01 Run = Level=Differential FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental Priority=49 1st,2nd,4th,5th sat at 00:01 Run = Level=Incremental FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental Priority=48 mon-fri at 00:01 } Schedule { Name = "WeeklyFourNoSun" Run = Level=Full FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental 4th sat at 00:01 Run = Level=Differential FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental Priority=49 1st-3rd,5th sat at 00:01 Run = Level=Incremental FullPool=Full DifferentialPool=Differential IncrementalPool=Incremental Priority=48 mon-fri at 00:01 } I hope this helps, and gives you some ideas on how to get your schedule to do what you want. :-) Cheers, GaryB-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 03/04/2020 03:58, Pierre Bernhardt wrote: It is exactly same job configured? A good thought but yes it is same simple job with no inherited defaults. Some extra detail this time using sed to reliably obfuscate :-) Running Jobs: Console connected at 29-Mar-20 12:36 JobId Type Level Files Bytes Name Status == 70172 Back Diff 120 36.70 M Backup-pc is running 70173 Back Incr 0 0 Backup-pc is waiting on max Job jobs Job { Name = Backup-pc Type = Backup Client = pc-fd FileSet = pc Schedule = Cycle Storage = pc Pool = pc Messages = Standard # Max Run Time = 3300 Allow Duplicate Jobs = no Cancel Queued Duplicates = yes Cancel Lower Level Duplicates = yes } Schedule { Name = Cycle Run = Level=Full 1st sun at 12:30 Run = Level=Differential 2nd-5th sun at 12:30 Run = Level=Incremental at 9:35 Run = Level=Incremental at 10:35 Run = Level=Incremental at 11:35 Run = Level=Incremental at 13:35 Run = Level=Incremental at 14:35 Run = Level=Incremental at 15:35 Run = Level=Incremental at 16:35 Run = Level=Incremental at 12:35 } My google fu has failed to find ways to debug this oddity .. Bernie ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
Am 01.04.20 um 17:41 schrieb Bernie Elbourn: > Oops... > > On 01/04/2020 16:10, Bernie Elbourn wrote: >> Hi, >> >> Oddly, jobs run sequentially as follows rather than duplicates being >> cancelled: >> >> Running Jobs: >> Console connected at 29-Mar-20 12:36 >> JobId Type Level Files Bytes Name Status >> == >> 70172 Back Diff12036.70 M Backup-pc is running >> 70173 Back Incr 0 0 Backup--pc is waiting on max Job jobs >> >> >> Are there any pointers to trace why the duplicate job 70173 is not >> cancelled? > > Obfuscation error both above should read Backup-pc - they were same name. It is exactly same job configured? I cannot see an error. By the way here my JobDefs configuration for my daily jobs which should work nearly as you expect: 34 JobDefs { 35 Name = "RotateJob" 36 Type = Backup 37 Level = Incremental 38 Schedule = "Cycle" 39 Max Start Delay = 124 40 Max Wait Time = 360 41 Max Run Time = 360 42 Spool Data = yes 43 Spool Attributes = yes 44 Messages = Standard 45 Pool = Daily 46 Storage = "Disk2" 47 Incremental Backup Pool = Daily 48 Differential Backup Pool = Weekly 49 Full Backup Pool = Monthly 50 Rerun Failed Levels = yes 51 Allow Duplicate Jobs = no 52 Cancel lower level duplicates = yes 53 Cancel Queued Duplicates = yes 54 Cancel Running Duplicates = no 55 Accurate = yes 56 Priority = 9 57 Allow Mixed Priority = yes 58 } By the way against my meaning I remember it cancel also lower level running jobs (Inc if Diff is started) althoug I should first finish the Inc and then start the diff job. Only if the Inc is waiting in the queue it should be cancel. I must check it again later. Cheers, Pierre ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
Oops... On 01/04/2020 16:10, Bernie Elbourn wrote: Hi, Oddly, jobs run sequentially as follows rather than duplicates being cancelled: Running Jobs: Console connected at 29-Mar-20 12:36 JobId Type Level Files Bytes Name Status == 70172 Back Diff 120 36.70 M Backup-pc is running 70173 Back Incr 0 0 Backup--pc is waiting on max Job jobs Are there any pointers to trace why the duplicate job 70173 is not cancelled? Obfuscation error both above should read Backup-pc - they were same name. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users