Re: [Bacula-users] waiting on max Job jobs - not cancelled?
> On Thu, 28 May 2020 06:10:36 +0100, Bernie Elbourn said: > > On 01/04/2020 16:10, Bernie Elbourn wrote: > > Oddly, jobs run sequentially as follows rather than duplicates being > > cancelled: > > > > Running Jobs: > > Console connected at 29-Mar-20 12:36 > > JobId Type Level Files Bytes Name Status > > == > > 70172 Back Diff 120 36.70 M Backup-pc is running > > 70173 Back Incr 0 0 Backup-pc is waiting on max Job jobs > > > > > > Are there any pointers to trace why the duplicate job 70173 is not > > cancelled? > > So I have cloned the system (less backup volumes) to test system with a test > pc. That test system has exactly the same > database and same bacula sever and same pc setup. > > On the test system the second incremental job IS cancelled. The reason is > pretty clear > > 12-May 15:35 sv-dir JobId 71204: Fatal error: JobId 71202 already running. > Duplicate job not allowed. > > Is there any logging or tracing information available on an actual live > system that might reveal why the production > bacula system decides to wait on max job jobs? There is no useful logging for this, but the function that checks for duplicates is allow_duplicate_job in src/dird/job.c. You could either add some more logging to it or run the Director under gdb and step through it. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 01/04/2020 16:10, Bernie Elbourn wrote: Oddly, jobs run sequentially as follows rather than duplicates being cancelled: Running Jobs: Console connected at 29-Mar-20 12:36 JobId Type Level Files Bytes Name Status == 70172 Back Diff 120 36.70 M Backup-pc is running 70173 Back Incr 0 0 Backup-pc is waiting on max Job jobs Are there any pointers to trace why the duplicate job 70173 is not cancelled? So I have cloned the system (less backup volumes) to test system with a test pc. That test system has exactly the same database and same bacula sever and same pc setup. On the test system the second incremental job IS cancelled. The reason is pretty clear 12-May 15:35 sv-dir JobId 71204: Fatal error: JobId 71202 already running. Duplicate job not allowed. Is there any logging or tracing information available on an actual live system that might reveal why the production bacula system decides to wait on max job jobs? Thanks ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 19/04/2020 20:49, Pierre Bernhardt wrote: [SNIP] For running jobs only that the running job with same level or higher will be abortet the running. For queued job it is ok, but for running job it should not start a new at same level. Better want to wait or want to cancel. If there is a way that a higher job will wait for each running job has to been finished would be also welcome. So I can start a new full after a inc has to been canceled. But for queued jobs the behavior is exactly as expected and needed. So any idea what I must change to have the effect on running jobs without changing the queued behavior? Are your full/differential/incremental jobs running with different priorities? You haven't shown anything that sets them differently, there is no priority difference between full/differential/incremental jobs unless you set it. Cheers, GaryB-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
Am 03.04.20 um 04:58 schrieb Pierre Bernhardt:
> I cannot see an error. By the way here my JobDefs configuration for my daily
> jobs
> which should work nearly as you expect:
>
> 34 JobDefs {
> 35 Name = "RotateJob"
> 36 Type = Backup
> 37 Level = Incremental
> 38 Schedule = "Cycle"
> 39 Max Start Delay = 124
> 40 Max Wait Time = 360
> 41 Max Run Time = 360
> 42 Spool Data = yes
> 43 Spool Attributes = yes
> 44 Messages = Standard
> 45 Pool = Daily
> 46 Storage = "Disk2"
> 47 Incremental Backup Pool = Daily
> 48 Differential Backup Pool = Weekly
> 49 Full Backup Pool = Monthly
> 50 Rerun Failed Levels = yes
> 51 Allow Duplicate Jobs = no
> 52 Cancel lower level duplicates = yes
> 53 Cancel Queued Duplicates = yes
> 54 Cancel Running Duplicates = no
> 55 Accurate = yes
> 56 Priority = 9
> 57 Allow Mixed Priority = yes
> 58 }
>
> By the way against my meaning I remember it cancel also lower level
> running jobs (Inc if Diff is started) althoug I should first finish
> the Inc and then start the diff job. Only if the Inc is waiting in
> the queue it should be cancel. I must check it again later.
I testet something:
FULL job is running: Start of INC, DIFF and FULL will aborted immediately
FULL job is queued: Start of INC will aborted immediately. Start of FULL will
abort the
waiting FULL and New job will be queued.
DIFF job is running: Start of INC and DIFF will aborted immediately. Start of
FULL will abort the
DIFF and New FULL job will be queued.
DIFF job is queued: Start of INC will aborted immediately. Start of DIFF and
FULL will abort the
DIFF and New job will be queued.
INC job is running: Start of INC will aborted immediately. Start of DIFF and
FULL will abort the
INC and New FULL job will be queued
INC job is queued: Start of INC, DIFF and FULL will abort the INC and New job
will be queued.
For running jobs only that the running job with same level or higher will be
abortet the running.
For queued job it is ok, but for running job it should not start a new at same
level.
Better want to wait or want to cancel.
If there is a way that a higher job will wait for each running job has to been
finished
would be also welcome. So I can start a new full after a inc has to been
canceled.
But for queued jobs the behavior is exactly as expected and needed.
So any idea what I must change to have the effect on running jobs without
changing the
queued behavior?
Cheers,
Pierre
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 03/04/2020 13:19, Gary R. Schmidt wrote: And the reason it doesn't get cancelled is because they are *not* identical jobs, one is "Differential" and the other is "Incremental." You need to run the Differential, Incremental, and Full jobs at different priorities - that reflect how you want things done - and set "Cancel Lower Level Duplicate = Yes" and "Allow Mixed Priority = Yes" in them all. Hi, Its an intriguing thought but the bacula 7 reference manual only mentions the name... "Allow Duplicate Jobs = A duplicate job in the sense we use it here means a second or sub- sequent job with the same name starts. This happens most frequently when the first job runs longer than expected because no tapes are available." ... it then goes on to explain options for lower or higher jobs without reference to priority... "Cancel Lower Level Duplicates = If Allow Duplicate Jobs is set to no and this directive is set to yes, Bacula will choose between duplicated jobs the one with the highest level. For example, it will cancel a previous Incremental to run a Full backup. It works only for Backup jobs. The default is no. If the levels of the duplicated jobs are the same, nothing is done and the other Cancel XXX Duplicate directives will be examined." I am pretty sure that the configuration files are ok. I have now copied the configuration to another system, and on that setup lower level duplicates are cancelled as expected rather than queued "max job jobs". Have to say debugging this peculiarity is baffling me... Rgs Bernie ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
Am 03.04.20 um 14:19 schrieb Gary R. Schmidt:
> You need to run the Differential, Incremental, and Full jobs at different
> priorities - that reflect how you want things done - and set "Cancel Lower
> Level Duplicate = Yes" and "Allow Mixed Priority = Yes" in them all.
Hi,
that could not be really true because I did not use different priorities für
different levels,
but it is mostly working as expected and if an diff job is queued a new full
will be cancel
the diff, and the new inc will also be canceled.
For the moment I'm migrate some tapes so next daysI have no time to test it
again.
Schedule {
Name = "Cycle"
Run = Level=Full 1st sun at 23:50
Run = Level=Differential 2nd-5th sun at 23:50
Run = Level=Incremental mon-sat at 23:50
}
But I will recheck later.
Cheers,
Pierre
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 03/04/2020 19:38, Bernie Elbourn wrote:
On 03/04/2020 03:58, Pierre Bernhardt wrote:
It is exactly same job configured?
A good thought but yes it is same simple job with no inherited defaults.
Some extra detail this time using sed to reliably obfuscate :-)
Running Jobs:
Console connected at 29-Mar-20 12:36
JobId Type Level Files Bytes Name Status
==
70172 Back Diff 120 36.70 M Backup-pc is running
70173 Back Incr 0 0 Backup-pc is waiting on max Job
jobs
And the reason it doesn't get cancelled is because they are *not*
identical jobs, one is "Differential" and the other is "Incremental."
You need to run the Differential, Incremental, and Full jobs at
different priorities - that reflect how you want things done - and set
"Cancel Lower Level Duplicate = Yes" and "Allow Mixed Priority = Yes" in
them all.
I have my schedule set that Full jobs run at 50, Differential Jobs run
at 49, and Incremental Jobs run at 48, as some of my Full jobs last more
than 24 hours.
I have a several machines, and some of them cannot be fully backed-up in
twenty-four hours.
I run Differential and Full backups on Saturday mornings (at 00:01), but
nor for *all* machines, and no backup on Sunday morning. All machines
do an incremental backup from Monday to Friday.
And I have a complicated schedule:
Schedule {
Name = "WeeklyOneNoSun"
Run = Level=Full FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental 1st sat at 00:01
Run = Level=Differential FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental Priority=49 2nd-5th sat at 00:01
Run = Level=Incremental FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental Priority=48 mon-fri at 00:01
}
Schedule {
Name = "WeeklyTwoNoSun"
Run = Level=Full FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental 2nd sat at 00:01
Run = Level=Differential FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental Priority=49 1st,3rd-5th sat at 00:01
Run = Level=Incremental FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental Priority=48 mon-fri at 00:01
}
Schedule {
Name = "WeeklyThreeNoSun"
Run = Level=Full FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental 3rd sat at 00:01
Run = Level=Differential FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental Priority=49 1st,2nd,4th,5th sat at 00:01
Run = Level=Incremental FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental Priority=48 mon-fri at 00:01
}
Schedule {
Name = "WeeklyFourNoSun"
Run = Level=Full FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental 4th sat at 00:01
Run = Level=Differential FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental Priority=49 1st-3rd,5th sat at 00:01
Run = Level=Incremental FullPool=Full DifferentialPool=Differential
IncrementalPool=Incremental Priority=48 mon-fri at 00:01
}
I hope this helps, and gives you some ideas on how to get your schedule
to do what you want. :-)
Cheers,
GaryB-)
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
On 03/04/2020 03:58, Pierre Bernhardt wrote:
It is exactly same job configured?
A good thought but yes it is same simple job with no inherited defaults.
Some extra detail this time using sed to reliably obfuscate :-)
Running Jobs:
Console connected at 29-Mar-20 12:36
JobId Type Level Files Bytes Name Status
==
70172 Back Diff 120 36.70 M Backup-pc is running
70173 Back Incr 0 0 Backup-pc is waiting on max Job jobs
Job {
Name = Backup-pc
Type = Backup
Client = pc-fd
FileSet = pc
Schedule = Cycle
Storage = pc
Pool = pc
Messages = Standard
# Max Run Time = 3300
Allow Duplicate Jobs = no
Cancel Queued Duplicates = yes
Cancel Lower Level Duplicates = yes
}
Schedule {
Name = Cycle
Run = Level=Full 1st sun at 12:30
Run = Level=Differential 2nd-5th sun at 12:30
Run = Level=Incremental at 9:35
Run = Level=Incremental at 10:35
Run = Level=Incremental at 11:35
Run = Level=Incremental at 13:35
Run = Level=Incremental at 14:35
Run = Level=Incremental at 15:35
Run = Level=Incremental at 16:35
Run = Level=Incremental at 12:35
}
My google fu has failed to find ways to debug this oddity ..
Bernie
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
Am 01.04.20 um 17:41 schrieb Bernie Elbourn:
> Oops...
>
> On 01/04/2020 16:10, Bernie Elbourn wrote:
>> Hi,
>>
>> Oddly, jobs run sequentially as follows rather than duplicates being
>> cancelled:
>>
>> Running Jobs:
>> Console connected at 29-Mar-20 12:36
>> JobId Type Level Files Bytes Name Status
>> ==
>> 70172 Back Diff12036.70 M Backup-pc is running
>> 70173 Back Incr 0 0 Backup--pc is waiting on max Job jobs
>>
>>
>> Are there any pointers to trace why the duplicate job 70173 is not
>> cancelled?
>
> Obfuscation error both above should read Backup-pc - they were same name.
It is exactly same job configured?
I cannot see an error. By the way here my JobDefs configuration for my daily
jobs
which should work nearly as you expect:
34 JobDefs {
35 Name = "RotateJob"
36 Type = Backup
37 Level = Incremental
38 Schedule = "Cycle"
39 Max Start Delay = 124
40 Max Wait Time = 360
41 Max Run Time = 360
42 Spool Data = yes
43 Spool Attributes = yes
44 Messages = Standard
45 Pool = Daily
46 Storage = "Disk2"
47 Incremental Backup Pool = Daily
48 Differential Backup Pool = Weekly
49 Full Backup Pool = Monthly
50 Rerun Failed Levels = yes
51 Allow Duplicate Jobs = no
52 Cancel lower level duplicates = yes
53 Cancel Queued Duplicates = yes
54 Cancel Running Duplicates = no
55 Accurate = yes
56 Priority = 9
57 Allow Mixed Priority = yes
58 }
By the way against my meaning I remember it cancel also lower level
running jobs (Inc if Diff is started) althoug I should first finish
the Inc and then start the diff job. Only if the Inc is waiting in
the queue it should be cancel. I must check it again later.
Cheers,
Pierre
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] waiting on max Job jobs - not cancelled?
Oops... On 01/04/2020 16:10, Bernie Elbourn wrote: Hi, Oddly, jobs run sequentially as follows rather than duplicates being cancelled: Running Jobs: Console connected at 29-Mar-20 12:36 JobId Type Level Files Bytes Name Status == 70172 Back Diff 120 36.70 M Backup-pc is running 70173 Back Incr 0 0 Backup--pc is waiting on max Job jobs Are there any pointers to trace why the duplicate job 70173 is not cancelled? Obfuscation error both above should read Backup-pc - they were same name. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
