Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Bill Arlofski]

Hello everyone,

Sorry I am a few days late on this, but attached is a script called
"bacula_warn_on_zero-20171116.sh"

It takes one to three command line parameters:

USE:
./bacula_warn_on_zero-20171116.sh jobid [debug] [zero_incremental_ok]

A jobid is required, debug just logs everything to stdout, and
zero_incremental_ok allows Incremental backup jobs with zero files or zero
bytes to be ignored. A good example use case is a job I have that backs up a
directory of CDROM ISO files.   This does not change much so the nightly
incremental job for this often has zero files and zero bytes.

This script is probably best run from a RunScript {} (RunsWhen = after) in a
Job resource to alert admins as soon as a job is finished if it completed with
zero files or zero bytes. Of course it may also be run manually or from a cron
job.

This script ONLY considers jobs of type "Backup" which have completed "T" or
"W" - Bacula of course already warns about other backup job statuses, so I
ignore them all.

Currently, Full backup jobs that terminate "T" or "W" must not have zero files
or zero bytes, and this script will send a warning email in these cases.

Make sure to edit the system variables near the top.

Please let me know if this is useful, or even more importantly, please let me
know about any bugs or feature requests or enhancements. It will upload this
to my website shortly, but I wanted to get it to the list as soon as I could.

Best regards,

Bill

-- 
Bill Arlofski
http://www.revpol.com/bacula
-- Not responsible for anything below this line --


bacula_warn_on_zero-20171116.sh
Description: application/shellscript
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Martin Simmons]

> On Mon, 13 Nov 2017 08:33:26 -0700, Bill Arlofski said:
> 
> On 11/13/2017 03:49 AM, Martin Simmons wrote:
> 
> > Why not take this info from the job table, rather than having to parse it 
> > from
> > the joblog?
> > 
> > __Martin
> 
> 
> Hi Martin,
> 
> I usually try to avoid directly accessing the catalog if possible. In this
> case, what I am looking for is so simple to get from bconsole it is just one
> bconsole command, then a few trivial grep/cut/sed/tr statements.
> 
> Directly querying the db is usually straight forward with MySQL, but
> PostgreSQL has a slightly more more complex setup for permissions - which can
> cause people minor headaches. Also, because we have SQL "standards"™, I
> usually need to write 3 different queries so that the script will work with
> MySQL, MariaDB, and PostgreSQL (see www.revpol.com/baculasummaryemails for
> example)
> 
> Having said all that, I have still not 100% ruled out querying the catalog
> directly. :)   If I run into a feature idea that I want to implement which
> needs information not included in the job summary - but is available in the
> catalog - then I will probably just shift gears. :)

Yes, I see what you mean.  

You could try "llist jobid=" as way to access the catalog without SQL.

One reason for not reading the joblog is that it is optional (configured via
the Messages resource).

__Martin

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Bill Arlofski]

On 11/13/2017 03:49 AM, Martin Simmons wrote:

> Why not take this info from the job table, rather than having to parse it from
> the joblog?
> 
> __Martin


Hi Martin,

I usually try to avoid directly accessing the catalog if possible. In this
case, what I am looking for is so simple to get from bconsole it is just one
bconsole command, then a few trivial grep/cut/sed/tr statements.

Directly querying the db is usually straight forward with MySQL, but
PostgreSQL has a slightly more more complex setup for permissions - which can
cause people minor headaches. Also, because we have SQL "standards"™, I
usually need to write 3 different queries so that the script will work with
MySQL, MariaDB, and PostgreSQL (see www.revpol.com/baculasummaryemails for
example)

Having said all that, I have still not 100% ruled out querying the catalog
directly. :)   If I run into a feature idea that I want to implement which
needs information not included in the job summary - but is available in the
catalog - then I will probably just shift gears. :)

Thanks for the comment!

Bill


-- 
Bill Arlofski
http://www.revpol.com/bacula
-- Not responsible for anything below this line --

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Radosław Korzeniewski]

Hello,

>From Bacula point of view a zero files / zero bytes backup size is
perfectly fine unless no errors detected during backup (i.e. permission is
denied) which Bacula should show in the job log.

So, an extreme specific conditions and requirements, should be handled
outside Bacula, as it could confuse most users.

This is IMVHO, and as always YMMV.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Kern Sibbald]

Hello Alan,

No, nothing gets written to storage.  Yes, there are database entries 
written for the job.


Best regards,

Kern


On 11/13/2017 01:04 PM, Alan Brown wrote:

On 11/11/17 17:26, Kern Sibbald wrote:
Yes, that is one of the reasons why I am not very enthusiastic about 
it because it will require yet another directive :-(


Kern, please clarify:

If a backup is 0bytes/0files, does anything actually get written to 
storage (tape or disk), or is it just a database entry?


Thanks






On 11/11/2017 06:21 PM, Phil Stracchino wrote:

On 11/11/17 07:41, Kern Sibbald wrote:

Hello Jim,

You can submit it as a feature request, but I am not really 
inclined to
implement it, because it is a very special case, and it is very 
easy for
anyone to look at the number of files backed up that is produced in 
the

job report.

Best regards,


If implemented at all it should be implemented as an option, because
there are many kinds of possible admin jobs (and even incremental
backups) for which it is *perfectly normal* to complete with 0 files/0
bytes backed up.





-- 


Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users






--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Alan Brown]

On 11/11/17 17:26, Kern Sibbald wrote:
Yes, that is one of the reasons why I am not very enthusiastic about 
it because it will require yet another directive :-(


Kern, please clarify:

If a backup is 0bytes/0files, does anything actually get written to 
storage (tape or disk), or is it just a database entry?


Thanks






On 11/11/2017 06:21 PM, Phil Stracchino wrote:

On 11/11/17 07:41, Kern Sibbald wrote:

Hello Jim,

You can submit it as a feature request, but I am not really inclined to
implement it, because it is a very special case, and it is very easy 
for

anyone to look at the number of files backed up that is produced in the
job report.

Best regards,


If implemented at all it should be implemented as an option, because
there are many kinds of possible admin jobs (and even incremental
backups) for which it is *perfectly normal* to complete with 0 files/0
bytes backed up.





-- 


Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Martin Simmons]

> On Sun, 12 Nov 2017 10:43:47 -0700, Bill Arlofski said:
> 
> The idea:
> 
> - Call this script from a RunScript (RunsWhen = after) "Command ="
> - Pass the script the current jobid (%i) and a few other variables:
>   - allow_zero_files
>   - allow_zero_bytes
>   - zero_incremental_ok
> 
> So, my idea is that each backup job can call this script, the script calls
> bconsole and runs a "list joblog jobid=" then determines the FD
> (Files|Bytes) written, determines if the level is Full or Inc of Diff and then
> send a warning email if the files and/or bytes are zero and if they should not
> have been (for example, when the Level=full), or if the Level=inc|diff and
> zero files for these levels are not to be considered OK (zero_incremental_ok
> not set).

Why not take this info from the job table, rather than having to parse it from
the joblog?

__Martin

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Bill Arlofski]

Hi everyone,

Personally I think this is something that should be kept out the core code of
Bacula, but the topic interested me so last night I emailed Jim directly (off
list) and then wrote a script.

I have the script mostly completed which I will release shortly. I am just
working out the final logic test details. :)


The idea:

- Call this script from a RunScript (RunsWhen = after) "Command ="
- Pass the script the current jobid (%i) and a few other variables:
  - allow_zero_files
  - allow_zero_bytes
  - zero_incremental_ok

So, my idea is that each backup job can call this script, the script calls
bconsole and runs a "list joblog jobid=" then determines the FD
(Files|Bytes) written, determines if the level is Full or Inc of Diff and then
send a warning email if the files and/or bytes are zero and if they should not
have been (for example, when the Level=full), or if the Level=inc|diff and
zero files for these levels are not to be considered OK (zero_incremental_ok
not set).

I am of course accepting comments and idea/feature requests etc.

Hope this will be useful, and hope to release it in the coming day(s).

Best regards,

Bill

-- 
Bill Arlofski
http://www.revpol.com/bacula
-- Not responsible for anything below this line --

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Josip Deanovic]

On Saturday 2017-11-11 12:21:22 Phil Stracchino wrote:
> If implemented at all it should be implemented as an option, because
> there are many kinds of possible admin jobs (and even incremental
> backups) for which it is *perfectly normal* to complete with 0 files/0
> bytes backed up.

If implemented it would have to be implemented only for the jobs of
the type that could have number of files bigger than 0, like backup
and copy job for example.

Or it could be implemented for all the job types and leave the user
to decide whether to use it or not but I don't see the point to fail
the job because of the zero-files if the particular type of job is
always going to have zero files.

I believe that sometimes this feature could be useful if people are
sure that there has to be non-zero files on particular backup jobs
they have defined.
If they start to get false positives it's on them. :-)
Then again sometimes false positive is better than undetected incident.

Of course. the check could be done outside the bacula directly on the
database but then again we could say the same for many features
already implemented into bacula.

-- 
Josip Deanovic

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Jim Richardson]

I completely understand.  The argument from my team is why get notified unless 
it is actionable.  We do have normal zero byte zero file jobs with incremental 
jobs. Some jobs should never be zero/zero.  A job directive would be a 
reasonable option.

Thanks.

Jim Richardson

From: Kern Sibbald [mailto:k...@sibbald.com]
Sent: Saturday, November 11, 2017 6:42 AM
To: Jim Richardson <j...@securit360.com>; bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Feature Request - job complete status of with 
warnings on zero files zero bytes


Hello Jim,

You can submit it as a feature request, but I am not really inclined to 
implement it, because it is a very special case, and it is very easy for anyone 
to look at the number of files backed up that is produced in the job report.

Best regards,

Kern

On 11/09/2017 09:35 PM, Jim Richardson wrote:
Kern,

Any chance I can get a feature request in?  I would like to have all jobs that 
complete with zero files and / or zero bytes to complete with Backup OK – with 
warnings.  Currently these jobs complete with Backup OK.

Thank you

Jim Richardson

CONFIDENTIALITY: This email (including any attachments) may contain 
confidential, proprietary and privileged information, and unauthorized 
disclosure or use is prohibited. If you received this email in error, please 
notify the sender and delete this email from your system. Thank you.



--

Check out the vibrant tech community on one of the world's most

engaging tech sites, Slashdot.org! http://sdm.link/slashdot




___

Bacula-users mailing list

Bacula-users@lists.sourceforge.net<mailto:Bacula-users@lists.sourceforge.net>

https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Dan Langille]

> On Nov 9, 2017, at 3:35 PM, Jim Richardson  wrote:
> 
> 
> Any chance I can get a feature request in?  I would like to have all jobs 
> that complete with zero files and / or zero bytes to complete with Backup OK 
> – with warnings.  Currently these jobs complete with Backup OK. 
> 
>  


Instead, why not make add this check to your monitoring system?

I have this in use:

$ cat /usr/home/dan/bin/size-check-files
#!/bin/sh

BACKUPDIR="/usr/home/dan/backups"

# nothing to report
response=""

# all good
success=0

FILES="`/usr/bin/find ${BACKUPDIR} -size 0 | /usr/bin/grep -v /.NOBACKUP`"

# if any of these files are of size zero, bad.
for file in ${FILES}
do
  success=2
  response="$response ${file} has size zero. "
done

if [ $success -eq  0 ]
then
  echo OK. No zero-size files found.
else
  echo $response
fi

return $success

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Kern Sibbald]

Yes, that is one of the reasons why I am not very enthusiastic about it 
because it will require yet another directive :-(



On 11/11/2017 06:21 PM, Phil Stracchino wrote:

On 11/11/17 07:41, Kern Sibbald wrote:

Hello Jim,

You can submit it as a feature request, but I am not really inclined to
implement it, because it is a very special case, and it is very easy for
anyone to look at the number of files backed up that is produced in the
job report.

Best regards,


If implemented at all it should be implemented as an option, because
there are many kinds of possible admin jobs (and even incremental
backups) for which it is *perfectly normal* to complete with 0 files/0
bytes backed up.





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Phil Stracchino]

On 11/11/17 07:41, Kern Sibbald wrote:
> Hello Jim,
> 
> You can submit it as a feature request, but I am not really inclined to
> implement it, because it is a very special case, and it is very easy for
> anyone to look at the number of files backed up that is produced in the
> job report. 
> 
> Best regards,


If implemented at all it should be implemented as an option, because
there are many kinds of possible admin jobs (and even incremental
backups) for which it is *perfectly normal* to complete with 0 files/0
bytes backed up.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Feature Request - job complete status of with warnings on zero files zero bytes

[Kern Sibbald]

  
  
Hello Jim,
You can submit it as a feature request, but I am not really
  inclined to implement it, because it is a very special case, and
  it is very easy for anyone to look at the number of files backed
  up that is produced in the job report.  

Best regards,
Kern


On 11/09/2017 09:35 PM, Jim Richardson
  wrote:


  
  
  
  
Kern,
 
Any chance I can get a feature request in? 
  I would like to have all jobs that complete with zero files
  and / or zero bytes to complete with Backup OK – with
  warnings.  Currently these jobs complete with Backup OK. 
  
 
Thank you
 
Jim
  Richardson
 
  
  CONFIDENTIALITY: This
email (including any attachments) may contain confidential,
proprietary and privileged information, and unauthorized
disclosure or use is prohibited. If you received this email in
error, please notify the sender and delete this email from your
system. Thank you.
  
  
  
  --
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
  
  
  
  ___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users



  


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users