Re: [Bacula-users] Not Receiving Messages for Read-Only file system
On 10/17/2017 09:57 AM, Josh Fisher wrote: > Bill, this seems very likely related to the issue you are seeing when the > volume is not in-changer. The re-attempt of the mount in mount.c should > probably not be happening at 1 second intervals, or if it is, then it should > not log every attempt. Also, at some point it should be pausing the job and > waiting on operator intervention. So this definitely looks like a bug in > 9.0.4. Hi Josh, Just to be clear, it is not once per second, it is thousands of times per minute. :-/ Best regards, Bill -- Bill Arlofski http://www.revpol.com/bacula -- Not responsible for anything below this line -- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Not Receiving Messages for Read-Only file system
Bill, this seems very likely related to the issue you are seeing when the volume is not in-changer. The re-attempt of the mount in mount.c should probably not be happening at 1 second intervals, or if it is, then it should not log every attempt. Also, at some point it should be pausing the job and waiting on operator intervention. So this definitely looks like a bug in 9.0.4. On 10/4/2017 6:54 PM, Jim Richardson wrote: Bill, Thank you for your response. The command output you asked for is 183MB, it is the message from below about 7 times a second every second from 10:21:30 10/2/2017 until we caught it and canceled the job at 22:30:05 10/02/2017. I attached a file with the heading and ending information. The primary problem is that the job is waiting on media intervention, but does not produce an email for the warning. With the write protect on, it is just like the tape is never there. All jobs queue, and we end up with a huge mess. If we get a message, then we can remove the tape, remove the write protect, and then wait for all the jobs to finish. I would expect that the Read-Only volume warning from mount.c would be at a severity level to produce the same media intervention email message as "operator = al...@somewhere.com = mount". Let me know if that additional context helps with another solution. Jim Richardson CISSP CISA SecurIT360 -Original Message- From: Bill Arlofski [mailto:waa-bac...@revpol.com] Sent: Wednesday, October 4, 2017 11:42 AM To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Not Receiving Messages for Read-Only file system On 10/04/2017 09:35 AM, Jim Richardson wrote: All, We are using Bacula 9.0.4. We accidentally placed a tape in our library with the write-protect on. The following message was written in bconsole, but we didn’t receive notifications. 02-Oct 22:30 bacula-sd JobId 2801: Warning: mount.c:210 Open of Tape device "ULT3580" (/dev/nst0) Volume "05L7" failed: ERR=tape_dev.c:170 Unable to open device "ULT3580" (/dev/nst0): ERR=Read-only file system Below are all of our messages directives. Thank you in advance. Hi Jim, The job summary should have some non-zero number of "SD Errors:", and the Termination status should (probably) be "Backup OK -- with warnings" Take a look with: * llist joblog jobid=2801 Since it is only some "minor" issue during the backup, Bacula does not fail the job, it just logs the issue, and moves on. Because Bacula will set the JobStatus in the catalog the same as a "Backup OK" job (JobStatus=T), there is no way to trigger warnings by simply querying the catalog for JobStatus=W (for example), so I have all backup jobs send the emails to my helpdesk, which is configured to look for "Backup OK -- with warnings" and flag those jobs as 'bad' which need to be investigated. Hope this helps... Best regards, Bill -- Bill Arlofski http://www.revpol.com/bacula -- Not responsible for anything below this line -- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users CONFIDENTIALITY: This email (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this email in error, please notify the sender and delete this email from your system. Thank you. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Not Receiving Messages for Read-Only file system
Jim, I think I have (accidentally) reproduced the same issue but in a different way. http://bugs.bacula.org/view.php?id=2332 In my case, I had a zero byte file volume. Bacula noticed that there was no label on it, and kept looping and reloading this volume and did not abandon it until I marked it "Disabled" Best regards, Bill -- Bill Arlofski http://www.revpol.com/bacula -- Not responsible for anything below this line -- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Not Receiving Messages for Read-Only file system
On 10/05/2017 09:00 AM, Jim Richardson wrote: > Thanks Bill, is my action from here to submit the two bug reports? > > Jim Richardson > CISSP CISA Hi Jim, yeah, I think so. Especially since Kern is traveling we might as well get this one into the bugs database so it is not lost. ;) Best regards, Bill -- Bill Arlofski http://www.revpol.com/bacula -- Not responsible for anything below this line -- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Not Receiving Messages for Read-Only file system
Thanks Bill, is my action from here to submit the two bug reports? Jim Richardson CISSP CISA SecurIT360 -Original Message- From: Bill Arlofski [mailto:waa-bac...@revpol.com] Sent: Thursday, October 5, 2017 7:09 AM To: Jim Richardson ; bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Not Receiving Messages for Read-Only file system On 10/04/2017 04:54 PM, Jim Richardson wrote: > Bill, > > Thank you for your response. The command output you asked for is 183MB, it > is the message from below about 7 times a second every second from 10:21:30 > 10/2/2017 until we caught it and canceled the job at 22:30:05 10/02/2017. Ug... OK, now I understand. Yes, this surely seems to be a bug regarding some timing loop, along with the another possible bug where this media is never abandoned and another media is never selected once there is a "problem" with the first. > I attached a file with the heading and ending information. Yes, thanks... Nothing to see there. Even my theory about SD Errors being non-zero did not pan out. I did not expect the output was going to be so long, and was only interested in seeing the Summary blocks at the end. Sorry for the extra trouble. :) > The primary problem is that the job is waiting on media intervention, > but does not produce an email for the warning. With the write protect > on, it is just like the tape is never there. All jobs queue, and we > end up with a huge mess. I totally understand the issue now, but it really looks to be a new bug and it seems to resemble the one I have opened recently here: http://bugs.bacula.org/view.php?id=2329 > Let me know if that additional context helps with another solution. It helped me to understand what you were explaining, but I have no solution, sorry. I think this looks like two possible bug reports. One for the the issue where it logs so many times when it runs into this situation, then a feature request to automatically log a message and unload a tape and choose another when it detects that the one loaded is marked read-only - or send an operator email if it is not an autoloader. The SD Errors counter should probably be incremented in this case too... Best regards, Bill -- Bill Arlofski http://www.revpol.com/bacula -- Not responsible for anything below this line -- CONFIDENTIALITY: This email (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this email in error, please notify the sender and delete this email from your system. Thank you. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Not Receiving Messages for Read-Only file system
On 10/04/2017 04:54 PM, Jim Richardson wrote: > Bill, > > Thank you for your response. The command output you asked for is 183MB, it > is the message from below about 7 times a second every second from 10:21:30 > 10/2/2017 until we caught it and canceled the job at 22:30:05 10/02/2017. Ug... OK, now I understand. Yes, this surely seems to be a bug regarding some timing loop, along with the another possible bug where this media is never abandoned and another media is never selected once there is a "problem" with the first. > I attached a file with the heading and ending information. Yes, thanks... Nothing to see there. Even my theory about SD Errors being non-zero did not pan out. I did not expect the output was going to be so long, and was only interested in seeing the Summary blocks at the end. Sorry for the extra trouble. :) > The primary problem is that the job is waiting on media intervention, but > does not produce an email for the warning. With the write protect on, it > is just like the tape is never there. All jobs queue, and we end up > with a huge mess. I totally understand the issue now, but it really looks to be a new bug and it seems to resemble the one I have opened recently here: http://bugs.bacula.org/view.php?id=2329 > Let me know if that additional context helps with another solution. It helped me to understand what you were explaining, but I have no solution, sorry. I think this looks like two possible bug reports. One for the the issue where it logs so many times when it runs into this situation, then a feature request to automatically log a message and unload a tape and choose another when it detects that the one loaded is marked read-only - or send an operator email if it is not an autoloader. The SD Errors counter should probably be incremented in this case too... Best regards, Bill -- Bill Arlofski http://www.revpol.com/bacula -- Not responsible for anything below this line -- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Not Receiving Messages for Read-Only file system
Bill, Thank you for your response. The command output you asked for is 183MB, it is the message from below about 7 times a second every second from 10:21:30 10/2/2017 until we caught it and canceled the job at 22:30:05 10/02/2017. I attached a file with the heading and ending information. The primary problem is that the job is waiting on media intervention, but does not produce an email for the warning. With the write protect on, it is just like the tape is never there. All jobs queue, and we end up with a huge mess. If we get a message, then we can remove the tape, remove the write protect, and then wait for all the jobs to finish. I would expect that the Read-Only volume warning from mount.c would be at a severity level to produce the same media intervention email message as "operator = al...@somewhere.com = mount". Let me know if that additional context helps with another solution. Jim Richardson CISSP CISA SecurIT360 -Original Message- From: Bill Arlofski [mailto:waa-bac...@revpol.com] Sent: Wednesday, October 4, 2017 11:42 AM To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Not Receiving Messages for Read-Only file system On 10/04/2017 09:35 AM, Jim Richardson wrote: > All, > > > > We are using Bacula 9.0.4. We accidentally placed a tape in our > library with the write-protect on. The following message was written > in bconsole, but we didn’t receive notifications. > > > > 02-Oct 22:30 bacula-sd JobId 2801: Warning: mount.c:210 Open of Tape > device "ULT3580" (/dev/nst0) Volume "05L7" failed: > ERR=tape_dev.c:170 Unable to open device "ULT3580" (/dev/nst0): > ERR=Read-only file system > > > > Below are all of our messages directives. Thank you in advance. Hi Jim, The job summary should have some non-zero number of "SD Errors:", and the Termination status should (probably) be "Backup OK -- with warnings" Take a look with: * llist joblog jobid=2801 Since it is only some "minor" issue during the backup, Bacula does not fail the job, it just logs the issue, and moves on. Because Bacula will set the JobStatus in the catalog the same as a "Backup OK" job (JobStatus=T), there is no way to trigger warnings by simply querying the catalog for JobStatus=W (for example), so I have all backup jobs send the emails to my helpdesk, which is configured to look for "Backup OK -- with warnings" and flag those jobs as 'bad' which need to be investigated. Hope this helps... Best regards, Bill -- Bill Arlofski http://www.revpol.com/bacula -- Not responsible for anything below this line -- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users CONFIDENTIALITY: This email (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this email in error, please notify the sender and delete this email from your system. Thank you. onnecting to Director localhost:9101 1000 OK: 103 bacula-dir Version: 9.0.4 (06 September 2017) Enter a period to cancel a command. llist joblog jobid=2801 Automatically selected Catalog: MyCatalog Using Catalog "MyCatalog" time: 2017-10-02 03:05:18 logtext: bacula-dir JobId 2801: Start Backup JobId 2801, Job=C2T-DiskBackups.2017-10-02_03.00.00_57 time: 2017-10-02 03:05:18 logtext: bacula-sd JobId 2801: 3307 Issuing autochanger "unload Volume 08L7, Slot 3, Drive 0" command. time: 2017-10-02 03:05:42 logtext: bacula-dir JobId 2801: Using Device "ULT3580" to write. time: 2017-10-02 03:05:43 logtext: bacula-sd JobId 2801: 3304 Issuing autochanger "load Volume 04L7, Slot 9, Drive 0" command. time: 2017-10-02 03:06:15 logtext: bacula-sd JobId 2801: 3305 Autochanger "load Volume 04L7, Slot 9, Drive 0", status is OK. time: 2017-10-02 03:06:18 logtext: bacula-sd JobId 2801: Volume "04L7" previously written, moving to end of data. time: 2017-10-02 03:07:06 logtext: bacula-sd JobId 2801: Ready to append to end of Volume "04L7" at file=7184. time: 2017-10-02 04:26:37 logtext: bacula-sd JobId 2801: End of Volume "04L7" at 7696:15127 on device "ULT3580" (/dev/nst0). Write of 64512 bytes got -1. time: 2017-10-02 04:26:43 logtext: bacula-sd JobId 2801: Re-read of last block succeeded. time: 2017-10-02 04:26:43 logtext: bacula-sd JobId 2801: End of medium on Volume "04L7" Bytes=7,
Re: [Bacula-users] Not Receiving Messages for Read-Only file system
On 10/04/2017 09:35 AM, Jim Richardson wrote: > All, > > > > We are using Bacula 9.0.4. We accidentally placed a tape in our library with > the write-protect on. The following message was written in bconsole, but we > didn’t receive notifications. > > > > 02-Oct 22:30 bacula-sd JobId 2801: Warning: mount.c:210 Open of Tape device > "ULT3580" (/dev/nst0) Volume "05L7" failed: ERR=tape_dev.c:170 Unable to > open device "ULT3580" (/dev/nst0): ERR=Read-only file system > > > > Below are all of our messages directives. Thank you in advance. Hi Jim, The job summary should have some non-zero number of "SD Errors:", and the Termination status should (probably) be "Backup OK -- with warnings" Take a look with: * llist joblog jobid=2801 Since it is only some "minor" issue during the backup, Bacula does not fail the job, it just logs the issue, and moves on. Because Bacula will set the JobStatus in the catalog the same as a "Backup OK" job (JobStatus=T), there is no way to trigger warnings by simply querying the catalog for JobStatus=W (for example), so I have all backup jobs send the emails to my helpdesk, which is configured to look for "Backup OK -- with warnings" and flag those jobs as 'bad' which need to be investigated. Hope this helps... Best regards, Bill -- Bill Arlofski http://www.revpol.com/bacula -- Not responsible for anything below this line -- -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Not Receiving Messages for Read-Only file system
All, We are using Bacula 9.0.4. We accidentally placed a tape in our library with the write-protect on. The following message was written in bconsole, but we didn't receive notifications. 02-Oct 22:30 bacula-sd JobId 2801: Warning: mount.c:210 Open of Tape device "ULT3580" (/dev/nst0) Volume "05L7" failed: ERR=tape_dev.c:170 Unable to open device "ULT3580" (/dev/nst0): ERR=Read-only file system Below are all of our messages directives. Thank you in advance. Messages { Name = Standard mailcommand = "/usr/sbin/bsmtp -h smtp.mailserver.com -f \"\(Bacula\) \\" -s \"Bacula: %t %e of %c %n\" %r" operatorcommand = "/usr/sbin/bsmtp -h smtp.mailserver.com -f \"\(Bacula\) \\" -s \"Bacula: Intervention needed for %j %n\" %r" mail = al...@somewhere.com = all, !skipped operator = al...@somewhere.com = mount console = all, !skipped, !saved append = "/var/log/bacula/bacula.log" = all, !skipped catalog = all } Messages { Name = Daemon mailcommand = "/usr/sbin/bsmtp -h smtp.mailserver.com -f \"\(Bacula\) \\" -s \"Bacula daemon message\" %r" mail = al...@somewhere.com = all, !skipped console = all, !skipped, !saved append = "/var/log/bacula/bacula.log" = all, !skipped } Messages { Name = Messages_Verify mailcommand = "/usr/sbin/bsmtp -h smtp.mailserver.com -f \"\(Bacula\) \\" -s \"Bacula: %t %e of %c %n\" %r" operatorcommand = "/usr/sbin/bsmtp -h smtp.mailserver.com -f \"\(Bacula\) \\" -s \"Bacula: %t %e of %c %n\" %r" mail = al...@somewhere.com = all, !skipped operator = al...@somewhere.com = mount console = all, !skipped, !saved append = "/var/log/bacula/verify.log" = all, !skipped catalog = all, !skipped, !saved } Jim Richardson CONFIDENTIALITY: This email (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this email in error, please notify the sender and delete this email from your system. Thank you. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users