Re: [Veritas-bu] Priority w/ an External Scheduler?
Date: Fri, 27 Jul 2007 09:14:57 -0400 [...] But, aren't we getting away from the OP's question? I don't think Kate's really looking for advice about whether running an external scheduler is recommended or even acceptable, but if you DO run an external scheduler, are the policy priorities respected?? Summary: yes. :-) Man, was this thread as painful to sit through day after day as it was/is to try to catch up on? To Paul's Most Excellent point: The Policy Priority field rules. An immediate backup--_obviously_ what Kate meant since user backups aren't prioritized/run by NetBackup's scheduler in the first place--is, AFAIK, identical to a manual backup or one kicked off by the scheduler. Two scheduling functions apply: policy priority and schedule priority (from the frequency interval). But one test is worth a thousand expert opinions, so I did one. TEST-pri1, TEST-pri2 and TEST-pri3 are identical classes except that Policy Priority is 1, 2, 3 respectively. They back up to a test DSU which allows only one concurrent job. They have full/cinc/diff schedules, where the frequency of the full is 7 days and the others are one day. This is how I ran them to exercise the scheduling logic, if any (execution jobids appended for this writeup): # cat /tmp/testpri #!/bin/sh date cd /usr/openv/netbackup/bin ./bpbackup -i -p TEST-pri1 14829 ./bpbackup -i -p TEST-pri1 14830 ./bpbackup -i -p TEST-pri2 14831 ./bpbackup -i -p TEST-pri2 14832 ./bpbackup -i -p TEST-pri3 14833 ./bpbackup -i -p TEST-pri3 14834 # The first instance of TEST-pri1 (jobid 14829) ran, as the DSU was idle. In order, the five that the scheduler handled were run in priority order, then first-entered order, and the full schedule always won: 14833, 14834, 14831, 14832, 14830. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Priority w/ an External Scheduler?
But, aren't we getting away from the OP's question? I don't think Kate's really looking for advice about whether running an external scheduler is recommended or even acceptable, but if you DO run an external scheduler, are the policy priorities respected?? Paul -- -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Curtis Preston Sent: July 25, 2007 3:39 PM To: Steven L. Sesar Cc: veritas-bu@mailman.eng.auburn.edu; Greenberg, Katherine (ISD,IT) Subject: Re: [Veritas-bu] Priority w/ an External Scheduler? I have a different way of handling that requirement. --- W. Curtis Preston Backup Blog @ www.backupcentral.com VP Data Protection, GlassHouse Technologies From: Steven L. Sesar [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 25, 2007 12:31 PM To: Curtis Preston Cc: Greenberg, Katherine (ISD, IT); veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Priority w/ an External Scheduler? The only other justification I can think of for using an external scheduler, is, when there are batch processing operations against, say, Oracle databases, which need to run during your backup window. Using the same batch processing application, you can add a backup to the chain. This is a very specific situation - backups of Oracle databases and batch processing against said databases which need to occur during the same backup window. We toyed with the idea, but in the end, we pushed back on the apps folks to fix their apps, so chains would run quicker and run either before our backups begin, or after they complete. We are currently considering hardware-based snapshot capability and array-based replication, which will give our apps folks back some time for chains to run. --Steve Curtis Preston wrote: While I agree with Justin's response, I really don't like using external schedulers for NetBackup unless absolutely necessary. The only real requirement I can think of that would force me to an external scheduler is multi-host dependencies, because external schedulers are really good at that. I've seen the following justifications: 1. I want a backup job to run exactly at 3 AM. a. You can do that with NBU, good windows, and dedicated resources. 2. I want to tell all my jobs exactly when to run. a. Why? Stop over engineering your backup environment. Let go and let Symantec. ;) You'll be amazed at how well run a NBU environment can be if you just give all your backups the same window, the same resources, and just set priorities. It's a beautiful thing and requires VERY little maintenance. 2. An external scheduler is already hooked into our overall reporting mechanism. a. Lousy excuse to cripple the NBU scheduler. Spend the time necessary to hook NBU into your overall reporting mechanism. Short of the one valid reason I've seen (multi-host dependencies), you'll never approach the level of resource utilization and efficiency that the NetBackup scheduler can give you by using an external scheduler. For example, while NBU can have thousands of queued jobs waiting for resource (to make sure that something is always using the tape drives) without consuming any additional resources, doing that with an external scheduler is impossible. --- W. Curtis Preston Backup Blog @ www.backupcentral.com VP Data Protection, GlassHouse Technologies From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Greenberg, Katherine (ISD, IT) Sent: Wednesday, July 25, 2007 10:55 AM To: veritas-bu@mailman.eng.auburn.edu Subject: [Veritas-bu] Priority w/ an External Scheduler? Have you guys ever used an external scheduler? I'm trying to figure out if that impacts Policy Prioritization or not. Since our backups aren't scheduled thru NetBackup (there are schedules, but no windows) does prioritization come into play or no? We're running 5.1 MP6. TIA, Kate La version française suit le texte anglais. This email may contain privileged and/or confidential information, and the Bank of Canada does not waive any related rights. Any distribution, use, or copying of this email or the information it contains by other than the intended recipient is unauthorized. If you received
Re: [Veritas-bu] Priority w/ an External Scheduler?
Doesn't that depend on whether you use the -i option? If you want to pass the policy a file list you run bpbackup without the -i -Original Message- From: Greenberg, Katherine (ISD, IT) [mailto:[EMAIL PROTECTED] Sent: 25 July 2007 20:35 To: Shekel, Tal; Justin Piszcz Cc: veritas-bu@mailman.eng.auburn.edu Subject: RE: [Veritas-bu] Priority w/ an External Scheduler? They aren't user jobs, they are run with bpbackup -i. -Original Message- From: Shekel, Tal [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 25, 2007 3:26 PM To: Justin Piszcz; Greenberg, Katherine (ISD, IT) Cc: veritas-bu@mailman.eng.auburn.edu Subject: RE: [Veritas-bu] Priority w/ an External Scheduler? Yes - It will still work The jobs will run as user jobs which are prioritised Regards Tal -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Justin Piszcz Sent: 25 July 2007 19:10 To: Greenberg, Katherine (ISD, IT) Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Priority w/ an External Scheduler? On Wed, 25 Jul 2007, Greenberg, Katherine (ISD, IT) wrote: Have you guys ever used an external scheduler? I'm trying to figure out if that impacts Policy Prioritization or not. Since our backups aren't scheduled thru NetBackup (there are schedules, but no windows) does prioritization come into play or no? We're running 5.1 MP6. TIA, Kate * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * Policy Prioritization is based on the policy itself, so if you kickoff the policy externally it should still keep its priority state (0..999/priority number correct?) Justin. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu * The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way. The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing [EMAIL PROTECTED] and delete the message and any attachments without retaining any copies. Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses. No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates (COLT) and any other party by email Communications unless expressly agreed in writing with such other party. Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900. * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * * The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way. The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing [EMAIL PROTECTED] and delete the message and any attachments without retaining any copies. Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses. No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates (COLT) and any other party by email Communications unless expressly agreed in writing with such other party. Please note that incoming emails
Re: [Veritas-bu] Priority w/ an External Scheduler?
I have a different way of handling that requirement. --- W. Curtis Preston Backup Blog @ www.backupcentral.com VP Data Protection, GlassHouse Technologies From: Steven L. Sesar [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 25, 2007 12:31 PM To: Curtis Preston Cc: Greenberg, Katherine (ISD, IT); veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Priority w/ an External Scheduler? The only other justification I can think of for using an external scheduler, is, when there are batch processing operations against, say, Oracle databases, which need to run during your backup window. Using the same batch processing application, you can add a backup to the chain. This is a very specific situation - backups of Oracle databases and batch processing against said databases which need to occur during the same backup window. We toyed with the idea, but in the end, we pushed back on the apps folks to fix their apps, so chains would run quicker and run either before our backups begin, or after they complete. We are currently considering hardware-based snapshot capability and array-based replication, which will give our apps folks back some time for chains to run. --Steve Curtis Preston wrote: While I agree with Justin's response, I really don't like using external schedulers for NetBackup unless absolutely necessary. The only real requirement I can think of that would force me to an external scheduler is multi-host dependencies, because external schedulers are really good at that. I've seen the following justifications: 1. I want a backup job to run exactly at 3 AM. a. You can do that with NBU, good windows, and dedicated resources. 2. I want to tell all my jobs exactly when to run. a. Why? Stop over engineering your backup environment. Let go and let Symantec. ;) You'll be amazed at how well run a NBU environment can be if you just give all your backups the same window, the same resources, and just set priorities. It's a beautiful thing and requires VERY little maintenance. 2. An external scheduler is already hooked into our overall reporting mechanism. a. Lousy excuse to cripple the NBU scheduler. Spend the time necessary to hook NBU into your overall reporting mechanism. Short of the one valid reason I've seen (multi-host dependencies), you'll never approach the level of resource utilization and efficiency that the NetBackup scheduler can give you by using an external scheduler. For example, while NBU can have thousands of queued jobs waiting for resource (to make sure that something is always using the tape drives) without consuming any additional resources, doing that with an external scheduler is impossible. --- W. Curtis Preston Backup Blog @ www.backupcentral.com VP Data Protection, GlassHouse Technologies From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Greenberg, Katherine (ISD, IT) Sent: Wednesday, July 25, 2007 10:55 AM To: veritas-bu@mailman.eng.auburn.edu Subject: [Veritas-bu] Priority w/ an External Scheduler? Have you guys ever used an external scheduler? I'm trying to figure out if that impacts Policy Prioritization or not. Since our backups aren't scheduled thru NetBackup (there are schedules, but no windows) does prioritization come into play or no? We're running 5.1 MP6. TIA, Kate * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu -- === Steven L. Sesar Lead Operating Systems Programmer/Analyst UNIX Application Services R101 The MITRE Corporation 202 Burlington Road - MS K101 Bedford, MA 01730 tel: (781) 271-7702 fax: (781) 271-2600 mobile: (617) 519-8933 email: [EMAIL PROTECTED] === ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Priority w/ an External Scheduler?
Yes - It will still work The jobs will run as user jobs which are prioritised Regards Tal -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Justin Piszcz Sent: 25 July 2007 19:10 To: Greenberg, Katherine (ISD, IT) Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Priority w/ an External Scheduler? On Wed, 25 Jul 2007, Greenberg, Katherine (ISD, IT) wrote: Have you guys ever used an external scheduler? I'm trying to figure out if that impacts Policy Prioritization or not. Since our backups aren't scheduled thru NetBackup (there are schedules, but no windows) does prioritization come into play or no? We're running 5.1 MP6. TIA, Kate * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * Policy Prioritization is based on the policy itself, so if you kickoff the policy externally it should still keep its priority state (0..999/priority number correct?) Justin. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu * The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way. The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing [EMAIL PROTECTED] and delete the message and any attachments without retaining any copies. Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses. No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates (COLT) and any other party by email Communications unless expressly agreed in writing with such other party. Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
[Veritas-bu] Priority w/ an External Scheduler?
Have you guys ever used an external scheduler? I'm trying to figure out if that impacts Policy Prioritization or not. Since our backups aren't scheduled thru NetBackup (there are schedules, but no windows) does prioritization come into play or no? We're running 5.1 MP6. TIA, Kate * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Priority w/ an External Scheduler?
They aren't user jobs, they are run with bpbackup -i. -Original Message- From: Shekel, Tal [mailto:[EMAIL PROTECTED] Sent: Wednesday, July 25, 2007 3:26 PM To: Justin Piszcz; Greenberg, Katherine (ISD, IT) Cc: veritas-bu@mailman.eng.auburn.edu Subject: RE: [Veritas-bu] Priority w/ an External Scheduler? Yes - It will still work The jobs will run as user jobs which are prioritised Regards Tal -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Justin Piszcz Sent: 25 July 2007 19:10 To: Greenberg, Katherine (ISD, IT) Cc: veritas-bu@mailman.eng.auburn.edu Subject: Re: [Veritas-bu] Priority w/ an External Scheduler? On Wed, 25 Jul 2007, Greenberg, Katherine (ISD, IT) wrote: Have you guys ever used an external scheduler? I'm trying to figure out if that impacts Policy Prioritization or not. Since our backups aren't scheduled thru NetBackup (there are schedules, but no windows) does prioritization come into play or no? We're running 5.1 MP6. TIA, Kate * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * Policy Prioritization is based on the policy itself, so if you kickoff the policy externally it should still keep its priority state (0..999/priority number correct?) Justin. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu * The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way. The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing [EMAIL PROTECTED] and delete the message and any attachments without retaining any copies. Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses. No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates (COLT) and any other party by email Communications unless expressly agreed in writing with such other party. Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900. * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Priority w/ an External Scheduler?
On Wed, 25 Jul 2007, Greenberg, Katherine (ISD, IT) wrote: Have you guys ever used an external scheduler? I'm trying to figure out if that impacts Policy Prioritization or not. Since our backups aren't scheduled thru NetBackup (there are schedules, but no windows) does prioritization come into play or no? We're running 5.1 MP6. TIA, Kate * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * Policy Prioritization is based on the policy itself, so if you kickoff the policy externally it should still keep its priority state (0..999/priority number correct?) Justin. ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
Re: [Veritas-bu] Priority w/ an External Scheduler?
While I agree with Justin's response, I really don't like using external schedulers for NetBackup unless absolutely necessary. The only real requirement I can think of that would force me to an external scheduler is multi-host dependencies, because external schedulers are really good at that. I've seen the following justifications: 1. I want a backup job to run exactly at 3 AM. a. You can do that with NBU, good windows, and dedicated resources. 2. I want to tell all my jobs exactly when to run. a. Why? Stop over engineering your backup environment. Let go and let Symantec. ;) You'll be amazed at how well run a NBU environment can be if you just give all your backups the same window, the same resources, and just set priorities. It's a beautiful thing and requires VERY little maintenance. 3. An external scheduler is already hooked into our overall reporting mechanism. a. Lousy excuse to cripple the NBU scheduler. Spend the time necessary to hook NBU into your overall reporting mechanism. Short of the one valid reason I've seen (multi-host dependencies), you'll never approach the level of resource utilization and efficiency that the NetBackup scheduler can give you by using an external scheduler. For example, while NBU can have thousands of queued jobs waiting for resource (to make sure that something is always using the tape drives) without consuming any additional resources, doing that with an external scheduler is impossible. --- W. Curtis Preston Backup Blog @ www.backupcentral.com VP Data Protection, GlassHouse Technologies From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Greenberg, Katherine (ISD, IT) Sent: Wednesday, July 25, 2007 10:55 AM To: veritas-bu@mailman.eng.auburn.edu Subject: [Veritas-bu] Priority w/ an External Scheduler? Have you guys ever used an external scheduler? I'm trying to figure out if that impacts Policy Prioritization or not. Since our backups aren't scheduled thru NetBackup (there are schedules, but no windows) does prioritization come into play or no? We're running 5.1 MP6. TIA, Kate * This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies. * ___ Veritas-bu maillist - Veritas-bu@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
