Jerry, I went to a DR test, where before hand myself and another old hand tried to tell some younger staff members make sure they inventory their tapes..well the didn't and forgot 14 of them
Scott ford www.identityforge.com On Jun 27, 2012, at 1:11 PM, Jerry Whitteridge <[email protected]> wrote: > I also use the residual time from our DR tests for "Destructive testing". > It's a great way to test procedures at no risk and do some training that > normally can't be done due to risk. > > Jerry Whitteridge > Lead Systems Programmer > Safeway Inc. > 925 951 4184 > > If you feel in control > you just aren't going fast enough. > > > -----Original Message----- > From: IBM Mainframe Discussion List [mailto:[email protected]] On > Behalf Of Tom Ambros > Sent: Wednesday, June 27, 2012 10:07 AM > To: [email protected] > Subject: Re: 'Inexperienced' RBS tech operative's blunder led to banking > meltdown > > We just happened to upgrade our scheduler earlier this year and the tech > doing it took advantage of a DR test to practice the production upgrade > and backout, he had done the backout in the sandbox but wanted to see how > long production would take. He happens to be the tech with the least > tenure as a tech but with many years of operations and batch support > experience... sort of how a lot of people used to end up in tech, I guess. > > > Thomas Ambros > Operating Systems and Connectivity Engineering > 518-436-6433 > > > > > > From: Jerry Whitteridge <[email protected]> > To: [email protected] > Date: 06/27/2012 12:48 > Subject: Re: 'Inexperienced' RBS tech operative's blunder led to > banking meltdown > Sent by: IBM Mainframe Discussion List <[email protected]> > > > > One relates to a conditioning PTF required on the back level system to be > able to backout an upgrade from a higher using a backup (DMPQ/MOVQ) > process native to CA-7. As I read it they attempted to backout using the > standard documented process but the downlevel system couldn't use the data > formatted for the uplevel system and queue corruption might have occurred. > > It's shutting the door after the event - but also points to a code not a > process problem. > > Jerry Whitteridge > Lead Systems Programmer > Safeway Inc. > 925 951 4184 > > If you feel in control > you just aren't going fast enough. > > > -----Original Message----- > From: IBM Mainframe Discussion List [mailto:[email protected]] On > Behalf Of Bill Ashton > Sent: Wednesday, June 27, 2012 9:39 AM > To: [email protected] > Subject: Re: 'Inexperienced' RBS tech operative's blunder led to banking > meltdown > > Without having seen the HIPERs, it could be that they are trying to keep > others from blowing off their feet if they have staff who are > "operationally challenged." > B > > On Wed, Jun 27, 2012 at 12:34 PM, Jerry Whitteridge < > [email protected]> wrote: > >> I've just seen 3 HIPER PTF's from CA related to CA-7 that also fit the >> reported information on the RBS outage. I'd not be too hasty in blaming >> staff at this point. >> >> Jerry Whitteridge >> Lead Systems Programmer >> Safeway Inc. >> 925 951 4184 >> >> If you feel in control >> you just aren't going fast enough. >> >> >> -----Original Message----- >> From: IBM Mainframe Discussion List [mailto:[email protected]] On >> Behalf Of Knutson, Sam >> Sent: Wednesday, June 27, 2012 6:49 AM >> To: [email protected] >> Subject: Re: 'Inexperienced' RBS tech operative's blunder led to banking >> meltdown >> >> To someone who is inexperienced on platform I imagine managing a CA-7 >> upgrade is like having the world's smartest dog but it only responds to >> commands in Latin. Miscommunication is likely to be a source of >> dissatisfaction. >> >> The average person <30 is not stupid because they haven't been editing >> command decks and JCL for 25 years or lazy but software on our platform >> tends to be very particular and sometimes unforgiving of mistakes i.e. >> failure to specify QUEUE=NOFORMAT or some non-obvious phrase as part of >> START command. Since business is no longer content to bring in newbies >> and let them spend 5-10 years to reach Journeyman status under the > guidance >> of more senior folks software is going to have to get smarter and more >> forgiving, documentation is going to have to be written with less >> assumptions about the expertise of the "systems programmer" and zNextGen >> (generic) is going to operate "our" mainframes weather we think they are >> ready or not. Some of those next generation folks are not going to be >> based in the home office anymore either whether we like that or not. >> >> The Register is fun reading sometimes but they rarely have enough detail >> to get real insight into what actually happened. So really anything we > say >> here is just idle speculation. Feels like Friday so that's my .02 > worth >> of idle speculation :-) >> >> Best Regards, >> >> Sam Knutson, GEICO >> System z Team Leader >> mailto:[email protected] >> (office) 301.986.3574 >> (cell) 301.996.1318 >> >> "Think big, act bold, start simple, grow fast..." >> >> >> -----Original Message----- >> From: IBM Mainframe Discussion List [mailto:[email protected]] On >> Behalf Of Tom Ambros >> Sent: Wednesday, June 27, 2012 8:25 AM >> To: [email protected] >> Subject: Re: 'Inexperienced' RBS tech operative's blunder led to banking >> meltdown >> >> "Complicated legacy mainframe system"? It's CA-7! If they think that's >> complicated they probably can't work a Mr. Coffee. >> >> Thomas Ambros >> Operating Systems and Connectivity Engineering >> 518-436-6433 >> >> >> >> >> >> From: Ed Gould <[email protected]> >> To: [email protected] >> Date: 06/27/2012 00:44 >> Subject: 'Inexperienced' RBS tech operative's blunder led to >> banking meltdown >> Sent by: IBM Mainframe Discussion List <[email protected]> >> >> >> >> >> > http://www.theregister.co.uk/2012/06/26/rbs_natwest_ca_technologies_outsourcing/ > >> >> >> >> ==================== >> This email/fax message is for the sole use of the intended >> recipient(s) and may contain confidential and privileged information. >> Any unauthorized review, use, disclosure or distribution of this >> email/fax is prohibited. If you are not the intended recipient, please >> destroy all paper and electronic copies of the original message. >> >> ---------------------------------------------------------------------- >> For IBM-MAIN subscribe / signoff / archive access instructions, >> send email to [email protected] with the message: INFO IBM-MAIN >> >> >> "Email Firewall" made the following annotations. >> >> > ------------------------------------------------------------------------------ >> >> Warning: >> All e-mail sent to this address will be received by the corporate e-mail >> system, and is subject to archival and review by someone other than the >> recipient. This e-mail may contain proprietary information and is > intended >> only for the use of the intended recipient(s). If the reader of this >> message is not the intended recipient(s), you are notified that you have >> received this message in error and that any review, dissemination, >> distribution or copying of this message is strictly prohibited. If you >> have received this message in error, please notify the sender > immediately. >> >> >> > ============================================================================== >> >> ---------------------------------------------------------------------- >> For IBM-MAIN subscribe / signoff / archive access instructions, >> send email to [email protected] with the message: INFO IBM-MAIN >> > > > > -- > Thank you and best regards, > *Billy Ashton* > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO IBM-MAIN > > > "Email Firewall" made the following annotations. > ------------------------------------------------------------------------------ > > Warning: > All e-mail sent to this address will be received by the corporate e-mail > system, and is subject to archival and review by someone other than the > recipient. This e-mail may contain proprietary information and is > intended only for the use of the intended recipient(s). If the reader of > this message is not the intended recipient(s), you are notified that you > have received this message in error and that any review, dissemination, > distribution or copying of this message is strictly prohibited. If you > have received this message in error, please notify the sender immediately. > > > ============================================================================== > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO IBM-MAIN > > This communication may contain privileged and/or confidential information. It > is intended solely for the use of the addressee. If you are not the intended > recipient, you are strictly prohibited from disclosing, copying, distributing > or using any of this information. If you received this communication in > error, please contact the sender immediately and destroy the material in its > entirety, whether electronic or hard copy. This communication may contain > nonpublic personal information about consumers subject to the restrictions of > the Gramm-Leach-Bliley Act. You may not directly or indirectly reuse or > redisclose such information for any purpose other than to provide the > services for which you are receiving the information. > > 127 Public Square, Cleveland, OH 44114 > If you prefer not to receive future e-mail offers for products or services > from Key > send an e-mail to mailto:[email protected] with 'No Promotional E-mails' in > the > SUBJECT line. > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO IBM-MAIN > > > "Email Firewall" made the following annotations. > ------------------------------------------------------------------------------ > > Warning: > All e-mail sent to this address will be received by the corporate e-mail > system, and is subject to archival and review by someone other than the > recipient. This e-mail may contain proprietary information and is intended > only for the use of the intended recipient(s). If the reader of this message > is not the intended recipient(s), you are notified that you have received > this message in error and that any review, dissemination, distribution or > copying of this message is strictly prohibited. If you have received this > message in error, please notify the sender immediately. > > ============================================================================== > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
