We just happened to upgrade our scheduler earlier this year and the tech doing it took advantage of a DR test to practice the production upgrade and backout, he had done the backout in the sandbox but wanted to see how long production would take. He happens to be the tech with the least tenure as a tech but with many years of operations and batch support experience... sort of how a lot of people used to end up in tech, I guess.
Thomas Ambros Operating Systems and Connectivity Engineering 518-436-6433 From: Jerry Whitteridge <jerry.whitteri...@safeway.com> To: IBM-MAIN@LISTSERV.UA.EDU Date: 06/27/2012 12:48 Subject: Re: 'Inexperienced' RBS tech operative's blunder led to banking meltdown Sent by: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> One relates to a conditioning PTF required on the back level system to be able to backout an upgrade from a higher using a backup (DMPQ/MOVQ) process native to CA-7. As I read it they attempted to backout using the standard documented process but the downlevel system couldn't use the data formatted for the uplevel system and queue corruption might have occurred. It's shutting the door after the event - but also points to a code not a process problem. Jerry Whitteridge Lead Systems Programmer Safeway Inc. 925 951 4184 If you feel in control you just aren't going fast enough. -----Original Message----- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Bill Ashton Sent: Wednesday, June 27, 2012 9:39 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: 'Inexperienced' RBS tech operative's blunder led to banking meltdown Without having seen the HIPERs, it could be that they are trying to keep others from blowing off their feet if they have staff who are "operationally challenged." B On Wed, Jun 27, 2012 at 12:34 PM, Jerry Whitteridge < jerry.whitteri...@safeway.com> wrote: > I've just seen 3 HIPER PTF's from CA related to CA-7 that also fit the > reported information on the RBS outage. I'd not be too hasty in blaming > staff at this point. > > Jerry Whitteridge > Lead Systems Programmer > Safeway Inc. > 925 951 4184 > > If you feel in control > you just aren't going fast enough. > > > -----Original Message----- > From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On > Behalf Of Knutson, Sam > Sent: Wednesday, June 27, 2012 6:49 AM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Re: 'Inexperienced' RBS tech operative's blunder led to banking > meltdown > > To someone who is inexperienced on platform I imagine managing a CA-7 > upgrade is like having the world's smartest dog but it only responds to > commands in Latin. Miscommunication is likely to be a source of > dissatisfaction. > > The average person <30 is not stupid because they haven't been editing > command decks and JCL for 25 years or lazy but software on our platform > tends to be very particular and sometimes unforgiving of mistakes i.e. > failure to specify QUEUE=NOFORMAT or some non-obvious phrase as part of > START command. Since business is no longer content to bring in newbies > and let them spend 5-10 years to reach Journeyman status under the guidance > of more senior folks software is going to have to get smarter and more > forgiving, documentation is going to have to be written with less > assumptions about the expertise of the "systems programmer" and zNextGen > (generic) is going to operate "our" mainframes weather we think they are > ready or not. Some of those next generation folks are not going to be > based in the home office anymore either whether we like that or not. > > The Register is fun reading sometimes but they rarely have enough detail > to get real insight into what actually happened. So really anything we say > here is just idle speculation. Feels like Friday so that's my .02 worth > of idle speculation :-) > > Best Regards, > > Sam Knutson, GEICO > System z Team Leader > mailto:sknut...@geico.com > (office) 301.986.3574 > (cell) 301.996.1318 > > "Think big, act bold, start simple, grow fast..." > > > -----Original Message----- > From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On > Behalf Of Tom Ambros > Sent: Wednesday, June 27, 2012 8:25 AM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Re: 'Inexperienced' RBS tech operative's blunder led to banking > meltdown > > "Complicated legacy mainframe system"? It's CA-7! If they think that's > complicated they probably can't work a Mr. Coffee. > > Thomas Ambros > Operating Systems and Connectivity Engineering > 518-436-6433 > > > > > > From: Ed Gould <edgould1...@comcast.net> > To: IBM-MAIN@LISTSERV.UA.EDU > Date: 06/27/2012 00:44 > Subject: 'Inexperienced' RBS tech operative's blunder led to > banking meltdown > Sent by: IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> > > > > > http://www.theregister.co.uk/2012/06/26/rbs_natwest_ca_technologies_outsourcing/ > > > > ==================== > This email/fax message is for the sole use of the intended > recipient(s) and may contain confidential and privileged information. > Any unauthorized review, use, disclosure or distribution of this > email/fax is prohibited. If you are not the intended recipient, please > destroy all paper and electronic copies of the original message. > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > > > "Email Firewall" made the following annotations. > > ------------------------------------------------------------------------------ > > Warning: > All e-mail sent to this address will be received by the corporate e-mail > system, and is subject to archival and review by someone other than the > recipient. This e-mail may contain proprietary information and is intended > only for the use of the intended recipient(s). If the reader of this > message is not the intended recipient(s), you are notified that you have > received this message in error and that any review, dissemination, > distribution or copying of this message is strictly prohibited. If you > have received this message in error, please notify the sender immediately. > > > ============================================================================== > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > -- Thank you and best regards, *Billy Ashton* ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN "Email Firewall" made the following annotations. ------------------------------------------------------------------------------ Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately. ============================================================================== ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN This communication may contain privileged and/or confidential information. It is intended solely for the use of the addressee. If you are not the intended recipient, you are strictly prohibited from disclosing, copying, distributing or using any of this information. If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act. You may not directly or indirectly reuse or redisclose such information for any purpose other than to provide the services for which you are receiving the information. 127 Public Square, Cleveland, OH 44114 If you prefer not to receive future e-mail offers for products or services from Key send an e-mail to mailto:dnereque...@key.com with 'No Promotional E-mails' in the SUBJECT line. ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
