Re: I would love to know what went wrong at NAB
Chris, If I remember rightly it was a bug in IMS 2.2 or 2.3. If I remember correctly NAB (where I worked at the time) had found the bug in stress and regression testing (TPNS for those that remember it) and were waiting for the fix that hit Westpac. Funny how times have changed. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Chris Craddock Sent: Tuesday, November 30, 2010 8:56 PM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] I would love to know what went wrong at NAB On Tue, Nov 30, 2010 at 10:39 PM, Chase, John jch...@ussco.com wrote: -Original Message- From: IBM Mainframe Discussion List On Behalf Of Shane As if. Can't you just imagine a major Aussie Bank doing that. You were at Bank of NSW when they had the IMS fiasco Steve - how much info on that got out ? (via Bank press releases I mean :-) Yeh It's probably a near-universal trait that dirty laundry is washed discreetly :-) This would be one of those rare cases where the story was big enough that there wasn't a lot of room for discrete laundering. It was actually Westpac by then (btw) and I was there too. That event wasn't a single outage, but rather a series of them wound around failed restart/recovery processing that eventually took several days to fully recover from. There wasn't any news coverage at first, but the scale of the problem had made the press by the second day. Once it had there was plenty of blame storming to go around. I don't recall whether the bank actually issued press releases but their point of view certainly did make it into the press coverage. -- This email might be from the artist formerly known as CC (or not) You be the judge. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
Wayne, Hmm I seem to remember a past-tapes-processed file that prevented CEMTEX and other exchange files, value and non-value being processed twice in the File Exchange part of NAB's batch. The tape was just a holdover from an old naming convention, but it used to prevent files from being processed twice in exactly the way you describe. I wonder what happened to that part of the process? Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Wayne Bickerdike Sent: Tuesday, November 30, 2010 5:01 PM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] I would love to know what went wrong at NAB Some information from a close source... It was a plain old S0C7 during their batch process. All Aussie banks use Cemtex ABA format which has been around for years as a transfer format between organisations. You would think that there is a validate step before running the transactions against their databases. A couple of things: Most of the support is now with the sub-continent... The previous grey-haired support staff were either laid off or moved to greener pastures Not many local staff are competent to manually reprocess or react to this situation. As a previous poster noted, probably the new transaction batch goes to a GDG, the failed process was missed and batch read yesterday's GDG and hence all the double transactions. My same source related the tale of a person in same bank who received an EBCDIC file, opened it in Windows and saved it back as ASCII. The file was duly transferred for processing on the mainframe. Most of the packed data was garbaged .. Oh my, dumb and dumber On Wed, Dec 1, 2010 at 11:19 AM, Shane ibm-m...@tpg.com.au wrote: As if. Can't you just imagine a major Aussie Bank doing that. You were at Bank of NSW when they had the IMS fiasco Steve - how much info on that got out ? (via Bank press releases I mean :-) Shane ... On Tue, 30 Nov 2010 18:52:08 +1100 Stephen Mednick wrote: One wonders if a detailed explanation of what transpired will be forthcoming as was the case back in July when the DBS Bank in Singapore had a major outage. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- Wayne V. Bickerdike -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
Nostalgia, don't you just love it. Stephen Mednick Computer Supervisory Services Sydney, Australia Asia/Pacific representatives for Innovation Data Processing, Inc. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Ron Hawkins Sent: Wednesday, 1 December 2010 7:12 PM To: IBM-MAIN@bama.ua.edu Subject: Re: I would love to know what went wrong at NAB Chris, If I remember rightly it was a bug in IMS 2.2 or 2.3. If I remember correctly NAB (where I worked at the time) had found the bug in stress and regression testing (TPNS for those that remember it) and were waiting for the fix that hit Westpac. Funny how times have changed. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Chris Craddock Sent: Tuesday, November 30, 2010 8:56 PM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] I would love to know what went wrong at NAB On Tue, Nov 30, 2010 at 10:39 PM, Chase, John jch...@ussco.com wrote: -Original Message- From: IBM Mainframe Discussion List On Behalf Of Shane As if. Can't you just imagine a major Aussie Bank doing that. You were at Bank of NSW when they had the IMS fiasco Steve - how much info on that got out ? (via Bank press releases I mean :-) Yeh It's probably a near-universal trait that dirty laundry is washed discreetly :-) This would be one of those rare cases where the story was big enough that there wasn't a lot of room for discrete laundering. It was actually Westpac by then (btw) and I was there too. That event wasn't a single outage, but rather a series of them wound around failed restart/recovery processing that eventually took several days to fully recover from. There wasn't any news coverage at first, but the scale of the problem had made the press by the second day. Once it had there was plenty of blame storming to go around. I don't recall whether the bank actually issued press releases but their point of view certainly did make it into the press coverage. -- This email might be from the artist formerly known as CC (or not) You be the judge. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
Steve, Actually it is sort of disappointing that some of the practices that made this stuff bullet proof are being pigeon holed as legacy systems and removed from the process. A bit like throwing the baby out with the bath water. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Stephen Mednick Sent: Wednesday, December 01, 2010 12:19 AM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] I would love to know what went wrong at NAB Nostalgia, don't you just love it. Stephen Mednick Computer Supervisory Services Sydney, Australia Asia/Pacific representatives for Innovation Data Processing, Inc. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Ron Hawkins Sent: Wednesday, 1 December 2010 7:12 PM To: IBM-MAIN@bama.ua.edu Subject: Re: I would love to know what went wrong at NAB Chris, If I remember rightly it was a bug in IMS 2.2 or 2.3. If I remember correctly NAB (where I worked at the time) had found the bug in stress and regression testing (TPNS for those that remember it) and were waiting for the fix that hit Westpac. Funny how times have changed. Ron -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Chris Craddock Sent: Tuesday, November 30, 2010 8:56 PM To: IBM-MAIN@bama.ua.edu Subject: Re: [IBM-MAIN] I would love to know what went wrong at NAB On Tue, Nov 30, 2010 at 10:39 PM, Chase, John jch...@ussco.com wrote: -Original Message- From: IBM Mainframe Discussion List On Behalf Of Shane As if. Can't you just imagine a major Aussie Bank doing that. You were at Bank of NSW when they had the IMS fiasco Steve - how much info on that got out ? (via Bank press releases I mean :-) Yeh It's probably a near-universal trait that dirty laundry is washed discreetly :-) This would be one of those rare cases where the story was big enough that there wasn't a lot of room for discrete laundering. It was actually Westpac by then (btw) and I was there too. That event wasn't a single outage, but rather a series of them wound around failed restart/recovery processing that eventually took several days to fully recover from. There wasn't any news coverage at first, but the scale of the problem had made the press by the second day. Once it had there was plenty of blame storming to go around. I don't recall whether the bank actually issued press releases but their point of view certainly did make it into the press coverage. -- This email might be from the artist formerly known as CC (or not) You be the judge. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
On 11/30/2010 10:34 PM, Jim Phoenix wrote: Mike Schwab wrote: On Tue, Nov 30, 2010 at 7:01 PM, Wayne Bickerdike wayn...@gmail.com wrote: deleted My same source related the tale of a person in same bank who received an EBCDIC file, opened it in Windows and saved it back as ASCII. The file was duly transferred for processing on the mainframe. Most of the packed data was garbaged .. Oh my, dumb and dumber Here is an idea to bounce around. z/OS Unix System Services does a lot of work converting ASCII to EBCDIC and back. z/Linux works all in ASCII. Why not get 4 new instructions that work with PD= ASCII like the PD = EBCDIC instructions PACK, UNPK, ED, EDMK, but with an A suffix to denote ASCII character. Conversion from Packed to binary would be the same. Assembler would get new instructions. z/OS would need to know if a file was ASCII for proper translation when printing it. Mike, Introduced with the z900 were the PKA (Pack ASCII), PKU (Pack Unicode), UNPKA (Unpack ASCII), and UNPKU (Unpack Unicode) instructions. These instructions, along with TP (TestPacked: set condition code to indicate if a memory location contains valid packed decimal data) are part of the extended-translation facility 2 and supported in z/OS 1.2 on. Our course z/OS Assembler Programming Part 4: z/Architecture and z/OS includes a discusion of all the non-privileged, non-floating-point instructions introduced with z/Architecture, including the new hardware instructions on the z9, z10, and z186 machines. One lab includes using PKA, PKU, and TP instructions. And, tying back to another thread, a different lab includes writing code to run in AMODE64. See this link for details: http://www.trainersfriend.com/Assembler_%20courses/C500descrpt.htm -- Kind regards, -Steve Comstock The Trainer's Friend, Inc. 303-393-8716 http://www.trainersfriend.com * To get a good Return on your Investment, first make an investment! + Training your people is an excellent investment * Try our new tool for calculating your Return On Investment for training dollars at http://www.trainersfriend.com/ROI/roi.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
On 30 November 2010 22:57, Mike Schwab mike.a.sch...@gmail.com wrote: Here is an idea to bounce around. z/OS Unix System Services does a lot of work converting ASCII to EBCDIC and back. I'm puzzled by this. What is all this conversion work that takes to much time and effort? To the extent that character coding is relevant, z/OS UNIX operates in EBCDIC, and it is typically only some external interfaces that require character translation. And that is often enough, e.g. in the case of TN3270, done at the client end on the desktop. To be sure there are some ugly cases, such as various sorts of archives containing a mix of binary and character data, but I don't see that there is a lot of expensive or conceptually difficult character conversion going on. z/Linux works all in ASCII. Why not get 4 new instructions that work with PD= ASCII like the PD = EBCDIC instructions PACK, UNPK, ED, EDMK, but with an A suffix to denote ASCII character. Conversion from Packed to binary would be the same. Assembler would get new instructions. z/OS would need to know if a file was ASCII for proper translation when printing it. It's all there. Shoot, do we want to limit this to just ASCII? Could we do some sort of Unicode translation? Some of that is there too. Tony H. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
On Wed, Dec 1, 2010 at 2:11 AM, Ron Hawkins ron.hawkins1...@sbcglobal.netwrote: If I remember rightly it was a bug in IMS 2.2 or 2.3. If I remember correctly NAB (where I worked at the time) had found the bug in stress and regression testing (TPNS for those that remember it) and were waiting for the fix that hit Westpac. Funny how times have changed. Yes it was IMS 2.2 and it did not actually fail in testing. It was in test for quite a while and there was a lot of pressure to put it into production. It crashed mid-morning as the national branch network came online on the monday after it went live. It came up again, went down again, yo-yo'd a few times and then stayed down. The real problem was in the restart and recovery processing which changed in 2.2. -- This email might be from the artist formerly known as CC (or not) You be the judge. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
In aanlkti=29p26ome3vbcev6ldfhu-jpwd09cv5sgxt...@mail.gmail.com, on 11/30/2010 at 09:57 PM, Mike Schwab mike.a.sch...@gmail.com said: Here is an idea to bounce around. z/OS Unix System Services does a lot of work converting ASCII to EBCDIC and back. z/Linux works all in ASCII. I doubt it. Could we do some sort of Unicode translation? We already do. Have GPR2 point to a memory area that indicates how many bytes per digit, then the 10 characters for 0-9? Why introduce a lot of unnecessary complexity for something that can be done easily. The existing instructions are perfectly adequate for dealing with decimal data in Unicode. Would we want the number of digits? Doing arithmetic in arbitrary baes is a different issue than doing decimal arithmetic in arbitrary character sets. Silly wabbit, trits are for kids! -- Shmuel (Seymour J.) Metz, SysProg and JOAT ISO position; see http://patriot.net/~shmuel/resume/brief.html We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
On 30/11/2010 6:12 PM, Mike Schwab wrote: That was not payroll. That was a bank. They screwed up all transactions for a week after a conversion and fallback. I would be really curious as to how the database was functioning without crashing while processing all those bad transactions. Almost like they were processing the same old (duplicate) transactions (?same GDG?) on one system while another system was putting new transactions into newer files (GDG +1) which the other system did not see (many missing transactions). Maybe if they said which days were repeated and which days were missed we could learn more? Does the sysplex carry catalog updates (new file names) across systems? Are there Coupling Facility links that spread catalog updates to other systems? My speculation based on what has been in the news is that they had some sort of error processing their overnight batch transactions, and their error recovery also failed. Then they would have had the problem of reversing the updates from the failed batch update and rerunning it, while still allowing new transactions into the database so that the bank could keep functioning. It's the sort of problem that you would hope that banks are prepared for, but it still sounds like it could give you headaches. I am also curious to know more. -- Andrew Rowley Black Hill Software Pty. Ltd. Phone: +61 413 302 386 EasySMF for z/OS: Interactive SMF Reports on Your PC http://www.smfreports.com -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
On 11/30/2010 12:52 AM, Stephen Mednick wrote: One wonders if a detailed explanation of what transpired will be forthcoming as was the case back in July when the DBS Bank in Singapore had a major outage. http://www.dbs.com/newsroom/2010/press100804.aspx Stephen Mednick Computer Supervisory Services Sydney, Australia Wow! I've never seen anything like that. Thanks for the link. -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Sheldon Davis Sent: Tuesday, 30 November 2010 5:54 PM To: IBM-MAIN@bama.ua.edu Subject: I would love to know what went wrong at NAB I would love to know how a corrupt system file in a parrallel sysplex can affect a payroll system http://www.channelregister.co.uk/2010/11/29/nab_mainframe_cockup/ -- Kind regards, -Steve Comstock The Trainer's Friend, Inc. 303-393-8716 http://www.trainersfriend.com * To get a good Return on your Investment, first make an investment! + Training your people is an excellent investment * Try our new tool for calculating your Return On Investment for training dollars at http://www.trainersfriend.com/ROI/roi.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
A detailed explanation would be nice. The way this article reads it's the mainframe that's at fault. As though this wouldn't have happened if the platform had been something other than the mainframe. However, I'd bet that the upgrade was to the application system, not the mainframe, and something was wrong in a new application program that corrupted the file. I'm wondering why it took 5 days to recover. It sounds like a proper backup/backout procedure wasn't in place in case of a failure like this. Tom Kelman Capacity Planning Commerce Bank, Kansas City -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Stephen Mednick Sent: Tuesday, November 30, 2010 1:52 AM To: IBM-MAIN@bama.ua.edu Subject: Re: I would love to know what went wrong at NAB One wonders if a detailed explanation of what transpired will be forthcoming as was the case back in July when the DBS Bank in Singapore had a major outage. http://www.dbs.com/newsroom/2010/press100804.aspx Stephen Mednick Computer Supervisory Services Sydney, Australia -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Sheldon Davis Sent: Tuesday, 30 November 2010 5:54 PM To: IBM-MAIN@bama.ua.edu Subject: I would love to know what went wrong at NAB I would love to know how a corrupt system file in a parrallel sysplex can affect a payroll system http://www.channelregister.co.uk/2010/11/29/nab_mainframe_cockup/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html * If you wish to communicate securely with Commerce Bank and its affiliates, you must log into your account under Online Services at http://www.commercebank.com or use the Commerce Bank Secure Email Message Center at https://securemail.commercebank.com NOTICE: This electronic mail message and any attached files are confidential. The information is exclusively for the use of the individual or entity intended as the recipient. If you are not the intended recipient, any use, copying, printing, reviewing, retention, disclosure, distribution or forwarding of the message or any attached file is not authorized and is strictly prohibited. If you have received this electronic mail message in error, please advise the sender by reply electronic mail immediately and permanently delete the original transmission, any attachments and any copies of this message from your computer system. * -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
In listserv%201011300054096099.0...@bama.ua.edu, on 11/30/2010 at 12:54 AM, Sheldon Davis sda...@isracard.co.il said: I would love to know how a corrupt system file in a parrallel sysplex can affect a payroll system Why? That's not what the article claims. a corrupted file on an IBM mainframe system is quite different from a corrupt system file. From the article it was a payment processing file. -- Shmuel (Seymour J.) Metz, SysProg and JOAT ISO position; see http://patriot.net/~shmuel/resume/brief.html We don't care. We don't have to care, we're Congress. (S877: The Shut up and Eat Your spam act of 2003) -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
I'd bet that the upgrade was to the application system, not the mainframe, and something was wrong in a new application program that corrupted the file. I wouldn't take the opposite of that bet! I'm wondering why it took 5 days to recover. Since, upon reading the article, I see that INFOSYS was I don't have to wonder. It sounds like a proper backup/backout procedure wasn't in place in case of a failure like this. In my experience, these off-shore application providers treat the application lightly, and don't seem to have the urgency regarding the security and integrity of either the programmes or the data that the original IT staff did. - Ted MacNEIL eamacn...@yahoo.ca -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
As if. Can't you just imagine a major Aussie Bank doing that. You were at Bank of NSW when they had the IMS fiasco Steve - how much info on that got out ? (via Bank press releases I mean :-) Shane ... On Tue, 30 Nov 2010 18:52:08 +1100 Stephen Mednick wrote: One wonders if a detailed explanation of what transpired will be forthcoming as was the case back in July when the DBS Bank in Singapore had a major outage. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
Some information from a close source... It was a plain old S0C7 during their batch process. All Aussie banks use Cemtex ABA format which has been around for years as a transfer format between organisations. You would think that there is a validate step before running the transactions against their databases. A couple of things: Most of the support is now with the sub-continent... The previous grey-haired support staff were either laid off or moved to greener pastures Not many local staff are competent to manually reprocess or react to this situation. As a previous poster noted, probably the new transaction batch goes to a GDG, the failed process was missed and batch read yesterday's GDG and hence all the double transactions. My same source related the tale of a person in same bank who received an EBCDIC file, opened it in Windows and saved it back as ASCII. The file was duly transferred for processing on the mainframe. Most of the packed data was garbaged .. Oh my, dumb and dumber On Wed, Dec 1, 2010 at 11:19 AM, Shane ibm-m...@tpg.com.au wrote: As if. Can't you just imagine a major Aussie Bank doing that. You were at Bank of NSW when they had the IMS fiasco Steve - how much info on that got out ? (via Bank press releases I mean :-) Shane ... On Tue, 30 Nov 2010 18:52:08 +1100 Stephen Mednick wrote: One wonders if a detailed explanation of what transpired will be forthcoming as was the case back in July when the DBS Bank in Singapore had a major outage. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- Wayne V. Bickerdike -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
On Tue, Nov 30, 2010 at 5:01 PM, Wayne Bickerdike wayn...@gmail.com wrote: Some information from a close source... It was a plain old S0C7 during their batch process. All Aussie banks use Cemtex ABA format which has been around for years as a transfer format between organisations. You would think that there is a validate step before running the transactions against their databases. A couple of things: Most of the support is now with the sub-continent... The previous grey-haired support staff were either laid off or moved to greener pastures Not many local staff are competent to manually reprocess or react to this situation. As a previous poster noted, probably the new transaction batch goes to a GDG, the failed process was missed and batch read yesterday's GDG and hence all the double transactions. My same source related the tale of a person in same bank who received an EBCDIC file, opened it in Windows and saved it back as ASCII. The file was duly transferred for processing on the mainframe. Most of the packed data was garbaged .. Oh my, dumb and dumber Frightening. On Wed, Dec 1, 2010 at 11:19 AM, Shane ibm-m...@tpg.com.au wrote: As if. Can't you just imagine a major Aussie Bank doing that. You were at Bank of NSW when they had the IMS fiasco Steve - how much info on that got out ? (via Bank press releases I mean :-) Shane ... On Tue, 30 Nov 2010 18:52:08 +1100 Stephen Mednick wrote: One wonders if a detailed explanation of what transpired will be forthcoming as was the case back in July when the DBS Bank in Singapore had a major outage. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- Wayne V. Bickerdike -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
On Tue, Nov 30, 2010 at 7:01 PM, Wayne Bickerdike wayn...@gmail.com wrote: deleted My same source related the tale of a person in same bank who received an EBCDIC file, opened it in Windows and saved it back as ASCII. The file was duly transferred for processing on the mainframe. Most of the packed data was garbaged .. Oh my, dumb and dumber Here is an idea to bounce around. z/OS Unix System Services does a lot of work converting ASCII to EBCDIC and back. z/Linux works all in ASCII. Why not get 4 new instructions that work with PD= ASCII like the PD = EBCDIC instructions PACK, UNPK, ED, EDMK, but with an A suffix to denote ASCII character. Conversion from Packed to binary would be the same. Assembler would get new instructions. z/OS would need to know if a file was ASCII for proper translation when printing it. Shoot, do we want to limit this to just ASCII? Could we do some sort of Unicode translation? Have GPR2 point to a memory area that indicates how many bytes per digit, then the 10 characters for 0-9? EDMK would overlay this address anyway, so ok for input. Would we want the number of digits? I.E. 10 for decimal, 8 for octal, 16 for hexadecimal (good for dumps!), up to 36 to encode numbers into a character string in some way (a basic encoding technique). Would Oriental languages want more than 255 numbers? Save the UNICODE value in the owner field, or a new field in the VTOC? 2 number of Bytes per digit, 2 number of digits, Digit 0, Digit 1, Digit 2, ..., Digit N. -- Mike A Schwab, Springfield IL USA Where do Forest Rangers go to get away from it all? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
-Original Message- From: IBM Mainframe Discussion List On Behalf Of Shane As if. Can't you just imagine a major Aussie Bank doing that. You were at Bank of NSW when they had the IMS fiasco Steve - how much info on that got out ? (via Bank press releases I mean :-) Yeh It's probably a near-universal trait that dirty laundry is washed discreetly :-) -jc- -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
On Tue, Nov 30, 2010 at 10:39 PM, Chase, John jch...@ussco.com wrote: -Original Message- From: IBM Mainframe Discussion List On Behalf Of Shane As if. Can't you just imagine a major Aussie Bank doing that. You were at Bank of NSW when they had the IMS fiasco Steve - how much info on that got out ? (via Bank press releases I mean :-) Yeh It's probably a near-universal trait that dirty laundry is washed discreetly :-) This would be one of those rare cases where the story was big enough that there wasn't a lot of room for discrete laundering. It was actually Westpac by then (btw) and I was there too. That event wasn't a single outage, but rather a series of them wound around failed restart/recovery processing that eventually took several days to fully recover from. There wasn't any news coverage at first, but the scale of the problem had made the press by the second day. Once it had there was plenty of blame storming to go around. I don't recall whether the bank actually issued press releases but their point of view certainly did make it into the press coverage. -- This email might be from the artist formerly known as CC (or not) You be the judge. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
Mike Schwab wrote: On Tue, Nov 30, 2010 at 7:01 PM, Wayne Bickerdike wayn...@gmail.com wrote: deleted My same source related the tale of a person in same bank who received an EBCDIC file, opened it in Windows and saved it back as ASCII. The file was duly transferred for processing on the mainframe. Most of the packed data was garbaged .. Oh my, dumb and dumber Here is an idea to bounce around. z/OS Unix System Services does a lot of work converting ASCII to EBCDIC and back. z/Linux works all in ASCII. Why not get 4 new instructions that work with PD= ASCII like the PD = EBCDIC instructions PACK, UNPK, ED, EDMK, but with an A suffix to denote ASCII character. Conversion from Packed to binary would be the same. Assembler would get new instructions. z/OS would need to know if a file was ASCII for proper translation when printing it. Mike, Introduced with the z900 were the PKA (Pack ASCII), PKU (Pack Unicode), UNPKA (Unpack ASCII), and UNPKU (Unpack Unicode) instructions. -- | Jim Phoenix | Voice: (310) 338-0400 x316 | | Senior Software Developer| Fax: (310) 338-0801| | Phoenix Software International || | 831 Parkview Drive North | jimphoe...@phoenixsoftware.com | | El Segundo, CA 90245 | http://www.phoenixsoftware.com | Opinions expressed by this individual are not necessarily those of the Company. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
A comment here:- http://www.theaustralian.com.au/australian-it/human-error-triggered-nab- software-corruption/story-e6frgakx-1225962953523 The file was actually software code containing instructions on how systems should operate in the batch processing cycle. would lead me to think someone had incorrectly modified the batch processing rules...like in TWS/OPC or CA-7. Regards Bruce Hewson -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
On Tue, Nov 30, 2010 at 12:54 AM, Sheldon Davis sda...@isracard.co.il wrote: I would love to know how a corrupt system file in a parrallel sysplex can affect a payroll system http://www.channelregister.co.uk/2010/11/29/nab_mainframe_cockup/ That was not payroll. That was a bank. They screwed up all transactions for a week after a conversion and fallback. I would be really curious as to how the database was functioning without crashing while processing all those bad transactions. Almost like they were processing the same old (duplicate) transactions (?same GDG?) on one system while another system was putting new transactions into newer files (GDG +1) which the other system did not see (many missing transactions). Maybe if they said which days were repeated and which days were missed we could learn more? Does the sysplex carry catalog updates (new file names) across systems? Are there Coupling Facility links that spread catalog updates to other systems? -- Mike A Schwab, Springfield IL USA Where do Forest Rangers go to get away from it all? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
Re: I would love to know what went wrong at NAB
One wonders if a detailed explanation of what transpired will be forthcoming as was the case back in July when the DBS Bank in Singapore had a major outage. http://www.dbs.com/newsroom/2010/press100804.aspx Stephen Mednick Computer Supervisory Services Sydney, Australia -Original Message- From: IBM Mainframe Discussion List [mailto:ibm-m...@bama.ua.edu] On Behalf Of Sheldon Davis Sent: Tuesday, 30 November 2010 5:54 PM To: IBM-MAIN@bama.ua.edu Subject: I would love to know what went wrong at NAB I would love to know how a corrupt system file in a parrallel sysplex can affect a payroll system http://www.channelregister.co.uk/2010/11/29/nab_mainframe_cockup/ -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
