We had added another raised floor area into the data center and installed a 3090J mod 5, the one that looked like a huge H. We also had a 3081K in another room on the raised floor. DASD farm, print land, break down room with bursters, trimmers, and decolators, tape land, Network, and Command Center. Each was a separate raised floor Halon zone.
One fine Saturday, I was in to give a tour to a Boy Scout group for a merit badge they were working on. Left the group in the reception area while I went up on the raised floor to check in with Operations staff. There was a contractor from the Halon fire suppression company in working on the Halon panel, and another contractor in working on our large telephone switch. The Halon panel was disabled, with all of the proper disabled lights showing, and I started the tour. We were all in the Tape Library when the horn went off. The Halon system skipped the bell and went right to the 30 second horn. I raced through the Boy Scouts and made a run for the Halon panel. My hand was a couple of inches from the master abort when the Halon started to dump. Every zone that had an automatic door dumped because the electricity to the door didn't drop (the doors could be opened by pushing them open - that part had been tested ) and the door sensors saw the approaching Halon and opened the door. The Halon sensors in the next zone (and next, etc.) saw the Halon cloud and dumped that zone too. To top it off, the Halon panel wouldn't release control, so it had to be taken out of service completely. We were down for 12 hours before all of the Halon could be properly vented. Coming back up was difficult with quite a few crashed DASD. One side of the 3090 wouldn't come up, so I brought up just the one side that still worked. The 3081 came up, but it took several more hours with the field engineers working on the DASD to get us ready to re-ipl. IBM field service did a great job getting us back to working order. The root cause of this? A wiring oops in the Halon panel itself and the assumption that the doors would drop power like they were supposed to. Linda ----- Original Message ----- From: "Anne & Lynn Wheeler" <[email protected]> To: [email protected] Sent: Sunday, December 13, 2015 12:04:39 PM Subject: Re: OT: Electrician cuts wrong wire and downs 25,000 square foot data centre [email protected] (Pete Lancashire) writes: > Showing my age .... > > I worked for Burroughs as an engineering technician. > > A customer with 360/65 instantaneous loss of power. I was there only for a > couple hours to drop off some equipment. Later heard they lost a couple > disk packs. separate from power failures can precipitate disk drive failure. IBM CKD dasd had power loss failure mode ... where there wasn't enough power to maintain memory contents ... but there was enough power left for the controller to complete a write operation ... problem was that the channel had stopped transferring data ... so the controller continued writting all zeros. The result was that after recovery, a subsequent read would show no errors ... for the record that had write operation ("correctly") complete with all zeros (this was especially troublesome when things like VTOC record was being written) FBA introduced that a physical record would not be written unless all data was available to correctly complete a write. This philosophy continued for RAID (write "failure" either completes correctly or at least results in error indication for subsequent read). During the 80s, there was lots of work trying to figure out how to retrofit such a fix to CKD dasd ... or at least provide a way for system to recognize an incorrect trailing zeros write. -- virtualization experience starting Jan1968, online at home since Mar1970 ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
