Business continuity plans and disaster recovery plans should always be developed for all mission critical systems. Your story helps to remind us all of that fact.
Sent from my iPhone On Mar 11, 2010, at 1:06 PM, "Jason Bondy" <jbondy at okhistory.org> wrote: > Melissa, > > I'm sorry to hear of all that trouble, but I thank you for sharing > your > story. A lot of times we take the technology for granted and don't > realize > we need to be proactive to make sure it is always available. > Although in > your case, with a lightning strike that close, even with better > backup/surge > solutions you may have still had some damage. > > Thanks again for the reminder. > > Jason > > _______________________________________________________ > Jason Bondy > Exhibit AV/IT Systems > Oklahoma History Center > 2401 N. Laird Ave. > Oklahoma City, OK 73105 > 405-522-0783 - Office > 405-522-5402 - Fax > www.okhistory.org > > -----Original Message----- > From: mcn-l-bounces at mcn.edu [mailto:mcn-l-bounces at mcn.edu] On Behalf > Of > Melissa Johnson > Sent: Wednesday, March 10, 2010 6:21 PM > To: mcn-l at mcn.edu > Subject: [MCN-L] Technology Disaster > > Hello MCN Listserv, > > I'm not sure what I am hoping to gain by posting this message other > than > sympathy. Well, and maybe, to share our own technology disaster > experience. > This is somewhat long but entertaining. Or, at least, shocking. > > Disclaimer: I am one of the many accidental techies in the museum > world. So, > please excuse any improper use of the words: router, switch, shared > drive, > UPS, network drive, phone computer, network cable, etc. > > On January 19, 2010 there was a pretty big thunderstorm in Silicon > Valley, > which does get very much lightning. In the early afternoon there was a > bright flash of light and a big clap of thunder, not surprisingly > the power > went out. After power was restored, we discovered that History Park > had been > struck by lighting and suffered a massive power surge. It took us a > few days > to discover the full extent of the damage. > > We found that the UPS connected to our server was not only dead but > had > scorch marks on the back. Our hope was that the UPS had done it's > job and > protected the server. No luck. The server wouldn't start up. (And, > to make > the lesson even more valuable, I was going to order a new, real backup > system that afternoon.) We kept our fingers crossed that the data > would be > okay. After about a week our part-time IT contractor was able to > gain access > to the hard drive and the data was fine. Now, we just had to wait > for the > new server to arrive. > > Next we checked the UPS that is connected to our telephone > computer/firewall/switch set up. The UPS had survived but did not do > it' s > job. Phone computer dead. When our vendor came out to fix it, he was > very > surprised that the UPS hadn't prevented the power surge. The > firewall was > fried but we had an extra (don't ask) and the switch was dead. > Luckily, it > was still under warranty. Over the next few days we got the phones and > network back up and running for most of the staff. > > History San Jos? has two sites that are across the street from each > other > and connected with a T1 connection. After we got the network back up > the > second site (where my office is) still did not have access to phones > or > internet. We didn't realize that the power surge had traveled down the > network cable to the other side of the building and fried a router > which > connects the phones and network to the second site. No vendor would > claim > knowledge of the routers and I could not find an invoice for them. I > finally > remembered that the routers were credited (aka given) to us as an > incentive > to upgrade our service. We didn't know the passwords to these > routers and > the vendor who installed them has since closed. The password to the > routers > was eventually cracked and connection was restored. The staff at the > second > site was without communication for 30 days. > > Lessons Learned: > --Have a real backup system and procedures. > --Good relationship with vendors so they will come to your rescue. > --Beware of vendors bearing free equipment. > --Google Apps saved our sanity. > > I hope this was informative, or at least, entertaining. In addition > to our > technology problems, the lightning strike also took out 3 elevators, > 1 light > tower, and 1 refractor (?) that runs our trolley. As of today, we > have our > new server and backup system installed, working phones, and access > to email > and the internet. It's nice to be back in the 21st century! > > -- Melissa > ------------------------- > Melissa Johnson > Curator of Interactive Media > history|san jos? > 408.521.5021 > _______________________________________________ > You are currently subscribed to mcn-l, the listserv of the Museum > Computer > Network (http://www.mcn.edu) > > To post to this list, send messages to: mcn-l at mcn.edu > > To unsubscribe or change mcn-l delivery options visit: > http://toronto.mediatrope.com/mailman/listinfo/mcn-l > > The MCN-L archives can be found at: > http://toronto.mediatrope.com/pipermail/mcn-l/ > > _______________________________________________ > You are currently subscribed to mcn-l, the listserv of the Museum > Computer Network (http://www.mcn.edu) > > To post to this list, send messages to: mcn-l at mcn.edu > > To unsubscribe or change mcn-l delivery options visit: > http://toronto.mediatrope.com/mailman/listinfo/mcn-l > > The MCN-L archives can be found at: > http://toronto.mediatrope.com/pipermail/mcn-l/
