This was about 6 years ago. I took over as senior admin at a client that does boat/rv loans. Our firm did their outsourced IT. They had a recent implementation of Cisco Voip including Unity 2.4.something (voicemail).
We were in the process of doing major app/infrastructure upgrades. I had been pushing to get them off the standalone unity implementation. For those who don't remember unity 2.4, it had, all on one box, windows 2000 AD, Exchange 5.5, and Unity. That update was on the planning board... One day we had a power outage in the office around 10 AM. I frantically shut down all the servers as quickly as I could, since our UPSs were, shall we say, less than optimal for the infrastructure. I got everything shut down and was wiping the sweat off my brow when the CEO walked in and said "I need the voicemail server on so I can check my voicemail once more; I'm expecting a call". I said no. Too risky. Bad things could happen. You have to trust me on this. He said "I really need this. I understand; turn it back on NOW." OK; you're the boss. I got it back up and running and just as he was connecting back to his vm, the UPS battery died. Sigh. A couple of hours later the power came back on and I went through my restart checklist. When we got to the VM server, it came up "can't find NTLDR" or something similar. Nice... The OS was hosed. None of my tricks worked, so we got MS on the line. After several hours with them, we finally got the OS back somehow; don't recall all the details. Then we tried to start AD and the DB was hosed. So several more hours with MS. Finally, about 6 pm, we got AD back. Then the exchange DB was toast. Of course, we'd just installed a new backup server a few days previously and didn't have good backups yet. So we had to fix the DB. That took more hours. Finally, about 11:30, we had the MS stuff working. We then tried to launch Unity. No such luck. Called Cisco. "Oh; good thing you called when you did; this product is EOL at midnight tonight". They said no prob though; they'd work with us as long as we needed since we opened the case before midnight. Spent the next 10 hours on the phone with Cisco getting Unity back up. Finally had it all working again. By now it had been over 24 hours and I was a bit fried. My boss was in the conference room with the CEO when I walked in. The CEO looked sheepishly at me; he knew what had happened and why. I pointed my finger at him and said, probably a bit more forcefully than I should have "The next time I tell you something's a bad idea, you'd better pay attention. And when I tell you no, I really mean no". My boss hid a shocked smile behind his laptop screen and the CEO just blushed and nodded. I left the building totally wiped out, ready to rebuild my resume once I realized what I'd done. But instead of getting fired I got a nice bonus for putting everything back together. Probably used up ALL of my CEO luck with that one episode... :-) *********************** Charlie Kaiser [email protected] Kingman, AZ *********************** > -----Original Message----- > From: John Aldrich [mailto:[email protected]] > Sent: Wednesday, February 24, 2010 12:46 PM > To: NT System Admin Issues > Subject: RE: Stupid user tricks! > > Now THAT would be an interesting story! :-) > > > > > -----Original Message----- > From: Charlie Kaiser [mailto:[email protected]] > Sent: Wednesday, February 24, 2010 2:42 PM > To: NT System Admin Issues > Subject: RE: Stupid user tricks! > > You mean 1100? Yeah, but I'm still shaking my head about the > whole coke on top of the JBOD thing in the first place... ;-) > > How the trays all had liquid coke in them and the unit is > still functional is beyond my comprehension. I'm just glad > it's my boss there and not me. I tend to open my can of WA > when that sort of thing happens. He's more diplomatic... > > Maybe sometime I'll tell the story of the time I chewed the > CEO a new one and kept my job... <G> > > *********************** > Charlie Kaiser > [email protected] > Kingman, AZ > *********************** > > > -----Original Message----- > > From: Kurt Buff [mailto:[email protected]] > > Sent: Wednesday, February 24, 2010 12:34 PM > > To: NT System Admin Issues > > Subject: Re: Stupid user tricks! > > > > You can't count to 12? > > > > On Wed, Feb 24, 2010 at 10:52, Charlie Kaiser > > <[email protected]> wrote: > > > Speaking of SUT... Just got a call from my boss at one of > > our clients. > > > We've been tracking down the cause of broken disk caddies > on a JBOD > > > (all of the caddy handles were broken). Turns out someone > > spilled coke > > > on the unit and couldn't figure out how to release the > > handles. Broke all 12... > > > > > > That's just wrong in more ways than I can count... > > > > > > *********************** > > > Charlie Kaiser > > > [email protected] > > > Kingman, AZ > > > *********************** > > > > > > > > > ~ Finally, powerful endpoint security that ISN'T a resource > > hog! ~ ~ > > > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > > > > > > ~ Finally, powerful endpoint security that ISN'T a resource > hog! ~ ~ > > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > > > > ~ Finally, powerful endpoint security that ISN'T a resource > hog! ~ ~ > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > > > ~ Finally, powerful endpoint security that ISN'T a resource > hog! ~ ~ > <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~
