I once worked for a provider who had a company next door that ran a small
datacenter of about a dozen or so racks. They had been sold and all of their
infrastructure had been virtualized and moved to the new owner’s network.
The last task of the local admin was to just get rid of everything. They
didn’t care how just get it gone. So he came over and asked us to come take
a look and we could have anything we wanted. I picked up a few servers for
lab a bunch of racks and stuff.
While we were working I asked the guy “So there is absolutely nothing that’s
in production in here anymore?” He said “Yep” so I asked “Then if the power
went off in here it wouldn’t be a big deal” and he said “Not at all”. Then
I asked “Can I hit the red button?” He said “Sure, I always wondered what
happened”. I hit the button and with a loud booming sound the room went
dead silent and then the UPS started beeping. It was at that moment
everyone realized that you just don’t pull the button out to restart the room.
It took us 20 minutes to figure out how to turn it all back on.
And with that when I got back to our office I made sure someone knew how to
restart everything if we ever had to hit our red button.
-richey
From: NANOG <nanog-bounces+richey.goldberg=gmail....@nanog.org> on behalf of
Roy <r.engehau...@gmail.com>
Date: Thursday, September 16, 2021 at 12:41 AM
To: nanog <nanog@nanog.org>
Subject: Re: Never push the Big Red Button (New York City subway failure)
Miy story in the late 1970s I was working in a large computer facility
with both mainframes and mil-spec 400hz computers.
Management decided that the EPO should be tested. So we powered down
the disk and tapes. The electrician pressed
the EPO button and NOTHING. Everything kept running.
Turns out a wire had come loose and the fuse in the EPO circuit had blown.
Roy
