Re: Never push the Big Red Button (New York City subway failure)
True EPO story; maintenance crew carrying new drywall into the data center backed into the EPO that didn't have a cover on it. One of the most eerie sounds in networking...a completely silent data center. -chris On Fri, Sep 10, 2021 at 2:48 PM Christopher Morrow wrote: > > > On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff wrote: > >> Reminds me of something that happened about 25 years ago when an >> elementary school visited our data center of the insurance company where I >> worked. One of our operators strategically positioned himself between the >> kids and the mainframe, leaned back and hit it's EPO button. >> >> > Or when your building engineering team cuts themselves a new key for the > 'main breaker' for the facility... and tests it at 2pm on a tuesday. > Or when that same team cuts a second key (gotta have 2 keys!) and tests > that key on the same 'main breaker' ... at 2pm on the following tuesday. > > > > not fakenews, a real story from a large building full of gov't employees > and computers and all manner of 'critical infrastructure' for the agency > occupying said building. > > Matthew Huff | Director of Technical Operations | OTA Management LLC >> >> Office: 914-460-4039 >> mh...@ox.com | www.ox.com >> >> ... >> >> -Original Message- >> From: NANOG On Behalf Of Sean >> Donelan >> Sent: Friday, September 10, 2021 12:38 PM >> To: nanog@nanog.org >> Subject: Never push the Big Red Button (New York City subway failure) >> >> NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER >> OUTAGE ISSUE ON AUGUST 29, 2021 >> Key Findings >> September 8, 2021 >> >> >> >> https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf >> >> Key Findings >> [...] >> >> 3. Based on the electrical equipment log readings and the manufacturer’s >> official assessment, it was determined that the most likely cause of RCC >> shutdown was the “Emergency Power Off” button being manually activated. >> >> Secondary Findings >> >> 1. The “Emergency Power Off” button did not have a protective cover at >> the >> time of the shutdown or the following WSP investigation. >> >> [...] >> Mitigation Steps >> >> 1. Set up the electrical equipment Control and Communication systems >> properly to stay active so that personnel can monitor RCC electrical >> system operations. >> >> [...] >> > -- Chris Kane
Re: MTU
This topic seems to come up more lately. Much like it did often during IPSec related deployments. I simplify on 9,000 as an easy number and I don't have to split hairs (read 9,214 v 9,216) that some vendors have. My experience has been making a view phone calls and agreeing on 9,000 is simple enough. I've only experienced one situation for which the MTU must match and that is on OSPF neighbor relationships, for which John T. Moy's book (OSPF - Anatomy of an Internet Routing Protocol) clearly explains why MTU became an issue during development of that protocol. As more and more of us choose or are forced to support 'jumbo' frames to accommodate Layer 2 extensions (DCI [Data Center Interconnects]) I find myself helping my customers work with their carriers to ensure that jumbo frames are supported. And frequently remind them to inquire that they be enabled not only on the primary path/s but any possible back up path as well. I've had customers experience DCI-related outages because their provider performed maintenance on the primary path and the re-route was sent across a path that did not support jumbo frames. As always, YMMV but I personally feel having the discussions and implementation with your internal network team as well as all of your providers is time well spent. Later, -chris On Fri, Jul 22, 2016 at 8:53 AM, Mark Tinka <mark.ti...@seacom.mu> wrote: > > > On 22/Jul/16 14:01, Baldur Norddahl wrote: > > > > > Obviously I only need to increase my MTU by the size of the GRE header. > But > > I am thinking is there any reason not to go all in and ask every peer to > go > > to whatever max MTU they can support? My own equipment will do MTU of > 9600 > > bytes. > > See the below: > > http://mailman.nanog.org/pipermail/nanog/2016-March/084598.html > > You can reliably run Jumbo frames in your own network core, and also to > another network that can guarantee you the same (which would typically > be under some form of commercial, private arrangement like an NNI). > > Across the Internet, 1,500 bytes is still safest, simply because that is > pretty much the standard. Trying to achieve Jumbo frames across an > Internet link (which includes links to your upstreams, links to your > peers and links to your customers) is an exercise in pain. > > Mark. > -- Chris Kane CCIE 14430 614 329 1906
