I got notification of their maintenance window, albeit with < 24 hours notice. Notice came in at 11:00GMT-5 yesterday, maintenance was scheduled for 00:00GMT-5 this morning.
That said, the notice said that the maintenance was in Phoenix but I got a notice about my IPT circuit at 60 Hudson which I found confusing. Based on my logs, our BGP session with them went down at 03:06GMT-5 and back up at 03:15GMT-5. Down again at 03:37GMT-5 until 04:20GMT-5. A third time at 06:41GMT-5 and back at 06:45GMT-5. Traffic graphs tell a bit of a different story. Just before 05:00GMT-5, our outbound traffic to Level 3 dropped substantially. About that time, I started getting reports about issues to Level 3 destinations. Traces seemed to indicate a black hole condition within Level 3's network in NYC, seemingly at, or just past csw3.NewYork1.Level3.net. Stuff seemed to correct itself by about 06:45GMT-5, but due to Level 3 sending only about 180k routes. About 20 minutes later, the table was back to ~431K and all's been fine since. On 2013-02-05, at 10:39 AM, Josh Reynolds <ess...@gmail.com> wrote: > I know a lot of you are out of the office right now, but does anybody have > any info on what happened with L3 this morning? They went into a 5 hour > maintenance window with expected downtime of about 30 minutes while they > upgraded something like *40* of their "core routers" (their words), but > also did this during some fiber work and completely cut off several of > their east coast peers for the entirety of the 5 hour window. > > If anybody has any more info on this, on a NOC contact for them on the East > Coast for future issues, you can hit me off off-list if you don't feel > comfortable replying with that info here. > > Thanks, and I hope hope you guys are enjoying Orlando. > > -- > *Josh Reynolds* > ess...@gmail.com - (270) 302-3552