L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Josh Reynolds
I know a lot of you are out of the office right now, but does anybody have any info on what happened with L3 this morning? They went into a 5 hour maintenance window with expected downtime of about 30 minutes while they upgraded something like *40* of their core routers (their words), but also did

Re: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Viral Vira
We also noticed outage due to L3 Maintenance that went into the outage. We were not even notified about the Maintenance itself. We also noticed black hauling in their network. -Thanks, Viral On 5 February 2013 21:09, Josh Reynolds ess...@gmail.com wrote: I know a lot of you are out of the

RE: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread David Hubbard
about some route withdrawals but by that time things were back to normal and remained stable. -Original Message- From: Josh Reynolds [mailto:ess...@gmail.com] Sent: Tuesday, February 05, 2013 10:40 AM To: nanog@nanog.org Subject: L3 East cost maint / fiber 05FEB2012 maintenance I

Re: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Jon Lewis
On Tue, 5 Feb 2013, Josh Reynolds wrote: I know a lot of you are out of the office right now, but does anybody have any info on what happened with L3 this morning? They went into a 5 hour maintenance window with expected downtime of about 30 minutes while they upgraded something like *40* of

RE: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Nick Olsen
dhubb...@dino.hostasaurus.com Sent: Tuesday, February 05, 2013 10:53 AM To: nanog@nanog.org Subject: RE: L3 East cost maint / fiber 05FEB2012 maintenance We saw the same thing out of their Tampa location; there was a brief drop around 2am EST and a more severe one around 4:05 AM which lasted about 10

Re: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Jason Lixfeld
I got notification of their maintenance window, albeit with 24 hours notice. Notice came in at 11:00GMT-5 yesterday, maintenance was scheduled for 00:00GMT-5 this morning. That said, the notice said that the maintenance was in Phoenix but I got a notice about my IPT circuit at 60 Hudson

RE: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread 2asx1y702
I acknowledge sliding past the maintenance window, and we're seeing similar bumps, 09:42 - 09:46 CST is most recent. This are with our Wisconsin and Netherlands locations. They seem to be having a bad day all around. KG Hi Andrey!

Re: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Ben Bartsch
We lost our peering with them in Baton Rouge (Houston) but not in Jackson MS (Atlanta). It was less than 10 minutes. No advanced notification. On Tue, Feb 5, 2013 at 10:06 AM, 2asx1y...@sneakemail.com wrote: I acknowledge sliding past the maintenance window, and we're seeing similar bumps,

Re: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Jonathan Lassoff
My hunch is that this is fallout and repairs from Juniper PR839412. Only fix is an upgrade. Not sure why they're not able to do a hitless upgrade though; that's unfortunate. Specially-crafted TCP packets that can get past RE/loopback filters can crash the box. --j On Tue, Feb 5, 2013 at 7:39

Re: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Jason Biel
Workaround is proper filtering and other techniques on the RE/Loopback to prevent the issue from happening. Should an upgrade be performed? Yes, but certainly doesn't have to have right away or without notice to customers. On Tue, Feb 5, 2013 at 11:23 AM, Jonathan Lassoff j...@thejof.com wrote:

Re: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Jonathan Lassoff
On Tue, Feb 5, 2013 at 9:33 AM, Jason Biel ja...@biel-tech.com wrote: Workaround is proper filtering and other techniques on the RE/Loopback to prevent the issue from happening. Agreed. However, if it only takes one packet, what if an attacker sources the traffic from your management address

Re: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread Jason Biel
Agree as well. Bad assumption on my part that Level3 would doing the items listed in the workaround already. On Tue, Feb 5, 2013 at 11:41 AM, Jonathan Lassoff j...@thejof.com wrote: On Tue, Feb 5, 2013 at 9:33 AM, Jason Biel ja...@biel-tech.com wrote: Workaround is proper filtering and other

Re: L3 East cost maint / fiber 05FEB2012 maintenance

2013-02-05 Thread joel jaeggli
On 2/5/13 10:02 AM, Jason Biel wrote: Agree as well. Bad assumption on my part that Level3 would doing the items listed in the workaround already. On Tue, Feb 5, 2013 at 11:41 AM, Jonathan Lassoff j...@thejof.com wrote: On Tue, Feb 5, 2013 at 9:33 AM, Jason Biel ja...@biel-tech.com wrote: