Re: [WISPA] Level3 Explosion?
* Matt Hoppes mhop...@indigowireless.com wrote: Did anyone else see an explosion on the Level3 network this morning? They've been up and down all morning since around 2:30 eastern time... just recovered recently. From the e-mail notification I received from Level 3 (~13 hours before the start of the maintenance window): Emergency Maintenance Advisement - This maintenance is scheduled. ... and ... Level 3 Communications will perform a mandatory network upgrade that will be service impacting and will impact devices in multiple locations. We are upgrading the code on portions of the global network to increase stability for the overall network. During this maintenance activity customers may be impacted for approximately 30 minutes. My own guess is that they wanted to quickly get their gear upgraded so that they weren't caught with their pants down like the last time such a vulnerability was announced. In that case, someone posted (previously unknown) exploit details to the full-disclosure mailing list, forcing many to to delay patching any longer: http://seclists.org/fulldisclosure/2010/Jan/136 On a side note: if you ever want to receive a ton of hate mail from engineers at large companies and telcos, that's a good way to do it. -- Jeremy L. Gaddis e: jer...@as54225.net Network Engineer m: +1.812.865.0581 ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless
Re: [WISPA] Level3 Explosion?
Here we had to upgrade our Juniper´s router because of these exploit. Juniper advertised all customers to update asap. Gustavo Santos Analista de Redes CCNA , MTCNA , MTCRE, MTCINE, JUNCIA-ER 2013/2/7 Jeremy L. Gaddis jer...@as54225.net * Matt Hoppes mhop...@indigowireless.com wrote: Did anyone else see an explosion on the Level3 network this morning? They've been up and down all morning since around 2:30 eastern time... just recovered recently. From the e-mail notification I received from Level 3 (~13 hours before the start of the maintenance window): Emergency Maintenance Advisement - This maintenance is scheduled. ... and ... Level 3 Communications will perform a mandatory network upgrade that will be service impacting and will impact devices in multiple locations. We are upgrading the code on portions of the global network to increase stability for the overall network. During this maintenance activity customers may be impacted for approximately 30 minutes. My own guess is that they wanted to quickly get their gear upgraded so that they weren't caught with their pants down like the last time such a vulnerability was announced. In that case, someone posted (previously unknown) exploit details to the full-disclosure mailing list, forcing many to to delay patching any longer: http://seclists.org/fulldisclosure/2010/Jan/136 On a side note: if you ever want to receive a ton of hate mail from engineers at large companies and telcos, that's a good way to do it. -- Jeremy L. Gaddis e: jer...@as54225.net Network Engineer m: +1.812.865.0581 ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless
Re: [WISPA] Level3 Explosion?
Matt, We didn't see anything here on the Sonet ring around the Philadelphia area. I have noticed that the past couple of weeks they have been doing BGP changes, seems like they are slowly integrating with GLBX's ASN3549. We lost IPv6 connectivity for short periods of time. -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Matt Hoppes Sent: Tuesday, February 05, 2013 7:05 AM To: WISPA General List; memb...@wispa.org Subject: [WISPA] Level3 Explosion? Did anyone else see an explosion on the Level3 network this morning? They've been up and down all morning since around 2:30 eastern time... just recovered recently. -- Matt Hoppes Director of Information Technology Indigo Wireless +1 (570) 723-7312 ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless
Re: [WISPA] Level3 Explosion?
One of our customers near NYC had their Level3 BGP session flap a couple times between 3:30am - 3:50am this morning. We also noticed some issues around 1:15am - 1:30am on a circuit we have to a provider that peers with Level3. We're in upstate NY but I believe the peering occurs in NYC. On 2/5/2013 8:49 AM, Eric Tykwinski wrote: Matt, We didn't see anything here on the Sonet ring around the Philadelphia area. I have noticed that the past couple of weeks they have been doing BGP changes, seems like they are slowly integrating with GLBX's ASN3549. We lost IPv6 connectivity for short periods of time. -Original Message- From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Matt Hoppes Sent: Tuesday, February 05, 2013 7:05 AM To: WISPA General List; memb...@wispa.org Subject: [WISPA] Level3 Explosion? Did anyone else see an explosion on the Level3 network this morning? They've been up and down all morning since around 2:30 eastern time... just recovered recently. ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless
Re: [WISPA] Level3 Explosion?
I was told yesterday at about 3:30pm EST that they were going to have scheduled emergency maintenance. We saw the BGP session flap. NANOG had a few words about it too. Nick Olsen Network Operations (855) FLSPEED x106 From: Matt Hoppes mhop...@indigowireless.com Sent: Tuesday, February 05, 2013 7:05 AM To: WISPA General List wireless@wispa.org, memb...@wispa.org memb...@wispa.org Subject: [WISPA] Level3 Explosion? Did anyone else see an explosion on the Level3 network this morning? They've been up and down all morning since around 2:30 eastern time... just recovered recently. -- Matt Hoppes Director of Information Technology Indigo Wireless +1 (570) 723-7312 ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless
Re: [WISPA] Level3 Explosion?
We had emergency maint from Level3 at 05:00 GMT to 10:00 GMT, but expected outage was 30 min. I didn't get up to see if we were flapping, but we expected it. Eric From: wireless-boun...@wispa.org [mailto:wireless-boun...@wispa.org] On Behalf Of Nick Olsen Sent: Tuesday, February 5, 2013 11:04 AM To: WISPA General List Subject: Re: [WISPA] Level3 Explosion? I was told yesterday at about 3:30pm EST that they were going to have scheduled emergency maintenance. We saw the BGP session flap. NANOG had a few words about it too. Nick Olsen Network Operations (855) FLSPEED x106 http://www.flhsi.com/files/emaillogo.jpg From: Matt Hoppes mhop...@indigowireless.com Sent: Tuesday, February 05, 2013 7:05 AM To: WISPA General List wireless@wispa.org, memb...@wispa.org memb...@wispa.org Subject: [WISPA] Level3 Explosion? Did anyone else see an explosion on the Level3 network this morning? They've been up and down all morning since around 2:30 eastern time... just recovered recently. -- Matt Hoppes Director of Information Technology Indigo Wireless +1 (570) 723-7312 ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless ___ Wireless mailing list Wireless@wispa.org http://lists.wispa.org/mailman/listinfo/wireless
