L3 Issues
Hello and Good Morning, Are there reports of L3 having issues this morning ? Starting at about 10:10 A Pacific, I started seeing huge drops in traffic at various sites, including San Diego, Houston, San Antonio, Charlotte, NC, Philadelphia, etc. Anyone seeing a similar behavior ?
Re: L3 Issues
On Mon, 01 Aug 2011 12:39:43 -0500, Khurram Khan brokenf...@gmail.com wrote: Hello and Good Morning, Are there reports of L3 having issues this morning ? Starting at about 10:10 A Pacific, I started seeing huge drops in traffic at various sites, including San Diego, Houston, San Antonio, Charlotte, NC, Philadelphia, etc. Anyone seeing a similar behavior ? Yes we are seeing Loss from Houston To LA (not NYC to LA) dropping out in Dallas -=Tom -- Using Opera's revolutionary email client: http://www.opera.com/mail/
RE: L3 Issues
Seeing a big drop in outbound traffic on our L3 link starting about that time (we're a web host). Getting calls about sites down too; all the traces I've walked customers through so far have died on L3 networks. David -Original Message- From: Khurram Khan [mailto:brokenf...@gmail.com] Sent: Monday, August 01, 2011 1:40 PM To: nanog@nanog.org Subject: L3 Issues Hello and Good Morning, Are there reports of L3 having issues this morning ? Starting at about 10:10 A Pacific, I started seeing huge drops in traffic at various sites, including San Diego, Houston, San Antonio, Charlotte, NC, Philadelphia, etc. Anyone seeing a similar behavior ?
Re: L3 Issues
On Mon, Aug 1, 2011 at 10:39 AM, Khurram Khan brokenf...@gmail.com wrote: Hello and Good Morning, Are there reports of L3 having issues this morning ? Starting at about 10:10 A Pacific, I started seeing huge drops in traffic at various sites, including San Diego, Houston, San Antonio, Charlotte, NC, Philadelphia, etc. Anyone seeing a similar behavior ? Seeing similar issues here. Traffic going over Level3 seems to be dying. -- Brent Jones br...@servuhome.net
Re: L3 Issues
Yes. I am getting customer calls from people not able to reach us via Level3. --Original Message-- From: Khurram Khan To: nanog@nanog.org Subject: L3 Issues Sent: Aug 1, 2011 10:39 AM Hello and Good Morning, Are there reports of L3 having issues this morning ? Starting at about 10:10 A Pacific, I started seeing huge drops in traffic at various sites, including San Diego, Houston, San Antonio, Charlotte, NC, Philadelphia, etc. Anyone seeing a similar behavior ? Sent from my Verizon Wireless BlackBerry
RE: L3 Issues
We were seeing issues here as well, we have BGP to Level 3 down until they stabilize. We were seeing a number of sites as unreachable, but ping tests from the Level3 IP address on that interface were working. Looks like perhaps they stopped advertising our addresses or were advertising them through an incorrect path. -Original Message- From: David Hubbard [mailto:dhubb...@dino.hostasaurus.com] Sent: Monday, August 01, 2011 1:43 PM To: nanog@nanog.org Subject: RE: L3 Issues Seeing a big drop in outbound traffic on our L3 link starting about that time (we're a web host). Getting calls about sites down too; all the traces I've walked customers through so far have died on L3 networks. David -Original Message- From: Khurram Khan [mailto:brokenf...@gmail.com] Sent: Monday, August 01, 2011 1:40 PM To: nanog@nanog.org Subject: L3 Issues Hello and Good Morning, Are there reports of L3 having issues this morning ? Starting at about 10:10 A Pacific, I started seeing huge drops in traffic at various sites, including San Diego, Houston, San Antonio, Charlotte, NC, Philadelphia, etc. Anyone seeing a similar behavior ?
Re: L3 Issues
В 11:39 -0600 на 01.08.2011 (пн), Khurram Khan написа: Hello and Good Morning, Are there reports of L3 having issues this morning ? Starting at about 10:10 A Pacific, I started seeing huge drops in traffic at various sites, including San Diego, Houston, San Antonio, Charlotte, NC, Philadelphia, etc. Anyone seeing a similar behavior ? Yes, l3 in Dallas is dropping about 40% of the traffic we're sending them. -- Regards, Vasil Kolev signature.asc Description: This is a digitally signed message part
Re: L3 Issues
On Mon, 1 Aug 2011, Khurram Khan wrote: Hello and Good Morning, Are there reports of L3 having issues this morning ? Starting at about 10:10 A Pacific, I started seeing huge drops in traffic at various sites, including San Diego, Houston, San Antonio, Charlotte, NC, Philadelphia, etc. Anyone seeing a similar behavior ? Yes...about the same time here...maybe a few minutes later. I was seeing traces stopping at ae-63-63.ebr3.Atlanta2.Level3.net. Things seem to be moving again. I just unshut our L3 BGP session. Interestingly, while Level3 was having problems, our local Akamai cluster was apparently not working properly. images.apple.com resolves to our local Akamai cluster, but requests there were timing out. -- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Community troubleshooting étiquette/BCP (was: L3 Issues)
On 2011-08-01, at 1:48 PM, Jon Lewis wrote: Things seem to be moving again. I happen to have an L3 link out of NYC, but unfortunately I don't have a list of on-net L3 prefixes in any of the reportedly affected regions, so I'm unable to provide any data from my vantage point up here. I'm sure others are in my position as well. Is there any sort of etiquette/BCP for reporting issues like this to the community? Something that might specify a method of providing information a little more specific than just specifying the affected region(s)? Maybe a list of a few affected hosts/prefixes/URLS/etc? (incidentally, images.apple.com also resolves to our local Akamai cluster)
RE: Community troubleshooting étiquette/BCP (was: L3 Issues)
I did finally see a Level 3 network event posted about this in their portal. Actually they list two separate ones: A routing issue failure between Dallas, TX and Los Angeles, CA is impacting IP services. Impacted for: 1 hour 29 minutes A loss of connectivity to servers in Dallas, TX, Tustin, CA, and Tokyo, Japan caused an impact to CDN services. The second one probably explains the Akamai issues one poster mentioned. -Original Message- From: Jason Lixfeld [mailto:ja...@lixfeld.ca] Sent: Monday, August 01, 2011 2:03 PM To: nanog@nanog.org Subject: Community troubleshooting étiquette/BCP (was: L3 Issues) On 2011-08-01, at 1:48 PM, Jon Lewis wrote: Things seem to be moving again. I happen to have an L3 link out of NYC, but unfortunately I don't have a list of on-net L3 prefixes in any of the reportedly affected regions, so I'm unable to provide any data from my vantage point up here. I'm sure others are in my position as well. Is there any sort of etiquette/BCP for reporting issues like this to the community? Something that might specify a method of providing information a little more specific than just specifying the affected region(s)? Maybe a list of a few affected hosts/prefixes/URLS/etc? (incidentally, images.apple.com also resolves to our local Akamai cluster)
