Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

Master Ticket # is 10009886, current ETA is 1700 GMT per Level 3's portal Nick Colton Director of Network Operations ALLO Communications 610 Broadway - PO Box 1123 Imperial, NE 69033 308-633-7814 On Fri, Nov 20, 2015 at 9:25 AM, Erik Sundberg via Outages < outages@outages.org> wrote: > Level3

Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

If anyone get's a master TT# or anything like that, let me know. I'm sure I'll be following up with them all day for updates for my clients. On Fri, Nov 20, 2015 at 10:19 AM, Mike Swier wrote: > Seeing slowness: > > > > [root@info10 data]# traceroute compass.act.org >

Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

As far as I can tell, the route we are using to Chicago is back up, but am having a surprising number of ping timeouts/latency within the level3 path. I know they de-prioritize ICMP, but this degree of it is just silly and makes trying to understand what's going on to cause the performance issues

Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

I’m seeing loss at Dallas to Cleveland on level3. If Chicago is out of route because of a fiber cut guessing Cleveland is the redundant path but doesn’t have the capacity. Joseph From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Erik Sundberg via Outages

Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

Level3 has another fiber cut today this time in Dallas, went down around 9:00Am. From: Joseph Jackson [mailto:jjack...@aninetworks.net] Sent: Friday, November 20, 2015 10:20 AM To: Erik Sundberg ; Thom Cleary ; frnk...@iname.com;

Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

Level3 Just said there is a another fiber cut outside Dallas. From: Erik Sundberg Sent: Friday, November 20, 2015 10:01 AM To: 'Thom Cleary' ; frnk...@iname.com Cc: 'outages@outages.org' Subject: RE: [outages] XO / Level 3 Outage | Chicago - St Louis

[outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

*** CASCADED EXTERNAL NOTES 20-Nov-2015 17:38:27 GMT From CASE: 10009886 - Event Field Services are at the fault site evaluating the fiber damage. ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

*** CASCADED EXTERNAL NOTES 20-Nov-2015 16:31:59 GMT From CASE: 10009886 - Event Field Services arrived onsite in Dallas, TX to proceed with investigations. Testing indicated an issue approximately 43 kilometers from the site. Field Services also reported they were contacted by a road construction

[outages] Test

NOTICE: This electronic mail transmission may contain confidential information and is intended only for the person(s) named. Any use, copying, or disclosure by any other person is strictly prohibited. If you have received this transmission in error, please notify the sender via e-mail.

Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

Latest *** CASCADED EXTERNAL NOTES 20-Nov-2015 18:06:17 GMT From CASE: 10009886 - Event The fiber cut has been identified, caused by drilling works during road construction. A lot of cable has become wound around an auger. Field Services are excavating the 96-count fiber cable for repair.

Re: [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

*11/20/2015 8:54 PM GMT* Field Services continue with restoration efforts at the site. As more information is gathered about the extent of the damage, repair plans are being evaluated to determine the best method for restoral. Further details will be provided as they are received. *11/20/2015

[outages] packet loss to us-east-1

Anyone else seeing packet log into us-east-1? 54.240.243.121 (AS16509) is Amazon MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0%102.5 8.3 2.5 37.5 10.7 5. AS8121 199.127.107.189 0.0%103.8 7.9 3.4 34.9 9.5 6.

Re: [outages] packet loss to us-east-1

Saw that, thanks! -Grant On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field wrote: > Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can > tell. This was experienced via both layer42 and XO, but Amazon appears to > have resolved their issues at this time. >

Re: [outages] packet loss to us-east-1

Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/ 4. AS46375 50.ae4.gw.pao1.sonic.net1.2%813.5 12.0 2.6 91.0 16.2 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com 6.2%816.6 10.5 3.2 79.9

Re: [outages] packet loss to us-east-1

Hmmm, It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8: traceroute amazon.com traceroute: Warning: amazon.com has multiple addresses; using 176.32.103.205 traceroute to amazon.com (176.32.103.205), 64 hops max, 52 byte packets 1 192.168.20.1