Re: [outages] Zayo Transit

[mike.lyon--- via Outages]

Looks like im.aol.com and webmail.aol.com (which both resolve to Akamai IPs) 
die in the Zayo network still...

> On Sep 29, 2017, at 13:07, Andy Litzinger via Outages  
> wrote:
> 
> We use Zayo for IP access and are still experiencing issues.  Here's the 
> response we've recived from Zayo Support:
> 
> We are experiencing a major network event across multiple peering points from 
> the global internet table that is impacting traffic throughout the internet. 
> We are working to mitigate the incoming traffic across these peers at this 
> time to lessen the impact until the root-cause of the influx can be 
> identified and halted. We expect to have a resolution within the hour, 
> however this appears to be incoming to the Zayo global internet routing table 
> from our major peering partners. We appreciate your patience as our technical 
> teams are fully engaged to cease impact to our customers at this time and we 
> will update you as the status changes. We have opened the following 
> customer-facing bridge to assist with real-time updates as they are known to 
> our NCC team: 
> 
> -andy
> 
>> On Fri, Sep 29, 2017 at 12:59 PM, Adam Gregory via Outages 
>>  wrote:
>> Looks like zayo went above their 60k prefix limit causing peers to drop if 
>> configured to do so. Looks like they updated to 65k per peeringdb. 
>> 
>> On Sep 29, 2017, at 3:12 PM, Mike Davis via Outages  
>> wrote:
>> 
>>> :|   sigh...
>>> 
>>>  created by Eric Zalesak on Sep 29, 2017 12:02:03 PM
>>> Turn up customer was Singtel. We are still seeing sessions dropping after 
>>> mitigating Singtel.
>>> 
>>>  created by Eric Zalesak on Sep 29, 2017 11:55:59 AM
>>> Suspected turn up with open policy.
>>> 
 On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages 
  wrote:
 We are still seeing problems with Zayo (indirectly) as of now.
 
 
 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages
>>> 
>>> ___
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>> 
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>> 
> 
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Transit

[Sean Pedersen via Outages]

Last update we received:

 

Dear Zayo Customer, The majority of our peering links with our affected peering 
partners have restored with ancillary points still left. We are working to 
clear damping and reset those sessions left and some customers may still 
experience latency until this is completed. Thank you for your patience and if 
you are still experiencing a hard down issue, let us know so we can work to 
resolve it in a timely manner. 

 

 

From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Andy Litzinger 
via Outages
Sent: Friday, September 29, 2017 1:07 PM
To: Adam Gregory 
Cc: outages@outages.org
Subject: Re: [outages] Zayo Transit

 

We use Zayo for IP access and are still experiencing issues.  Here's the 
response we've recived from Zayo Support:

We are experiencing a major network event across multiple peering points from 
the global internet table that is impacting traffic throughout the internet. We 
are working to mitigate the incoming traffic across these peers at this time to 
lessen the impact until the root-cause of the influx can be identified and 
halted. We expect to have a resolution within the hour, however this appears to 
be incoming to the Zayo global internet routing table from our major peering 
partners. We appreciate your patience as our technical teams are fully engaged 
to cease impact to our customers at this time and we will update you as the 
status changes. We have opened the following customer-facing bridge to assist 
with real-time updates as they are known to our NCC team: 

-andy

 

On Fri, Sep 29, 2017 at 12:59 PM, Adam Gregory via Outages  > wrote:

Looks like zayo went above their 60k prefix limit causing peers to drop if 
configured to do so. Looks like they updated to 65k per peeringdb. 


On Sep 29, 2017, at 3:12 PM, Mike Davis via Outages  > wrote:

:|   sigh...

 

 created by Eric Zalesak on Sep 29, 2017 12:02:03 PM

Turn up customer was Singtel. We are still seeing sessions dropping after 
mitigating Singtel.

 

 created by Eric Zalesak on Sep 29, 2017 11:55:59 AM

Suspected turn up with open policy.

 

On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages 
 > wrote:

We are still seeing problems with Zayo (indirectly) as of now.



___
Outages mailing list
Outages@outages.org  
https://puck.nether.net/mailman/listinfo/outages

 

___
Outages mailing list
Outages@outages.org  
https://puck.nether.net/mailman/listinfo/outages


___
Outages mailing list
Outages@outages.org  
https://puck.nether.net/mailman/listinfo/outages

 

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Transit

[Andy Litzinger via Outages]

We use Zayo for IP access and are still experiencing issues.  Here's the
response we've recived from Zayo Support:

We are experiencing a major network event across multiple peering points
from the global internet table that is impacting traffic throughout the
internet. We are working to mitigate the incoming traffic across these
peers at this time to lessen the impact until the root-cause of the influx
can be identified and halted. We expect to have a resolution within the
hour, however this appears to be incoming to the Zayo global internet
routing table from our major peering partners. We appreciate your patience
as our technical teams are fully engaged to cease impact to our customers
at this time and we will update you as the status changes. We have opened
the following customer-facing bridge to assist with real-time updates as
they are known to our NCC team: 

-andy

On Fri, Sep 29, 2017 at 12:59 PM, Adam Gregory via Outages <
outages@outages.org> wrote:

> Looks like zayo went above their 60k prefix limit causing peers to drop if
> configured to do so. Looks like they updated to 65k per peeringdb.
>
> On Sep 29, 2017, at 3:12 PM, Mike Davis via Outages 
> wrote:
>
> :|   sigh...
>
>  created by Eric Zalesak on Sep 29, 2017 12:02:03 PM
> Turn up customer was Singtel. We are still seeing sessions dropping after
> mitigating Singtel.
>
>  created by Eric Zalesak on Sep 29, 2017 11:55:59 AM
> Suspected turn up with open policy.
>
> On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages <
> outages@outages.org> wrote:
>
>> We are still seeing problems with Zayo (indirectly) as of now.
>>
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Transit

[Luke Rockwell via Outages]

Zayo Master Ticket TTN-0001950534

On Fri, Sep 29, 2017 at 12:59 PM, Adam Gregory via Outages <
outages@outages.org> wrote:

> Looks like zayo went above their 60k prefix limit causing peers to drop if
> configured to do so. Looks like they updated to 65k per peeringdb.
>
> On Sep 29, 2017, at 3:12 PM, Mike Davis via Outages 
> wrote:
>
> :|   sigh...
>
>  created by Eric Zalesak on Sep 29, 2017 12:02:03 PM
> Turn up customer was Singtel. We are still seeing sessions dropping after
> mitigating Singtel.
>
>  created by Eric Zalesak on Sep 29, 2017 11:55:59 AM
> Suspected turn up with open policy.
>
> On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages <
> outages@outages.org> wrote:
>
>> We are still seeing problems with Zayo (indirectly) as of now.
>>
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Transit

[Mike Davis via Outages]

:|   sigh...

 created by Eric Zalesak on Sep 29, 2017 12:02:03 PM
Turn up customer was Singtel. We are still seeing sessions dropping after
mitigating Singtel.

 created by Eric Zalesak on Sep 29, 2017 11:55:59 AM
Suspected turn up with open policy.

On Fri, Sep 29, 2017 at 11:22 AM, Steven Raymond via Outages <
outages@outages.org> wrote:

> We are still seeing problems with Zayo (indirectly) as of now.
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Transit

[Steven Raymond via Outages]

We are still seeing problems with Zayo (indirectly) as of now.


___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Down?

[Alvin Chan via Outages]

We saw a latency spike at 10:40a pacific daylight time, located in Las
Vegas zColo.

On Fri, Sep 29, 2017 at 11:12 AM, Erik Wooding via Outages <
outages@outages.org> wrote:

> I've got about a dozen Zayo sites that are down for the last 20 minutes or
> so, anyone else seeing significant outages in the midwest/west?
>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Down?

[John Neiberger via Outages]

We just saw them trip our max prefix limits. I wonder if "something bad"
happened and sessions are being torn down with multiple providers because
of prefix limit violations.

On Fri, Sep 29, 2017 at 12:12 PM, Erik Wooding via Outages <
outages@outages.org> wrote:

> I've got about a dozen Zayo sites that are down for the last 20 minutes or
> so, anyone else seeing significant outages in the midwest/west?
>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Transit

[Nicholas Suan via Outages]

Yeah, seeing a huge spike in updates, especially in Chicago.

On Fri, Sep 29, 2017 at 2:11 PM, Mike Davis via Outages
 wrote:
> Anyone noticing problems with Zayo transit within the last 15 minutes?
>
> Peers are about 125k routes lower in Chicago and Amsterdam.
>
> Mike
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Transit

[Ryan Collins via Outages]

Yes,

We are seeing tracert’s fail to our destination in New York from France, 
Sweden, Denmark, Toronto, CA, and Philadelphia, PA.

Regards,
Ryan



> On Sep 29, 2017, at 14:11, Mike Davis via Outages  wrote:
> 
> Anyone noticing problems with Zayo transit within the last 15 minutes?  
> 
> Peers are about 125k routes lower in Chicago and Amsterdam.
> 
> Mike
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Zayo Down?

[Erik Wooding via Outages]

I've got about a dozen Zayo sites that are down for the last 20 minutes or
so, anyone else seeing significant outages in the midwest/west?
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Zayo Transit

[Mike Davis via Outages]

Anyone noticing problems with Zayo transit within the last 15 minutes?

Peers are about 125k routes lower in Chicago and Amsterdam.

Mike
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages