RE: Anyone having issues with Equinix IX out of Ashburn?

[Graham Johnston]

I think we saw an issue like this a few weeks back in Chicago.  It took them 
longer than I would have expected to fix it, later they ultimately ended up 
upgrading software I think.

Graham Johnston
Network Planner
Westman Communications Group
204.717.2829
johnst...@westmancom.com
think green; don't print this email.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Baker, Byrn
Sent: Friday, November 27, 2015 8:32 AM
To: nanog@nanog.org
Subject: RE: Anyone having issues with Equinix IX out of Ashburn?


All of our IX peers dropped around 11 UTC. All recovered about 30 mins later.


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jeroen Wunnink
Sent: Friday, November 27, 2015 8:13 AM
To: nanog@nanog.org
Subject: Re: Anyone having issues with Equinix IX out of Ashburn?

We've seen issues as well. We've just started to turn the exchange up again and 
check if it's fixed.


On 27/11/15 15:03, Nick Ellermann wrote:
> At about 4:15 am eastern we lost our bgp peers on the Ashburn IX at Equinix. 
> Equinix is not responding to our support requests, either they are overloaded 
> with support requests or all on holiday. Curious if others know if there are 
> known issues at this site or is it just us.
>
>
> Sincerely,
> Nick Ellermann - CTO & VP Cloud Services BroadAspect
>
> E: nellerm...@broadaspect.com<mailto:nellerm...@broadaspect.com>
> P: 703-297-4639
> F: 703-996-4443
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
> MATERIAL and is thus for use only by the intended recipient. If you received 
> this in error, please contact the sender and delete the e-mail and its 
> attachments from all computers.
>


--

Jeroen Wunnink
IP Engineering Manager - Hibernia Networks Main numbers (Ext: 1011): USA 
+1.908.516.4200 | UK +44.1704.322.300 Netherlands +31.208.200.622 | 24/7 IP NOC 
Phone: +31.20.82.00.623 jeroen.wunn...@hibernianetworks.com
www.hibernianetworks.com

This e-mail and any attachments thereto is intended only for use by the 
addressee(s) named herein and may be proprietary and/or legally privileged. If 
you are not the intended recipient of this e-mail, you are hereby notified that 
any dissemination, distribution or copying of this email, and any attachments 
thereto, without the prior written permission of the sender is strictly 
prohibited. If you receive this e-mail in error, please immediately telephone 
or e-mail the sender and permanently delete the original copy and any copy of 
this e-mail, and any printout thereof. All documents, contracts or agreements 
referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an 
attachment to this e-mail may contain software viruses that could damage your 
own computer system. While Hibernia Networks has taken every reasonable 
precaution to minimize this risk, we cannot accept liability for any damage 
that you sustain as a result of software viruses. You should carry out your own 
virus checks before opening any attachment.

RE: Anyone having issues with Equinix IX out of Ashburn?

[Frank Bulk]

Storm caused by an L2 loop, malicious attack, bug in router code, or something 
else?

Frank

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Xavier Beaudouin
Sent: Friday, November 27, 2015 8:22 AM
To: nanog@nanog.org
Subject: Re: Anyone having issues with Equinix IX out of Ashburn?

Hi,

Seems it was some icmpv6 neighbor solicitation multicast storm.

Got about > 800kpps this morning for about 4hours...

Strange there is no rate limits somewhere...

Regards

Re: Anyone having issues with Equinix IX out of Ashburn?

[Jeroen Wunnink]

We've seen issues as well. We've just started to turn the exchange up
again and check if it's fixed.


On 27/11/15 15:03, Nick Ellermann wrote:

At about 4:15 am eastern we lost our bgp peers on the Ashburn IX at Equinix. 
Equinix is not responding to our support requests, either they are overloaded 
with support requests or all on holiday. Curious if others know if there are 
known issues at this site or is it just us.


Sincerely,
Nick Ellermann - CTO & VP Cloud Services
BroadAspect

E: nellerm...@broadaspect.com
P: 703-297-4639
F: 703-996-4443

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.




--

Jeroen Wunnink
IP Engineering Manager - Hibernia Networks
Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300
Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623
jeroen.wunn...@hibernianetworks.com
www.hibernianetworks.com

This e-mail and any attachments thereto is intended only for use by the 
addressee(s) named herein and may be proprietary and/or legally privileged. If 
you are not the intended recipient of this e-mail, you are hereby notified that 
any dissemination, distribution or copying of this email, and any attachments 
thereto, without the prior written permission of the sender is strictly 
prohibited. If you receive this e-mail in error, please immediately telephone 
or e-mail the sender and permanently delete the original copy and any copy of 
this e-mail, and any printout thereof. All documents, contracts or agreements 
referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an 
attachment to this e-mail may contain software viruses that could damage your 
own computer system. While Hibernia Networks has taken every reasonable 
precaution to minimize this risk, we cannot accept liability for any damage 
that you sustain as a result of software viruses. You should carry out your own 
virus checks before opening any attachment.

RE: Anyone having issues with Equinix IX out of Ashburn?

[Eric Van Tol]

Received a notification shortly after reporting the problem at 6am this morning 
about "issues" on the Exchange. We saw packet input on our routers in upwards 
of 350K pps, with all peers disabled. Looked to be a broadcast/multicast storm.

Seems to be good now, but no explanation of what was found.

-evt

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jeroen Wunnink
> Sent: Friday, November 27, 2015 9:13 AM
> To: nanog@nanog.org
> Subject: Re: Anyone having issues with Equinix IX out of Ashburn?
> 
> We've seen issues as well. We've just started to turn the exchange up
> again and check if it's fixed.
> 
> 
> On 27/11/15 15:03, Nick Ellermann wrote:
> > At about 4:15 am eastern we lost our bgp peers on the Ashburn IX at
> Equinix. Equinix is not responding to our support requests, either they are
> overloaded with support requests or all on holiday. Curious if others know
> if there are known issues at this site or is it just us.
> >
> >
> > Sincerely,
> > Nick Ellermann - CTO & VP Cloud Services
> > BroadAspect
> >
> > E: nellerm...@broadaspect.com<mailto:nellerm...@broadaspect.com>
> > P: 703-297-4639
> > F: 703-996-4443
> >
> > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
> >
> 
> 
> --
> 
> Jeroen Wunnink
> IP Engineering Manager - Hibernia Networks
> Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300
> Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623
> jeroen.wunn...@hibernianetworks.com
> www.hibernianetworks.com
> 
> This e-mail and any attachments thereto is intended only for use by the
> addressee(s) named herein and may be proprietary and/or legally privileged.
> If you are not the intended recipient of this e-mail, you are hereby
> notified that any dissemination, distribution or copying of this email, and
> any attachments thereto, without the prior written permission of the sender
> is strictly prohibited. If you receive this e-mail in error, please
> immediately telephone or e-mail the sender and permanently delete the
> original copy and any copy of this e-mail, and any printout thereof. All
> documents, contracts or agreements referred or attached to this e-mail are
> SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may
> contain software viruses that could damage your own computer system. While
> Hibernia Networks has taken every reasonable precaution to minimize this
> risk, we cannot accept liability for any damage that you sustain as a result
> of software viruses. You should carry out your own virus checks before
> opening any attachment.

Re: Anyone having issues with Equinix IX out of Ashburn?

[Xavier Beaudouin]

Hi,

Seems it was some icmpv6 neighbor solicitation multicast storm.

Got about > 800kpps this morning for about 4hours...

Strange there is no rate limits somewhere...

Regards

RE: Anyone having issues with Equinix IX out of Ashburn?

[Baker, Byrn]

All of our IX peers dropped around 11 UTC. All recovered about 30 mins later.


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jeroen Wunnink
Sent: Friday, November 27, 2015 8:13 AM
To: nanog@nanog.org
Subject: Re: Anyone having issues with Equinix IX out of Ashburn?

We've seen issues as well. We've just started to turn the exchange up again and 
check if it's fixed.


On 27/11/15 15:03, Nick Ellermann wrote:
> At about 4:15 am eastern we lost our bgp peers on the Ashburn IX at Equinix. 
> Equinix is not responding to our support requests, either they are overloaded 
> with support requests or all on holiday. Curious if others know if there are 
> known issues at this site or is it just us.
>
>
> Sincerely,
> Nick Ellermann - CTO & VP Cloud Services BroadAspect
>
> E: nellerm...@broadaspect.com<mailto:nellerm...@broadaspect.com>
> P: 703-297-4639
> F: 703-996-4443
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
> MATERIAL and is thus for use only by the intended recipient. If you received 
> this in error, please contact the sender and delete the e-mail and its 
> attachments from all computers.
>


--

Jeroen Wunnink
IP Engineering Manager - Hibernia Networks Main numbers (Ext: 1011): USA 
+1.908.516.4200 | UK +44.1704.322.300 Netherlands +31.208.200.622 | 24/7 IP NOC 
Phone: +31.20.82.00.623 jeroen.wunn...@hibernianetworks.com
www.hibernianetworks.com

This e-mail and any attachments thereto is intended only for use by the 
addressee(s) named herein and may be proprietary and/or legally privileged. If 
you are not the intended recipient of this e-mail, you are hereby notified that 
any dissemination, distribution or copying of this email, and any attachments 
thereto, without the prior written permission of the sender is strictly 
prohibited. If you receive this e-mail in error, please immediately telephone 
or e-mail the sender and permanently delete the original copy and any copy of 
this e-mail, and any printout thereof. All documents, contracts or agreements 
referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an 
attachment to this e-mail may contain software viruses that could damage your 
own computer system. While Hibernia Networks has taken every reasonable 
precaution to minimize this risk, we cannot accept liability for any damage 
that you sustain as a result of software viruses. You should carry out your own 
virus checks before opening any attachment.