Re: East coast outage

[Seth Mattinen]

On 2/2/17 14:12, Baldur Norddahl wrote:

Should we not be able to expect that these kind of failures are handled
seamlessly by rerouting through redundant paths? A little extra latency ok.
Packet loss, you failed to plan and upgrade your redundancy correctly.



Cost savings.

~Seth

Re: East coast outage

[Baldur Norddahl]

Should we not be able to expect that these kind of failures are handled
seamlessly by rerouting through redundant paths? A little extra latency ok.
Packet loss, you failed to plan and upgrade your redundancy correctly.

Regards

Baldur

Den 2. feb. 2017 21.34 skrev "Scott Farber" :

>From our ticket with Zayo:

*Reason for Outage Summary*: The cause of this issue was traced to an
active construction site. 50' of 432 ct cable in 8 conduits were damaged.
OSP was able to repair all conduits and pull and splice new fiber cable to
make this repair.

On Wed, Feb 1, 2017 at 10:06 AM, Jeroen Wunnink 
wrote:

> According to our optical noc they’re shooting new fiber as we speak. So
> now we wait.
>
>
>
>
> Jeroen Wunnink
> IP Engineering manager
> office: +31.208.200.622 ext. 1011
> Amsterdam Office
> www.gtt.net 
>
>
>
>
> On 01/02/2017, 15:41, "NANOG on behalf of Scott Farber" <
> nanog-boun...@nanog.org on behalf of sco...@gmail.com> wrote:
>
> I know Zayo has a cut somewhere between MD and DC, it's knocked a few
> things offline. Not sure if it's a shared conduit and who else may be
> affected. ETA for splicing crew is 1030 Eastern.
>
> On Wed, Feb 1, 2017 at 9:30 AM, Tom Beecher 
> wrote:
>
> > I see a couple things that leads me to believe there's something
> afoot in
> > NOVA as well.
> >
> > But not done with my first coffee, so unable to process any
> specifics yet.
> > :)
> >
> > On Wed, Feb 1, 2017 at 9:25 AM, Raymond Dijkxhoorn <
> > raym...@prolocation.net>
> > wrote:
> >
> > > Hello Ben,
> > >
> > >
> > > Is anyone else seeing connectivity issues along the east coast?
> Our pipe
> > >> through HE in NYC is showing loss to things behind most of
> Level3, and
> > >> Qwest below Washington.
> > >>
> > >> *Ben Hatton*
> > >>
> > >> Network Engineer
> > >>
> > >> Haefele TV Inc.
> > >>
> > >> d:(607)589-8000
> > >>
> > >> bhat...@htva.net
> > >>
> > >> www.htva.net
> > >>
> > >
> > > We see the same, traffic going from Amsterdam towards HE heading
> USA is
> > > experiencing big packetloss at the moment.
> > >
> > > Traffic heading towards Ashburn seems affected from our point of
> view.
> > >
> > > Bye,
> > > Raymond.
> > >
> > >
> >
>
>
>

Re: East coast outage

[Scott Farber]

>From our ticket with Zayo:

*Reason for Outage Summary*: The cause of this issue was traced to an
active construction site. 50' of 432 ct cable in 8 conduits were damaged.
OSP was able to repair all conduits and pull and splice new fiber cable to
make this repair.

On Wed, Feb 1, 2017 at 10:06 AM, Jeroen Wunnink 
wrote:

> According to our optical noc they’re shooting new fiber as we speak. So
> now we wait.
>
>
>
>
> Jeroen Wunnink
> IP Engineering manager
> office: +31.208.200.622 ext. 1011
> Amsterdam Office
> www.gtt.net 
>
>
>
>
> On 01/02/2017, 15:41, "NANOG on behalf of Scott Farber" <
> nanog-boun...@nanog.org on behalf of sco...@gmail.com> wrote:
>
> I know Zayo has a cut somewhere between MD and DC, it's knocked a few
> things offline. Not sure if it's a shared conduit and who else may be
> affected. ETA for splicing crew is 1030 Eastern.
>
> On Wed, Feb 1, 2017 at 9:30 AM, Tom Beecher 
> wrote:
>
> > I see a couple things that leads me to believe there's something
> afoot in
> > NOVA as well.
> >
> > But not done with my first coffee, so unable to process any
> specifics yet.
> > :)
> >
> > On Wed, Feb 1, 2017 at 9:25 AM, Raymond Dijkxhoorn <
> > raym...@prolocation.net>
> > wrote:
> >
> > > Hello Ben,
> > >
> > >
> > > Is anyone else seeing connectivity issues along the east coast?
> Our pipe
> > >> through HE in NYC is showing loss to things behind most of
> Level3, and
> > >> Qwest below Washington.
> > >>
> > >> *Ben Hatton*
> > >>
> > >> Network Engineer
> > >>
> > >> Haefele TV Inc.
> > >>
> > >> d:(607)589-8000
> > >>
> > >> bhat...@htva.net
> > >>
> > >> www.htva.net
> > >>
> > >
> > > We see the same, traffic going from Amsterdam towards HE heading
> USA is
> > > experiencing big packetloss at the moment.
> > >
> > > Traffic heading towards Ashburn seems affected from our point of
> view.
> > >
> > > Bye,
> > > Raymond.
> > >
> > >
> >
>
>
>

Re: East coast outage

[Jeroen Wunnink]

According to our optical noc they’re shooting new fiber as we speak. So now we 
wait.


 
 
Jeroen Wunnink
IP Engineering manager
office: +31.208.200.622 ext. 1011 
Amsterdam Office
www.gtt.net 


 

On 01/02/2017, 15:41, "NANOG on behalf of Scott Farber" 
 wrote:

I know Zayo has a cut somewhere between MD and DC, it's knocked a few
things offline. Not sure if it's a shared conduit and who else may be
affected. ETA for splicing crew is 1030 Eastern.

On Wed, Feb 1, 2017 at 9:30 AM, Tom Beecher  wrote:

> I see a couple things that leads me to believe there's something afoot in
> NOVA as well.
>
> But not done with my first coffee, so unable to process any specifics yet.
> :)
>
> On Wed, Feb 1, 2017 at 9:25 AM, Raymond Dijkxhoorn <
> raym...@prolocation.net>
> wrote:
>
> > Hello Ben,
> >
> >
> > Is anyone else seeing connectivity issues along the east coast?  Our 
pipe
> >> through HE in NYC is showing loss to things behind most of Level3, and
> >> Qwest below Washington.
> >>
> >> *Ben Hatton*
> >>
> >> Network Engineer
> >>
> >> Haefele TV Inc.
> >>
> >> d:(607)589-8000
> >>
> >> bhat...@htva.net
> >>
> >> www.htva.net
> >>
> >
> > We see the same, traffic going from Amsterdam towards HE heading USA is
> > experiencing big packetloss at the moment.
> >
> > Traffic heading towards Ashburn seems affected from our point of view.
> >
> > Bye,
> > Raymond.
> >
> >
>

Re: East coast outage

[Scott Farber]

I know Zayo has a cut somewhere between MD and DC, it's knocked a few
things offline. Not sure if it's a shared conduit and who else may be
affected. ETA for splicing crew is 1030 Eastern.

On Wed, Feb 1, 2017 at 9:30 AM, Tom Beecher  wrote:

> I see a couple things that leads me to believe there's something afoot in
> NOVA as well.
>
> But not done with my first coffee, so unable to process any specifics yet.
> :)
>
> On Wed, Feb 1, 2017 at 9:25 AM, Raymond Dijkxhoorn <
> raym...@prolocation.net>
> wrote:
>
> > Hello Ben,
> >
> >
> > Is anyone else seeing connectivity issues along the east coast?  Our pipe
> >> through HE in NYC is showing loss to things behind most of Level3, and
> >> Qwest below Washington.
> >>
> >> *Ben Hatton*
> >>
> >> Network Engineer
> >>
> >> Haefele TV Inc.
> >>
> >> d:(607)589-8000
> >>
> >> bhat...@htva.net
> >>
> >> www.htva.net
> >>
> >
> > We see the same, traffic going from Amsterdam towards HE heading USA is
> > experiencing big packetloss at the moment.
> >
> > Traffic heading towards Ashburn seems affected from our point of view.
> >
> > Bye,
> > Raymond.
> >
> >
>

Re: East coast outage

[Tom Beecher]

That'll do it.

/refills coffee

On Wed, Feb 1, 2017 at 9:26 AM, Jeroen Wunnink 
wrote:

> There’s a major fiber outage between Ashburn, VA and Philadelphia, PA.
>
>
>
> Jeroen Wunnink
> IP Engineering manager
> office: +31.208.200.622 ext. 1011
> Amsterdam Office
> www.gtt.net 
>
>
>
>
> On 01/02/2017, 15:22, "NANOG on behalf of Daniel Brisson" <
> nanog-boun...@nanog.org on behalf of dbris...@uvm.edu> wrote:
>
> Definitely seeing problems here in Vermont.  Reachability issues to
> amazon and other sites.  Cogent BGP sessions have bounced, but still up.
>
> Right now things look normal, but it was definitely rocky over the
> last hour.
>
> -dan
>
>
>
> Dan Brisson
> Network Engineer
> University of Vermont
>
>
> > -Original Message-
> > From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Benjamin
> > Hatton
> > Sent: Wednesday, February 01, 2017 9:17 AM
> > To: nanog@nanog.org
> > Subject: East coast outage
> >
> > Is anyone else seeing connectivity issues along the east coast?  Our
> pipe
> > through HE in NYC is showing loss to things behind most of Level3,
> and Qwest
> > below Washington.
> >
> > *Ben Hatton*
> >
> > Network Engineer
> >
> > Haefele TV Inc.
> >
> > d:(607)589-8000
> >
> > bhat...@htva.net
> >
> > www.htva.net
>
>
>

Re: East coast outage

[Benjamin Hatton]

>
> Definitely seeing problems here in Vermont.  Reachability issues to amazon
> and other sites.  Cogent BGP sessions have bounced, but still up.
> Right now things look normal, but it was definitely rocky over the last
> hour.
> -dan


Amazon was definitely on the list of sites with issues.  I threw the switch
over to a different provider so my CSR's wouldn't kill me. If you are
seeing it start to seem more normal, might throw a test prefix back in an
hour or so and see what it is doing.

*Ben Hatton*

Network Engineer

Haefele TV Inc.

d:(607)589-8000

bhat...@htva.net

www.htva.net

On Wed, Feb 1, 2017 at 9:22 AM, Daniel Brisson  wrote:

> Definitely seeing problems here in Vermont.  Reachability issues to amazon
> and other sites.  Cogent BGP sessions have bounced, but still up.
>
> Right now things look normal, but it was definitely rocky over the last
> hour.
>
> -dan
>
>
>
> Dan Brisson
> Network Engineer
> University of Vermont
>
>
> > -Original Message-
> > From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Benjamin
> > Hatton
> > Sent: Wednesday, February 01, 2017 9:17 AM
> > To: nanog@nanog.org
> > Subject: East coast outage
> >
> > Is anyone else seeing connectivity issues along the east coast?  Our pipe
> > through HE in NYC is showing loss to things behind most of Level3, and
> Qwest
> > below Washington.
> >
> > *Ben Hatton*
> >
> > Network Engineer
> >
> > Haefele TV Inc.
> >
> > d:(607)589-8000
> >
> > bhat...@htva.net
> >
> > www.htva.net
>

Re: East coast outage

[Tom Beecher]

I see a couple things that leads me to believe there's something afoot in
NOVA as well.

But not done with my first coffee, so unable to process any specifics yet.
:)

On Wed, Feb 1, 2017 at 9:25 AM, Raymond Dijkxhoorn 
wrote:

> Hello Ben,
>
>
> Is anyone else seeing connectivity issues along the east coast?  Our pipe
>> through HE in NYC is showing loss to things behind most of Level3, and
>> Qwest below Washington.
>>
>> *Ben Hatton*
>>
>> Network Engineer
>>
>> Haefele TV Inc.
>>
>> d:(607)589-8000
>>
>> bhat...@htva.net
>>
>> www.htva.net
>>
>
> We see the same, traffic going from Amsterdam towards HE heading USA is
> experiencing big packetloss at the moment.
>
> Traffic heading towards Ashburn seems affected from our point of view.
>
> Bye,
> Raymond.
>
>

Re: East coast outage

[Jeroen Wunnink]

There’s a major fiber outage between Ashburn, VA and Philadelphia, PA.

 
 
Jeroen Wunnink
IP Engineering manager
office: +31.208.200.622 ext. 1011 
Amsterdam Office
www.gtt.net 


 

On 01/02/2017, 15:22, "NANOG on behalf of Daniel Brisson" 
 wrote:

Definitely seeing problems here in Vermont.  Reachability issues to amazon 
and other sites.  Cogent BGP sessions have bounced, but still up.

Right now things look normal, but it was definitely rocky over the last 
hour.

-dan



Dan Brisson
Network Engineer
University of Vermont


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Benjamin
> Hatton
> Sent: Wednesday, February 01, 2017 9:17 AM
> To: nanog@nanog.org
> Subject: East coast outage
> 
> Is anyone else seeing connectivity issues along the east coast?  Our pipe
> through HE in NYC is showing loss to things behind most of Level3, and 
Qwest
> below Washington.
> 
> *Ben Hatton*
> 
> Network Engineer
> 
> Haefele TV Inc.
> 
> d:(607)589-8000
> 
> bhat...@htva.net
> 
> www.htva.net

Re: East coast outage

[Raymond Dijkxhoorn]

Hello Ben,


Is anyone else seeing connectivity issues along the east coast?  Our pipe
through HE in NYC is showing loss to things behind most of Level3, and
Qwest below Washington.

*Ben Hatton*

Network Engineer

Haefele TV Inc.

d:(607)589-8000

bhat...@htva.net

www.htva.net


We see the same, traffic going from Amsterdam towards HE heading USA is 
experiencing big packetloss at the moment.


Traffic heading towards Ashburn seems affected from our point of view.

Bye,
Raymond.

RE: East coast outage

[Daniel Brisson]

Definitely seeing problems here in Vermont.  Reachability issues to amazon and 
other sites.  Cogent BGP sessions have bounced, but still up.

Right now things look normal, but it was definitely rocky over the last hour.

-dan



Dan Brisson
Network Engineer
University of Vermont


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Benjamin
> Hatton
> Sent: Wednesday, February 01, 2017 9:17 AM
> To: nanog@nanog.org
> Subject: East coast outage
> 
> Is anyone else seeing connectivity issues along the east coast?  Our pipe
> through HE in NYC is showing loss to things behind most of Level3, and Qwest
> below Washington.
> 
> *Ben Hatton*
> 
> Network Engineer
> 
> Haefele TV Inc.
> 
> d:(607)589-8000
> 
> bhat...@htva.net
> 
> www.htva.net