[outages] Spectrum (and possibly AT and VZW) outage in Dane County WI

[Jeremy Charles via Outages]

At about 11:20pm central time, we lost certain Charter/Spectrum fiber services 
at our corporate campus in Verona WI.  The two fiber circuits that went down 
are used as backhaul for Verizon and AT from our in-house cellular system (I 
don't have any visibility in to the sizes or where they go).  Other fiber 
services that we have with Spectrum, including Internet and SIP, are still up.

I just received an update from Spectrum stating that they have engaged Spectrum 
Construction to locate a possible fiber cut.

I'm also receiving anecdotal reports from coworkers of coinciding problems with:

  *   Verizon and AT cell phone services in the area surrounding but outside 
of our corporate campus
  *   Spectrum home Internet outages, including in neighboring Mount Horeb, WI

If you've got any intel or accompanying observations, I'd be interested in 
hearing from you either directly or via outages-discuss.

===
Jeremy Charles
Epic's Computer and Technology Services Division
jchar...@epic.com
608-271-9000

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] TDS Telecom outage?

[Jeremy Charles via Outages]

We lost our 10-gig Internet service in Verona WI with TDS Telecom at 15:05 
central time today.   Another business unit also lost a 100-gig with TDS 
Telecom that runs between Verona WI and Chicago IL around the same time.   My 
teammate called TDS to open a ticket at about 15:25 and, at that point, the 
front line person who took the call didn't mention any awareness of a larger 
issue.

Anyone seeing anything potentially related?

At this point we're just chilling and waiting to hear back.  Redundant carriers 
FTW.

===
Jeremy Charles
Epic's Computer and Technology Services Division
jchar...@epic.com
608-271-9000

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Fiber cut possibly along I-90/94 corridor in WI

[Jeremy Charles via Outages]

Our affected circuits came back up at 10:32am.  WIN subsequently confirmed that 
there was a fiber cut, but that their fiber provider was able to move all 
services on to spare fibers in order to restore traffic while the cut fibers 
are being worked on.


From: Jeremy Charles
Sent: Wednesday, June 5, 2019 10:34 AM
To: 'outages@outages.org' 
Subject: Fiber cut possibly along I-90/94 corridor in WI

We lost a couple (unprotected) circuits with Wisconsin Independent Network at 
about 9:50 CDT this morning, running from Verona WI to Rochester MN and MSP.  
One of our team members called WIN's NMC and the tech who answered said his 
phones just started lighting up like Christmas trees and their network 
engineers were beginning the troubleshooting process.

I'm pretty sure our two circuits are groomed along the I-90/94 corridor between 
Madison and Tomah, and then they split to follow 90 and 94 separately.  If this 
is a fiber cut in that area, it's likely to impact a number of carriers.

Obviously, it was too early for WIN to give us a diagnosis and ETR at this 
point.  They're pretty good about sending updates when they've got info to 
share.

Anyone else seeing impact?

==
Jeremy Charles
Epic's Computer and Technology Services Division (CatNet)
jchar...@epic.com
608-271-9000

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Fiber cut possibly along I-90/94 corridor in WI

[Jeremy Charles via Outages]

We lost a couple (unprotected) circuits with Wisconsin Independent Network at 
about 9:50 CDT this morning, running from Verona WI to Rochester MN and MSP.  
One of our team members called WIN's NMC and the tech who answered said his 
phones just started lighting up like Christmas trees and their network 
engineers were beginning the troubleshooting process.

I'm pretty sure our two circuits are groomed along the I-90/94 corridor between 
Madison and Tomah, and then they split to follow 90 and 94 separately.  If this 
is a fiber cut in that area, it's likely to impact a number of carriers.

Obviously, it was too early for WIN to give us a diagnosis and ETR at this 
point.  They're pretty good about sending updates when they've got info to 
share.

Anyone else seeing impact?

==
Jeremy Charles
Epic's Computer and Technology Services Division (CatNet)
jchar...@epic.com
608-271-9000

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Fiber cut in Wisconsin

[Jeremy Charles via Outages]

There is a fiber cut in Central Wisconsin that affects, at a minimum, Wisconsin 
independent Network. No ETR at this point. The event started a little after 
11:30. I don't know what impact this may have on other carriers. I do know that 
WIN usually rentS strands in other peoples cables.

Sent from my phone; disregard quality of message.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Charter issues this morning?

[Jeremy Charles via Outages]

It seems like the Charter issue this morning was "regional" (perhaps a rather 
small "region").  It sounds like it primarily affected the Madison, WI area, 
which we hang off of (we're in Verona, WI).

We re-enabled our BGP advertisements via Charter about 15 minutes ago and 
everything seems to be operating normally.  Yes, we like to hold off for a 
while after the up-stream gives the all-clear, just in case there are some 
lingering issues that still need to shake out.

Thanks to those who replied with observations and offers of assistance, both on 
and off-list.


From: Jeremy Charles
Sent: Friday, February 24, 2017 10:01 AM
To: 'outages@outages.org' 
Subject: RE: Charter issues this morning?

Update received from Charter just a moment ago   I think we'll give it a 
bit for things to settle down before we put traffic back on them.

STATUS: The SE NOC has been advised that the bad fiber has been re-spliced at 
this time. Following the completion of these actions, both fibers have been 
verified and the original links have restored. If you have any questions or 
concerns please call 866-603-3199 and reference your ticket number.


From: Jeremy Charles
Sent: Friday, February 24, 2017 8:45 AM
To: 'outages@outages.org' >
Subject: Charter issues this morning?

Anyone else using Charter Internet have issues with connectivity this morning?

We (AS 10359) have site-to-site VPN tunnels to about 150 customers across the 
US.  We started seeing loss of connectivity to about 20% of them at about 06:25 
CST.  When we filtered out all advertisements sent/received via Charter, we 
regained connectivity to everyone.  (We have two ISPs in addition to Charter.)

Additionally, our total aggregate Internet consumption jumped from about 200 
Mbps to 500 Mbps as soon as Charter was taken out of the mix.  It looks like a 
lot of traffic was preferring to come to us via Charter (which is normal) but 
just wasn't making it here.

One of our engineers waited on hold for Charter for about 20 minutes and then 
gave up to go get coffee.   :-)

[https://mrtg1.epic.com/mrtg/internet/inbound-borders.png]

==
Jeremy Charles
Epic's Computer and Technology Services Division
jchar...@epic.com
608-271-9000

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Charter issues this morning?

[Jeremy Charles via Outages]

Update received from Charter just a moment ago   I think we'll give it a 
bit for things to settle down before we put traffic back on them.

STATUS: The SE NOC has been advised that the bad fiber has been re-spliced at 
this time. Following the completion of these actions, both fibers have been 
verified and the original links have restored. If you have any questions or 
concerns please call 866-603-3199 and reference your ticket number.


From: Jeremy Charles
Sent: Friday, February 24, 2017 8:45 AM
To: 'outages@outages.org' 
Subject: Charter issues this morning?

Anyone else using Charter Internet have issues with connectivity this morning?

We (AS 10359) have site-to-site VPN tunnels to about 150 customers across the 
US.  We started seeing loss of connectivity to about 20% of them at about 06:25 
CST.  When we filtered out all advertisements sent/received via Charter, we 
regained connectivity to everyone.  (We have two ISPs in addition to Charter.)

Additionally, our total aggregate Internet consumption jumped from about 200 
Mbps to 500 Mbps as soon as Charter was taken out of the mix.  It looks like a 
lot of traffic was preferring to come to us via Charter (which is normal) but 
just wasn't making it here.

One of our engineers waited on hold for Charter for about 20 minutes and then 
gave up to go get coffee.   :-)

[https://mrtg1.epic.com/mrtg/internet/inbound-borders.png]

==
Jeremy Charles
Epic's Computer and Technology Services Division
jchar...@epic.com
608-271-9000

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Charter issues this morning?

[Jeremy Charles via Outages]

Anyone else using Charter Internet have issues with connectivity this morning?

We (AS 10359) have site-to-site VPN tunnels to about 150 customers across the 
US.  We started seeing loss of connectivity to about 20% of them at about 06:25 
CST.  When we filtered out all advertisements sent/received via Charter, we 
regained connectivity to everyone.  (We have two ISPs in addition to Charter.)

Additionally, our total aggregate Internet consumption jumped from about 200 
Mbps to 500 Mbps as soon as Charter was taken out of the mix.  It looks like a 
lot of traffic was preferring to come to us via Charter (which is normal) but 
just wasn't making it here.

One of our engineers waited on hold for Charter for about 20 minutes and then 
gave up to go get coffee.   :-)

[https://mrtg1.epic.com/mrtg/internet/inbound-borders.png]

==
Jeremy Charles
Epic's Computer and Technology Services Division
jchar...@epic.com
608-271-9000

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Level 3 Chicago

[Jeremy Charles via Outages]

We connect to Level3’s MPLS cloud and WebEx SIP service in Chicago (Equinix) 
and I’m not seeing any evidence of trouble on either service.


From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Andrew Bunde 
via Outages
Sent: Wednesday, February 8, 2017 3:15 PM
To: 'outages@outages.org' 
Subject: [outages] Level 3 Chicago

Lost a circuit around 4PM EST today. Anyone else?
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Level3 Internet in Amsterdam

[Jeremy Charles via Outages]

Did anyone else in Amsterdam lose Level3 Internet from about 16:42 UTC to about 
16:55 UTC today?

The response on our ticket seems to suggest it was a bigger problem than just 
for us...


12/14/2016 17:06:47 GMT - The Level 3 IP Network Operations Center (NOC) 
responded to alarms indicating a router issue in Amsterdam, Netherlands that 
impacted IP traffic. The IP NOC investigated and details are pending.  Please 
be advised that updates for this event will be relayed at a minimum of hourly 
unless otherwise noted. The information conveyed hereafter is associated to 
live troubleshooting effort and as the discovery process evolves through to 
service resolution, ticket closure, or post incident review, details may evolve.


==
Jeremy Charles
Epic's Computer and Technology Services Division
jchar...@epic.com
608-271-9000

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] TW/Level 3 issues?

[Jeremy Charles via Outages]

We lost Level 3 MPLS between Verona WI and Amsterdam from about 09:50am CDT to 
about 10:00am CDT.   It bounced several times during that ten-minute time 
frame.  We’re still seeing some evidence of instability or packet loss, but no 
hard downs.


From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Jayson Navitsky 
via Outages
Sent: Wednesday, November 2, 2016 10:01 AM
To: Brad Riemann 
Cc: outages@outages.org
Subject: Re: [outages] TW/Level 3 issues?

Confirmed - we saw some MPLS circuits drop between NJ, NY and Houston along 
with some IP circuits BGP sessions drop in NYC.Looks like it's clearing up now.

On Wed, Nov 2, 2016 at 10:57 AM, Brad Riemann via Outages 
> wrote:
Anyone seeing anything, have links with high (80-100%) packetloss in DC and 
Chicago, our P2P just dropped as well.

[https://ci3.googleusercontent.com/proxy/MI3Tr0-S1A9lnN4pbtzF7dl0IN4-Vcue5z3XqIWdQQR4sDljaGNwzEaPyLYO0ThmlKci55t-7Fg49PW-J3pRM0Mm=s0-d-e1-ft#https://www.nmi.com/images/nmi-signature.png]

Brad Riemann
Senior Systems and Network Engineer
Tel: 847.352.4850 x125
briem...@nmi.com | www.nmi.com




___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages