Re: [outages] Level3 VoIP Issues?

Time to move it to outages-discussion, so CCing that list. Don't reply-all and keep it on both lists... : Fix Action: Configuration adjustments were implemented to allow calls : to reach the switches and complete appropriately. Reason for Outage : (RFO) Summary: ...The Voice NOC was able to

Re: [outages] Level3 VoIP Issues?

On 10/04/2016 12:08 PM, Michael Cargile via Outages wrote: Both Bandwidth and Vitelity are just VoIP resellers. They use larger carriers like Level3 for their termination and origination. As such Level3's outage is effecting their services as well. Bandwidth started out as a Level3 resale

Re: [outages] Level3 VoIP Issues?

g] On Behalf Of Rob Hall via Outages Sent: Tuesday, October 04, 2016 3:35 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3 VoIP Issues? Call flow appears to be normal as of 4:00PM Eastern across Charter, Comcast and Cox. Thanks! -- Rob Hall, CCNP R,

Re: [outages] Level3 VoIP Issues?

Here is my last update from Level 3: Outage Start: October 4 , 2016 14:06 GMT Outage Stop: October 4 , 2016 15:31 GMT Root Cause: Calls were not present in Level 3 voice switches, impacting voice services in multiple markets of the United States. Fix Action: Configuration adjustments were

Re: [outages] Level3 VoIP Issues?

place until a root cause analysis is complete." Frank -Original Message- From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Rob Hall via Outages Sent: Tuesday, October 04, 2016 3:35 PM To: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Call flow appears to

Re: [outages] Level3 VoIP Issues?

: Outages [mailto:outages-boun...@outages.org] On Behalf Of Jeff Shultz > via Outages > Sent: Tuesday, October 04, 2016 5:41 PM > To: Randall (Randy) Raitz <randy.ra...@readytalk.com>; outages@outages.org > Subject: Re: [outages] Level3 VoIP Issues? > > I have a feeling

Re: [outages] Level3 VoIP Issues?

<randy.ra...@readytalk.com>; outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? I have a feeling that "fire from a fiber cut" email was a bit of snark (playing on the usual evasive answers you get about many outages) and not meant to be taken seriously. But who kn

Re: [outages] Level3 VoIP Issues?

ons will take place to ensure incidents of > this nature do not recur in the future. Services are currently restored and > stable. > > > > > > *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *Ryan > McGinnis via Outages > *Sent:* Tuesday, October 4, 201

Re: [outages] Level3 VoIP Issues?

ly restored and stable. > > > > > > From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Ryan > McGinnis via Outages > Sent: Tuesday, October 4, 2016 1:44 PM > To: outages@outages.org > > > Subject: Re: [outages] Level3 VoIP Issues? > > > >

Re: [outages] Level3 VoIP Issues?

innis via Outages Sent: Tuesday, October 4, 2016 1:44 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3 VoIP Issues? Man, I hope that sentence is backwards, because I don't even want to know how much light you have to pump down fiber to cause a fire when

Re: [outages] Level3 VoIP Issues?

4, 2016 1:44 PM To: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Man, I hope that sentence is backwards, because I don't even want to know how much light you have to pump down fiber to cause a fire when you cut it. On Tue, Oct 4, 2016 at 12:35 PM, Mike Castoro via Outages

Re: [outages] Level3 VoIP Issues?

: Outages [mailto:outages-boun...@outages.org] On Behalf Of Rob Hall via Outages Sent: Tuesday, October 4, 2016 2:17 PM To: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Also confirmed with Comcast and Cox voice service in VA. Thanks! -- Rob Hall, CCNP R, CCP-V I.T. Manager @Work

Re: [outages] Level3 VoIP Issues?

...@outages.org] On Behalf Of Rob Hall via Outages Sent: Tuesday, October 4, 2016 2:08 PM To: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? FWIW, we're still seeing an outage via Charter's voice service. Local calling seems to be unaffected, but all inbound/outbound long distance

Re: [outages] Level3 VoIP Issues?

, 2016 1:36 PM To: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Unofficially, I'm hearing that the Level 3 voice outage was caused by a "fire at a large switching center caused by a major fiber cut" -Mike -Original Message- From: Outages [mailto:outages-boun...@o

Re: [outages] Level3 VoIP Issues?

4, 2016 12:09 PM To: Peter Beckman <beck...@angryox.com> Cc: Aubrey Wells via Outages <outages@outages.org> Subject: Re: [outages] Level3 VoIP Issues? Both Bandwidth and Vitelity are just VoIP resellers. They use larger carriers like Level3 for their termination and origination. A

Re: [outages] Level3 VoIP Issues?

Bandwidth has their own CLEC operations, and also uses Level 3 in regions where they do not have their own network. We received outage notifications from Bandwidth directly during this incident. ___ Outages mailing list Outages@outages.org

Re: [outages] Level3 VoIP Issues?

Bandwidth also has its own CLEC operation, and we didn't see any impact to origination services through them since the majority of our DIDs are on their own network. No impact to termination through them that I'm aware of, though it could be that we had some traffic going out through other

Re: [outages] Level3 VoIP Issues?

Both Bandwidth and Vitelity are just VoIP resellers. They use larger carriers like Level3 for their termination and origination. As such Level3's outage is effecting their services as well. On 2016-10-04 11:57, Peter Beckman via Outages wrote: Hey Aubrey -- Did you see origination or

Re: [outages] Level3 VoIP Issues?

: (423)312-6003 [cisco-certified-ccnp][CCP-V] From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Jeremie Chism via Outages Sent: Tuesday, October 4, 2016 11:45 AM To: Guillermo <g.sando...@chasedatacorp.com> Cc: Outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? I

Re: [outages] Level3 VoIP Issues?

Hey Aubrey -- Did you see origination or termination issues with Bandwidth and Vitelity? Or both? Beckman On Tue, 4 Oct 2016, Aubrey Wells via Outages wrote: Atlanta Ga, our trunks terminate in McLean VA and NYC. Same problem. Seeing issues with other providers like Bandwidth.com and

Re: [outages] Level3 VoIP Issues?

4, 2016 11:39 AM To: Outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Starting to see improvement inbound and outbound. A few errors here and there but I see a light at the end of the tunnel. Hopefully its not a train. - Aubrey Wells Manager, Network Engineer

Re: [outages] Level3 VoIP Issues?

eir customers. >> >> >> >> >> >> >> >> *From:* Outages [mailto:outages-boun...@outages.org >> <javascript:_e(%7B%7D,'cvml','outages-boun...@outages.org');>] *On >> Behalf Of *Lucky 225 via Outages >> *Sent:* Tuesday, October 0

Re: [outages] Level3 VoIP Issues?

t; >>> >>> >>> >>> >>> *From:* Outages [mailto:outages-boun...@outages.org >>> <outages-boun...@outages.org>] *On Behalf Of *Lucky 225 via Outages >>> *Sent:* Tuesday, October 04, 2016 11:07 AM >>> *To:* David Wessell <da..

Re: [outages] Level3 VoIP Issues?

t" for info for their customers. >> >> FROM: Outages [mailto:outages-boun...@outages.org] ON BEHALF OF Lucky 225 >> via Outages >> SENT: Tuesday, October 04, 2016 11:07 AM >> TO: David Wessell <da...@ringfree.com> >> CC: outages@outages.org >> SUBJECT: Re:

Re: [outages] Level3 VoIP Issues?

t effort” for info for their customers. > > > > > > > > *From:* Outages [mailto:outages-boun...@outages.org > <outages-boun...@outages.org>] *On Behalf Of *Lucky 225 via Outages > *Sent:* Tuesday, October 04, 2016 11:07 AM > *To:* David Wessell <da...@ringfree.com> >

Re: [outages] Level3 VoIP Issues?

Functioning inbound, outbound, and inbound toll-free in MKE now as well. From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Kevin Shymkiw via Outages Sent: Tuesday, October 04, 2016 10:37 AM To: Mark Harrison Cc: Outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Our

Re: [outages] Level3 VoIP Issues?

Our Level3 DIDs are working in Phila metro area. From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Kevin Shymkiw via Outages Sent: Tuesday, October 04, 2016 11:37 AM To: Mark Harrison <ma...@fltg.com> Cc: Outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Our

Re: [outages] Level3 VoIP Issues?

t;> >> >> >> >> *From:* Outages [mailto:outages-boun...@outages.org >> <outages-boun...@outages.org>] *On Behalf Of *Lucky 225 via Outages >> *Sent:* Tuesday, October 04, 2016 11:07 AM >> *To:* David Wessell <da...@ringfree.com> >> *Cc:* o

Re: [outages] Level3 VoIP Issues?

Posted in an earlier email. http://downdetector.com/status/level3/map/ From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Dennis Burgess via Outages Sent: Tuesday, October 04, 2016 11:21 AM To: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Does anyone have a link

Re: [outages] Level3 VoIP Issues?

nd issues in Central California as well. > > > > > > *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *Nathan > J. Lichtenstein via Outages > *Sent:* Tuesday, October 04, 2016 8:08 AM > *To:* outages@outages.org > *Subject:* Re: [outages] Level3 VoIP Issues? &

Re: [outages] Level3 VoIP Issues?

VIA the L3 Support Portal: Impact to Voice Services Oct 04, 2016 1:00 PM GMT We are aware of a situation impacting voice services. Our technical team is looking into this issue to determine the cause. Our priority is to ensure the reliability of our network and services. We will provide

Re: [outages] Level3 VoIP Issues?

Los Angeles area affected as well. From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Jeremie Chism via Outages Sent: Tuesday, October 04, 2016 9:38 AM To: Will Wylie <willjwy...@gmail.com> Cc: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Seeing the same

Re: [outages] Level3 VoIP Issues?

SIP / PRI inbound-outbound issues in Central California as well. From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Nathan J. Lichtenstein via Outages Sent: Tuesday, October 04, 2016 8:08 AM To: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? We're seeing issues

Re: [outages] Level3 VoIP Issues?

We're seeing issues with SIP/PRI in ROC inbound. Also seeing throughout NE and upstate NY. On Tue, Oct 4, 2016 at 10:27 AM, Kevin Miller via Outages < outages@outages.org> wrote: > All of our Level3 DIDs in PHL and JFK do not seem to working inbound. > Outgoing seems to be fine. Waiting for

Re: [outages] Level3 VoIP Issues?

local and long distance still gets an intercept message. >>>> >>>> >>>> >>>> >>>> >>>> *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *Kevin >>>> Kastelic via Outag

Re: [outages] Level3 VoIP Issues?

still appears to be working. >>>>> Outbound local and long distance still gets an intercept message. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *From:* Outages [mailto:outages-boun...@outages.org] *On

Re: [outages] Level3 VoIP Issues?

> office number does the same, but our inbound still appears to be working. >>>>> Outbound local and long distance still gets an intercept message. >>>>> >>>>> >>>>> >>>>> >>>>> >>>&g

Re: [outages] Level3 VoIP Issues?

From: Outages [mailto:outages-boun...@outages.org > <javascript:_e(%7B%7D,'cvml','outages-boun...@outages.org');>] On Behalf Of > Kevin Kastelic via Outages > Sent: Tuesday, October 04, 2016 9:37 AM > To: Will Wylie > Cc: outages@outages.org <javascript:_e(%7

Re: [outages] Level3 VoIP Issues?

Outages [mailto:outages-boun...@outages.org >>>> <javascript:_e(%7B%7D,'cvml','outages-boun...@outages.org');>] *On >>>> Behalf Of *Kevin Kastelic via Outages >>>> *Sent:* Tuesday, October 04, 2016 9:37 AM >>>> *To:* Will Wylie >>>> *Cc:*

Re: [outages] Level3 VoIP Issues?

Outbound dialing is failing for the majority of our trunks in AZ. On Tue, Oct 4, 2016 at 7:27 AM, Kevin Miller via Outages < outages@outages.org> wrote: > All of our Level3 DIDs in PHL and JFK do not seem to working inbound. > Outgoing seems to be fine. Waiting for someone to pick up at Level3.

Re: [outages] Level3 VoIP Issues?

We are seeing it intermittently in all of our offices throughout the major Texas cities as well. Thanks, Jeff Hershberger | Telecom Manager BCNE, ACMA 2323 Ross Avenue, Suite 600 | Dallas, TX | 75201 V: (214) 953-5748 | F: (214) 661-6665 | jhershber...@jw.com

Re: [outages] Level3 VoIP Issues?

Not that we likely need more confirmation at this point, but I'm seeing it on Cleveland-area DIDs. Definitely some kind of SHTF situation at Level3. On Tue, Oct 4, 2016 at 10:28 AM Kevin Miller via Outages < outages@outages.org> wrote: > All of our Level3 DIDs in PHL and JFK do not seem to

Re: [outages] Level3 VoIP Issues?

pears to be working. > Outbound local and long distance still gets an intercept message. > > > > > > *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *Kevin > Kastelic via Outages > *Sent:* Tuesday, October 04, 2016 9:37 AM > *To:* Will Wylie > *Cc:

Re: [outages] Level3 VoIP Issues?

;>> Outbound local and long distance still gets an intercept message. >>> >>> >>> >>> >>> >>> *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *Kevin >>> Kastelic via Outages >>> *Sent:* Tuesday, October 04, 2016 9:37

Re: [outages] Level3 VoIP Issues?

ls with Level3 from MKE, but incoming is fine >> as far as we can tell. This is on what was formerly a TWTC connection. I've >> been waiting about 10 minutes on hold with support as well. >> > >> > >> > >> > Nicholas Sideras >> > Director of Technic

Re: [outages] Level3 VoIP Issues?

ages-boun...@outages.org] *On Behalf Of *Kevin > Kastelic via Outages > *Sent:* Tuesday, October 04, 2016 9:37 AM > *To:* Will Wylie > *Cc:* outages@outages.org > *Subject:* Re: [outages] Level3 VoIP Issues? > > > > We are also seeing it in Sussex, WI on Level3. Local calli

Re: [outages] Level3 VoIP Issues?

ike > > > > *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *Jeremie > Chism via Outages > *Sent:* Tuesday, October 04, 2016 10:38 AM > *To:* Will Wylie <willjwy...@gmail.com> > *Cc:* outages@outages.org > *Subject:* Re: [outages] Level3 VoIP Issues?

Re: [outages] Level3 VoIP Issues?

...@outages.org] On Behalf Of Jayson Navitsky via Outages Sent: Tuesday, October 04, 2016 10:30 To: Kevin Miller <kmil...@vantage.com> Cc: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? Confirmed, we're seeing issues at multiple sites in northern NJ and NYC. Outbound seems to wor

Re: [outages] Level3 VoIP Issues?

@outages.org Subject: Re: [outages] Level3 VoIP Issues? The support line just hung up on me while on hold, and calling back gets an "all circuits are busy" message. Calling our sales rep's local MKE office number does the same, but our inbound still appears to be working. Outbound local a

Re: [outages] Level3 VoIP Issues?

ge. From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Kevin Kastelic via Outages Sent: Tuesday, October 04, 2016 9:37 AM To: Will Wylie Cc: outages@outages.org Subject: Re: [outages] Level3 VoIP Issues? We are also seeing it in Sussex, WI on Level3. Local calling seems to work, but no

Re: [outages] Level3 VoIP Issues?

l Operations > Holland Bulb Farms > > From: Outages > [mailto:outages-boun...@outages.org<mailto:outages-boun...@outages.org>] On > Behalf Of Kevin Miller via Outages > Sent: Tuesday, October 04, 2016 9:27 AM > To: outages@outages.org<mailto:outages@outages.org> >

Re: [outages] Level3 VoIP Issues?

lt;mailto:outages-boun...@outages.org>] On > Behalf Of Kevin Miller via Outages > Sent: Tuesday, October 04, 2016 9:27 AM > To: outages@outages.org<mailto:outages@outages.org> > Subject: [outages] Level3 VoIP Issues? > > All of our Level3 DIDs in PHL and JFK do not seem

Re: [outages] Level3 VoIP Issues?

t; Director of Technical Operations >> > Holland Bulb Farms >> > >> > From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Kevin >> > Miller via Outages >> > Sent: Tuesday, October 04, 2016 9:27 AM >> > To: outages@outages.org >

Re: [outages] Level3 VoIP Issues?

>> > Holland Bulb Farms >> > >> > From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Kevin >> Miller via Outages >> > Sent: Tuesday, October 04, 2016 9:27 AM >> > To: outages@outages.org >> > Subject: [outages] Level3

Re: [outages] Level3 VoIP Issues?

rector of Technical Operations > > Holland Bulb Farms > > > > From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Kevin > Miller via Outages > > Sent: Tuesday, October 04, 2016 9:27 AM > > To: outages@outages.org > > Subject: [outages] Level3 VoIP

Re: [outages] Level3 VoIP Issues?

a Outages > Sent: Tuesday, October 04, 2016 9:27 AM > To: outages@outages.org > Subject: [outages] Level3 VoIP Issues? > > All of our Level3 DIDs in PHL and JFK do not seem to working inbound. > Outgoing seems to be fine. Waiting for someone to pick up at Level3. Anyone > else se

Re: [outages] Level3 VoIP Issues?

[mailto:outages-boun...@outages.org] On Behalf Of Kevin Miller via Outages Sent: Tuesday, October 04, 2016 9:27 AM To: outages@outages.org Subject: [outages] Level3 VoIP Issues? All of our Level3 DIDs in PHL and JFK do not seem to working inbound. Outgoing seems to be fine. Waiting for someone

Re: [outages] Level3 VoIP Issues?

Confirmed, we're seeing issues at multiple sites in northern NJ and NYC. Outbound seems to work most of the time, incoming standard DIDs and 8xx service seems impacted. On Tue, Oct 4, 2016 at 10:27 AM, Kevin Miller via Outages < outages@outages.org> wrote: > All of our Level3 DIDs in PHL and JFK