Re: [outages] Comcast outages in Denver metro area

[frnkblk--- via Outages]

We saw IPv6-related timeouts (ICMPv6 and HTTPv6) to postmaster.comcast.net 
(multiple hosts), xfinity.comast.net, www.comcast.net <http://www.comcast.net> 
, mydeviceinfo.comcast.net, www.comcast6.net <http://www.comcast6.net>  between 
~2:35 and to 5:49 am (U.S. Central).

 

Frank

 

From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Steve Meuse via 
Outages
Sent: Tuesday, October 11, 2016 7:49 AM
To: Dave Pascoe <davek...@gmail.com>
Cc: outages@outages.org
Subject: Re: [outages] Comcast outages in Denver metro area

 

 

There is a Backbone device that is/was mis-behaving in Denver that is causing 
some traffic issues. We're in the process of getting corrected.

 

-Steve

 

 

 

On Tue, Oct 11, 2016 at 3:31 AM, Dave Pascoe via Outages <outages@outages.org 
<mailto:outages@outages.org> > wrote:

There does appear to be some sort of Comcast outage. I have a remote system out 
there that also went down in the last hour.

Dave

On Tue, Oct 11, 2016 at 3:26 AM Chris Stone via Outages <outages@outages.org 
<mailto:outages@outages.org> > wrote:

Anyone aware of any Comcast outages in the Denver area? Have a number of
VoIP clients that are on Comcast circuits around the Denver metro area that
are all down right now. I monitor their gateways in Nagios too and they are
all 100% down at present. Trying to access them via Cogent and Level3 with
the same results - 100% failure.

Thanks.


Chris

___
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org> 
https://puck.nether.net/mailman/listinfo/outages


___
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org> 
https://puck.nether.net/mailman/listinfo/outages





 

-- 


-Steve

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[Steve Meuse via Outages]

There is a Backbone device that is/was mis-behaving in Denver that is
causing some traffic issues. We're in the process of getting corrected.

-Steve



On Tue, Oct 11, 2016 at 3:31 AM, Dave Pascoe via Outages <
outages@outages.org> wrote:

> There does appear to be some sort of Comcast outage. I have a remote
> system out there that also went down in the last hour.
>
> Dave
>
> On Tue, Oct 11, 2016 at 3:26 AM Chris Stone via Outages <
> outages@outages.org> wrote:
>
>> Anyone aware of any Comcast outages in the Denver area? Have a number of
>> VoIP clients that are on Comcast circuits around the Denver metro area
>> that
>> are all down right now. I monitor their gateways in Nagios too and they
>> are
>> all 100% down at present. Trying to access them via Cogent and Level3 with
>> the same results - 100% failure.
>>
>> Thanks.
>>
>>
>> Chris
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>


-- 

-Steve
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[B.J. Ash via Outages]

We have noticed an issue with a single Comcast connection in Michigan (in the 
SW corner of Michigan) that has been intermittently up/down since Friday.  Same 
story as the rest (goes through Level 3 around Chicago with issues before 
coming into Michigan).


-Original Message-
From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Jeremy Chadwick 
via Outages
Sent: Tuesday, October 11, 2016 7:34
To: outages@outages.org
Subject: Re: [outages] Comcast outages in Denver metro area

Issue in question was extremely widespread -- a large portion of the US east 
coast (from Maine to Florida), as well as Colorado, Washington, Texas, and 
California; I can confirm the latter.

Issue lasted from 00:56 to 04:08 PDT (UTC-0700), according to my periodic mtrs 
between Mountain View (Silicon Valley) and Seattle.
Sadly, my return-path traceroutes (SEA->MV) weren't working during the outage 
due to a configuration mistake on my part, so I don't have a full picture of 
what happened.

For general end-user complaints, the #comcastoutage hashtag on Twitter has 
quite a lot.  There is strong speculation this was Level 3 botching something 
-- which, if true, makes 2 major outages in 7 days.

--
| Jeremy Chadwick   j...@koitsu.org |
| UNIX Systems Administratorhttp://jdc.koitsu.org/ |
| Making life hard for others since 1977. PGP 4BD6C0CB |

On Tue, Oct 11, 2016 at 01:25:55AM -0600, Chris Stone via Outages wrote:
> Anyone aware of any Comcast outages in the Denver area? Have a number
> of VoIP clients that are on Comcast circuits around the Denver metro
> area that are all down right now. I monitor their gateways in Nagios
> too and they are all 100% down at present. Trying to access them via
> Cogent and Level3 with the same results - 100% failure.
>
> Thanks.
>
>
> Chris
>

> begin:vcard
> fn:Chris Stone
> n:Stone;Chris
> org:AxisInternet, Inc.
> adr:Suite 1110;;910 16th St.;Denver;Colorado;80202;USA
> email;internet:cst...@axint.net title:VP/CTO
> tel;work:303.592.2947 x302
> tel;fax:303.893.2947
> tel;home:303.838.2386
> tel;cell:303.570.6947
> x-mozilla-html:FALSE
> url:www.axint.net
> version:2.1
> end:vcard
>

> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Disclaimer: This email and its attachments may be confidential and are intended 
solely for the use of the individual to whom it is addressed. Any views or 
opinions expressed are solely those of the author and do not necessarily 
represent those of Kent Record Management Inc. If you are not the intended 
recipient of this email and its attachments, you must take no action based upon 
them, nor must you copy or show them to anyone. Please contact the sender if 
you believe you have received this email in error.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[Jeremy Chadwick via Outages]

Issue in question was extremely widespread -- a large portion of the US
east coast (from Maine to Florida), as well as Colorado, Washington,
Texas, and California; I can confirm the latter.

Issue lasted from 00:56 to 04:08 PDT (UTC-0700), according to my
periodic mtrs between Mountain View (Silicon Valley) and Seattle.
Sadly, my return-path traceroutes (SEA->MV) weren't working during the
outage due to a configuration mistake on my part, so I don't have a full
picture of what happened.

For general end-user complaints, the #comcastoutage hashtag on Twitter
has quite a lot.  There is strong speculation this was Level 3 botching
something -- which, if true, makes 2 major outages in 7 days.

-- 
| Jeremy Chadwick   j...@koitsu.org |
| UNIX Systems Administratorhttp://jdc.koitsu.org/ |
| Making life hard for others since 1977. PGP 4BD6C0CB |

On Tue, Oct 11, 2016 at 01:25:55AM -0600, Chris Stone via Outages wrote:
> Anyone aware of any Comcast outages in the Denver area? Have a number of
> VoIP clients that are on Comcast circuits around the Denver metro area that
> are all down right now. I monitor their gateways in Nagios too and they are
> all 100% down at present. Trying to access them via Cogent and Level3 with
> the same results - 100% failure.
> 
> Thanks.
> 
> 
> Chris
> 

> begin:vcard
> fn:Chris Stone
> n:Stone;Chris
> org:AxisInternet, Inc.
> adr:Suite 1110;;910 16th St.;Denver;Colorado;80202;USA
> email;internet:cst...@axint.net
> title:VP/CTO
> tel;work:303.592.2947 x302
> tel;fax:303.893.2947
> tel;home:303.838.2386
> tel;cell:303.570.6947
> x-mozilla-html:FALSE
> url:www.axint.net
> version:2.1
> end:vcard
> 

> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[Sneath, Dustin via Outages]

Monitoring shows I lost ping to all 6 of my remote sites in the Chicago area on 
Comcast overnight (fiber and coax) from various internet ping services, but we 
appear to be back up at this time. I would have to agree with the previous 
email that the packet loss seemed to originate at 350 E Cermak.

Dustin Sneath
Systems Administrator
Elk Grove Park District
(847)228-3846 (desk)
(224)800-2146 (cell)

Sent from my Verizon Wireless 4G LTE DROID


Dave Pascoe via Outages <outages@outages.org> wrote:

FWIW...seems like Comcast traffic going west from Comcast in the New England 
area is dying at 350 E Cermak in Chicago. This is also affecting ISPs that get 
transit from Comcast out here, at least for Comcast destinations west of 
Chicago. Can't tell if it is limited to only certain Comcast markets out 
west...but it is definitely affecting Comcast transit in the Bay Area AFAICT.

On Tue, Oct 11, 2016 at 4:13 AM, James Laszko via Outages 
<outages@outages.org<mailto:outages@outages.org>> wrote:
We are seeing Comcast IP cable modem reachability issues across the entire US 
right now from AT  Traffic appears to be dying between Level3 and Comcast 
from our AT based sensors.

Regards,


James

-Original Message-
From: Outages 
[mailto:outages-boun...@outages.org<mailto:outages-boun...@outages.org>] On 
Behalf Of Chris Stone via Outages
Sent: Tuesday, October 11, 2016 0:26
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] Comcast outages in Denver metro area

Anyone aware of any Comcast outages in the Denver area? Have a number of VoIP 
clients that are on Comcast circuits around the Denver metro area that are all 
down right now. I monitor their gateways in Nagios too and they are all 100% 
down at present. Trying to access them via Cogent and Level3 with the same 
results - 100% failure.

Thanks.


Chris

___
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[Randall (Randy) Raitz via Outages]

I'm seeing Comcast cable modem issues from my house here in Denver. Details 
from my account portal indicate an ETR of 7AM Denver time

Sent from my iPhone

On Oct 11, 2016, at 3:01 AM, Dave Pascoe via Outages 
<outages@outages.org<mailto:outages@outages.org>> wrote:

FWIW...seems like Comcast traffic going west from Comcast in the New England 
area is dying at 350 E Cermak in Chicago. This is also affecting ISPs that get 
transit from Comcast out here, at least for Comcast destinations west of 
Chicago. Can't tell if it is limited to only certain Comcast markets out 
west...but it is definitely affecting Comcast transit in the Bay Area AFAICT.

On Tue, Oct 11, 2016 at 4:13 AM, James Laszko via Outages 
<outages@outages.org<mailto:outages@outages.org>> wrote:
We are seeing Comcast IP cable modem reachability issues across the entire US 
right now from AT  Traffic appears to be dying between Level3 and Comcast 
from our AT based sensors.

Regards,


James

-Original Message-
From: Outages 
[mailto:outages-boun...@outages.org<mailto:outages-boun...@outages.org>] On 
Behalf Of Chris Stone via Outages
Sent: Tuesday, October 11, 2016 0:26
To: outages@outages.org<mailto:outages@outages.org>
Subject: [outages] Comcast outages in Denver metro area

Anyone aware of any Comcast outages in the Denver area? Have a number of VoIP 
clients that are on Comcast circuits around the Denver metro area that are all 
down right now. I monitor their gateways in Nagios too and they are all 100% 
down at present. Trying to access them via Cogent and Level3 with the same 
results - 100% failure.

Thanks.


Chris

___
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org<mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[Dave Pascoe via Outages]

FWIW...seems like Comcast traffic going west from Comcast in the New
England area is dying at 350 E Cermak in Chicago. This is also affecting
ISPs that get transit from Comcast out here, at least for Comcast
destinations west of Chicago. Can't tell if it is limited to only certain
Comcast markets out west...but it is definitely affecting Comcast transit
in the Bay Area AFAICT.

On Tue, Oct 11, 2016 at 4:13 AM, James Laszko via Outages <
outages@outages.org> wrote:

> We are seeing Comcast IP cable modem reachability issues across the entire
> US right now from AT  Traffic appears to be dying between Level3 and
> Comcast from our AT based sensors.
>
> Regards,
>
>
> James
>
> -Original Message-
> From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Chris
> Stone via Outages
> Sent: Tuesday, October 11, 2016 0:26
> To: outages@outages.org
> Subject: [outages] Comcast outages in Denver metro area
>
> Anyone aware of any Comcast outages in the Denver area? Have a number of
> VoIP clients that are on Comcast circuits around the Denver metro area that
> are all down right now. I monitor their gateways in Nagios too and they are
> all 100% down at present. Trying to access them via Cogent and Level3 with
> the same results - 100% failure.
>
> Thanks.
>
>
> Chris
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[James Laszko via Outages]

We are seeing Comcast IP cable modem reachability issues across the entire US 
right now from AT  Traffic appears to be dying between Level3 and Comcast 
from our AT based sensors.

Regards,


James

-Original Message-
From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Chris Stone via 
Outages
Sent: Tuesday, October 11, 2016 0:26
To: outages@outages.org
Subject: [outages] Comcast outages in Denver metro area

Anyone aware of any Comcast outages in the Denver area? Have a number of VoIP 
clients that are on Comcast circuits around the Denver metro area that are all 
down right now. I monitor their gateways in Nagios too and they are all 100% 
down at present. Trying to access them via Cogent and Level3 with the same 
results - 100% failure.

Thanks.


Chris

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[Chris Stone via Outages]

Thanks - I did see one client come back up, but then immediately went back 
down again.



Chris


On 10/11/2016 01:31 AM, Dave Pascoe wrote:
There does appear to be some sort of Comcast outage. I have a remote 
system out there that also went down in the last hour.


Dave

On Tue, Oct 11, 2016 at 3:26 AM Chris Stone via Outages 
> wrote:


Anyone aware of any Comcast outages in the Denver area? Have a number of
VoIP clients that are on Comcast circuits around the Denver metro area
that
are all down right now. I monitor their gateways in Nagios too and
they are
all 100% down at present. Trying to access them via Cogent and Level3 with
the same results - 100% failure.

Thanks.


Chris

___
Outages mailing list
Outages@outages.org 
https://puck.nether.net/mailman/listinfo/outages



___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[Chris Stone via Outages]

I should clarify. The VoIP clients are on my VoIP services - not Comcast or 
L3. I also have some client that are not VoIP clients that use Comcast 
circuits for Internet access and they are all down too. What I am asking 
about is Comcast Internet access circuits being down.



Chris


On 10/11/2016 01:31 AM, Russell Zen wrote:

I just got off the phone with Level 3 TSC. Was on the phone for a totally 
separate issue, but then I saw your remarks and included this on my phone 
conversation with Level 3 just now.

Me: "Are you having any issues right now with phones in the Denver area?"
TSC: "Not that I'm aware of."
Me: "Oh well, there was some reporting of VOIP issues in the Denver area. Plus, 
while waiting on hold with you the phone kept ringing and ringing for the last 20 
minutes, no standard Level 3 auto attendant or anything."
TSC: "My phone only range twice and then I picked up."

Russell Zen
Sprouts Farmers Market IT



-Original Message-
From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Chris Stone via 
Outages
Sent: Tuesday, October 11, 2016 12:26 AM
To: outages@outages.org
Subject: [outages] Comcast outages in Denver metro area

Anyone aware of any Comcast outages in the Denver area? Have a number of VoIP 
clients that are on Comcast circuits around the Denver metro area that are all 
down right now. I monitor their gateways in Nagios too and they are all 100% 
down at present. Trying to access them via Cogent and Level3 with the same 
results - 100% failure.

Thanks.


Chris


___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outages in Denver metro area

[Dave Pascoe via Outages]

There does appear to be some sort of Comcast outage. I have a remote system
out there that also went down in the last hour.

Dave

On Tue, Oct 11, 2016 at 3:26 AM Chris Stone via Outages 
wrote:

> Anyone aware of any Comcast outages in the Denver area? Have a number of
> VoIP clients that are on Comcast circuits around the Denver metro area that
> are all down right now. I monitor their gateways in Nagios too and they are
> all 100% down at present. Trying to access them via Cogent and Level3 with
> the same results - 100% failure.
>
> Thanks.
>
>
> Chris
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Comcast outages in Denver metro area

[Chris Stone via Outages]

Anyone aware of any Comcast outages in the Denver area? Have a number of 
VoIP clients that are on Comcast circuits around the Denver metro area that 
are all down right now. I monitor their gateways in Nagios too and they are 
all 100% down at present. Trying to access them via Cogent and Level3 with 
the same results - 100% failure.


Thanks.


Chris

<>___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages