Re: [outages] Verizon to Comcast Routing Issue - East Coast

[John Neiberger via Outages]

I'd be happy to take a look. Can you provide more details?

John

On Fri, Mar 15, 2024, 9:09 AM Austin Fisher via Outages 
wrote:

> Anyone else noticing issues with connections from Verizon to Comcast and
> Vice Versa? On the East Coast and seeing issues across some of our
> customers VPNing from Comcast homes to Verizon offices, and same the other
> way. I ran a few tracerts and noticed that Comcast could get to Alter but
> dropped there, and Verizon never left its network.
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] T-Mobile Down Nationwide?

[John Neiberger via Outages]

I'm not one to refer to Downdetector often, but in this case it's
interesting that pretty much every mobile carrier in the US took a hit at
the same time. However, the same is true for Apple support, Google,
Samsung, American Express, and a handful of others. Feels sorta cloud-ish.

On Mon, Feb 13, 2023 at 9:15 PM Nate via Outages 
wrote:

> I'm in Seattle, Tmobile services keep coming and going, and at one point
> it rolled me over to TMobile Edge 2G. Now I have no service whatsoever,
> having to use my home internet on a different ISP.
>
> Tmo hasn't acknowledged they are down hard at all. Curious what is going
> on.
>
> On Mon, Feb 13, 2023, 8:07 PM Benjamin Monjay via Outages <
> outages@outages.org> wrote:
>
>> Seeing a lot of reports on Twitter, FB and Reddit. My personal phone is
>> out as well. Showing no service at all.
>>
>> Interestingly enough, my Ziply Fiber went down at the same time as well
>> but came up about 5 minutes later.
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Experiencing VzW LTE outage across the US

[John Neiberger via Outages]

Confirmed in a suburb of Denver. My phone has a complete loss of
connectivity right now.

On Wed, Nov 10, 2021, 7:30 AM Nicholas Saraniti via Outages <
outages@outages.org> wrote:

> Can confirm the same from South Florida
>
>
> *Nicholas Saraniti*
>
>
>
> On Nov 10, 2021, at 9:02 AM, Bob Harman - Huntersville via Outages <
> outages@outages.org> wrote:
>
> We're experiencing lack of Internet connectivity on devices using Verizon
> Wireless LTE service across various parts of the US.  This appears to have
> started around 2AM and is ongoing.
>
> We have a similar hardware setup for tertiary Internet access using Telus
> in Canada, which is not affected, so we're pretty sure this is either a
> Verizon Wireless, or an upstream dependency issue.
>
> --
> Bob Harman
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast

[John Neiberger via Outages]

If there is anything about this specific to Comcast, please let me know
some details and I'll check into it.

Thanks,
John

On Tue, Apr 27, 2021 at 8:00 AM George Ly via Outages 
wrote:

> We also are seeing random Microsoft outages.
>
>
>
> Thanks,
>
> George Ly
>
> Business Analyst
>
> The Living Legacy Foundation of Maryland
>
> 1730 Twin Springs Road |Suite 200 |Baltimore, MD 21227
>
> Phone: (443) 833-1497 | Mobile: (443) 498-3532
>
> E-Mail: g...@thellf.org
>
> https://www.thellf.org
>
>
>
> *From:* Outages  *On Behalf Of *Bruce
> Freshwater via Outages
> *Sent:* Tuesday, April 27, 2021 9:36 AM
> *To:* outages 
> *Subject:* [outages] Comcast
>
>
>
> [CAUTION:External Email]
>
> Seeing routing issues to AWS and Microsoft in total from Pittsburgh.
>
>
>
> Cogent and Centurylink both work correctly.
>
>
>
> Regards,
>
> *Bruce Freshwater*
>
>
>
> *Direct:  *412.722.1701
>
> *Mobile:  *412.292.0282
>
> *Main:  *866.707.5869
>
> *Fax: * 412.774.2469
>
> *Email: * *bfreshwa...@sierraexperts.com *
>
> *Request Help: *Support Portal (SIM)
> 
>
>
>
> *Sierra Experts *
>
> 2 Robinson Plaza, Suite 300, Pittsburgh, PA 15205-1017
>
> Our Services: Managed Services
>  | Development & Design
>  | Telephony & VoIP
>  | Hardware & Software
>  | Data Center
> 
>
> Stay Connected: Website  | Awards
>  | Facebook
>  | Twitter
>  | LinkedIn
>  | Blog
> 
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast - New England region

[John Neiberger via Outages]

Looks like a malfunctioning card on one of the large routers in that
region. Nothing to do with peering. Ops is working it.


On Thu, Sep 3, 2020 at 6:13 PM William Peebles via Outages <
outages@outages.org> wrote:

> Appears as Comcast customers throughout New England are having
> intermittent packet loss, guessing it's a peering misconfig at this point.
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outage

[John Neiberger via Outages]

This isn't quite accurate. There was a maintenance in one part of a
regional network, but there was an optical line failure in another part of
that same network that happened during the maintenance. It was too late
into the maintenance to back out, so they finished as quickly as they could
to restore service while another team was working the transport issue.

On Fri, Dec 20, 2019 at 11:52 AM Ross Tajvar via Outages <
outages@outages.org> wrote:

> Here's what I got from Comcast for RFO:
>
> We tracked this as an outage ticket SI039747811 but it was not an outage
> it was a core maintenance SM001713155 that was failed notified and link to
> all the locations That is why we received it as an outage due to the fail
> to notify and link affected elements by the maintenance submitter.
>
> Maintenance was to migrate Whitemarsh AR from New York Core to Newark Core
> and add POR capacity.  Had an estimated impact of 1 hour with ETR of 3am
> EST.
>
> On Fri, Dec 20, 2019, 1:07 AM Christopher Conley <
> ccon...@forsmarshgroup.com> wrote:
>
>> Same, just came back up.
>>
>> On Dec 20, 2019 12:53 AM, Ross Tajvar via Outages 
>> wrote:
>> I'm now seeing my service back up.
>>
>> On Fri, Dec 20, 2019 at 12:38 AM Ross Tajvar  wrote:
>>
>>> I'm seeing a major Comcast outage in the MD/DC/VA area. Coax, DIA, and
>>> transit services are all down. We do have some ENS (L2) circuits that are
>>> up. Haven't been able to reach anyone on the phone yet.
>>>
>> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Internet Issues in Southeast

[John Neiberger via Outages]

Could you share some traceroutes?

On Thu, Jan 31, 2019 at 8:52 AM Michael Gates via Outages <
outages@outages.org> wrote:

> My users, geographically nearby, are having issues accessing resources at
> our facility in South Alabama and another facility in Columbus Georgia from
> residential connections on Comcast and Verizon, possibly others.
>
> I have confirmed it from my own home connection with Comcast as well. High
> latency (from 180 ms to a peak of 13400 ms) and periods of complete packet
> loss as well, all happening when traversing Level3 or GTT in Atlanta.
>
> Is there a known issue occurring or could someone weigh in on this?
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast routing / peering issues in Ashburn, VA

[John Neiberger via Outages]

Can you unicast one of their source IPs to me? I'll take a look and see
what's up.

Thanks,
John

On Mon, Nov 12, 2018 at 12:41 PM Kenneth Padgett via Outages <
outages@outages.org> wrote:

> Seems they're having issues, no internet routing for several coax based
> customers in Northern VA:
>
> Tracing route to google-public-dns-a.google.com [8.8.8.8]
> over a maximum of 30 hops:
>
>   415 ms19 ms12 ms  96.108.140.117
>   5 *** Request timed out.
>   657 ms28 ms20 ms  be-21508-cr02.ashburn.va.ibone.comcast.net
> [68.8
> 6.91.53]
>   723 ms33 ms16 ms  be-10189-pe07.ashburn.va.ibone.comcast.net
> [68.8
> 6.83.66]
>   8 *** Request timed out.
>   9 *** Request timed out.
>  10 *** Request timed out.
>  11 *** Request timed out.
>  12 *** Request timed out.
>  13 *** Request timed out.
>  14 *** Request timed out.
>  15 *** Request timed out.
>  16 *** Request timed out.
>
> KP
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Facebook

[John Neiberger via Outages]

FB is working perfectly for me in Denver on Comcast.

On Mon, Sep 3, 2018 at 3:13 PM Jon Lewis via Outages 
wrote:

> Can't handle some light humor?  They are down though.  At least from here
> (both web and mobile) via Spectrum cable and AT wireless.
>
> On Mon, 3 Sep 2018, Dominick Colangelo wrote:
>
> > I didn't realize this network operator outage listserv converted to a
> political satire listserv.
> >
> > On Mon, Sep 3, 2018, 17:02 Jon Lewis via Outages 
> wrote:
> >   So...did Trump shut them down?  :)
> >
> >
>  --
> > Jon Lewis, MCP :)   |  I route
> > |  therefore you are
> >   _ http://www.lewis.org/~jlewis/pgp for PGP public
> key_
> >   ___
> >   Outages mailing list
> >   Outages@outages.org
> >   https://puck.nether.net/mailman/listinfo/outages
> >
> >
> >
>
> --
>   Jon Lewis, MCP :)   |  I route
>   |  therefore you are
> _ http://www.lewis.org/~jlewis/pgp for PGP public
> key
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Facebook Outage

[John Neiberger via Outages]

I absolutely and vehemently disagree. The first signs of a network outage
*are* application outages. If you tell people not to report major
application outages--and let's face it, Facebook's 200 million US customers
is major--then we'll miss out on crucial early detection and
troubleshooting opportunities. And what applications are we talking about,
exactly? Does VoIP become an application, something that should no longer
be discussed? It doesn't make sense to me to narrow the focus of this list
that much.

My $0.02. With that said, this really should be on the discussion list. I
guess I need to make sure I'm even subscribed to that one.

John

On Fri, Aug 3, 2018 at 1:21 PM Chris Mayhew via Outages 
wrote:

> I agree that this is the wrong place for application outages, be it Slack,
> Facebook or Google.  For those who are concerned when these applications go
> down, it would be more appropriate to start an application outages list.
>
>
>
>
>
> *From:* Outages  *On Behalf Of *Neal Lawson
> via Outages
> *Sent:* August 3, 2018 15:16
> *To:* Josh Luthman 
> *Cc:* OUTAGES Mailing List 
> *Subject:* Re: [outages] Facebook Outage
>
>
>
> Point of order.. should this not belong on outages-discussion?
>
>
>
>
>
> On Aug 3, 2018, at 12:11 PM, Josh Luthman via Outages 
> wrote:
>
>
>
> As an ISP we get tons of calls when "the Internet is down" because
> Facebook.com doesn't load.  It is the third most visited website in the
> world after all.  I'm sure there are others that benefit from knowing this
> information.
>
>
>
> It's disappointing you're starting a discussion on the non-discussion list
> this odd complaint is not appropriate here.
>
>
>
>
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
>
>
>
> On Fri, Aug 3, 2018 at 3:07 PM, James via Outages 
> wrote:
>
> I’m surprised a Facebook outage rises to the level that it would be
> included in this list.  I guess disappointed is more appropriate.
>
>
>
> James
>
>
>
> On Fri, Aug 3, 2018 at 1:53 PM Eduardo Schoedler via Outages <
> outages@outages.org> wrote:
>
> 2018-08-03 14:04 GMT-03:00 Saymon Fernandes via Outages <
> outages@outages.org>:
>
> Hello,
>
>
>
> Anyone knows what caused this ?
>
>
>
>
>
>
>
> n...@fb.com
>
>
>
>
>
> --
>
> Eduardo Schoedler
>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> --
>
> _
> James Mitchell
> 636-634-0196
> jame...@gmail.com
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast data issues

[John Neiberger via Outages]

Sounds accurate.

http://abc7ny.com/crews-respond-to-manhole-fire-outside-chelsea-market/3574425/

We're trying to find some alternate fiber to get some of our circuits back
up.

On Thu, Jun 7, 2018, 2:22 PM Nikolas Geyer  wrote:

> What I have heard (anecdotally, not confirmed) is someone’s cut a gas pipe
> and some fiber at (or near) 111 8th. Gas leak in the area so physical
> access has been restricted.
>
> Sent from my iPhone
>
> On 7 Jun 2018, at 3:16 pm, John Neiberger via Outages 
> wrote:
>
> We're checking into this. We suspect a fiber cut in NY affecting multiple
> core and edge routers in that region.
>
> On Thu, Jun 7, 2018 at 1:03 PM Chuck Anderson via Outages <
> outages@outages.org> wrote:
>
>> Has anyone noticed any Comcast data issues?  I could't ping 8.8.8.8 a few
>> minutes ago although that came back.  Also issues with Google Docs.  My
>> IPv6 connectivity was also down.  Things are slowly recovering.  Perhaps
>> there was another core link flap in NYC.
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast data issues

[John Neiberger via Outages]

We're checking into this. We suspect a fiber cut in NY affecting multiple
core and edge routers in that region.

On Thu, Jun 7, 2018 at 1:03 PM Chuck Anderson via Outages <
outages@outages.org> wrote:

> Has anyone noticed any Comcast data issues?  I could't ping 8.8.8.8 a few
> minutes ago although that came back.  Also issues with Google Docs.  My
> IPv6 connectivity was also down.  Things are slowly recovering.  Perhaps
> there was another core link flap in NYC.
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast IPv6 outages

[John Neiberger via Outages]

Yeah, I see there were some transport flaps in the area earlier, but it
seems to be stable now. I suspect those were the cause of this problem.

On Wed, Apr 4, 2018 at 11:22 AM, Dave Pascoe <davek...@gmail.com> wrote:

> This went beyond IPv6. IPv4 routing was dying at Comcast in NYC to Linode
> and Choopa/Vultr, among possibly other destination networks.
>
> This happened twice, once earlier this morning and once just recently.
> Cleared up in less than 10 minutes.
>
> Sorry for the lack of detail...not near laptop right now.
>
> Dave
>
> On Wed, Apr 4, 2018 at 1:04 PM, Michael via Outages <outages@outages.org>
> wrote:
>
>> Had to turn off IPv6 on my home router.  Connections to Ravellosystems
>> and Atlassian were dropping for me intermittently.
>>
>>
>> -- Original Message --
>> From: "John Neiberger via Outages" <outages@outages.org>
>> To: "Chuck Anderson" <c...@wpi.edu>; outages@outages.org
>> Sent: 4/4/2018 11:57:54 AM
>> Subject: Re: [outages] Comcast IPv6 outages
>>
>> What IPv6 destination are you tracing to? I'll take a look.
>>
>> On Wed, Apr 4, 2018 at 10:51 AM, Chuck Anderson via Outages <
>> outages@outages.org> wrote:
>>
>>> Comcast in New England/Massachusetts has been experiencing intermittent
>>> IPv6 packet loss and outages for the past hour or so on and off.
>>>
>>> Here is a traceroute when it isn't going through:
>>>
>>>Packets   Pings
>>>  HostLoss%   Snt   Last   Avg  Best
>>> Wrst StDev
>>>  1. 2601:19c:4880:5121::1 0.0%532.0   1.9   1.3
>>>  4.4   0.4
>>>  2. 2001:558:4023:147::1  1.9%53   11.8  22.0  10.4
>>> 131.7  21.5
>>>  3. 2001:558:202:410::1   0.0%53   13.8  15.5  10.0
>>> 70.9  11.6
>>>  4. 2001:558:200:364::1   0.0%53   68.7  18.6  10.4
>>> 190.7  25.9
>>>  5. 2001:558:200:4020::1 11.5%53   49.1  21.4  12.5
>>> 170.3  23.8
>>>  6. be-7015-cr02.newyork.ny.ibone.co 26.4%53  100.8  33.9  18.5
>>> 116.8  26.7
>>>  7. ???
>>>
>>> The last working hop above resolves to: 2001:558:0:f6b6::1.  It has been
>>> down for the last 5-10 minutes now.
>>> ___
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast IPv6 outages

[John Neiberger via Outages]

What IPv6 destination are you tracing to? I'll take a look.

On Wed, Apr 4, 2018 at 10:51 AM, Chuck Anderson via Outages <
outages@outages.org> wrote:

> Comcast in New England/Massachusetts has been experiencing intermittent
> IPv6 packet loss and outages for the past hour or so on and off.
>
> Here is a traceroute when it isn't going through:
>
>Packets   Pings
>  HostLoss%   Snt   Last   Avg  Best  Wrst
> StDev
>  1. 2601:19c:4880:5121::1 0.0%532.0   1.9   1.3   4.4
>  0.4
>  2. 2001:558:4023:147::1  1.9%53   11.8  22.0  10.4 131.7
> 21.5
>  3. 2001:558:202:410::1   0.0%53   13.8  15.5  10.0  70.9
> 11.6
>  4. 2001:558:200:364::1   0.0%53   68.7  18.6  10.4 190.7
> 25.9
>  5. 2001:558:200:4020::1 11.5%53   49.1  21.4  12.5 170.3
> 23.8
>  6. be-7015-cr02.newyork.ny.ibone.co 26.4%53  100.8  33.9  18.5
> 116.8  26.7
>  7. ???
>
> The last working hop above resolves to: 2001:558:0:f6b6::1.  It has been
> down for the last 5-10 minutes now.
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast - Houston Outage

[John Neiberger via Outages]

I don't know that it is resolved yet, but it has been stable for ten
minutes. I'm keeping an eye on it and trying to find out some more details.

On Tue, Mar 27, 2018 at 9:51 AM, Darin Steffl via Outages <
outages@outages.org> wrote:

> Ok before 6 emails turns into 87 emails, the outage sounds like it's
> resolved. No need for more "me too" emails.
>
> On Tue, Mar 27, 2018, 8:49 AM Thom Blew via Outages 
> wrote:
>
>> we went down - Comcast Business - Cable for us.
>>
>> On Tue, Mar 27, 2018 at 10:23 AM, Neader, Brent via Outages <
>> outages@outages.org> wrote:
>>
>>> We just had 5 sites in Houston with Comcast all go offline at once.
>>> Anyone else?
>>>
>>> ___
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast - Houston Outage

[John Neiberger via Outages]

Word on the street is multiple fiber issues happening at the same time,
including a fiber cut. I'm trying to find out more.

On Tue, Mar 27, 2018 at 9:43 AM, Thom Blew via Outages 
wrote:

> we went down - Comcast Business - Cable for us.
>
> On Tue, Mar 27, 2018 at 10:23 AM, Neader, Brent via Outages <
> outages@outages.org> wrote:
>
>> We just had 5 sites in Houston with Comcast all go offline at once.
>> Anyone else?
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] DNS/Routing issues for Comcast/Xfinity customers?

[John Neiberger via Outages]

If you can, send me some details and I'll see if I can find someone to
check into this. I haven't heard of any widespread issues today, but maybe
you're catching something just as it's beginning.

-John

On Thu, Mar 1, 2018 at 1:23 PM, Wilda, Ben via Outages 
wrote:

> Anyone else with users on Comcast or Xfinity having issues reaching
> certain websites? It’s not consistent so far (not all users on Comcast
> having issues), but everyone who has reported issues connecting with some
> of our public-facing services is on Comcast or Xfinity. Looks to us like a
> DNS or routing issue for Comcast.
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] cocmast packetloss and high latency

[John Neiberger via Outages]

It's looking like a problem with Level 3, which would affect us (Comcast).
The "me too" emails aren't that helpful, but traceroutes would be very
helpful! So far, every trace/mtr showing a problem is showing loss within
Level 3 and I've heard three reports of "some problem" within Level 3.
Whatever is going on is pretty major, but we don't have much good data yet.

On Mon, Nov 6, 2017 at 11:39 AM, Neil Hanlon via Outages <
outages@outages.org> wrote:

> We just got a notification from Bluejeans warning of degraded service for
> Comcast customers due to "issues comcast is having in several regions".
>
> So, sounds like it's a known thing. Can probably direct the "me toos" to
> the discussion list/trash? They don't really add much more to the thread
> except notifications on our devices :)
>
> On Nov 6, 2017 13:36, "Guillermo via Outages"  wrote:
>
>>
>>
>>
>> Can confirm, we see it affecting our NJ Data center right now
>>
>>
>>
>>
>>
>> On 2017-11-06 13:23, Audrius Peciukas via Outages wrote:
>>
>> anyone else is seeing lots of packet loss and 100ms+ latency mostly in
>> east coast on comcast network?
>>
>>
>> --
>>
>> Audrius Peciukas
>> 1Wire Communications
>> http://www.1wirecom.com
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] cocmast packetloss and high latency

[John Neiberger via Outages]

Anyone having packet loss, please send traceroutes to the list. There is
clearly something happening, but not much useful data yet. Some traceroutes
might help narrow down the cause of the problem.

On Mon, Nov 6, 2017 at 11:29 AM, Mr. NPP via Outages 
wrote:

> I'm also getting lots of reports from customers with massive amounts of
> packet loss.
>
> http://downdetector.com/status/comcast-xfinity
>
> seems to be a comcast issue for sure.
>
> On Mon, Nov 6, 2017 at 10:25 AM, Jim Limmer via Outages <
> outages@outages.org> wrote:
>
>> Yes. And whats worse is my deli nearby uses Comcast business for their
>> credit card reader.. guess I owe them for lunch tomorrow..
>>
>>
>>
>>
>>
>> Jim Limmer, CISSP
>>
>> < Information Security Officer
>>
>>  Systems and Network Architect >
>>
>> VNA Community Healthcare
>>
>> www.connecticuthomecare.org
>>
>>
>>
>> [image: cid:image002.jpg@01D332E1.E5C48B90] [image: smCISSP-logo-2lines]
>>
>>
>>
>>
>>
>>
>>
>>
>> *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *Audrius
>> Peciukas via Outages
>> *Sent:* Monday, November 06, 2017 1:24 PM
>> *To:* outages@outages.org
>> *Subject:* [outages] cocmast packetloss and high latency
>>
>>
>>
>> anyone else is seeing lots of packet loss and 100ms+ latency mostly in
>> east coast on comcast network?
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>> Audrius Peciukas
>>
>> 1Wire Communications
>>
>> http://www.1wirecom.com
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Zayo Down?

[John Neiberger via Outages]

We just saw them trip our max prefix limits. I wonder if "something bad"
happened and sessions are being torn down with multiple providers because
of prefix limit violations.

On Fri, Sep 29, 2017 at 12:12 PM, Erik Wooding via Outages <
outages@outages.org> wrote:

> I've got about a dozen Zayo sites that are down for the last 20 minutes or
> so, anyone else seeing significant outages in the midwest/west?
>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] smugmug.com (on cloudfront) is down

[John Neiberger via Outages]

​There are some serious fiber issues happening out east tonight with Level
3. I'm not sure why there would be a reachability issue, though. Can you
send me a traceroute?

Thanks​

On Fri, Sep 15, 2017 at 8:45 PM, Patrick W. Gilmore via Outages <
outages@outages.org> wrote:

> It seems I have forgotten to say I am coming from Comcast in Boston.
> (Sorry.)
>
> RDP’ed to my office (One Summer in Boston), works fine.
>
> Strange that downforeveryoneorjustme says it is down as well.
>
> --
> TTFN,
> patrick
>
>
> On Sep 15, 2017, at 10:41 PM, Jay Farrell  wrote:
>
> Working from Conshohocken/Philadelphia PA; doesn't redirect.
>
> Jay
>
> On Fri, Sep 15, 2017 at 10:32 PM, Patrick W. Gilmore via Outages <
> outages@outages.org> wrote:
>
>> Thanks. Wish I were in Milwaukee. :-)
>>
>> DownForEveryoneOrJustMe says:
>> It's not just you! wc.smugmug.com.cdn.cloudflare.net looks down
>> from here.
>>
>> Ditto for dq33tynpwunh.cloudfront.net.
>>
>> --
>> TTFN,
>> patrick
>>
>>
>> > On Sep 15, 2017, at 10:30 PM, Nicholas Sideras <
>> nside...@hollandbulbfarms.com> wrote:
>> >
>> > Site and Android app are both working as expected from Milwaukee.
>> >
>> > From: "Patrick W. Gilmore via Outages" 
>> > Sent: Sep 15, 2017 9:19 PM
>> > To: outages@outages.org
>> > Subject: [outages] smugmug.com (on cloudfront) is down
>> >
>> > [This email is from an outside sender. Think twice before opening
>> attachments or replying with confidential information.]
>> >
>> > Couldn’t get to smugmug (where I keep my photos).
>> >
>> > The host “www.smugmug.com” is up, but it redirects to cloudfront:
>> >
>> > TiggerBook-C:~ patrick$ host www.smugmug.com
>> > www.smugmug.com is an alias for dq33tynpwunh.cloudfront.net.
>> >
>> > That host is down. As is the specific hostname that serves my personal
>> photos.
>> >
>> > --
>> > TTFN,
>> > patrick
>> >
>> > ___
>> > Outages mailing list
>> > Outages@outages.org
>> > https://puck.nether.net/mailman/listinfo/outages
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast NYC area

[John Neiberger via Outages]

I'll take a look. It goes without saying that a fiber cut out east has
nothing to do with packet loss in the Seattle regional network, so it's
off-topic for this thread. I'll look and see what I can find. That does
look like there is some real loss between that AR and CR.

John

On Wed, Aug 2, 2017 at 8:25 AM, John Sage <js...@finchhaven.com> wrote:

> On 08/02/2017 06:58 AM, John Neiberger via Outages wrote:
>
>> I'll take a look and see what's going on.
>>
>> Thanks,
>> John
>>
>> On Wed, Aug 2, 2017 at 6:42 AM, Mark Price via Outages
>> <outages@outages.org <mailto:outages@outages.org>> wrote:
>>
>>
>> I'm seeing consistent packet loss getting to Comcast endpoints in
>> the northeast US.
>>
>
> Welcome to my world. I see this from Comcast just about any time I run mtr
> against just about any destination you'd care to name.
>
> Retail "high speed Internet" at its ... worst.
>
>
> - John
> --
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast NYC area

[John Neiberger via Outages]

Additional remediation was required. Should be good now. Traffic is fully
routed around the problem area.

Thanks,
John

On Wed, Aug 2, 2017 at 7:58 AM, John Neiberger  wrote:

> I'll take a look and see what's going on.
>
> Thanks,
> John
>
> On Wed, Aug 2, 2017 at 6:42 AM, Mark Price via Outages <
> outages@outages.org> wrote:
>
>>
>> I'm seeing consistent packet loss getting to Comcast endpoints in the
>> northeast US.
>>
>> If you can, please forward to someone w/ a clue at Comcast?
>>
>> $ mtr --report-wide 50.232.27.1
>> ...
>>   3.|-- e1-5.er01.ral01.alwaysdedicated.net 0.0%10
>>  0.7   0.7   0.6   0.8   0.0
>>   4.|-- 208.79.80.254   0.0%107.0
>>   7.1   7.0   7.2   0.0
>>   5.|-- 246.10.225.104.rdns.hostvirtual.com 0.0%10
>>  7.0   7.0   6.8   7.2   0.0
>>   6.|-- 1200.ae8.er4.iad10.us.zip.zayo.com  0.0%10
>>  7.1   7.2   6.9   7.5   0.0
>>   7.|-- ae16.er5.iad10.us.zip.zayo.com  0.0%10
>>  7.4   7.4   7.3   7.5   0.0
>>   8.|-- ae15.cr2.dca2.us.zip.zayo.com   0.0%10
>>  8.5   8.5   8.2   9.0   0.0
>>   9.|-- ae1.er2.iad10.us.zip.zayo.com   0.0%10
>>  9.6  11.3   9.2  29.3   6.3
>>  10.|-- zayo-comcast.iad10.us.zip.zayo.com  0.0%10
>> 11.1   9.1   8.5  11.1   0.7
>>  11.|-- hu-0-3-0-6-cr02.ashburn.va.ibone.comcast.net0.0%10
>> 10.1  10.5  10.1  11.4   0.0
>>  12.|-- be-10102-cr01.newark.nj.ibone.comcast.net  20.0%10
>> 16.2  15.8  15.3  16.2   0.0
>>  13.|-- be-10203-cr02.newyork.ny.ibone.comcast.net 30.0%10
>> 16.3  15.8  15.0  16.3   0.0
>>  14.|-- be-7922-ar01.needham.ma.boston.comcast.net 20.0%10
>> 21.6  21.9  21.3  23.0   0.4
>>  15.|-- te-0-7-0-0-sur02.foxboro.ma.boston.comcast.net 10.0%10
>> 22.0  21.9  21.8  22.1   0.0
>>  16.|-- 68.85.88.253   30.0%10   21.9
>>  21.9  21.8  22.0   0.0
>>  17.|-- 50-232-27-1-static.hfc.comcastbusiness.net30.0%10   21.6
>>  21.5  21.4  21.7   0.0
>>
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast NYC area

[John Neiberger via Outages]

I'll take a look and see what's going on.

Thanks,
John

On Wed, Aug 2, 2017 at 6:42 AM, Mark Price via Outages 
wrote:

>
> I'm seeing consistent packet loss getting to Comcast endpoints in the
> northeast US.
>
> If you can, please forward to someone w/ a clue at Comcast?
>
> $ mtr --report-wide 50.232.27.1
> ...
>   3.|-- e1-5.er01.ral01.alwaysdedicated.net 0.0%100.7
>   0.7   0.6   0.8   0.0
>   4.|-- 208.79.80.254   0.0%107.0
>   7.1   7.0   7.2   0.0
>   5.|-- 246.10.225.104.rdns.hostvirtual.com 0.0%107.0
>   7.0   6.8   7.2   0.0
>   6.|-- 1200.ae8.er4.iad10.us.zip.zayo.com  0.0%107.1
>   7.2   6.9   7.5   0.0
>   7.|-- ae16.er5.iad10.us.zip.zayo.com  0.0%107.4
>   7.4   7.3   7.5   0.0
>   8.|-- ae15.cr2.dca2.us.zip.zayo.com   0.0%108.5
>   8.5   8.2   9.0   0.0
>   9.|-- ae1.er2.iad10.us.zip.zayo.com   0.0%109.6
>  11.3   9.2  29.3   6.3
>  10.|-- zayo-comcast.iad10.us.zip.zayo.com  0.0%10   11.1
>   9.1   8.5  11.1   0.7
>  11.|-- hu-0-3-0-6-cr02.ashburn.va.ibone.comcast.net0.0%10   10.1
>  10.5  10.1  11.4   0.0
>  12.|-- be-10102-cr01.newark.nj.ibone.comcast.net  20.0%10   16.2
>  15.8  15.3  16.2   0.0
>  13.|-- be-10203-cr02.newyork.ny.ibone.comcast.net 30.0%10   16.3
>  15.8  15.0  16.3   0.0
>  14.|-- be-7922-ar01.needham.ma.boston.comcast.net 20.0%10   21.6
>  21.9  21.3  23.0   0.4
>  15.|-- te-0-7-0-0-sur02.foxboro.ma.boston.comcast.net 10.0%10   22.0
>  21.9  21.8  22.1   0.0
>  16.|-- 68.85.88.253   30.0%10   21.9
>  21.9  21.8  22.0   0.0
>  17.|-- 50-232-27-1-static.hfc.comcastbusiness.net30.0%10   21.6
>  21.5  21.4  21.7   0.0
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast

[John Neiberger via Outages]

​If you have any more details you can provide, please send them to me. I
checked our national NOC email list and didn't see anything jumping out at
me about that area, but if you have more details, I can see what I can find
out.

John​

On Thu, Mar 30, 2017 at 10:35 AM, Chris Bromley via Outages <
outages@outages.org> wrote:

> Noticed that Comcast in the DC area appears to be having some up/down
> connectivity issues across various routers this morning, every 15-20
> minutes.  Having customers traveling through that network having trouble,
> and when I ping each hop, only Comcast's drop for extended periods of time.
>
> Call has been put in on their side to rectify.
>
>
> On 3/30/2017 12:13 PM, Scott Whyte via Outages wrote:
>
>> Dreamhost is having some widespread problems this morning
>>
>> https://www.dreamhoststatus.com/
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
> --
> Chris Bromley
> Cloud Services Systems Administrator
> TMW Systems Inc. | A TRIMBLE COMPANY
> www.tmwsystems.com
>
>
> --
> This email and any files transmitted with it are confidential and intended
> solely for the use of the addressee. If you are not the intended addressee,
> then you have received this email in error and any use, dissemination,
> forwarding, printing, or copying of this email is strictly prohibited.
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast Outages

[John Neiberger via Outages]

I haven't heard of any issues this morning. Can you be more specific? I'll
see what I can find out.

Thanks,
John

On Wed, Nov 16, 2016 at 10:42 AM, Matthew Carse via Outages <
outages@outages.org> wrote:

> Anyone else seeing Comcast down in California?
>
> We are seeing all our Comcast connections offline.
>
> Matt
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast cable outage?

[John Neiberger via Outages]

Can you send me some traceroutes? I'll check with our NOC to see if they've
heard of anything.

Thanks,
John

On Thu, Oct 20, 2016 at 6:29 AM, Rob Hall via Outages 
wrote:

> John,
>
> Every office we have on Comcast in NE TN, SW VA, and
> Westen Kentucky.
>
>
>
> Thanks!
>
> --
> Rob
>
>
>
> *From:* John Neiberger [mailto:jneiber...@gmail.com]
> *Sent:* Thursday, October 20, 2016 8:27 AM
> *To:* Rob Hall 
> *Cc:* outages@outages.org
> *Subject:* Re: [outages] Comcast cable outage?
>
>
>
> Can you provide some more details? I'm not aware of any regional issues. I
> just checked our NOC email list and chat and nothing jumped out at me.
>
>
>
> Thanks,
>
> John
>
>
>
> On Thu, Oct 20, 2016 at 6:20 AM, Rob Hall via Outages 
> wrote:
>
> Sorry, it’s early and I haven’t had any coffee yet. Comcast is the only
> one in question.
>
>
>
> --
>
> Rob
>
>
>
> *From:* Wyn Bryant [mailto:w...@vistaits.com]
> *Sent:* Thursday, October 20, 2016 8:16 AM
> *To:* Rob Hall 
> *Subject:* RE: [outages] Comcast cable outage?
>
>
>
> None in various FL markets (so far) for Comcast.
>
>
>
> Your message references both Comcast and Charter though?
>
>
>
>
>
>  Original message 
>
> From: Rob Hall via Outages 
>
> Date: 10/20/16 8:12 AM (GMT-05:00)
>
> To: outages@outages.org
>
> Subject: [outages] Comcast cable outage?
>
>
>
> I’ve gotten reports from my monitoring system for offices in 3 states that
> are offline. Is anyone else seeing this happening with Charter in the
> southeastern region?
>
>
>
> Thanks!
>
> --
>
> Rob Hall, CCNP R, CCP-V
>
> I.T. Manager
>
> @Work Personnel Service
>
> 3815 Hwy 66 S, Suite 4
>
> Rogersville, TN 37857
>
> Office: (423)272-7897, x10113
>
> Mobile: (423)312-6003
>
> [image: cisco-certified-ccnp][image: CCP-V]
>
>
>
>
>
> --
>
> This email was Malware checked by UTM 9. http://www.sophos.com
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast cable outage?

[John Neiberger via Outages]

Can you provide some more details? I'm not aware of any regional issues. I
just checked our NOC email list and chat and nothing jumped out at me.

Thanks,
John

On Thu, Oct 20, 2016 at 6:20 AM, Rob Hall via Outages 
wrote:

> Sorry, it’s early and I haven’t had any coffee yet. Comcast is the only
> one in question.
>
>
>
> --
>
> Rob
>
>
>
> *From:* Wyn Bryant [mailto:w...@vistaits.com]
> *Sent:* Thursday, October 20, 2016 8:16 AM
> *To:* Rob Hall 
> *Subject:* RE: [outages] Comcast cable outage?
>
>
>
> None in various FL markets (so far) for Comcast.
>
>
>
> Your message references both Comcast and Charter though?
>
>
>
>
>
>  Original message 
>
> From: Rob Hall via Outages 
>
> Date: 10/20/16 8:12 AM (GMT-05:00)
>
> To: outages@outages.org
>
> Subject: [outages] Comcast cable outage?
>
>
>
> I’ve gotten reports from my monitoring system for offices in 3 states that
> are offline. Is anyone else seeing this happening with Charter in the
> southeastern region?
>
>
>
> Thanks!
>
> --
>
> Rob Hall, CCNP R, CCP-V
>
> I.T. Manager
>
> @Work Personnel Service
>
> 3815 Hwy 66 S, Suite 4
>
> Rogersville, TN 37857
>
> Office: (423)272-7897, x10113
>
> Mobile: (423)312-6003
>
> [image: cisco-certified-ccnp][image: CCP-V]
>
>
>
>
>
> --
>
> This email was Malware checked by UTM 9. http://www.sophos.com
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast Issues in Miami?

[John Neiberger via Outages]

It looks like someone shifted an enormous amount of traffic to that path
and it's maxed out toward us. I don't know if Cogent shifted traffic or if
one of their customers is sending far, far more traffic than normal to us.
I'll check some other tools to see if I can figure it out. It's definitely
full, though. At this time of day, that particular bundle would be running
at around 30-35%, but around 16 hours ago, it hit 100% and hasn't dropped
since.

John

On Tue, Jun 14, 2016 at 8:47 AM, John Neiberger 
wrote:

> I'll take a look.
>
> Thanks,
> John
>
> On Tue, Jun 14, 2016 at 8:44 AM, Mason Moody via Outages <
> outages@outages.org> wrote:
>
>> Anyone else seeing issues with carrier handoff to Comcast in Miami? I'm
>> seeing dropped packets in the handoff between Cogent and Comcast there:
>>
>> HOST: orlando Loss%   Snt
>>  Last   Avg  Best  Wrst StDev
>>   1.|-- 10.10.200.20.0%25
>> 0.3   0.3   0.2   0.5   0.0
>>   2.|-- 10.10.200.252  8.0%25
>> 0.6   0.3   0.2   0.7   0.0
>>   3.|-- vl5-dis-01.orl-fl.atlantic.net 0.0%25
>> 0.5   0.9   0.5   3.0   0.6
>>   4.|-- 209.208.7.41   0.0%25
>> 0.4   0.6   0.4   0.8   0.0
>>   5.|-- te0-0-1-1.nr11.b006655-1.mco01.atlas.cogentco.com  0.0%25
>> 1.0   1.2   0.9   2.3   0.2
>>   6.|-- te0-0-1-1.rcr11.mco01.atlas.cogentco.com   0.0%25
>> 2.5   2.5   2.3   2.8   0.0
>>   7.|-- te0-0-1-0.rcr21.tpa01.atlas.cogentco.com   0.0%25
>> 6.4   4.6   4.3   6.4   0.4
>>   8.|-- be2744.ccr21.mia01.atlas.cogentco.com  0.0%25
>>  10.1  10.5  10.0  17.1   1.3
>>   9.|-- be2054.ccr21.mia03.atlas.cogentco.com  0.0%25
>>  11.0  11.2  10.8  13.0   0.4
>>  10.|-- be-200-pe03.nota.fl.ibone.comcast.net 20.0%25
>>  11.9  12.1  11.8  13.7   0.3
>>  11.|-- hu-0-3-0-1-cr02.miami.fl.ibone.comcast.net32.0%25
>>  12.1  11.4   9.9  12.4   0.5
>>  12.|-- be-11422-cr02.56marietta.ga.ibone.comcast.net 28.0%25
>>  19.8  19.3  18.0  20.7   0.4
>>  13.|-- be-10114-cr02.ashburn.va.ibone.comcast.net20.0%25
>>  32.2  31.3  30.3  32.2   0.3
>>  14.|-- et-15-0-1-0-ar01.capitolhghts.md.bad.comcast.net  20.0%25
>>  33.2  33.2  33.0  33.4   0.0
>>  15.|-- ae-0-0-sur03.arlington.va.bad.comcast.net 32.0%25
>>  34.0  34.4  33.8  41.0   1.7
>>  16.|-- te-7-0-acr02.arlington.va.bad.comcast.net 16.0%25
>>  34.7  34.8  34.7  35.4   0.0
>>  17.|-- 96-88-249-94-static.hfc.comcastbusiness.net   24.0%25
>>  46.5  45.8  42.3  49.2   1.9
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast Issues in Miami?

[John Neiberger via Outages]

I'll take a look.

Thanks,
John

On Tue, Jun 14, 2016 at 8:44 AM, Mason Moody via Outages <
outages@outages.org> wrote:

> Anyone else seeing issues with carrier handoff to Comcast in Miami? I'm
> seeing dropped packets in the handoff between Cogent and Comcast there:
>
> HOST: orlando Loss%   Snt
>  Last   Avg  Best  Wrst StDev
>   1.|-- 10.10.200.20.0%25
> 0.3   0.3   0.2   0.5   0.0
>   2.|-- 10.10.200.252  8.0%25
> 0.6   0.3   0.2   0.7   0.0
>   3.|-- vl5-dis-01.orl-fl.atlantic.net 0.0%25
> 0.5   0.9   0.5   3.0   0.6
>   4.|-- 209.208.7.41   0.0%25
> 0.4   0.6   0.4   0.8   0.0
>   5.|-- te0-0-1-1.nr11.b006655-1.mco01.atlas.cogentco.com  0.0%25
> 1.0   1.2   0.9   2.3   0.2
>   6.|-- te0-0-1-1.rcr11.mco01.atlas.cogentco.com   0.0%25
> 2.5   2.5   2.3   2.8   0.0
>   7.|-- te0-0-1-0.rcr21.tpa01.atlas.cogentco.com   0.0%25
> 6.4   4.6   4.3   6.4   0.4
>   8.|-- be2744.ccr21.mia01.atlas.cogentco.com  0.0%25
>  10.1  10.5  10.0  17.1   1.3
>   9.|-- be2054.ccr21.mia03.atlas.cogentco.com  0.0%25
>  11.0  11.2  10.8  13.0   0.4
>  10.|-- be-200-pe03.nota.fl.ibone.comcast.net 20.0%25
>  11.9  12.1  11.8  13.7   0.3
>  11.|-- hu-0-3-0-1-cr02.miami.fl.ibone.comcast.net32.0%25
>  12.1  11.4   9.9  12.4   0.5
>  12.|-- be-11422-cr02.56marietta.ga.ibone.comcast.net 28.0%25
>  19.8  19.3  18.0  20.7   0.4
>  13.|-- be-10114-cr02.ashburn.va.ibone.comcast.net20.0%25
>  32.2  31.3  30.3  32.2   0.3
>  14.|-- et-15-0-1-0-ar01.capitolhghts.md.bad.comcast.net  20.0%25
>  33.2  33.2  33.0  33.4   0.0
>  15.|-- ae-0-0-sur03.arlington.va.bad.comcast.net 32.0%25
>  34.0  34.4  33.8  41.0   1.7
>  16.|-- te-7-0-acr02.arlington.va.bad.comcast.net 16.0%25
>  34.7  34.8  34.7  35.4   0.0
>  17.|-- 96-88-249-94-static.hfc.comcastbusiness.net   24.0%25
>  46.5  45.8  42.3  49.2   1.9
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Cannot access 132.60.0.0/16 from 141.219.0.0/16

[John Neiberger via Outages]

My trace looked just like this. It's as if some probes reach the
destination, but other probes spin off into Neverland.

On Wed, Jun 8, 2016 at 9:38 AM, Jay Farrell via Outages  wrote:

> Here's an odd trace from Verizon DSL in phl; traceroute *passes thru*
> the target IP (18th & 19th hop) and keeps on going
>
> $ traceroute 132.60.10.110
>
> traceroute to 132.60.10.110 (132.60.10.110), 64 hops max, 52 byte packets
>
>  1  192.168.1.1 (192.168.1.1)  4.847 ms  0.775 ms  0.729 ms
>
>  2  10.7.120.1 (10.7.120.1)  24.318 ms  23.881 ms  23.799 ms
>
>  3  g0-5-3-2.phlapa-lcr-21.verizon-gni.net (130.81.193.156)  25.839 ms
>  27.322 ms  28.054 ms
>
>  4  * * *
>
>  5  * * *
>
>  6  140.222.230.249 (140.222.230.249)  28.971 ms  30.500 ms  29.107 ms
>
>  7  63-235-40-49.dia.static.qwest.net (63.235.40.49)  29.777 ms
> 30.166 ms  30.333 ms
>
>  8  frp-edge-06.inet.qwest.net (67.14.31.222)  64.505 ms  65.422 ms
> 64.512 ms
>
>  9  67.133.228.2 (67.133.228.2)  66.473 ms  66.183 ms  65.948 ms
>
> 10  * * *
>
> 11  * * *
>
> 12  * * *
>
> 13  * * *
>
> 14  * * *
>
> 15  100.50.1.1 (100.50.1.1)  68.667 ms  68.542 ms  68.992 ms
>
> 16  * * *
>
> 17  * * *
>
> 18  132.60.10.110 (132.60.10.110)  66.489 ms  66.270 ms  66.555 ms
>
> 19  132.60.10.110 (132.60.10.110)  67.895 ms  68.194 ms  68.009 ms
>
> 20  * * *
>
> 21  * * *
>
> 22  140.14.52.33 (140.14.52.33)  88.105 ms  67.567 ms  67.976 ms
>
> 23  * * *
>
> 24  * * *
>
> 25  214.40.23.10 (214.40.23.10)  85.446 ms  84.989 ms  85.437 ms
>
> 26  * * *
>
> 27  * * *
>
> 28  * * *
>
> 29  * * *
>
> 30  * * *
>
> 31  * * *
>
>
> On Wed, Jun 8, 2016 at 10:57 AM, Matt Freitag via Outages
>  wrote:
> > Actually, I can't seem to get to any of the prefixes at the US
> Department of
> > Defense...
> >
> > Matt Freitag
> > Network Engineer I
> > Information Technology
> > Michigan Technological University
> > (906) 487-3696
> > https://www.mtu.edu/
> > https://www.it.mtu.edu/
> >
> >
> > On Wed, Jun 8, 2016 at 10:54 AM, Matt Freitag  wrote:
> >>
> >> Looks like that's the DoD Network Information Center:
> >> http://bgp.he.net/AS395#_graph4
> >>
> >> Matt Freitag
> >> Network Engineer I
> >> Information Technology
> >> Michigan Technological University
> >> (906) 487-3696
> >> https://www.mtu.edu/
> >> https://www.it.mtu.edu/
> >>
> >>
> >> On Wed, Jun 8, 2016 at 10:53 AM, David Palmerton via Outages
> >>  wrote:
> >>>
> >>> Looks like the peering point Qwest is handing off to is the culprit
> >>>
> >>> Sent from my iPhone
> >>>
> >>> On Jun 8, 2016, at 10:51 AM, Dovid Bender via Outages
> >>>  wrote:
> >>>
> >>> From HE and NTT it dies on quests network. The last responding hop is
> >>> 67.133.228.2
> >>>
> >>> On Wed, Jun 8, 2016 at 10:43 AM, Jim Limmer via Outages
> >>>  wrote:
> 
>  I die at the same router, coming in from Level3
> 
> 
> 
>  Jim Limmer, CISSP
> 
>  < Information Security Officer
> 
>   Systems and Network Architect >
> 
> 
> 
> 
> 
> 
> 
>  From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Matt
>  Freitag via Outages
>  Sent: Wednesday, June 08, 2016 10:39 AM
>  To: outages@outages.org
>  Subject: [outages] Cannot access 132.60.0.0/16 from 141.219.0.0/16
> 
> 
> 
>  We're having problems getting to anything in the 132.60.0.0/16 net
> block
>  from 141.219.0.0/16. A traceroute shows traffic gets lost somewhere
> in
>  Qwest's network. Trace is below. Is anyone else having this issue? Is
>  someone from Qwest listening who has insight into what's going on?
> 
> 
> 
>  Specifically we're trying to get to 132.60.10.110:
> 
> 
> 
>  PS H:\> tracert ausis.maxwell.af.mil
> 
> 
> 
>  Tracing route to ausis.maxwell.af.mil [132.60.10.110]
> 
>  over a maximum of 30 hops:
> 
> 
> 
>    1 1 ms<1 ms<1 ms  mx480-07-001-staff-it-z16.tc.mtu.edu
>  [141.219.40.1]
> 
>    2<1 ms<1 ms<1 ms  mx80-07-002.tc.mtu.edu
> [141.219.183.97]
> 
>    3 1 ms 1 ms<1 ms  xe-0-0-3.hgtn-cor-mtu.mich.net
>  [207.75.40.9]
> 
>    4 4 ms 4 ms 4 ms  irbx70.pwrs-cor-powers.mich.net
>  [198.108.22.77]
> 
>    510 ms10 ms10 ms  ae1x18.eq-chi2.mich.net
> [198.108.22.37]
> 
>    611 ms11 ms10 ms  12.250.16.17
> 
>    714 ms11 ms11 ms  cr2.cgcil.ip.att.net [12.122.132.138]
> 
>    814 ms11 ms15 ms  cgr1.cgcil.ip.att.net [12.122.132.157
> ]
> 
>    911 ms11 ms11 ms  chp-brdr-03.inet.qwest.net
>  [63.146.26.217]
> 
>   10 *** Request timed out.
> 
>   1133 ms33 ms33 ms  67.133.228.2
> 
>   12 *** Request timed out.
> 

Re: [outages] Comcast issues in the Bay Area for the last 24 hours

[John Neiberger via Outages]

Sounds like it's something local to your neighborhood. Is this Metro E or
DOCSIS?

John

On Wed, May 11, 2016 at 10:44 AM, Robbie Trencheny <m...@robbiet.us> wrote:

> This outage is still on going. Comcast is continually pushing back the
> ETOR. I was also informed that Pandora.com HQ is affected by the same issue
> (they are located across the street from me). Just seeing random packet
> loss. Will try to get a traceroute today.
>
>
> On Monday, May 9, 2016, Michael Sinatra <michael+outa...@burnttofu.net>
> wrote:
>
>> I am still seeing significant loss between my work (ESnet in Berkeley)
>> and my home (Comcast Business Class in Uptown Oakland):
>>
>> [noc5] ~% ping manasquan.burnttofu.net
>> PING manasquan.burnttofu.net (98.210.60.123): 56 data bytes
>> 64 bytes from 98.210.60.123: icmp_seq=0 ttl=247 time=15.060 ms
>> 64 bytes from 98.210.60.123: icmp_seq=1 ttl=247 time=12.531 ms
>> 64 bytes from 98.210.60.123: icmp_seq=2 ttl=247 time=13.307 ms
>> 64 bytes from 98.210.60.123: icmp_seq=3 ttl=247 time=12.680 ms
>> 64 bytes from 98.210.60.123: icmp_seq=8 ttl=247 time=12.489 ms
>> 64 bytes from 98.210.60.123: icmp_seq=9 ttl=247 time=12.865 ms
>> 64 bytes from 98.210.60.123: icmp_seq=10 ttl=247 time=22.611 ms
>> 64 bytes from 98.210.60.123: icmp_seq=11 ttl=247 time=22.628 ms
>> 64 bytes from 98.210.60.123: icmp_seq=12 ttl=247 time=14.488 ms
>> 64 bytes from 98.210.60.123: icmp_seq=13 ttl=247 time=22.506 ms
>> 64 bytes from 98.210.60.123: icmp_seq=14 ttl=247 time=12.523 ms
>> 64 bytes from 98.210.60.123: icmp_seq=16 ttl=247 time=22.536 ms
>> 64 bytes from 98.210.60.123: icmp_seq=17 ttl=247 time=23.900 ms
>> 64 bytes from 98.210.60.123: icmp_seq=18 ttl=247 time=12.726 ms
>> 64 bytes from 98.210.60.123: icmp_seq=19 ttl=247 time=13.563 ms
>> 64 bytes from 98.210.60.123: icmp_seq=20 ttl=247 time=22.434 ms
>> ^C
>> --- manasquan.burnttofu.net ping statistics ---
>> 21 packets transmitted, 16 packets received, 23.8% packet loss
>> round-trip min/avg/max/stddev = 12.489/16.803/23.900/4.681 ms
>>
>> ESnet has a number of perfSONAR hosts (e.g. sunn-owamp.es.net) from/to
>> which you can do ping and traceroute if that helps.
>>
>> Similarly, I am seeing loss from UC San Francisco to my home network:
>>
>> [tulare:~] ms% ping manasquan.burnttofu.net
>> PING manasquan.burnttofu.net (98.210.60.123): 56 data bytes
>>
>> --- manasquan.burnttofu.net ping statistics ---
>> 26 packets transmitted, 23 packets received, 11.5% packet loss
>> round-trip min/avg/max/stddev = 16.218/22.903/50.951/7.144 ms
>>
>> I am not seeing any loss between UCSF and a VPS I have in Los Angeles.
>>
>> Traceroutes from UCSF to home:
>>
>> traceroute to manasquan.burnttofu.net (98.210.60.123), 64 hops max, 52
>> byte packets
>>  1  10.131.240.1 (10.131.240.1)  3.862 ms  4.669 ms  6.025 ms
>>  2  10.255.255.0 (10.255.255.0)  3.680 ms  3.583 ms  4.700 ms
>>  3  3002-s251-r01.ucsf.edu (169.230.252.9)  3.489 ms  3.166 ms  5.008 ms
>>  4  oak-agg4--ucsf--10ge.cenic.net (137.164.3.113)  5.633 ms  4.747 ms
>> 9.053 ms
>>  5  dc-svl-agg4--oak-agg4-100ge.cenic.net (137.164.46.145)  5.550 ms
>> 8.902 ms  8.859 ms
>>  6  xe-0-1-0.0.rtr.paix.net.internet2.edu (64.57.21.4)  6.259 ms
>> xe-1-1-0.0.rtr.paix.net.internet2.edu (64.57.21.6)  5.475 ms
>> xe-0-1-0.0.rtr.paix.net.internet2.edu (64.57.21.4)  5.857 ms
>>  7  te-0-3-0-0-pe02.529bryant.ca.ibone.comcast.net (66.208.233.121)
>> 6.002 ms  6.885 ms  6.002 ms
>>  8  * * *
>>  9  be-7922-ar01.santaclara.ca.sfba.comcast.net (68.86.90.94)  14.099
>> ms  6.931 ms  9.776 ms
>> 10  hu-0-5-0-1-sur03.oakland.ca.sfba.comcast.net (68.85.57.62)  8.405
>> ms  8.775 ms
>> hu-0-5-0-0-sur03.oakland.ca.sfba.comcast.net (68.85.57.58)  10.548 ms
>> 11  te-6-1-acr05.oakland.ca.sfba.comcast.net (68.85.217.178)  8.970 ms
>> 8.436 ms
>> te-6-0-acr05.oakland.ca.sfba.comcast.net (68.85.190.70)  8.556 ms
>> 12  c-98-210-60-123.hsd1.ca.comcast.net (98.210.60.123)  26.135 ms
>> 43.082 ms  32.402 ms
>>
>> Note, hop 8, when it does respond, looks like this:
>>
>>  8  68.86.86.29 (68.86.86.29)  7.791 ms
>> 68.86.86.145 (68.86.86.145)  7.724 ms
>> 68.86.86.69 (68.86.86.69)  8.259 ms
>>
>> michael
>>
>> On 5/9/16 2:02 PM, John Neiberger via Outages wrote:
>>
>>> Gotcha. Well, if it happens again, let me know. I'd be glad to take a
>>> look. I haven't heard of any widespread issues in that area. I was
>>> hoping we could narrow it down a bit.
>>>
>>> Thanks,
>>>

Re: [outages] Comcast issues in the Bay Area for the last 24 hours

[John Neiberger via Outages]

How are you verifying the packet loss? Do you have traceroutes?

Thanks,
John​

On Mon, May 9, 2016 at 2:42 PM, Robbie Trencheny via Outages <
outages@outages.org> wrote:

> There has been a Comcast outage/degradation of services in the Bay Area
> starting at 1403 PDT 05/08/2016. The current estimated restoration time is
> 1707 PDT. I couldn't get any details from Comcast about the issues. I've
> been seeing high packet loss. They did tell me that I will receive a bill
> credit.
>
> Robbie Trencheny (@robbie )
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast internet in Northern VA

[John Neiberger via Outages]

I just saw another report of someone at a colo who was noticing
intermittent bad response times from 8.8.8.8. They switched to 8.8.4.4 and
response times improved. When you notice it acting up again, try 8.8.4.4
and see if it behaves differently.

John

On Thu, Feb 25, 2016 at 12:41 PM, Kenneth Padgett via Outages <
outages@outages.org> wrote:

> Got that and PingPlotter running now, looks pretty good at the moment.
> It's highly intermittent with times of serious latency, 2000ms+ when it's
> bad, that's when we're getting the calls from users. Been that way since
> this AM.
>
> KP
>
> On Thu, Feb 25, 2016 at 2:20 PM, Mark Wiater via Outages <
> outages@outages.org> wrote:
>
>> On 2/25/2016 2:14 PM, Kenneth Padgett via Outages wrote:
>> > Yeah, I can run something similar. No *nix boxes at these sites.
>>
>> winmtr? (http://winmtr.net/)
>>
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast internet in Northern VA

[John Neiberger via Outages]

Thanks. Can you run MTR, as well? I've run thousands of ping tests to
8.8.8.8 from cr02.ashburn.va.ibone with no loss and an RTT averaging around
10ms. I'd like to what MTR looks like after it's been running for a bit.

Thanks again,
John

On Thu, Feb 25, 2016 at 9:37 AM, Kenneth Padgett 
wrote:

> It's sporadic, but here's an example:
>
> Tracing route to google-public-dns-a.google.com [8.8.8.8]
> over a maximum of 30 hops:
>
>   1 2 ms 1 ms 1 ms  [custIP]-static.hfc.comcastbusiness.net
>   263 ms23 ms29 ms  96.120.106.81
>   318 ms15 ms17 ms
> xe-1-3-0-32767-sur01.catharpin.va.bad.comcast.net [68.85.147.5]
>   424 ms22 ms22 ms  ae-40-0-ar01.whitemarsh.md.bad.comcast.net
> [68.85.133.53]
>   563 ms18 ms36 ms
> ae-0-0-ar01.capitolhghts.md.bad.comcast.net [68.85.67.201]
>   671 ms55 ms43 ms
> hu-0-1-0-3-cr02.ashburn.va.ibone.comcast.net [68.86.90.229]
>   7   394 ms   445 ms *
> he-0-14-0-1-pe04.ashburn.va.ibone.comcast.net [68.86.86.42]
>   874 ms72 ms65 ms  as36040-2-c.ashburn.va.ibone.comcast.net
> [75.149.229.86]
>   9   116 ms   143 ms   122 ms  209.85.242.140
>  1023 ms25 ms26 ms  209.85.143.210
>  11 *  432 ms26 ms  216.239.48.160
>  1228 ms26 ms26 ms  216.239.48.235
>  13 *** Request timed out.
>  1433 ms29 ms25 ms  google-public-dns-a.google.com [8.8.8.8]
>
> Trace complete.
>
>
>
> Pinging 8.8.8.8 with 32 bytes of data:
> Reply from 8.8.8.8: bytes=32 time=27ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=151ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=27ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=29ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=27ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=26ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=545ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=30ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=42ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=26ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=24ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=25ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=203ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=386ms TTL=42
> Request timed out.
> Reply from 8.8.8.8: bytes=32 time=25ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=24ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=171ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=25ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=28ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=26ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=27ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=29ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=33ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=34ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=28ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=125ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=24ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=28ms TTL=42
> Request timed out.
> Reply from 8.8.8.8: bytes=32 time=25ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=29ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=31ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=25ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=23ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=27ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=25ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=25ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=26ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=42ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=25ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=24ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=27ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=26ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=32ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=25ms TTL=42
> Request timed out.
> Reply from 8.8.8.8: bytes=32 time=42ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=26ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=24ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=409ms TTL=42
> Reply from 8.8.8.8: bytes=32 time=31ms TTL=42
>
> On Thu, Feb 25, 2016 at 11:20 AM, John Neiberger 
> wrote:
>
>> Can you provide some traceroutes?
>>
>> Thanks,
>> John
>>
>> On Thu, Feb 25, 2016 at 9:16 AM, Kenneth Padgett via Outages <
>> outages@outages.org> wrote:
>>
>>> Anyone else seeing packet loss and connectivity issues on Comcast in
>>> NoVA? We have several clients on Comcast Business with 1000ms+ pings and
>>> dropped packets.
>>>
>>> ___
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast internet in Northern VA

[John Neiberger via Outages]

Can you provide some traceroutes?

Thanks,
John

On Thu, Feb 25, 2016 at 9:16 AM, Kenneth Padgett via Outages <
outages@outages.org> wrote:

> Anyone else seeing packet loss and connectivity issues on Comcast in NoVA?
> We have several clients on Comcast Business with 1000ms+ pings and dropped
> packets.
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Intermedia/Comcast Outage

[John Neiberger via Outages]

That information is pretty vague. If you hear of any real details, pass
them along and I'll see what I can find out.

Thanks,
John

On Fri, Jan 29, 2016 at 8:26 AM, Tailor Mitchell <
tailor.mitch...@uptimelegal.com> wrote:

> I pulled this straight from our Intermedia dashboard; that’s all the info
> I have.
>
>
>
> *From:* John Neiberger [mailto:jneiber...@gmail.com]
> *Sent:* Friday, 29 January, 2016 09:24
> *To:* Tailor Mitchell 
> *Cc:* outages@outages.org
> *Subject:* Re: [outages] Intermedia/Comcast Outage
>
>
>
> Have any more details? I'm at Comcast and don't see anything about this on
> our network operations distro.
>
>
>
> Thanks,
>
> John
>
>
>
> On Fri, Jan 29, 2016 at 8:19 AM, Tailor Mitchell via Outages <
> outages@outages.org> wrote:
>
> *New Event: *1078 Connectivity issues to Intermedia services via Comcast
> *Started at: *01/29/2016 08:30 AM ET
> *Updated at: *01/29/2016 10:04 AM ET
> *Summary of Issue: *
> Some customers may experience connectivity issues when connecting to
> Intermedia servers due to an incident on Comcast ISP side.
> *Business Impact: *
> Some users on Comcast ISP may observe Outlook, OWA, Cloud Server and HPBX
> connectivity issues.
> *Technical Details*
> Customers whose trace route connections are passing through Comcast ISP
> may experience Outlook, OWA, Cloud Server and HPBX connectivity issues due
> to an outage on Comcast ISP side. Please contact your ISP provider for
> further troubleshooting.
>
>
>
>
>
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outage in Northern CA

[John Neiberger via Outages]

Keep in mind that a single head end serves thousands of customers, so
it's going to be a broad impact. This is only affecting our Fresno
head end.

John

On Wed, Nov 18, 2015 at 3:32 PM, chris <tknch...@gmail.com> wrote:
> Doesnt look that way
>
> http://downdetector.com/status/comcast-xfinity/map/
>
> On Wed, Nov 18, 2015 at 5:29 PM, John Neiberger via Outages
> <outages@outages.org> wrote:
>>
>> This is limited to customers connected to one particular head end in
>> Fresno. Everyone else should be fine.
>>
>> John
>>
>>
>>
>>
>> On Wed, Nov 18, 2015 at 3:24 PM, John Kinsella via Outages
>> <outages@outages.org> wrote:
>> > Looks like a fire took out part of Comcast's infrastructure in Fresno,
>> > CA,
>> > causing connectivity issues to customers in central/northern California.
>> > A
>> > few users I've talked to in the northern half of the state don't seem to
>> > be
>> > having issues, so I guess it's hit-and-miss.
>> >
>> >
>> > http://sanfrancisco.cbslocal.com/2015/11/18/electrical-fire-causes-comcast-outage-across-central-northern-california/
>> >
>> > ___
>> > Outages mailing list
>> > Outages@outages.org
>> > https://puck.nether.net/mailman/listinfo/outages
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast outage in Northern CA

[John Neiberger via Outages]

This is limited to customers connected to one particular head end in
Fresno. Everyone else should be fine.

John




On Wed, Nov 18, 2015 at 3:24 PM, John Kinsella via Outages
 wrote:
> Looks like a fire took out part of Comcast's infrastructure in Fresno, CA,
> causing connectivity issues to customers in central/northern California. A
> few users I've talked to in the northern half of the state don't seem to be
> having issues, so I guess it's hit-and-miss.
>
> http://sanfrancisco.cbslocal.com/2015/11/18/electrical-fire-causes-comcast-outage-across-central-northern-california/
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] comcast chicago blackholing

[John Neiberger via Outages]

Can you give me some more details please? I'll take a look.

Thanks,
John

On Tue, Nov 17, 2015 at 10:00 AM, Jared Mauch via Outages
 wrote:
>
> Anyone know what is going on here?  It seems to have started
> around 1600 utc and is still ongoing.
>
> - Jared
>
> --
> Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
> clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] comcast chicago blackholing

[John Neiberger via Outages]

Thanks, all. Our side should be resolved now. Had a combination of a
linecard fabric failure and a partial RSP failure. We're back in
working shape and BGP from our core to that edge router is back up.
Should be safe for peers to turn sessions back up.

John

On Tue, Nov 17, 2015 at 10:15 AM, Jared Mauch  wrote:
> This is both an issue over V6 and V4, eg:
>
> Jareds-Mac-mini:~% traceroute puck
> traceroute to puck.nether.net (204.42.254.5), 64 hops max, 52 byte packets
>  1  10.0.0.1 (10.0.0.1)  0.603 ms  0.418 ms  0.403 ms
>  2  173-167-0-110-michigan.hfc.comcastbusiness.net (173.167.0.110)  2.044 ms  
> 2.019 ms  2.179 ms
>  3  96.120.40.141 (96.120.40.141)  17.399 ms  12.408 ms  9.639 ms
>  4  68.85.85.37 (68.85.85.37)  14.371 ms  8.378 ms  15.355 ms
>  5  te-0-8-0-6-ar02.pontiac.mi.michigan.comcast.net (68.87.191.153)  14.025 ms
> te-0-8-0-8-ar02.pontiac.mi.michigan.comcast.net (69.139.255.69)  14.749 
> ms  18.706 ms
>  6  be-33668-cr02.350ecermak.il.ibone.comcast.net (68.86.90.45)  21.955 ms  
> 20.456 ms  19.259 ms
>  7  he-0-16-0-0-pe03.350ecermak.il.ibone.comcast.net (68.86.88.146)  18.333 
> ms  17.245 ms  16.952 ms
> ^C
>
> Jareds-Mac-mini:~% traceroute6 puck
> traceroute6 to puck.nether.net (2001:418:3f4::5) from 
> 2601:401:3:6a00:ad42:ce75:8a0:c370, 64 hops max, 12 byte packets
>  1  2601:401:3:6a00:20f:f7ff:feab:1ec0  1.135 ms  0.965 ms  0.948 ms
>  2  2601:401:3:6a00:4694:fcff:fecd:64bc  3.710 ms  3.544 ms  21.221 ms
>  3  2001:558:6007:24::1  30.051 ms  10.909 ms  10.868 ms
>  4  te8-2-ur02.wannarbor.mi.michigan.comcast.net  10.664 ms  10.451 ms  
> 10.570 ms
>  5  xe-11-1-0-0-sur01.rochestrhlls.mi.michigan.comcast.net  14.826 ms
> te-0-8-0-5-ar02.taylor.mi.michigan.comcast.net  13.890 ms  13.179 ms
>  6  te-0-7-0-13-ar01.pontiac.mi.michigan.comcast.net  18.917 ms  15.503 ms
> te-0-8-0-11-ar01.pontiac.mi.michigan.comcast.net  13.686 ms
>  7  be-33668-cr01.350ecermak.il.ibone.comcast.net  22.316 ms  22.014 ms  
> 28.086 ms
>  8  hu-0-16-0-1-pe03.350ecermak.il.ibone.comcast.net  20.933 ms  20.740 ms  
> 20.831 ms
> ^C
>
> - Jared
>
>
> On Tue, Nov 17, 2015 at 10:02:58AM -0700, John Neiberger wrote:
>> Can you give me some more details please? I'll take a look.
>>
>> Thanks,
>> John
>>
>> On Tue, Nov 17, 2015 at 10:00 AM, Jared Mauch via Outages
>>  wrote:
>> >
>> > Anyone know what is going on here?  It seems to have started
>> > around 1600 utc and is still ongoing.
>> >
>> > - Jared
>> >
>> > --
>> > Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
>> > clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
>> > ___
>> > Outages mailing list
>> > Outages@outages.org
>> > https://puck.nether.net/mailman/listinfo/outages
>
> --
> Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
> clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] comcast chicago blackholing

[John Neiberger via Outages]

May have spoken too soon. Something is still apparently sick, so we've
taken the router back out of routing for now.

John

On Tue, Nov 17, 2015 at 12:17 PM, John Neiberger  wrote:
> And by "partial" I mean that it was very sick, but not sick enough to
> trigger a switchover to the backup RSP. That had to be done manually.
>
> John
>
> On Tue, Nov 17, 2015 at 12:15 PM, John Neiberger  wrote:
>> Thanks, all. Our side should be resolved now. Had a combination of a
>> linecard fabric failure and a partial RSP failure. We're back in
>> working shape and BGP from our core to that edge router is back up.
>> Should be safe for peers to turn sessions back up.
>>
>> John
>>
>> On Tue, Nov 17, 2015 at 10:15 AM, Jared Mauch  wrote:
>>> This is both an issue over V6 and V4, eg:
>>>
>>> Jareds-Mac-mini:~% traceroute puck
>>> traceroute to puck.nether.net (204.42.254.5), 64 hops max, 52 byte packets
>>>  1  10.0.0.1 (10.0.0.1)  0.603 ms  0.418 ms  0.403 ms
>>>  2  173-167-0-110-michigan.hfc.comcastbusiness.net (173.167.0.110)  2.044 
>>> ms  2.019 ms  2.179 ms
>>>  3  96.120.40.141 (96.120.40.141)  17.399 ms  12.408 ms  9.639 ms
>>>  4  68.85.85.37 (68.85.85.37)  14.371 ms  8.378 ms  15.355 ms
>>>  5  te-0-8-0-6-ar02.pontiac.mi.michigan.comcast.net (68.87.191.153)  14.025 
>>> ms
>>> te-0-8-0-8-ar02.pontiac.mi.michigan.comcast.net (69.139.255.69)  14.749 
>>> ms  18.706 ms
>>>  6  be-33668-cr02.350ecermak.il.ibone.comcast.net (68.86.90.45)  21.955 ms  
>>> 20.456 ms  19.259 ms
>>>  7  he-0-16-0-0-pe03.350ecermak.il.ibone.comcast.net (68.86.88.146)  18.333 
>>> ms  17.245 ms  16.952 ms
>>> ^C
>>>
>>> Jareds-Mac-mini:~% traceroute6 puck
>>> traceroute6 to puck.nether.net (2001:418:3f4::5) from 
>>> 2601:401:3:6a00:ad42:ce75:8a0:c370, 64 hops max, 12 byte packets
>>>  1  2601:401:3:6a00:20f:f7ff:feab:1ec0  1.135 ms  0.965 ms  0.948 ms
>>>  2  2601:401:3:6a00:4694:fcff:fecd:64bc  3.710 ms  3.544 ms  21.221 ms
>>>  3  2001:558:6007:24::1  30.051 ms  10.909 ms  10.868 ms
>>>  4  te8-2-ur02.wannarbor.mi.michigan.comcast.net  10.664 ms  10.451 ms  
>>> 10.570 ms
>>>  5  xe-11-1-0-0-sur01.rochestrhlls.mi.michigan.comcast.net  14.826 ms
>>> te-0-8-0-5-ar02.taylor.mi.michigan.comcast.net  13.890 ms  13.179 ms
>>>  6  te-0-7-0-13-ar01.pontiac.mi.michigan.comcast.net  18.917 ms  15.503 ms
>>> te-0-8-0-11-ar01.pontiac.mi.michigan.comcast.net  13.686 ms
>>>  7  be-33668-cr01.350ecermak.il.ibone.comcast.net  22.316 ms  22.014 ms  
>>> 28.086 ms
>>>  8  hu-0-16-0-1-pe03.350ecermak.il.ibone.comcast.net  20.933 ms  20.740 ms  
>>> 20.831 ms
>>> ^C
>>>
>>> - Jared
>>>
>>>
>>> On Tue, Nov 17, 2015 at 10:02:58AM -0700, John Neiberger wrote:
 Can you give me some more details please? I'll take a look.

 Thanks,
 John

 On Tue, Nov 17, 2015 at 10:00 AM, Jared Mauch via Outages
  wrote:
 >
 > Anyone know what is going on here?  It seems to have started
 > around 1600 utc and is still ongoing.
 >
 > - Jared
 >
 > --
 > Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
 > clue++;  | http://puck.nether.net/~jared/  My statements are only 
 > mine.
 > ___
 > Outages mailing list
 > Outages@outages.org
 > https://puck.nether.net/mailman/listinfo/outages
>>>
>>> --
>>> Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
>>> clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] comcast chicago blackholing

[John Neiberger via Outages]

Okay, we should be good to go. For real this time. Had some lingering
fabric-related issues to clear up. Everything looks good now. Let me
know if anyone sees any more problems through that router.

Thanks,
John

On Tue, Nov 17, 2015 at 12:37 PM, John Neiberger  wrote:
> May have spoken too soon. Something is still apparently sick, so we've
> taken the router back out of routing for now.
>
> John
>
> On Tue, Nov 17, 2015 at 12:17 PM, John Neiberger  wrote:
>> And by "partial" I mean that it was very sick, but not sick enough to
>> trigger a switchover to the backup RSP. That had to be done manually.
>>
>> John
>>
>> On Tue, Nov 17, 2015 at 12:15 PM, John Neiberger  
>> wrote:
>>> Thanks, all. Our side should be resolved now. Had a combination of a
>>> linecard fabric failure and a partial RSP failure. We're back in
>>> working shape and BGP from our core to that edge router is back up.
>>> Should be safe for peers to turn sessions back up.
>>>
>>> John
>>>
>>> On Tue, Nov 17, 2015 at 10:15 AM, Jared Mauch  wrote:
 This is both an issue over V6 and V4, eg:

 Jareds-Mac-mini:~% traceroute puck
 traceroute to puck.nether.net (204.42.254.5), 64 hops max, 52 byte packets
  1  10.0.0.1 (10.0.0.1)  0.603 ms  0.418 ms  0.403 ms
  2  173-167-0-110-michigan.hfc.comcastbusiness.net (173.167.0.110)  2.044 
 ms  2.019 ms  2.179 ms
  3  96.120.40.141 (96.120.40.141)  17.399 ms  12.408 ms  9.639 ms
  4  68.85.85.37 (68.85.85.37)  14.371 ms  8.378 ms  15.355 ms
  5  te-0-8-0-6-ar02.pontiac.mi.michigan.comcast.net (68.87.191.153)  
 14.025 ms
 te-0-8-0-8-ar02.pontiac.mi.michigan.comcast.net (69.139.255.69)  
 14.749 ms  18.706 ms
  6  be-33668-cr02.350ecermak.il.ibone.comcast.net (68.86.90.45)  21.955 ms 
  20.456 ms  19.259 ms
  7  he-0-16-0-0-pe03.350ecermak.il.ibone.comcast.net (68.86.88.146)  
 18.333 ms  17.245 ms  16.952 ms
 ^C

 Jareds-Mac-mini:~% traceroute6 puck
 traceroute6 to puck.nether.net (2001:418:3f4::5) from 
 2601:401:3:6a00:ad42:ce75:8a0:c370, 64 hops max, 12 byte packets
  1  2601:401:3:6a00:20f:f7ff:feab:1ec0  1.135 ms  0.965 ms  0.948 ms
  2  2601:401:3:6a00:4694:fcff:fecd:64bc  3.710 ms  3.544 ms  21.221 ms
  3  2001:558:6007:24::1  30.051 ms  10.909 ms  10.868 ms
  4  te8-2-ur02.wannarbor.mi.michigan.comcast.net  10.664 ms  10.451 ms  
 10.570 ms
  5  xe-11-1-0-0-sur01.rochestrhlls.mi.michigan.comcast.net  14.826 ms
 te-0-8-0-5-ar02.taylor.mi.michigan.comcast.net  13.890 ms  13.179 ms
  6  te-0-7-0-13-ar01.pontiac.mi.michigan.comcast.net  18.917 ms  15.503 ms
 te-0-8-0-11-ar01.pontiac.mi.michigan.comcast.net  13.686 ms
  7  be-33668-cr01.350ecermak.il.ibone.comcast.net  22.316 ms  22.014 ms  
 28.086 ms
  8  hu-0-16-0-1-pe03.350ecermak.il.ibone.comcast.net  20.933 ms  20.740 ms 
  20.831 ms
 ^C

 - Jared


 On Tue, Nov 17, 2015 at 10:02:58AM -0700, John Neiberger wrote:
> Can you give me some more details please? I'll take a look.
>
> Thanks,
> John
>
> On Tue, Nov 17, 2015 at 10:00 AM, Jared Mauch via Outages
>  wrote:
> >
> > Anyone know what is going on here?  It seems to have started
> > around 1600 utc and is still ongoing.
> >
> > - Jared
> >
> > --
> > Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
> > clue++;  | http://puck.nether.net/~jared/  My statements are only 
> > mine.
> > ___
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages

 --
 Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
 clue++;  | http://puck.nether.net/~jared/  My statements are only mine.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] YouTube is down

[John Neiberger via Outages]

Works fine for me from Denver on Comcast. The link you posted doesn't
show anything out of the ordinary.

On Sat, Nov 7, 2015 at 10:50 AM, Frank Zhou via Outages
 wrote:
>
> https://downdetector.com/status/youtube
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast Atlanta

[John Neiberger via Outages]

​I'm not aware of any major outages in the Atlanta area and I don't see
anything about it on our NOC team's email distro. Do you have any details?

Thanks,
John​

On Wed, Aug 5, 2015 at 1:52 PM, Stagg Shelton via Outages 
outages@outages.org wrote:

 Anyone know how large the Comcast outage in Atlanta is?

 Stagg Shelton
 Digium Cloud Services

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast Atlanta

[John Neiberger via Outages]

Not much happening on Down Detector in that region, either.

https://downdetector.com/status/comcast-xfinity/map/

If you have more details, I can check into it further. At the moment, I
don't see anything unusual.

Thanks,
John

On Wed, Aug 5, 2015 at 1:55 PM, John Neiberger jneiber...@gmail.com wrote:

 ​I'm not aware of any major outages in the Atlanta area and I don't see
 anything about it on our NOC team's email distro. Do you have any details?

 Thanks,
 John​

 On Wed, Aug 5, 2015 at 1:52 PM, Stagg Shelton via Outages 
 outages@outages.org wrote:

 Anyone know how large the Comcast outage in Atlanta is?

 Stagg Shelton
 Digium Cloud Services

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages



___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast DNS broken?

[John Neiberger via Outages]

It's localized to the west coast. We're investigating now.

Thanks,
John

On Mon, Jun 1, 2015 at 9:10 PM, Aubrey Wells via Outages 
outages@outages.org wrote:

 Its working fine here in Atlanta, but 75.75.7[56].7[56] are anycasted so
 the cluster you are hitting from SFO is almost certainly not the same
 cluster I am hitting from ATL.


 -
 Aubrey Wells
 Manager, Network Operations
 Digium Cloud Services
 Main: 888.305.3850
 Support: 877.344.4861 or http://www.digium.com/en/support
 http://www.digium.com/en/support?elq=65516445a5964d3597e25eaf566bc2cfelqCampaignId=

 On Mon, Jun 1, 2015 at 11:06 PM, Sajal Kayan via Outages 
 outages@outages.org wrote:

 Hi,

 Comcast DNS servers (75.75.75.75, 75.75.76.76) seem to be down. Web pages
 were suck at dns resolution phase, switching to Google DNS solved the
 problem immediately.

 Tested from San Francisco.

 sajal@sajal-lappy:~$ mtr --report 75.75.75.75
 Start: Mon Jun  1 20:03:02 2015
 HOST: sajal-lappy Loss%   Snt   Last   Avg  Best  Wrst
 StDev
   1.|-- 10.0.1.1   0.0%101.2   3.1   1.0   5.9
 1.2
   2.|-- 50.174.142.1   0.0%10   15.9  20.7  14.6  49.7
  10.4
   3.|-- te-0-7-0-5-sur03.sffolsom  0.0%10   16.9  24.1  16.4  54.6
  11.6
   4.|-- be-209-ar01.santaclara.ca  0.0%10   18.2  33.4  17.0 146.1
  39.9
   5.|-- he-0-14-0-0-ar01.sfsutro.  0.0%10   22.4  26.9  20.0  42.9
 6.8
   6.|-- te-9-4-ur01-d.sanjose.ca.  0.0%10   22.9  23.1  20.6  28.4
 2.0
   7.|-- te-8-2-ur01-d.sanjose.ca.  0.0%10   21.5  23.4  20.6  29.2
 3.1
   8.|-- cdns01.comcast.net 0.0%10   32.2  47.3  20.5  89.7
  29.0
 sajal@sajal-lappy:~$ dig google.com @75.75.75.75

 ;  DiG 9.9.5-3ubuntu0.2-Ubuntu  google.com @75.75.75.75
 ;; global options: +cmd
 ;; connection timed out; no servers could be reached
 sajal@sajal-lappy:~$


 https://pulse.turbobytes.com/results/556d1b32ecbe4023ebab/ - Using
 our tool we see timeout from Seattle and Mountain View. Is this a national
 level outage or something?

 -Sajal

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages



 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast San Jose,CA.

[John Neiberger via Outages]

I'm not aware of any widespread issues in the area. I checked our internal
ops lists and see no mention of anything happening in San Jose right now.
What symptoms are you experiencing?

John

On Fri, May 15, 2015 at 7:21 PM, Michael Dilworth via Outages 
outages@outages.org wrote:

 Anyone have info on when I may return?


 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast LA issue?

[John Neiberger via Outages]

Those of you having issues, can you check to see if they're still occurring?

Thanks,
John

On Thu, Feb 19, 2015 at 4:25 PM, Jeremy Chadwick via Outages 
outages@outages.org wrote:

 Here's another destination showing the same problem (I can't even load
 http://www.dslreports.com/ at present -- handshake works, but either
 client-server GET request gets lost, or payload response gets lost,
 followed by TCP retrans, eventually RST from the server, rinse lather
 repeat), with things again looking centralised around LA.

 Hopefully having multiple test endpoints will come in handy.

 Source IP: 69.181.142.213
 Destination IP: 64.91.255.98


 Packets   Pings
  Host   Loss%
  Snt   Rcv  Last   Avg  Best  Wrst
  1. gw.home.lan (192.168.1.1)0.0%
  171   171   0.2   0.2   0.2   0.4
  2. 69.181.136.1 0.0%
  171   171  12.7   9.6   7.7  19.2
  3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
  0.0%   171   171   9.4   9.2   8.0  17.1
  4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134
 0.0%   171   171  12.1  12.2   9.5  18.4
  5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93)
 98.8%   171 2 50456 50095 49735 50456
  6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)
  2.9%   171   166  14.1  13.5  10.9  22.7
  7. he-2-12-0-0-cr01.seattle.wa.ibone.comcast.net (68.86.86.229)
  1.8%   171   168  29.2  30.2  27.4  36.3
  8. he-0-15-0-0-cr01.denver.co.ibone.comcast.net (68.86.86.213)
 9.9%   171   154  50.1  47.6  44.9  53.1
  9. he-5-13-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.85.245)
 4.1%   171   163  72.9  72.3  68.6  84.7
 10. hu-0-10-0-1.pe03.350ecermak.il.ibone.comcast.net (68.86.86.6)
 5.3%   170   161  69.8  69.4  68.1  84.7
 11. 50.242.150.126   7.1%
  170   158 106.0  89.7  67.4 400.5
 50.242.150.130
 12. lw-dc3-core1-te8-16.rtr.liquidweb.com (209.59.157.244)
  9.4%   170   154  75.7  80.9  75.2 242.7
 13. lw-dc3-dist15-po5.rtr.liquidweb.com (69.167.128.241)
  2.4%   170   166  75.0  75.9  74.7  83.6
 14. www.dslreports.com (64.91.255.98)
 5.3%   170   161  74.2  74.3  73.2  82.5

 --
 | Jeremy Chadwick   j...@koitsu.org |
 | UNIX Systems Administratorhttp://jdc.koitsu.org/ |
 | Making life hard for others since 1977. PGP 4BD6C0CB |

 On Thu, Feb 19, 2015 at 03:07:04PM -0800, Jeremy Chadwick wrote:
  Something weird is going on with Comcast in Los Angeles: small packets
  appear to be getting discarded with a loss rate of ~6-10% while larger
  packets (I've tried 1000 and 1200) seem to make it through without any
  loss.  Things like SSH and IMAP+SSL are impacted consistently
  (particularly during key exchange), while HTTP does not appear to be, so
  the issue appears to be affecting both TCP and ICMP (have not tested
  UDP).
 
  First sign of this issue, according to my logs, is at 11:51 PT (UTC-8),
  and has gradually gotten worse over the past few hours.
 
  Looks like the issue is with the forward path (Source-Destination
  listed below), as the return path only shows loss at the final
  destination.  (I currently cannot provide a return-path mtr because the
  only way I can access my box is through VNC, and I don't like doing
  things like screenshots of a VNC window, so you'll just have to trust
  me)
 
  Nothing anomalous shows up when using L3's looking glass to ping things,
  and the only Comcast LG I know of is in NYC.
 
  I most likely won't see Email responses from anyone (directly or to the
  list) until this problem is addressed, since my mail is hosted on shown
  box and cannot currently be fetched due to the problem.  Frustrating.
 
  Source IP: 69.181.142.213
  Destination IP: 174.136.108.146
 
  ICMP 56 bytes:
 
 
  Packets   Pings
   Host
  Loss%   Snt   Rcv  Last   Avg  Best  Wrst
   1. gw.home.lan (192.168.1.1)
 0.0%   185   185   0.2   0.2   0.2   0.3
   2. 69.181.136.1
  0.0%   185   185   8.9   9.9   7.0  21.1
   3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
  0.0%   185   185   8.7   9.9   7.9  17.0
   4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134
 0.0%   185   185  10.7  12.9   9.7  20.7
   5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93)
 98.4%   184 3 49496 49006 47906 49615
   6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)
  5.4%   184   174  11.9  14.1  10.9  23.5
   7. 50.242.148.34
 6.5%   184   172  11.7  13.4  10.4  69.9
   8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
  66.7%   18461  44.9  20.6  18.0  44.9
   9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
  59.0%   18475  18.7  19.8  18.0  26.0
  10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)
  2.2%   184   180  20.8  63.1  18.2 293.3
  11. 

Re: [outages] Comcast LA issue?

[John Neiberger via Outages]

Yep. A 100G link had lost its mind. Sounds like all is well now. Thanks for
the info!

John

On Thu, Feb 19, 2015 at 5:41 PM, Jeremy Chadwick j...@koitsu.org wrote:

 Looks resolved as of about 16:23 PT (UTC-8) -- everything appears
 reliable now (all my mail just came flooding in ;-) ).

 Looks like there was a link flap or some reconvergence on the forward
 path that happened at about 16:21, followed by recovery.  (I saw nothing
 on the return path).  Look about right?

 === Thu Feb 19 16:21:00 PST 2015  (1424391660)
 Start: Thu Feb 19 16:21:00 2015
 HOST: icarus.home.lan
 Loss%   Snt   Rcv  Last   Avg  Best  Wrst
   1.|-- gw.home.lan (192.168.1.1)
  0.0%3030   0.2   0.2   0.2   0.3
   2.|-- 69.181.136.1
 0.0%3030   9.5   8.8   7.8  15.4
   3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
 0.0%3030   9.4   8.9   8.2  15.7
   4.|-- te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134)
 0.0%3030  11.9  11.8   9.5  13.8
   5.|-- ???
 100.030 0   0.0   0.0   0.0   0.0
   6.|-- he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)
  16.7%3025  12.5  13.2  11.2  16.2
   7.|-- 50.242.148.34
 23.3%3023  11.6  13.8  10.9  47.6
   8.|-- ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
 16.7%3025  18.7  19.0  18.3  21.7
   9.|-- ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
 50.0%3015  18.6  19.7  18.2  26.2
  10.|-- ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)
  16.7%3025 370.2  87.7  25.9 370.2
  11.|-- GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)
 10.0%3027 133.6  26.8  19.1 133.6
  12.|-- mambo.koitsu.org (174.136.108.146)
  13.3%3026  19.1  19.4  18.4  26.9
 === END

 === Thu Feb 19 16:22:00 PST 2015  (1424391720)
 Start: Thu Feb 19 16:22:00 2015
 HOST: icarus.home.lan
 Loss%   Snt   Rcv  Last   Avg  Best  Wrst
   1.|-- gw.home.lan (192.168.1.1)
  0.0%3030   0.2   0.2   0.2   0.3
   2.|-- 69.181.136.1
 0.0%3030   9.1   9.0   8.0  13.6
   3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
 0.0%3030   9.0   9.2   8.1  15.9
   4.|-- te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134)
 0.0%3030  11.7  12.6   9.8  21.6
   5.|-- ???
 100.030 0   0.0   0.0   0.0   0.0
 === END

 === Thu Feb 19 16:23:00 PST 2015  (1424391780)
 Start: Thu Feb 19 16:23:00 2015
 HOST: icarus.home.lan
 Loss%   Snt   Rcv  Last   Avg  Best  Wrst
   1.|-- gw.home.lan (192.168.1.1)
  0.0%3030   0.2   0.2   0.2   0.3
   2.|-- 69.181.136.1
 0.0%3030   9.0   9.0   7.7  15.0
   3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
 0.0%3030   9.2   8.9   8.2  10.1
   4.|-- te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134)
 0.0%3030  12.5  12.4  10.0  19.2
   5.|-- ???
 100.030 0   0.0   0.0   0.0   0.0
   6.|-- he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)
 0.0%3030  14.6  14.0  11.1  23.2
   7.|-- 50.242.148.34
  0.0%3030  12.2  11.7  10.7  19.0
   8.|-- ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
 16.7%3025  19.5  24.6  18.2  53.5
   9.|-- ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)
 46.7%3016  19.2  18.8  18.2  19.9
  10.|-- ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)
 0.0%3030  76.1  53.7  20.2 109.3
  11.|-- GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)
  0.0%3030  22.2  21.4  18.9  32.7
  12.|-- mambo.koitsu.org (174.136.108.146)
 0.0%3030  18.6  19.3  18.5  21.5
 === END

 --
 | Jeremy Chadwick   j...@koitsu.org |
 | UNIX Systems Administratorhttp://jdc.koitsu.org/ |
 | Making life hard for others since 1977. PGP 4BD6C0CB |

 On Thu, Feb 19, 2015 at 05:24:08PM -0700, John Neiberger wrote:
  Those of you having issues, can you check to see if they're still
 occurring?
 
  Thanks,
  John
 
  On Thu, Feb 19, 2015 at 4:25 PM, Jeremy Chadwick via Outages 
  outages@outages.org wrote:
 
   Here's another destination showing the same problem (I can't even load
   http://www.dslreports.com/ at present -- handshake works, but either
   client-server GET request gets lost, or payload response gets lost,
   followed by TCP retrans, eventually RST from the server, rinse lather
   repeat), with things again looking centralised around LA.
  
   Hopefully having multiple test endpoints will come in handy.
  
   Source IP: 69.181.142.213
   Destination IP: 64.91.255.98
  
  
   Packets   Pings
Host
  Loss%
Snt   Rcv  Last   Avg  Best  Wrst
1. gw.home.lan (192.168.1.1)
 0.0%
171   171   0.2   0.2   0.2   0.4
2. 69.181.136.1
  0.0%
171   171  12.7   9.6   7.7  19.2
3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)
0.0%   171   171   9.4   9.2   8.0  17.1
4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134
   0.0%   171   

Re: [outages] Comcast packet loss

[John Neiberger via Outages]

I'm not seeing any loss on our side. This is from the first Comcast router
you hit:

#ping 192.30.252.129 size 1000 count 1000
Wed Oct 22 16:34:46.957 utc
Type escape sequence to abort.
Sending 1000, 1000-byte ICMP Echos to 192.30.252.129, timeout is 2 seconds:
!!
!!
!!
!!
!!
!!
!!
!!
!!
!!
!!
!!
!!
!!

Success rate is 100 percent (1000/1000), round-trip min/avg/max = 32/32/33
ms

From that same router back to the first hop in your trace:

#ping 200.147.29.137 count 500
Wed Oct 22 16:36:59.522 utc
Type escape sequence to abort.
Sending 500, 100-byte ICMP Echos to 200.147.29.137, timeout is 2 seconds:
!!
!!
!!
!!
!!
!!
!!
!!
Success rate is 100 percent (500/500), round-trip min/avg/max = 151/152/218
ms

Long RTT, but it's long distance, so totally what I'd expect. What's
interesting to me is the huge RTT in your trace. You might want to check
with Highwinds Network Group or Universo Online to see if they can
troubleshoot from their network back to you. The path on our side looks
good from initial observations.

John

On Wed, Oct 22, 2014 at 10:19 AM, Everton Marques via Outages 
outages@outages.org wrote:

 High packet loss from Highwinds towards github.com.
 It seems the issue is rooted within Comcast.

 Time is 16:16:44.371 Wed Oct 22 2014 GMT-0

 ping -t 192.30.252.129

 Pinging 192.30.252.129 with 32 bytes of data:
 Reply from 192.30.252.129: bytes=32 time=474ms TTL=44
 Request timed out.
 Reply from 192.30.252.129: bytes=32 time=480ms TTL=44
 Reply from 192.30.252.129: bytes=32 time=483ms TTL=44
 Reply from 192.30.252.129: bytes=32 time=488ms TTL=44
 Request timed out.
 Reply from 192.30.252.129: bytes=32 time=485ms TTL=44
 Reply from 192.30.252.129: bytes=32 time=485ms TTL=44
 Reply from 192.30.252.129: bytes=32 time=481ms TTL=44
 Reply from 192.30.252.129: bytes=32 time=484ms TTL=44
 Reply from 192.30.252.129: bytes=32 time=478ms TTL=44

 Ping statistics for 192.30.252.129:
 Packets: Sent = 11, Received = 9, Lost = 2 (18% loss),
 Approximate round trip times in milli-seconds:
 Minimum = 474ms, Maximum = 488ms, Average = 482ms
 Control-C

 #traceroute 192.30.252.129

 Type escape sequence to abort.
 Tracing the route to 192.30.252.129

   1 200.147.29.137 [AS 7162] 0 msec 0 msec 0 msec
   2 200.147.29.146 [AS 7162] 0 msec 0 msec 0 msec
   3  *  *  *
   4 173.167.57.1 [AS 7922] 120 msec 120 msec 120 msec
   5 68.86.82.249 [AS 7922] 120 msec 124 msec
 68.86.82.253 [AS 7922] 124 msec
   6  *  *  *
   7  *  *  *
   8 68.86.83.86 [AS 7922] 472 msec 472 msec 472 msec
   9  *  *  *
  10  *  *  *

 Everton

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast - Seattle (maybe other areas)

[John Neiberger via Outages]

Michael,

Can you share some traces?

Thanks,
John

On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages 
outages@outages.org wrote:

 Having serious issues getting outside of Comcast's network starting
 around 1250 Pacific time from Seattle area.


 --

 Genius might be described as a supreme capacity for getting its possessors
 into trouble of all kinds.
 -- Samuel Butler
 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Comcast - Seattle (maybe other areas)

[John Neiberger via Outages]

Okay, good to hear! That's very odd. Let me know if it happens again and
you need someone to check into it.

Thanks,
John

On Mon, Oct 20, 2014 at 2:24 PM, Michael Loftis mlof...@wgops.com wrote:

 It seems to have now mysteriously gone away/cleared up.  Not at all
 sure what was going on there.  I'd written it off initially as an
 issue with my laptop (OSX since a while back seems to need a reboot
 every few weeks if you've Parallels installed and use it - something
 inside the network stack goes nuts)

 On Mon, Oct 20, 2014 at 1:12 PM, John Neiberger jneiber...@gmail.com
 wrote:
  Michael,
 
  Can you share some traces?
 
  Thanks,
  John
 
  On Mon, Oct 20, 2014 at 1:57 PM, Michael Loftis via Outages
  outages@outages.org wrote:
 
  Having serious issues getting outside of Comcast's network starting
  around 1250 Pacific time from Seattle area.
 
 
  --
 
  Genius might be described as a supreme capacity for getting its
  possessors
  into trouble of all kinds.
  -- Samuel Butler
  ___
  Outages mailing list
  Outages@outages.org
  https://puck.nether.net/mailman/listinfo/outages
 
 



 --

 Genius might be described as a supreme capacity for getting its possessors
 into trouble of all kinds.
 -- Samuel Butler

___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Belkin [heartbeat.belkin.com]

[John Neiberger via Outages]

I think it was initially suspected to be a firmware upgrade, but I doubt
that's the case now. It's interesting to me that there are two seemingly
unrelated fixes:

1. Put the IP for heartbeat.belkin.com on a loopback
2. Manually add working DNS entries to affected PCs

Reports indicate that either fix works, but why? They seem unrelated to one
another unless the failing heartbeat messages mess up DNS handling on the
router.

John

On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages 
outages@outages.org wrote:

 Where?  I keep reading bad firmware problems but everything suggests
 there is no automatic upgrade and I think we can safely assume resi users
 didn't upgrade their routers over night.



 Josh Luthman
 Office: 937-552-2340
 Direct: 937-552-2343
 1100 Wayne St
 Suite 1337
 Troy, OH 45373

 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages 
 outages@outages.org wrote:

 Funny,

 Just saw something about Belkin routers and a botched firmware upgrade.

 I wonder if this is what they meant.

 That being said, it is working fine from Zito Media in Clanton, AL.

 --John

 On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:

 On one of my netblocks my subscribers can nolonger ping Belkin.  I can
 ping
 heartbeat.belkin.com from my webservers and from subscriber units on a
 different netblock.

 Apparently Belkin routers ping the home base every couple seconds to
 determine
 the correct color of the Internet light on the router (orange or blue
 *sigh*).
 Worse, when it can't ping the Belkin site the router seems to pack up and
 redirect my subscribers to their router configuration page.

 I found this pretty easy to work around.  I simply DNATed the pings
 intended
 for heartbeat.belkin.com to one of my local nameservers and that seems
 to get
 them back online.  It's a silly workaround, but if anyone else is seeing
 all
 the Belkin routers on their network pack up, this could be why.

 -Paul

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages



 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Belkin [heartbeat.belkin.com]

[John Neiberger via Outages]

Has anyone else tried the loopback/heartbeat workaround with success? I've
seen a few reports so far that this has temporarily solved the problem.

John

On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk frnk...@iname.com wrote:

 I can ping heartbeat.belkin.com and the DSL Reports link and other
 connected links are stating it was back up roughly 30 to 45 minutes ago, so
 I think we’re out of the woods.



 Frank



 *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *John
 Neiberger via Outages
 *Sent:* Tuesday, October 07, 2014 11:04 AM
 *To:* Josh Luthman
 *Cc:* outages@outages.org; John Barbieri
 *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]



 I think it was initially suspected to be a firmware upgrade, but I doubt
 that's the case now. It's interesting to me that there are two seemingly
 unrelated fixes:



 1. Put the IP for heartbeat.belkin.com on a loopback

 2. Manually add working DNS entries to affected PCs



 Reports indicate that either fix works, but why? They seem unrelated to
 one another unless the failing heartbeat messages mess up DNS handling on
 the router.



 John



 On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages 
 outages@outages.org wrote:

 Where?  I keep reading bad firmware problems but everything suggests
 there is no automatic upgrade and I think we can safely assume resi users
 didn't upgrade their routers over night.





 Josh Luthman
 Office: 937-552-2340
 Direct: 937-552-2343
 1100 Wayne St
 Suite 1337
 Troy, OH 45373



 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages 
 outages@outages.org wrote:

 Funny,

 Just saw something about Belkin routers and a botched firmware upgrade.

 I wonder if this is what they meant.

 That being said, it is working fine from Zito Media in Clanton, AL.

 --John


 On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:

 On one of my netblocks my subscribers can nolonger ping Belkin.  I can ping
 heartbeat.belkin.com from my webservers and from subscriber units on a
 different netblock.

 Apparently Belkin routers ping the home base every couple seconds to
 determine
 the correct color of the Internet light on the router (orange or blue
 *sigh*).
 Worse, when it can't ping the Belkin site the router seems to pack up and
 redirect my subscribers to their router configuration page.

 I found this pretty easy to work around.  I simply DNATed the pings
 intended
 for heartbeat.belkin.com to one of my local nameservers and that seems to
 get
 them back online.  It's a silly workaround, but if anyone else is seeing
 all
 the Belkin routers on their network pack up, this could be why.

 -Paul

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages




 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages



___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Belkin [heartbeat.belkin.com]

[John Neiberger via Outages]

Given the number of current social media postings, it doesn't appear that
they are back up yet, but I suppose it is hard to say for sure given the
number of people affected. Twitter is all aflutter with new reports of
people still having problems. Yours is the only report that indicated they
were working again. It could just be that it's taking a while for the
devices to correct themselves, or maybe they even need to be restarted or
something like that.

On Tue, Oct 7, 2014 at 11:57 AM, Frank Bulk frnk...@iname.com wrote:

 Why do that if Belkin is up and running again? Or are you suggesting that
 Belkin doesn’t have it all worked out?



 Frank



 *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *John
 Neiberger via Outages
 *Sent:* Tuesday, October 07, 2014 12:42 PM
 *To:* outages@outages.org

 *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]



 Has anyone else tried the loopback/heartbeat workaround with success? I've
 seen a few reports so far that this has temporarily solved the problem.



 John



 On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk frnk...@iname.com wrote:

 I can ping heartbeat.belkin.com and the DSL Reports link and other
 connected links are stating it was back up roughly 30 to 45 minutes ago, so
 I think we’re out of the woods.



 Frank



 *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *John
 Neiberger via Outages
 *Sent:* Tuesday, October 07, 2014 11:04 AM
 *To:* Josh Luthman
 *Cc:* outages@outages.org; John Barbieri
 *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]



 I think it was initially suspected to be a firmware upgrade, but I doubt
 that's the case now. It's interesting to me that there are two seemingly
 unrelated fixes:



 1. Put the IP for heartbeat.belkin.com on a loopback

 2. Manually add working DNS entries to affected PCs



 Reports indicate that either fix works, but why? They seem unrelated to
 one another unless the failing heartbeat messages mess up DNS handling on
 the router.



 John



 On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages 
 outages@outages.org wrote:

 Where?  I keep reading bad firmware problems but everything suggests
 there is no automatic upgrade and I think we can safely assume resi users
 didn't upgrade their routers over night.





 Josh Luthman
 Office: 937-552-2340
 Direct: 937-552-2343
 1100 Wayne St
 Suite 1337
 Troy, OH 45373



 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages 
 outages@outages.org wrote:

 Funny,

 Just saw something about Belkin routers and a botched firmware upgrade.

 I wonder if this is what they meant.

 That being said, it is working fine from Zito Media in Clanton, AL.

 --John


 On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:

 On one of my netblocks my subscribers can nolonger ping Belkin.  I can ping
 heartbeat.belkin.com from my webservers and from subscriber units on a
 different netblock.

 Apparently Belkin routers ping the home base every couple seconds to
 determine
 the correct color of the Internet light on the router (orange or blue
 *sigh*).
 Worse, when it can't ping the Belkin site the router seems to pack up and
 redirect my subscribers to their router configuration page.

 I found this pretty easy to work around.  I simply DNATed the pings
 intended
 for heartbeat.belkin.com to one of my local nameservers and that seems to
 get
 them back online.  It's a silly workaround, but if anyone else is seeing
 all
 the Belkin routers on their network pack up, this could be why.

 -Paul

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages




 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages





___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Belkin [heartbeat.belkin.com]

[John Neiberger via Outages]

I just saw a report from another ISP who says they had a customer whose
VoIP continued to work and they were able to RDP into their computer from
the outside. It appears that only web traffic was broken. I wonder if
parental controls or web filtering are a component in this fiasco.

John

On Tue, Oct 7, 2014 at 12:04 PM, Frank Bulk frnk...@iname.com wrote:

 No, we’re still getting customer reports about the light staying orange. I
 was just hoping they were having other issues.



 I’ve just added 67.20.176.130 as a loopback in our network, too.



 Frank



 *From:* John Neiberger [mailto:jneiber...@gmail.com]
 *Sent:* Tuesday, October 07, 2014 1:01 PM
 *To:* Frank Bulk
 *Cc:* outages@outages.org

 *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]



 Given the number of current social media postings, it doesn't appear that
 they are back up yet, but I suppose it is hard to say for sure given the
 number of people affected. Twitter is all aflutter with new reports of
 people still having problems. Yours is the only report that indicated they
 were working again. It could just be that it's taking a while for the
 devices to correct themselves, or maybe they even need to be restarted or
 something like that.



 On Tue, Oct 7, 2014 at 11:57 AM, Frank Bulk frnk...@iname.com wrote:

 Why do that if Belkin is up and running again? Or are you suggesting that
 Belkin doesn’t have it all worked out?



 Frank



 *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *John
 Neiberger via Outages
 *Sent:* Tuesday, October 07, 2014 12:42 PM
 *To:* outages@outages.org


 *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]



 Has anyone else tried the loopback/heartbeat workaround with success? I've
 seen a few reports so far that this has temporarily solved the problem.



 John



 On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk frnk...@iname.com wrote:

 I can ping heartbeat.belkin.com and the DSL Reports link and other
 connected links are stating it was back up roughly 30 to 45 minutes ago, so
 I think we’re out of the woods.



 Frank



 *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *John
 Neiberger via Outages
 *Sent:* Tuesday, October 07, 2014 11:04 AM
 *To:* Josh Luthman
 *Cc:* outages@outages.org; John Barbieri
 *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]



 I think it was initially suspected to be a firmware upgrade, but I doubt
 that's the case now. It's interesting to me that there are two seemingly
 unrelated fixes:



 1. Put the IP for heartbeat.belkin.com on a loopback

 2. Manually add working DNS entries to affected PCs



 Reports indicate that either fix works, but why? They seem unrelated to
 one another unless the failing heartbeat messages mess up DNS handling on
 the router.



 John



 On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages 
 outages@outages.org wrote:

 Where?  I keep reading bad firmware problems but everything suggests
 there is no automatic upgrade and I think we can safely assume resi users
 didn't upgrade their routers over night.





 Josh Luthman
 Office: 937-552-2340
 Direct: 937-552-2343
 1100 Wayne St
 Suite 1337
 Troy, OH 45373



 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages 
 outages@outages.org wrote:

 Funny,

 Just saw something about Belkin routers and a botched firmware upgrade.

 I wonder if this is what they meant.

 That being said, it is working fine from Zito Media in Clanton, AL.

 --John


 On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:

 On one of my netblocks my subscribers can nolonger ping Belkin.  I can ping
 heartbeat.belkin.com from my webservers and from subscriber units on a
 different netblock.

 Apparently Belkin routers ping the home base every couple seconds to
 determine
 the correct color of the Internet light on the router (orange or blue
 *sigh*).
 Worse, when it can't ping the Belkin site the router seems to pack up and
 redirect my subscribers to their router configuration page.

 I found this pretty easy to work around.  I simply DNATed the pings
 intended
 for heartbeat.belkin.com to one of my local nameservers and that seems to
 get
 them back online.  It's a silly workaround, but if anyone else is seeing
 all
 the Belkin routers on their network pack up, this could be why.

 -Paul

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages




 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages







___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Belkin [heartbeat.belkin.com]

[John Neiberger via Outages]

Frank,

I'd be very interested to hear if that helps. We're considering doing the
same thing since a couple of people have reported good results.

John

On Tue, Oct 7, 2014 at 12:04 PM, Frank Bulk frnk...@iname.com wrote:

 No, we’re still getting customer reports about the light staying orange. I
 was just hoping they were having other issues.



 I’ve just added 67.20.176.130 as a loopback in our network, too.



 Frank



 *From:* John Neiberger [mailto:jneiber...@gmail.com]
 *Sent:* Tuesday, October 07, 2014 1:01 PM
 *To:* Frank Bulk
 *Cc:* outages@outages.org

 *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]



 Given the number of current social media postings, it doesn't appear that
 they are back up yet, but I suppose it is hard to say for sure given the
 number of people affected. Twitter is all aflutter with new reports of
 people still having problems. Yours is the only report that indicated they
 were working again. It could just be that it's taking a while for the
 devices to correct themselves, or maybe they even need to be restarted or
 something like that.



 On Tue, Oct 7, 2014 at 11:57 AM, Frank Bulk frnk...@iname.com wrote:

 Why do that if Belkin is up and running again? Or are you suggesting that
 Belkin doesn’t have it all worked out?



 Frank



 *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *John
 Neiberger via Outages
 *Sent:* Tuesday, October 07, 2014 12:42 PM
 *To:* outages@outages.org


 *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]



 Has anyone else tried the loopback/heartbeat workaround with success? I've
 seen a few reports so far that this has temporarily solved the problem.



 John



 On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk frnk...@iname.com wrote:

 I can ping heartbeat.belkin.com and the DSL Reports link and other
 connected links are stating it was back up roughly 30 to 45 minutes ago, so
 I think we’re out of the woods.



 Frank



 *From:* Outages [mailto:outages-boun...@outages.org] *On Behalf Of *John
 Neiberger via Outages
 *Sent:* Tuesday, October 07, 2014 11:04 AM
 *To:* Josh Luthman
 *Cc:* outages@outages.org; John Barbieri
 *Subject:* Re: [outages] Belkin [heartbeat.belkin.com]



 I think it was initially suspected to be a firmware upgrade, but I doubt
 that's the case now. It's interesting to me that there are two seemingly
 unrelated fixes:



 1. Put the IP for heartbeat.belkin.com on a loopback

 2. Manually add working DNS entries to affected PCs



 Reports indicate that either fix works, but why? They seem unrelated to
 one another unless the failing heartbeat messages mess up DNS handling on
 the router.



 John



 On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages 
 outages@outages.org wrote:

 Where?  I keep reading bad firmware problems but everything suggests
 there is no automatic upgrade and I think we can safely assume resi users
 didn't upgrade their routers over night.





 Josh Luthman
 Office: 937-552-2340
 Direct: 937-552-2343
 1100 Wayne St
 Suite 1337
 Troy, OH 45373



 On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages 
 outages@outages.org wrote:

 Funny,

 Just saw something about Belkin routers and a botched firmware upgrade.

 I wonder if this is what they meant.

 That being said, it is working fine from Zito Media in Clanton, AL.

 --John


 On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:

 On one of my netblocks my subscribers can nolonger ping Belkin.  I can ping
 heartbeat.belkin.com from my webservers and from subscriber units on a
 different netblock.

 Apparently Belkin routers ping the home base every couple seconds to
 determine
 the correct color of the Internet light on the router (orange or blue
 *sigh*).
 Worse, when it can't ping the Belkin site the router seems to pack up and
 redirect my subscribers to their router configuration page.

 I found this pretty easy to work around.  I simply DNATed the pings
 intended
 for heartbeat.belkin.com to one of my local nameservers and that seems to
 get
 them back online.  It's a silly workaround, but if anyone else is seeing
 all
 the Belkin routers on their network pack up, this could be why.

 -Paul

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages




 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages







___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Facebook down

[John Neiberger via Outages]

It was only down for me in Denver on Comcast for a few minutes. It has been
back up for me for around 20 minutes. I'm connecting via v6.


On Wed, Sep 3, 2014 at 1:54 PM, James Cornman via Outages 
outages@outages.org wrote:

 Get back to work :)

 I'm getting the same 'Sorry, something went wrong' error too from NJ, and
 mobile is not working from ATT.

 -James


 On Wed, Sep 3, 2014 at 3:44 PM, Luke Rockwell via Outages 
 outages@outages.org wrote:

 Now getting this

 [image: Inline image 1]




 On Wed, Sep 3, 2014 at 12:43 PM, Luke Rockwell 
 luke.rockw...@alumni.berkeley.edu wrote:

 tracert facebook.com

 Tracing route to facebook.com [173.252.110.27]
 over a maximum of 30 hops:

   11 ms1 ms1 ms

   21 ms1 ms1 ms
   3 1 ms1 ms1 ms

   4 1 ms 1 ms 1 ms  dc-sfo-agg-1--ucb-10ge.cenic.net
 [137.164.50.16]

   5 4 ms 2 ms 3 ms  oak-agg2--sfo-agg1-10g.cenic.net
 [137.164.22.25]

   6 3 ms 3 ms 3 ms  dc-paix-px1--oak-core1-ge.cenic.net
 [137.164.47.
 18]
   7 3 ms 3 ms 3 ms  paix-px1--facebook-ge.cenic.net
 [198.32.251.133]

   8 4 ms 3 ms 4 ms  ae1.bb01.pao1.tfbnw.net [74.119.76.134]
   9 4 ms 4 ms12 ms  ae4.bb01.sjc1.tfbnw.net [74.119.76.137]
  1063 ms63 ms67 ms  be16.bb01.frc3.tfbnw.net [31.13.24.28]
  1167 ms68 ms66 ms  ae87.dr04.frc1.tfbnw.net [173.252.64.53]
  12 *** Request timed out.
  13  ^C



 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages




 --

 *James Cornman*


 *Chief Technology Officer*jcorn...@atlanticmetro.net
 212.792.9950 - ext 101

 *Atlantic Metro Communications, Inc.*

 *4 Century Drive, Parsippany NJ  07054*


 *Data Centers • Managed Hosting • Internet Access • Cloud Computing*
 Follow @atlanticmetro http://www.twitter.com/atlanticmetro on Twitter
 to learn more about daily specials.
 www.atlanticmetro.net

 ___
 Outages mailing list
 Outages@outages.org
 https://puck.nether.net/mailman/listinfo/outages


___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages