Re: Centurylink having a bad morning?

The CL portal loads for me, and I can log in, but it is slower than usual. Not seeing traffic issues on our CL circuits. -mel via cell On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG wrote:  Hello, Woke up this morning to a bunch of reports of issues with connectivity had to shut down

Re: Centurylink having a bad morning?

Started about 5:05am central, started clearing up for me about 7:15am.   My route from ATT in Chicago is still going through NYC to get back to Chicago but at least packet loss isn't 70-100% anymore. I also tried turning down sessions and still was seeing stale announcements on other LGs.

RE: Centurylink having a bad morning?

That site seems to be just for their cloud products, is there one of these for their actual network? From: R. Leigh Hennig Sent: Sunday, August 30, 2020 8:54 AM To: Drew Weaver ; nanog@nanog.org Subject: Re: Centurylink having a bad morning? Global impact with issues reported by Fastly,

Re: Centurylink having a bad morning?

> > They’re not reachable so who knows if they’re even working on it. > Gonna go out on a limb here and assume that a lot of phones were ringing and people are in fact working on whatever it is. :) On Sun, Aug 30, 2020 at 8:34 AM David Hubbard wrote: > Same. Also, as reported on outages list,

RE: Centurylink having a bad morning?

Yikes, they really need to get their reflectors fixed so that we can dis-engage from them. -Original Message- From: NANOG On Behalf Of Ben Russell Sent: Sunday, August 30, 2020 9:14 AM To: 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? One of my legacy L3 peers is

Re: Ipv6 help

In many situations it will make sense to keep the CPE provided by the ISP in a configuration equivalent to a "bridge" (I know is not a bridge, I'm trying to use a single word to describe it), so it runs things like the NAT for IPv4 and the CLAT for IPv6, even DHCP, or RA, etc. It all depends on

Re: Centurylink having a bad morning?

Looks like this is causing problems with Meraki Cloud connectivity since level3 is a transit provider for Meraki. From: NANOG on behalf of Mel Beckman Sent: Sunday, August 30, 2020, 08:47 To: Drew Weaver Cc: nanog@nanog.org Subject: Re: Centurylink having a

Re: Centurylink having a bad morning?

Reporting from Europe, any IP with them in the path is unreachable from various providers. I guess they wanted to try IPv6-only.. :P IPv6 is fine, working fine, IPv4 not at all.. Antonis > On 30 Aug 2020, at 14:58, Tomas Lynch wrote: > > Flapping in Miami, Dallas, Atlanta, Los Angeles,

Re: Centurylink having a bad morning?

I am seeing some odd traffic deflections in the EU via 3356, but nothing in the US so far. Some scattershot oddball reports landing in our NOC, but nothing conclusive. On Sun, Aug 30, 2020 at 8:41 AM Mel Beckman wrote: > The CL portal loads for me, and I can log in, but it is slower than

RE: Centurylink having a bad morning?

One of my legacy L3 peers is only advertising 498174 prefixes. Very strange. Ben Russell  -Original Message- From: NANOG On Behalf Of Drew Weaver Sent: Sunday, August 30, 2020 7:45 AM To: 'Mel Beckman' Cc: 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? Can't

Re: Centurylink having a bad morning?

I just brought one of my sessions back up to attempt to avoid the blackholing, should be full feed, getting all of 850 v4 routes and 106 v6. From: NANOG on behalf of Tomas Lynch Date: Sunday, August 30, 2020 at 9:41 AM To: Drew Weaver Cc: "nanog@nanog.org" Subject: Re: Centurylink having a

Re: Centurylink having a bad morning?

Well, When I tried calling I got a fast busy, so that's nice. On Sun, Aug 30, 2020 at 8:33 AM David Hubbard wrote: > Same. Also, as reported on outages list, what’s even worse is that they > appear to be continuing to propagate advertisements from circuits whose > sessions have been turned

Re: Centurylink having a bad morning?

I have checked all of our circuits in the LA area and are seeing no BGP issues, but we don’t provide transit for CL, only use them for DIA. Traffic flows appear normal, but early Sunday morning is normally the lowest time of the week. -mel beckman > On Aug 30, 2020, at 5:29 AM, Mel Beckman

Re: Centurylink having a bad morning?

Flapping in Miami, Dallas, Atlanta, Los Angeles, Seattle and San Jose. It is also affecting some data centers in Europe too. but haven't seen flaps there, just suboptimal routing. On Sun, Aug 30, 2020 at 8:53 AM Drew Weaver wrote: > Saw the flapping in Cleveland but not in Cincinnatti or

Re: Centurylink having a bad morning?

Now if you call into CL you get a message stating their technicians are working on an ip outage. On Sun, Aug 30, 2020 at 6:56 AM Chase Christian wrote: > Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00 > Pacific: > > Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent

RE: Centurylink having a bad morning?

Yeah, I am still seeing them announce our Ips even though we’ve shut down our sessions with them. From: NANOG On Behalf Of David Hubbard Sent: Sunday, August 30, 2020 8:28 AM To: nanog@nanog.org Subject: Re: Centurylink having a bad morning? Same. Also, as reported on outages list, what’s

Re: Centurylink having a bad morning?

Finally got through on their support line and spoke to level1. The only thing the tech could say was it was an issue with BGP route reflectors and it started about 3am(pacific). They were still trying to isolate the issue. I've tried failing over my circuits and no go, the traffic just dies as L3

RE: Centurylink having a bad morning?

Saw the flapping in Cleveland but not in Cincinnatti or Ashburn… From: Tomas Lynch Sent: Sunday, August 30, 2020 8:45 AM To: Mel Beckman Cc: Drew Weaver ; nanog@nanog.org Subject: Re: Centurylink having a bad morning? BGP sessions randomly flapping or having routing issues in different cities

RE: Centurylink having a bad morning?

I don’t think it is anywhere near back to normal but I can get to 2-3 more sites than I could when I got called in. From: NANOG On Behalf Of Andy Brezinsky Sent: Sunday, August 30, 2020 8:48 AM To: nanog@nanog.org Subject: Re: Centurylink having a bad morning? Started about 5:05am central,

RE: Centurylink having a bad morning?

Something just now changed in this situation and now it seems to have gotten worse. From: Jason Kuehl Sent: Sunday, August 30, 2020 10:00 AM To: Drew Weaver Cc: R. Leigh Hennig ; nanog@nanog.org Subject: Re: Centurylink having a bad morning? People are rebooting ghosting now.

Re: Centurylink having a bad morning?

Same. Also, as reported on outages list, what’s even worse is that they appear to be continuing to propagate advertisements from circuits whose sessions have been turned down. I validated ours still were via a couple looking glass portals. Down Detector shows nearly every major service

Re: Centurylink having a bad morning?

One anomaly I’ve just notice: we can’t reach 209.244.0.1 (ns1.level3.com) from anywhere on the Internet, including our CL connections. Traceroute fails at 4.16.30.129 (lag-101.ear2.LosAngeles1.Level3.net) with 65% packet loss: traceroute to 209.244.0.1 (209.244.0.1), 30 hops max, 56 byte

RE: Centurylink having a bad morning? [EXTERNAL]

Their route-servers just can’t keep up. They are not getting our community-string announcements at all From: NANOG On Behalf Of Drew Weaver Sent: Sunday, August 30, 2020 8:38 AM To: 'David Hubbard' ; 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? [EXTERNAL] Yeah, I am

RE: Centurylink having a bad morning?

Can't get to pretty much anything on the Internet from my Tmobile device... not really sure whats going on =) -Original Message- From: Mel Beckman Sent: Sunday, August 30, 2020 8:43 AM To: Drew Weaver Cc: nanog@nanog.org Subject: Re: Centurylink having a bad morning? One anomaly I’ve

Re: Centurylink having a bad morning?

We've been on hold for more than an hour trying to get an update. We see the same behavior where they continue to announce our blocks despite all the interfaces to them being hard down. Scott Helms On Sun, Aug 30, 2020 at 8:58 AM Jason Kuehl wrote: > > Well, When I tried calling I got a fast

Re: Centurylink having a bad morning?

People are rebooting ghosting now. [image: image.png] https://twitter.com/ir_kujoe/status/1300066569645707265 Seeing other reports of this too. On Sun, Aug 30, 2020 at 9:45 AM Drew Weaver wrote: > That site seems to be just for their cloud products, is there one of these > for their actual

Re: Centurylink having a bad morning?

Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00 Pacific: Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor 4.35.X.Y (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Aug 30 03:05:13 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old state

Re: Centurylink having a bad morning?

I'm over in MA in a CL building, it's very much still broken. I shut down the interfaces to CL and now just using Comcast. On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky wrote: > Started about 5:05am central, started clearing up for me about 7:15am. > My route from ATT in Chicago is still going

Re: Centurylink having a bad morning?

Once upon a time, Robert Blayzor said: > Just to confirm we're seeing this on AS3356 and not AS209, correct? Correct - we had problems with our 3356 connection but not our 209 connection. -- Chris Adams

Re: Centurylink having a bad morning?

On 30/08/2020 20:08, Baldur Norddahl wrote: https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/ Sounds like Flowspec possibly blocking tcp/179 might be the cause. But that is Cloudflare speculation.

Re: Centurylink having a bad morning?

On Sun, 30 Aug 2020 at 20:00, Baldur Norddahl wrote: > Not really the point. BGP is designed such that if I take down the link, the > prefixes MUST be withdrawn within reasonable time. The self healing aspect of > the internet entirely depends on this. Clearly they have some kind of system >

Centurylink having a bad morning?

Hello, Woke up this morning to a bunch of reports of issues with connectivity had to shut down some Level3/CTL connections to get it to return to normal. As of right now their support portal won't load: https://www.centurylink.com/business/login/ Just wondering what others are seeing.

Re: Centurylink having a bad morning?

Its dead JIM. I also can't get in starting around 7 est On Sun, Aug 30, 2020 at 8:19 AM Drew Weaver via NANOG wrote: > Hello, > > > > Woke up this morning to a bunch of reports of issues with connectivity had > to shut down some Level3/CTL connections to get it to return to normal. > > > > As

Re: Centurylink having a bad morning?

BGP sessions randomly flapping or having routing issues in different cities since ~5AM EST On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman wrote: > The CL portal loads for me, and I can log in, but it is slower than usual. > Not seeing traffic issues on our CL circuits. > > -mel via cell > > On Aug

Re: Centurylink having a bad morning?

On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote: > Woke up this morning to a bunch of reports of issues with connectivity > had to shut down some Level3/CTL connections to get it to return to normal. Just to confirm we're seeing this on AS3356 and not AS209, correct? We have links to both and

Re: Centurylink having a bad morning?

How is that acceptable behaviour? I shall remember never to make a contract with these guys until they can prove that they won't advertise my prefixes after I pull them. Under any circumstances. søn. 30. aug. 2020 15.14 skrev Joseph Jenkins : > Finally got through on their support line and spoke

RE: Centurylink having a bad morning?

I’m not defending them but I am sure it isn’t intentional. From: NANOG On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org Subject: Re: Centurylink having a bad morning? How is that acceptable behaviour? I shall remember never to make a contract with these

Re: Does anyone actually like CenturyLink?

This is on the Legacy Level 3 side which is generally much better and still largely separate (according to my former co-workers). So say what you want about CTL but this is an LVLT issue. - Mike Bolitho On Sun, Aug 30, 2020, 8:03 AM Ross Tajvar wrote: > I've never heard a single positive word

Re: Centurylink having a bad morning?

I saw customers behind AS209’s ILEC Residential network flapping all morning. It’s stable now. So LVLT 3356 is definitely feeding parts of 209 now. On Sun, Aug 30, 2020 at 8:57 AM Ross Tajvar wrote: > Not being intentional isn't really an excuseOutages are generally not > intentional but

Re: Centurylink having a bad morning?

Once upon a time, Baldur Norddahl said: > How is that acceptable behaviour? I shall remember never to make a contract > with these guys until they can prove that they won't advertise my prefixes > after I pull them. Under any circumstances. Umm, then I guess you won't sign a contract with

Re: Centurylink having a bad morning?

AS3356 is the Level3 internet. On Sun, Aug 30, 2020 at 8:09 AM Ian Bowers wrote: > AS3356 is the one I've seen all the chatter about this morning. > > On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor > wrote: > >> On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote: >> > Woke up this morning to a

Re: Centurylink having a bad morning?

Not being intentional isn't really an excuseOutages are generally not intentional but we still like to use services that stay up most of the time. On Sun, Aug 30, 2020 at 11:11 AM Drew Weaver wrote: > I’m not defending them but I am sure it isn’t intentional. > > > > *From:* NANOG *On

RE: Does anyone actually like CenturyLink?

CenturyLink a network which has a long history of troubles, merged with Level3/TWT/Telcove/etc which had, IMOHO, excellent technical history. They've created some bastard which is less than what Level3 once was. As legacy Level3, I still have most of my original team, but I'm confident that

Re: Centurylink having a bad morning?

Yep Regards Ray Ludendorff On Aug 30, 2020, at 16:16, Jason Kuehl wrote:  I'm over in MA in a CL building, it's very much still broken. I shut down the interfaces to CL and now just using Comcast. On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky mailto:a...@mbrez.com>> wrote: Started about

Re: Centurylink having a bad morning?

If you have to have connectivity to them, you could always just instruct them not to announce your routes beyond their AS; paid peering, and announce through more reliable ASs such as 2914 and 1299. Many people do this. Otherwise, cut ties with them and save yourself the headaches. > On Aug

Does anyone actually like CenturyLink?

I've never heard a single positive word about them, and I've had my fair share of issues myself (as an indirect customer). But it seems that lots of people put them in their transit blend. Other than lack of options, why would anyone use them? To me, it just seems like asking for trouble...but

Re: Centurylink having a bad morning?

How is that acceptable behavior? It's not, the best part. There RCA will be terrible. "Bad Regex" or the best I ever got was "Bad cable" just two words... My contact is ending soon... On Sun, Aug 30, 2020 at 10:29 AM Antonios Chariton wrote: > Reporting from Europe, any IP with them in the

Re: Centurylink having a bad morning?

Exactly. And asking that they somehow prove this won't happen again is impossible. - Mike Bolitho On Sun, Aug 30, 2020, 8:10 AM Drew Weaver wrote: > I’m not defending them but I am sure it isn’t intentional. > > > > *From:* NANOG *On Behalf > Of *Baldur Norddahl > *Sent:* Sunday, August 30,

Re: Centurylink having a bad morning? [EXTERNAL]

On 30/08/2020 18:22, Joseph Jenkins wrote: Well at least it looks like the issue is starting to resolve  and stuff is coming back up. On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes

Re: Centurylink having a bad morning?

An outage is what it is. I am not worried about outages. We have multiple transits to deal with that. It is the keep announcing prefixes after withdrawal from peers and customers that is the huge problem here. That is killing all the effort and money I put into having redundancy. It is sabotage

Re: Centurylink having a bad morning? [EXTERNAL]

Is this what happens when your entire network is database driven?

Re: Centurylink having a bad morning?

I believe from this moment forward things are converging back to normal. Kind regards, Job

Re: Does anyone actually like CenturyLink?

It’s just due to network size. Horrid service and reliability aside, if you have enough eyeballs, application providers will want to directly peer, and if you have enough app providers on net then access providers will want to peer. With all the acquisitions, they have a ton of fiber in the

Re: Does anyone actually like CenturyLink?

Also adding GLBC to this list but as a customer for several CL products I wish we would have more protection and isolation for our region. Even we are working in global BGP terms nowadays, that incident makes the idea of a global CSMA/CD switch more attractive. A similar event was on

Re: Centurylink having a bad morning? [EXTERNAL]

Well at least it looks like the issue is starting to resolve and stuff is coming back up. On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes < mattli...@rivervalleyinternet.net> wrote: > Is this what happens when your entire network is database driven? >

Re: Centurylink having a bad morning?

I've been burning before. I'll wait at least an hour before turning my links back on. On Sun, Aug 30, 2020 at 11:31 AM Job Snijders wrote: > I believe from this moment forward things are converging back to normal. > > Kind regards, > > Job > -- Sincerely, Jason W Kuehl Cell 920-419-8983

Re: Does anyone actually like CenturyLink?

I’ve got CL Fiber+ ent service with BGP for my own v4 and v6 space. There’s been two outages in the several years it’s been installed - once when the major nationwide outage happened, and the other when Track Utils touched our dedicated strand and left us without service for two days while CL

Re: Does anyone actually like CenturyLink?

Peace, On Sun, Aug 30, 2020, 6:02 PM Ross Tajvar wrote: > Other than lack of options, why would anyone use them? > Connectivity and latency (of Level3 which was acquired). -- Töma >

Re: Centurylink having a bad morning?

Big leaf SDWan is seeing the outage between gateways. https://status.bigleaf.net/incidents/31r4wts0jlrr Robert DeVita Managing Director Mejeticks c. 469-441-8864 e. radev...@mejeticks.com From: NANOG on behalf of Tom Beecher Sent: Sunday, August 30, 2020

Re: Centurylink having a bad morning?

Latest updates from my tickets: 08/30/2020 14:28:20 GMT - The IP NOC confirmed a routing issue and commenced with troubleshooting efforts. Routing configuration adjustments have been made and service affecting alarms are beginning to clear. 08/30/2020 11:38:15 GMT - The IP NOC is engaged in

RE: Centurylink having a bad morning?

My systems dropped CL about 03 this morning. Appears that a correction has been applied as I am seeing more normal traffic from CL. Still no access to their portal and the phone is a mess. LQ. Marshall | Sr. Network Engineer | RegEd -Original Message- From: NANOG On Behalf Of Drew

Re: Does anyone actually like CenturyLink?

No clue. They’ve been progressively getting worse since 2010. I have no idea why anyone chooses them and they shouldn’t be considered a Tier1 carrier with the level of issues they have. > On Aug 30, 2020, at 11:07 AM, Ross Tajvar wrote: > >  > I've never heard a single positive word about

Re: Centurylink having a bad morning?

AS3356 is the one I've seen all the chatter about this morning. On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor wrote: > On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote: > > Woke up this morning to a bunch of reports of issues with connectivity > > had to shut down some Level3/CTL connections

Re: Centurylink having a bad morning?

That might be because of this: The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the

Re: Does anyone actually like CenturyLink?

We (eyeball network in Germany) cancelled all our Level3 transit links last year since it consisted mostly of traffic of their own CDN and they refused to lower the price per mbit/s or set up peering for the CDN traffic. Now we receive their CDN traffic via a different Tier1, for a fraction of

Re: Centurylink having a bad morning?

On Sun, Aug 30, 2020 at 5:21 PM Chris Adams wrote: > Once upon a time, Baldur Norddahl said: > > How is that acceptable behaviour? I shall remember never to make a > contract > > with these guys until they can prove that they won't advertise my > prefixes > > after I pull them. Under any

Re: Centurylink having a bad morning?

This is what happens when the design of 'god power' automation tools doesn't take into account the concept of blast radius. It might be more inconvenient to internally partition automated change management systems, but it can also limit the effect of automation tools gone awry.