Re: Centurylink having a bad morning?

[Hank Nussbacher]

  
  
On 30/08/2020 20:08, Baldur Norddahl
  wrote:


https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/


Sounds like Flowspec possibly blocking
  tcp/179 might be the cause.


But that is Cloudflare speculation.


Regards,
  Hank
Caveat: The views expressed above are
  solely my own and do not express the views or opinions of my
  employer 




  
  
An outage is what it is. I am not worried about outages. We
  have multiple transits to deal with that.


It is the keep announcing prefixes after withdrawal from
  peers and customers that is the huge problem here. That is
  killing all the effort and money I put into having redundancy.
  It is sabotage of my network after I cut the ties. I do not
  want to be a customer at an outlet who has a system that will
  do that. Luckily we do not currently have a contract and now
  they will have to convince me it is safe for me to make a
  contract with them. If that is impossible I guess I won't be
  getting a contract with them.


But I disagree in that it would be impossible. They need to
  make a good report telling exactly what went wrong and how
  they changed the design, so something like this can not happen
  again. The basic design of BGP is such that this should not
  happen easily if at all. They did something unwise. Did they
  make a route reflector based on a database or something?


Regards,


Baldur


  On Sun, Aug 30, 2020 at 5:13
PM Mike Bolitho 
wrote:
  
  
Exactly. And asking that they somehow prove
  this won't happen again is impossible.
  
  - Mike Bolitho



  On Sun, Aug 30, 2020,
8:10 AM Drew Weaver 
wrote:
  
  

  
I’m not defending them but I am
  sure it isn’t intentional.
 

  From: NANOG
thenap@nanog.org>
On Behalf Of Baldur Norddahl
Sent: Sunday, August 30, 2020 9:28 AM
To: nanog@nanog.org
Subject: Re: Centurylink having a bad
morning?

 

  How is that acceptable
behaviour? I shall remember never to make a
contract with these guys until they can prove
that they won't advertise my prefixes after I
pull them. Under any circumstances. 

 

  
søn. 30. aug. 2020 15.14
  skrev Joseph Jenkins :
  
  

  Finally got through on
their support line and spoke to level1. The
only thing the tech could say was it was an
issue with BGP route reflectors and it
started about 3am(pacific). They were still
trying to isolate the issue. I've tried
failing over my circuits and no go, the
traffic just dies as L3 won't stop
advertising my routes.

 

  
On Sun, Aug 30, 2020 at
  5:21 AM Drew Weaver via NANOG 
  wrote:
  
  

  
Hello,
 
Woke up this
  morning to a bunch of reports of
  issues with connectivity had to shut
  down some Level3/CTL connections to
  get it to return to normal.
 
As of right now
  their support portal won’t load:
  https://www.centurylink.com/business/login/

Re: Centurylink having a bad morning?

[Saku Ytti]

On Sun, 30 Aug 2020 at 20:00, Baldur Norddahl  wrote:

> Not really the point. BGP is designed such that if I take down the link, the 
> prefixes MUST be withdrawn within reasonable time. The self healing aspect of 
> the internet entirely depends on this. Clearly they have some kind of system 
> that does not respect that by design. I am guessing they have something 
> homebrewn going on with their route reflectors?

Add scale and BGP implementations can take a lot of time, hours of it.
Best thing you can do is add contractual obligations so people at your
provider who agree with you have some ammo. Instant is not on the
table, I'm sure that is obvious after that it's less than obvious what
is good enough.

> It is like a plane. It is impossible to prove or even design a plane that can 
> never fall out of the sky. But now we had a plane that crashed in a very bad 
> way, so that plane (Centurylink) is grounded until they can prove that 
> something like this can not happen again. Which means they need to redesign 
> whatever the hell they have going on here.

Nothing ever works like this, it's naive to think any RCA leads to
something fixed so that it can never happen again. Only thing that can
be affected is the frequency of an event, removing it is not on the
cards. And usually affecting frequency is mostly about belief not
something provable. In addition to MTBF, questions should be raised
about MTTR, provable MTTR efforts are far more likely to exist than
provable MTBF efforts, but if we buy-in to the notion that it never
will happen again, because we is good, then no MTTR focus is needed,
why fix something that will never happen.
What if this outage took 5min to solve?

-- 
  ++ytti

Re: Centurylink having a bad morning?

[Eric Kuhnke]

This is what happens when the design of 'god power' automation tools
doesn't take into account the concept of blast radius. It might be more
inconvenient to internally partition automated change management systems,
but it can also limit the effect of automation tools gone awry.

https://www.ibm.com/garage/method/practices/manage/practice_limited_blast_radius/

https://principlesofchaos.org/

On Sun, Aug 30, 2020 at 10:09 AM Baldur Norddahl 
wrote:

> An outage is what it is. I am not worried about outages. We have multiple
> transits to deal with that.
>
> It is the keep announcing prefixes after withdrawal from peers and
> customers that is the huge problem here. That is killing all the effort and
> money I put into having redundancy. It is sabotage of my network after I
> cut the ties. I do not want to be a customer at an outlet who has a system
> that will do that. Luckily we do not currently have a contract and now they
> will have to convince me it is safe for me to make a contract with them. If
> that is impossible I guess I won't be getting a contract with them.
>
> But I disagree in that it would be impossible. They need to make a good
> report telling exactly what went wrong and how they changed the design, so
> something like this can not happen again. The basic design of BGP is such
> that this should not happen easily if at all. They did something unwise.
> Did they make a route reflector based on a database or something?
>
> Regards,
>
> Baldur
>
> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho 
> wrote:
>
>> Exactly. And asking that they somehow prove this won't happen again is
>> impossible.
>>
>> - Mike Bolitho
>>
>> On Sun, Aug 30, 2020, 8:10 AM Drew Weaver  wrote:
>>
>>> I’m not defending them but I am sure it isn’t intentional.
>>>
>>>
>>>
>>> *From:* NANOG  *On
>>> Behalf Of *Baldur Norddahl
>>> *Sent:* Sunday, August 30, 2020 9:28 AM
>>> *To:* nanog@nanog.org
>>> *Subject:* Re: Centurylink having a bad morning?
>>>
>>>
>>>
>>> How is that acceptable behaviour? I shall remember never to make a
>>> contract with these guys until they can prove that they won't advertise my
>>> prefixes after I pull them. Under any circumstances.
>>>
>>>
>>>
>>> søn. 30. aug. 2020 15.14 skrev Joseph Jenkins <
>>> j...@breathe-underwater.com>:
>>>
>>> Finally got through on their support line and spoke to level1. The only
>>> thing the tech could say was it was an issue with BGP route reflectors and
>>> it started about 3am(pacific). They were still trying to isolate the issue.
>>> I've tried failing over my circuits and no go, the traffic just dies as L3
>>> won't stop advertising my routes.
>>>
>>>
>>>
>>> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
>>> wrote:
>>>
>>> Hello,
>>>
>>>
>>>
>>> Woke up this morning to a bunch of reports of issues with connectivity
>>> had to shut down some Level3/CTL connections to get it to return to normal.
>>>
>>>
>>>
>>> As of right now their support portal won’t load:
>>> https://www.centurylink.com/business/login/
>>>
>>>
>>>
>>> Just wondering what others are seeing.
>>>
>>>
>>>
>>>

Re: Centurylink having a bad morning? [EXTERNAL]

[Hank Nussbacher]

  
  
On 30/08/2020 18:22, Joseph Jenkins
  wrote:


  
  Well at least it looks like the issue is starting
to resolve  and stuff is coming back up.
  
  
On Sun, Aug 30, 2020 at 8:21
  AM Matt Hoppes 
  wrote:

Is
  this what happens when your entire network is database driven?

  



See:
https://status.ctl.io/
and specifically:
https://status.ctl.io/history/f19a0555-abbd-4038-91cb-b55a7645c1f5


Regards,
  Hank
Caveat: The views expressed above are solely my own and do not
  express the views or opinions of my employer 

  

Re: Centurylink having a bad morning?

[JASON BOTHE via NANOG]

If you have to have connectivity to them, you could always just instruct them 
not to announce your routes beyond their AS; paid peering, and announce through 
more reliable ASs such as 2914 and 1299. Many people do this. Otherwise, cut 
ties with them and save yourself the headaches. 


> On Aug 30, 2020, at 12:09, Baldur Norddahl  wrote:
> 
> 
> An outage is what it is. I am not worried about outages. We have multiple 
> transits to deal with that.
> 
> It is the keep announcing prefixes after withdrawal from peers and customers 
> that is the huge problem here. That is killing all the effort and money I put 
> into having redundancy. It is sabotage of my network after I cut the ties. I 
> do not want to be a customer at an outlet who has a system that will do that. 
> Luckily we do not currently have a contract and now they will have to 
> convince me it is safe for me to make a contract with them. If that is 
> impossible I guess I won't be getting a contract with them.
> 
> But I disagree in that it would be impossible. They need to make a good 
> report telling exactly what went wrong and how they changed the design, so 
> something like this can not happen again. The basic design of BGP is such 
> that this should not happen easily if at all. They did something unwise. Did 
> they make a route reflector based on a database or something?
> 
> Regards,
> 
> Baldur
> 
>> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho  wrote:
>> Exactly. And asking that they somehow prove this won't happen again is 
>> impossible.
>> 
>> - Mike Bolitho
>> 
>>> On Sun, Aug 30, 2020, 8:10 AM Drew Weaver  wrote:
>>> I’m not defending them but I am sure it isn’t intentional.
>>> 
>>>  
>>> 
>>> From: NANOG  On Behalf Of 
>>> Baldur Norddahl
>>> Sent: Sunday, August 30, 2020 9:28 AM
>>> To: nanog@nanog.org
>>> Subject: Re: Centurylink having a bad morning?
>>> 
>>>  
>>> 
>>> How is that acceptable behaviour? I shall remember never to make a contract 
>>> with these guys until they can prove that they won't advertise my prefixes 
>>> after I pull them. Under any circumstances. 
>>> 
>>>  
>>> 
>>> søn. 30. aug. 2020 15.14 skrev Joseph Jenkins :
>>> 
>>> Finally got through on their support line and spoke to level1. The only 
>>> thing the tech could say was it was an issue with BGP route reflectors and 
>>> it started about 3am(pacific). They were still trying to isolate the issue. 
>>> I've tried failing over my circuits and no go, the traffic just dies as L3 
>>> won't stop advertising my routes.
>>> 
>>>  
>>> 
>>> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG  
>>> wrote:
>>> 
>>> Hello,
>>> 
>>>  
>>> 
>>> Woke up this morning to a bunch of reports of issues with connectivity had 
>>> to shut down some Level3/CTL connections to get it to return to normal.
>>> 
>>>  
>>> 
>>> As of right now their support portal won’t load: 
>>> https://www.centurylink.com/business/login/
>>> 
>>>  
>>> 
>>> Just wondering what others are seeing.
>>> 
>>>  

Re: Centurylink having a bad morning?

[Baldur Norddahl]

An outage is what it is. I am not worried about outages. We have multiple
transits to deal with that.

It is the keep announcing prefixes after withdrawal from peers and
customers that is the huge problem here. That is killing all the effort and
money I put into having redundancy. It is sabotage of my network after I
cut the ties. I do not want to be a customer at an outlet who has a system
that will do that. Luckily we do not currently have a contract and now they
will have to convince me it is safe for me to make a contract with them. If
that is impossible I guess I won't be getting a contract with them.

But I disagree in that it would be impossible. They need to make a good
report telling exactly what went wrong and how they changed the design, so
something like this can not happen again. The basic design of BGP is such
that this should not happen easily if at all. They did something unwise.
Did they make a route reflector based on a database or something?

Regards,

Baldur

On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho  wrote:

> Exactly. And asking that they somehow prove this won't happen again is
> impossible.
>
> - Mike Bolitho
>
> On Sun, Aug 30, 2020, 8:10 AM Drew Weaver  wrote:
>
>> I’m not defending them but I am sure it isn’t intentional.
>>
>>
>>
>> *From:* NANOG  *On
>> Behalf Of *Baldur Norddahl
>> *Sent:* Sunday, August 30, 2020 9:28 AM
>> *To:* nanog@nanog.org
>> *Subject:* Re: Centurylink having a bad morning?
>>
>>
>>
>> How is that acceptable behaviour? I shall remember never to make a
>> contract with these guys until they can prove that they won't advertise my
>> prefixes after I pull them. Under any circumstances.
>>
>>
>>
>> søn. 30. aug. 2020 15.14 skrev Joseph Jenkins > >:
>>
>> Finally got through on their support line and spoke to level1. The only
>> thing the tech could say was it was an issue with BGP route reflectors and
>> it started about 3am(pacific). They were still trying to isolate the issue.
>> I've tried failing over my circuits and no go, the traffic just dies as L3
>> won't stop advertising my routes.
>>
>>
>>
>> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
>> wrote:
>>
>> Hello,
>>
>>
>>
>> Woke up this morning to a bunch of reports of issues with connectivity
>> had to shut down some Level3/CTL connections to get it to return to normal.
>>
>>
>>
>> As of right now their support portal won’t load:
>> https://www.centurylink.com/business/login/
>>
>>
>>
>> Just wondering what others are seeing.
>>
>>
>>
>>

Re: Centurylink having a bad morning?

[Ray Ludendorff]

Yep

Regards
Ray Ludendorff


On Aug 30, 2020, at 16:16, Jason Kuehl  wrote:


I'm over in MA in a CL building, it's very much still broken. I shut down the 
interfaces to CL and now just using Comcast.

On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky 
mailto:a...@mbrez.com>> wrote:

Started about 5:05am central, started clearing up for me about 7:15am.   My 
route from ATT in Chicago is still going through NYC to get back to Chicago but 
at least packet loss isn't 70-100% anymore.

I also tried turning down sessions and still was seeing stale announcements on 
other LGs.

On 08/30/2020 07:27 AM, David Hubbard wrote:
Same.  Also, as reported on outages list, what’s even worse is that they appear 
to be continuing to propagate advertisements from circuits whose sessions have 
been turned down.  I validated ours still were via a couple looking glass 
portals.  Down Detector shows nearly every major service provider impacted.

They’re not reachable so who knows if they’re even working on it.  I feel like 
they’ve been cutting heavily on the network ops side in recent years…

From: NANOG 

 on behalf of Drew Weaver via NANOG 
Reply-To: Drew Weaver 
Date: Sunday, August 30, 2020 at 8:23 AM
To: "nanog@nanog.org" 

Subject: Centurylink having a bad morning?

Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.




--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

Re: Centurylink having a bad morning?

[Baldur Norddahl]

On Sun, Aug 30, 2020 at 5:21 PM Chris Adams  wrote:

> Once upon a time, Baldur Norddahl  said:
> > How is that acceptable behaviour? I shall remember never to make a
> contract
> > with these guys until they can prove that they won't advertise my
> prefixes
> > after I pull them. Under any circumstances.
>
> Umm, then I guess you won't sign a contract with anybody?  I sure
> wouldn't agree to that.  I don't personally write the routing software,
> so I can't guarantee there isn't a bug in there (actually, since it is
> software, I can guarantee there ARE bugs in there).
>
> We'll see if/when they issue an RFO, but software has bugs, and
> configuration errors have entirely unexpected consequences.  It's
> possible some poor design issue was exposed, or it could be some
> basically unforeseeable incident.
>

Not really the point. BGP is designed such that if I take down the link,
the prefixes MUST be withdrawn within reasonable time. The self healing
aspect of the internet entirely depends on this. Clearly they have some
kind of system that does not respect that by design. I am guessing they
have something homebrewn going on with their route reflectors?

It is like a plane. It is impossible to prove or even design a plane that
can never fall out of the sky. But now we had a plane that crashed in a
very bad way, so that plane (Centurylink) is grounded until they can prove
that something like this can not happen again. Which means they need to
redesign whatever the hell they have going on here.

Regards,

Baldur

Re: Does anyone actually like CenturyLink?

[Jörg Kost]

Also adding GLBC to this list but as a customer for several CL products 
I wish we would have more protection and isolation for our region. Even 
we are working in global BGP terms nowadays, that incident makes the 
idea of a global CSMA/CD switch more attractive.


A similar event was on 2019-10-07, when several Level3/CL global routers 
had rolling reboots.


In general and after the CL/L3 merge we have seen a lot of good changes 
locally. Time to build upon this, act local, think global - a company 
this size and that is in charge  of such "a lot of Internet", also needs 
to made aware of their responsibility from head to toe. And that can 
also be done in positive ways.


On 30 Aug 2020, at 17:56, Marshall, Quincy wrote:

CenturyLink a network which has a long history of troubles, merged 
with Level3/TWT/Telcove/etc which had, IMOHO, excellent technical 
history. They've created some bastard which is less than what Level3 
once was. As legacy Level3, I still have most of my original team, but 
I'm confident that whatever caused CL's messes has worked its way into 
the works.


LQ. Marshall | Sr. Network Engineer | RegEd

Re: Does anyone actually like CenturyLink?

[Töma Gavrichenkov]

Peace,

On Sun, Aug 30, 2020, 6:02 PM Ross Tajvar  wrote:

> Other than lack of options, why would anyone use them?
>

Connectivity and latency (of Level3 which was acquired).

--
Töma

>

Re: Does anyone actually like CenturyLink?

[David Hubbard]

It’s just due to network size.  Horrid service and reliability aside, if you 
have enough eyeballs, application providers will want to directly peer, and if 
you have enough app providers on net then access providers will want to peer.  
With all the acquisitions, they have a ton of fiber in the ground and they keep 
rolling the networks in, causing people to continue to buy their circuits.

The service has been horrible compared to the twtelecom days.  Fortunately 
their layer two stuff seems to be managed independently of their IP networks, 
so those remained up.

From: NANOG  on behalf 
of Ross Tajvar 
Date: Sunday, August 30, 2020 at 11:05 AM
To: North American Network Operators' Group 
Subject: Does anyone actually like CenturyLink?

I've never heard a single positive word about them, and I've had my fair share 
of issues myself (as an indirect customer). But it seems that lots of people 
put them in their transit blend. Other than lack of options, why would anyone 
use them? To me, it just seems like asking for trouble...but maybe I'm missing 
something?

Re: Centurylink having a bad morning?

[Jared Geiger]

I saw customers behind AS209’s ILEC Residential network flapping all
morning. It’s stable now.  So LVLT 3356 is definitely feeding parts of 209
now.

On Sun, Aug 30, 2020 at 8:57 AM Ross Tajvar  wrote:

> Not being intentional isn't really an excuseOutages are generally not
> intentional but we still like to use services that stay up most of the time.
>
> On Sun, Aug 30, 2020 at 11:11 AM Drew Weaver 
> wrote:
>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> I’m not defending them but I am sure it isn’t intentional.
>>
>>
>>
>>
>>
>>
>>
>> *From:* NANOG 
>>
>> *On Behalf Of *Baldur Norddahl
>>
>>
>> *Sent:* Sunday, August 30, 2020 9:28 AM
>>
>>
>> *To:* nanog@nanog.org
>>
>>
>> *Subject:* Re: Centurylink having a bad morning?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> How is that acceptable behaviour? I shall remember never to make a
>> contract with these guys until they can prove that they won't advertise my
>> prefixes after I pull them. Under any circumstances.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> søn. 30. aug. 2020 15.14 skrev Joseph Jenkins > >:
>>
>>
>>
>>
>>
>>
>>
>>
>> Finally got through on their support line and spoke to level1. The only
>> thing the tech could say was it was an issue with BGP route reflectors and
>> it started about 3am(pacific). They were still trying to isolate the issue.
>> I've tried failing
>>
>> over my circuits and no go, the traffic just dies as L3 won't stop
>> advertising my routes.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
>> wrote:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Hello,
>>
>>
>>
>>
>>
>> Woke up this morning to a bunch of reports of issues with connectivity
>> had to shut down some Level3/CTL connections to get it to return to normal.
>>
>>
>>
>>
>>
>> As of right now their support portal won’t load:
>>
>> https://www.centurylink.com/business/login/
>>
>>
>>
>>
>>
>> Just wondering what others are seeing.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>

Re: Does anyone actually like CenturyLink?

[Brielle]

I’ve got CL Fiber+ ent service with BGP for my own v4 and v6 space.

There’s been two outages in the several years it’s been installed - once when 
the major nationwide outage happened, and the other when Track Utils touched 
our dedicated strand and left us without service for two days while CL tried to 
figure out what they did and where we were plugged into now.

So, been fine here with us getting the level of service we paid for outside of 
those two incidents...

Sent from my iPad

> On Aug 30, 2020, at 9:04 AM, Ross Tajvar  wrote:
> 
> 
> I've never heard a single positive word about them, and I've had my fair 
> share of issues myself (as an indirect customer). But it seems that lots of 
> people put them in their transit blend. Other than lack of options, why would 
> anyone use them? To me, it just seems like asking for trouble...but maybe I'm 
> missing something?

RE: Does anyone actually like CenturyLink?

[Marshall, Quincy]

CenturyLink a network which has a long history of troubles, merged with 
Level3/TWT/Telcove/etc which had, IMOHO, excellent technical history. They've 
created some bastard which is less than what Level3 once was. As legacy Level3, 
I still have most of my original team, but I'm confident that whatever caused 
CL's messes has worked its way into the works.

LQ. Marshall | Sr. Network Engineer | RegEd 

From: NANOG  On Behalf Of 
Mike Bolitho
Sent: Sunday, August 30, 2020 11:06 AM
To: Ross Tajvar 
Cc: North American Network Operators' Group 
Subject: Re: Does anyone actually like CenturyLink?

This is on the Legacy Level 3 side which is generally much better and still 
largely separate (according to my former co-workers). So say what you want 
about CTL but this is an LVLT issue.
- Mike Bolitho

On Sun, Aug 30, 2020, 8:03 AM Ross Tajvar  wrote:
I've never heard a single positive word about them, and I've had my fair share 
of issues myself (as an indirect customer). But it seems that lots of people 
put them in their transit blend. Other than lack of options, why would anyone 
use them? To me, it just seems like asking for trouble...but maybe I'm missing 
something?
---
 This email has been scanned for email related threats and delivered safely by 
Mimecast.
 For more information please visit http://www.mimecast.com
---

Re: Centurylink having a bad morning?

[Jason Kuehl]

I've been burning before. I'll wait at least an hour before turning my
links back on.

On Sun, Aug 30, 2020 at 11:31 AM Job Snijders  wrote:

> I believe from this moment forward things are converging back to normal.
>
> Kind regards,
>
> Job
>


-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

Re: Centurylink having a bad morning? [EXTERNAL]

[Joseph Jenkins]

Well at least it looks like the issue is starting to resolve  and stuff is
coming back up.

On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes <
mattli...@rivervalleyinternet.net> wrote:

> Is this what happens when your entire network is database driven?
>

Re: Does anyone actually like CenturyLink?

[Matt Hoppes]

No clue. They’ve been progressively  getting worse since 2010. I have no idea 
why anyone chooses them and they shouldn’t be considered a Tier1 carrier with 
the level of issues they have. 

> On Aug 30, 2020, at 11:07 AM, Ross Tajvar  wrote:
> 
> 
> I've never heard a single positive word about them, and I've had my fair 
> share of issues myself (as an indirect customer). But it seems that lots of 
> people put them in their transit blend. Other than lack of options, why would 
> anyone use them? To me, it just seems like asking for trouble...but maybe I'm 
> missing something?

Re: Centurylink having a bad morning?

[Ross Tajvar]

Not being intentional isn't really an excuseOutages are generally not
intentional but we still like to use services that stay up most of the time.

On Sun, Aug 30, 2020 at 11:11 AM Drew Weaver  wrote:

> I’m not defending them but I am sure it isn’t intentional.
>
>
>
> *From:* NANOG  *On Behalf
> Of *Baldur Norddahl
> *Sent:* Sunday, August 30, 2020 9:28 AM
> *To:* nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> How is that acceptable behaviour? I shall remember never to make a
> contract with these guys until they can prove that they won't advertise my
> prefixes after I pull them. Under any circumstances.
>
>
>
> søn. 30. aug. 2020 15.14 skrev Joseph Jenkins  >:
>
> Finally got through on their support line and spoke to level1. The only
> thing the tech could say was it was an issue with BGP route reflectors and
> it started about 3am(pacific). They were still trying to isolate the issue.
> I've tried failing over my circuits and no go, the traffic just dies as L3
> won't stop advertising my routes.
>
>
>
> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
> wrote:
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>
>

Re: Centurylink having a bad morning?

[Mike Bolitho]

Exactly. And asking that they somehow prove this won't happen again is
impossible.

- Mike Bolitho

On Sun, Aug 30, 2020, 8:10 AM Drew Weaver  wrote:

> I’m not defending them but I am sure it isn’t intentional.
>
>
>
> *From:* NANOG  *On Behalf
> Of *Baldur Norddahl
> *Sent:* Sunday, August 30, 2020 9:28 AM
> *To:* nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> How is that acceptable behaviour? I shall remember never to make a
> contract with these guys until they can prove that they won't advertise my
> prefixes after I pull them. Under any circumstances.
>
>
>
> søn. 30. aug. 2020 15.14 skrev Joseph Jenkins  >:
>
> Finally got through on their support line and spoke to level1. The only
> thing the tech could say was it was an issue with BGP route reflectors and
> it started about 3am(pacific). They were still trying to isolate the issue.
> I've tried failing over my circuits and no go, the traffic just dies as L3
> won't stop advertising my routes.
>
>
>
> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
> wrote:
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>
>

Re: Does anyone actually like CenturyLink?

[vidister via NANOG]

We (eyeball network in Germany) cancelled all our Level3 transit links last 
year since it consisted mostly of traffic of their own CDN and they refused to 
lower the price per mbit/s or set up peering for the CDN traffic. Now we 
receive their CDN traffic via a different Tier1, for a fraction of the price.

Operating a CDN inside a Tier1 network is just shitty behaviour.

- vidister

> On 30. Aug 2020, at 16:17, Ross Tajvar  wrote:
> 
> I've never heard a single positive word about them, and I've had my fair 
> share of issues myself (as an indirect customer). But it seems that lots of 
> people put them in their transit blend. Other than lack of options, why would 
> anyone use them? To me, it just seems like asking for trouble...but maybe I'm 
> missing something?

Re: Centurylink having a bad morning?

[Joseph Jenkins]

AS3356 is the Level3 internet.

On Sun, Aug 30, 2020 at 8:09 AM Ian Bowers  wrote:

> AS3356 is the one I've seen all the chatter about this morning.
>
> On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor 
> wrote:
>
>> On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote:
>> > Woke up this morning to a bunch of reports of issues with connectivity
>> > had to shut down some Level3/CTL connections to get it to return to
>> normal.
>>
>>
>>
>> Just to confirm we're seeing this on AS3356 and not AS209, correct?
>>
>>
>> We have links to both and shut down AS3356 which seems to have cleared
>> "most" of the problems.
>>
>>
>> --
>> inoc.net!rblayzor
>> XMPP: rblayzor.AT.inoc.net
>> PGP:  https://pgp.inoc.net/rblayzor/
>>
>

Re: Does anyone actually like CenturyLink?

[Mike Bolitho]

This is on the Legacy Level 3 side which is generally much better and still
largely separate (according to my former co-workers). So say what you want
about CTL but this is an LVLT issue.

- Mike Bolitho

On Sun, Aug 30, 2020, 8:03 AM Ross Tajvar  wrote:

> I've never heard a single positive word about them, and I've had my fair
> share of issues myself (as an indirect customer). But it seems that lots of
> people put them in their transit blend. Other than lack of options, why
> would anyone use them? To me, it just seems like asking for trouble...but
> maybe I'm missing something?
>

RE: Centurylink having a bad morning?

[Marshall, Quincy]

My systems dropped CL about 03 this morning. Appears that a correction has been 
applied as I am seeing more normal traffic from CL. Still no access to their 
portal and the phone is a mess.

LQ. Marshall | Sr. Network Engineer | RegEd 

-Original Message-
From: NANOG  On Behalf Of 
Drew Weaver
Sent: Sunday, August 30, 2020 10:07 AM
To: 'Ben Russell' ; 'nanog@nanog.org' 
Subject: RE: Centurylink having a bad morning?

Yikes, they really need to get their reflectors fixed so that we can dis-engage 
from them.

-Original Message-
From: NANOG  On Behalf Of Ben 
Russell
Sent: Sunday, August 30, 2020 9:14 AM
To: 'nanog@nanog.org' 
Subject: RE: Centurylink having a bad morning?

One of my legacy L3 peers is only advertising 498174 prefixes.  Very strange.



Ben Russell  



-Original Message-
From: NANOG  On Behalf Of Drew 
Weaver
Sent: Sunday, August 30, 2020 7:45 AM
To: 'Mel Beckman' 
Cc: 'nanog@nanog.org' 
Subject: RE: Centurylink having a bad morning?

Can't get to pretty much anything on the Internet from my Tmobile device... not 
really sure whats going on =)

-Original Message-
From: Mel Beckman  
Sent: Sunday, August 30, 2020 8:43 AM
To: Drew Weaver 
Cc: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

One anomaly I’ve just notice: we can’t reach 209.244.0.1 (ns1.level3.com) from 
anywhere on the Internet, including our CL connections. Traceroute fails at 
4.16.30.129 (lag-101.ear2.LosAngeles1.Level3.net) with 65% packet loss:

traceroute to 209.244.0.1 (209.244.0.1), 30 hops max, 56 byte packets
 1  sent:79 loss:100% last:0 ms avg:0 ms
 2  sent:79 loss:0% last:15.483 ms avg:28.071 ms
172.102.107.166
 3  sent:79 loss:2.5% last:140.81 ms avg:33.374 ms
ae8---0.scr02.lsan.ca.frontiernet.net (74.40.3.49)
 4  sent:78 loss:0% last:11.695 ms avg:23.637 ms
ae1---0.cbr01.lsan.ca.frontiernet.net (74.40.3.214)
 5  sent:78 loss:65.4% last:13.844 ms avg:19.393 ms
lag-101.ear2.LosAngeles1.Level3.net (4.16.30.129)
 6  sent:78 loss:100% last:0 ms avg:0 ms
 7  sent:78 loss:100% last:0 ms avg:0 ms
 8  sent:78 loss:100% last:0 ms avg:0 ms
 9  sent:78 loss:100% last:0 ms avg:0 ms
10  sent:78 loss:100% last:0 ms avg:0 ms




 -mel beckman

> On Aug 30, 2020, at 5:36 AM, Mel Beckman  wrote:
> 
---
 This email has been scanned for email related threats and delivered safely by 
Mimecast.
 For more information please visit http://www.mimecast.com
---

Re: Centurylink having a bad morning?

[Job Snijders]

I believe from this moment forward things are converging back to normal.

Kind regards,

Job

Re: Centurylink having a bad morning?

[Joseph Jenkins]

That might be because of this:

The IP NOC with the assistance of the Operations Engineering team confirmed
a routing issue to be preventing BGP sessions from establishing correctly.
A configuration adjustment was deployed at a high level, and sessions began
to re-establish with stability. As the change propagates through the
affected devices, service affecting alarms continue to clear.
Due to the nature of this outage, it may be necessary to reset your
services locally at your equipment, or manually reset your BGP session. If
after that action has been performed a service issue prevails, please
contact the CenturyLink Repair Center for troubleshooting assistance

On Sun, Aug 30, 2020 at 7:42 AM Drew Weaver  wrote:

> Something just now changed in this situation and now it seems to have
> gotten worse.
>
>
>
> *From:* Jason Kuehl 
> *Sent:* Sunday, August 30, 2020 10:00 AM
> *To:* Drew Weaver 
> *Cc:* R. Leigh Hennig ; nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> People are rebooting ghosting now.
>
>
>
> https://twitter.com/ir_kujoe/status/1300066569645707265
>
>
>
> Seeing other reports of this too.
>
>
>
> On Sun, Aug 30, 2020 at 9:45 AM Drew Weaver 
> wrote:
>
> That site seems to be just for their cloud products, is there one of these
> for their actual network?
>
>
>
> *From:* R. Leigh Hennig 
> *Sent:* Sunday, August 30, 2020 8:54 AM
> *To:* Drew Weaver ; nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> Global impact with issues reported by Fastly, Cloudflare, OpenDNS.
>
>
>
> https://status.ctl.io/
>
>
> STARTED
>
> Sun Aug 30 2020 08:13 (EDT)
> Sun Aug 30 2020 12:13 (UTC)
> AFFECTED SERVICES
>
> External Cloud Network (CA3)
> DATE
> LATEST UPDATE
>
> Sun Aug 30 2020 08:13 (EDT)
> Sun Aug 30 2020 12:13 (UTC) Our technical teams are investigating an issue
> affecting some services in the CA3 data center. Ensuring the reliability of
> our services is our top priority. We will continue to provide status
> updates as this incident progresses. If you need further support, please
> contact us at h...@ctl.io.
>
>
>
>
>
>
>
> Sent from ProtonMail Mobile
>
>
>
>
>
> On Sun, Aug 30, 2020 at 8:14 AM, Drew Weaver via NANOG 
> wrote:
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>
>
>
>
>
>
>
>
> --
>
> Sincerely,
>
> Jason W Kuehl
> Cell 920-419-8983
> jason.w.ku...@gmail.com
>

Re: Centurylink having a bad morning?

[Jason Kuehl]

How is that acceptable behavior?

It's not, the best part. There RCA will be terrible.  "Bad Regex" or the
best I ever got was "Bad cable" just two words... My contact is ending
soon...

On Sun, Aug 30, 2020 at 10:29 AM Antonios Chariton 
wrote:

> Reporting from Europe, any IP with them in the path is unreachable from
> various providers. I guess they wanted to try IPv6-only.. :P IPv6 is fine,
> working fine, IPv4 not at all..
>
> Antonis
>
> > On 30 Aug 2020, at 14:58, Tomas Lynch  wrote:
> >
> > Flapping in Miami, Dallas, Atlanta, Los Angeles, Seattle and San Jose.
> It is also affecting some data centers in Europe too. but haven't seen
> flaps there, just suboptimal routing.
> >
> > On Sun, Aug 30, 2020 at 8:53 AM Drew Weaver 
> wrote:
> > Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…
> >
> >
> >
> > From: Tomas Lynch 
> > Sent: Sunday, August 30, 2020 8:45 AM
> > To: Mel Beckman 
> > Cc: Drew Weaver ; nanog@nanog.org
> > Subject: Re: Centurylink having a bad morning?
> >
> >
> >
> > BGP sessions randomly flapping or having routing issues in different
> cities since ~5AM EST
> >
> >
> >
> > On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman  wrote:
> >
> > The CL portal loads for me, and I can log in, but it is slower than
> usual. Not seeing traffic issues on our CL circuits.
> >
> > -mel via cell
> >
> >
> >
> >
> > On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG 
> wrote:
> >
> > 
> >
> > Hello,
> >
> >
> >
> > Woke up this morning to a bunch of reports of issues with connectivity
> had to shut down some Level3/CTL connections to get it to return to normal.
> >
> >
> >
> > As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
> >
> >
> >
> > Just wondering what others are seeing.
> >
> >
> >
>
>

-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

Re: Centurylink having a bad morning?

[Chris Adams]

Once upon a time, Baldur Norddahl  said:
> How is that acceptable behaviour? I shall remember never to make a contract
> with these guys until they can prove that they won't advertise my prefixes
> after I pull them. Under any circumstances.

Umm, then I guess you won't sign a contract with anybody?  I sure
wouldn't agree to that.  I don't personally write the routing software,
so I can't guarantee there isn't a bug in there (actually, since it is
software, I can guarantee there ARE bugs in there).

We'll see if/when they issue an RFO, but software has bugs, and
configuration errors have entirely unexpected consequences.  It's
possible some poor design issue was exposed, or it could be some
basically unforeseeable incident.
-- 
Chris Adams 

Re: Centurylink having a bad morning? [EXTERNAL]

[Matt Hoppes]

Is this what happens when your entire network is database driven?

Re: Centurylink having a bad morning?

[Joseph Jenkins]

Latest updates from my tickets:
08/30/2020 14:28:20 GMT - The IP NOC confirmed a routing issue and
commenced with troubleshooting efforts. Routing configuration adjustments
have been made and service affecting alarms are beginning to clear.

08/30/2020 11:38:15 GMT - The IP NOC is engaged in cooperative escalated
investigations to isolate and troubleshoot the fault at this time.

08/30/2020 11:03:09 GMT - On August 30, 2020 at 10:00 GMT, CenturyLink
identified a Market Wide service impact. As this network fault is impacting
multiple clients, the event has increased visibility with CenturyLink
leadership. As such, client trouble tickets associated to this fault have
been automatically escalated to higher priority.

The NOC is engaged and investigating in order to isolate the cause. Please
be advised that updates for this event will be relayed at a minimum of
hourly unless otherwise noted. The information conveyed hereafter is
associated to live troubleshooting effort and as the discovery process
evolves through to service resolution, ticket closure, or post incident
review, details may evolve.



On Sun, Aug 30, 2020 at 7:30 AM Antonios Chariton 
wrote:

> Reporting from Europe, any IP with them in the path is unreachable from
> various providers. I guess they wanted to try IPv6-only.. :P IPv6 is fine,
> working fine, IPv4 not at all..
>
> Antonis
>
> > On 30 Aug 2020, at 14:58, Tomas Lynch  wrote:
> >
> > Flapping in Miami, Dallas, Atlanta, Los Angeles, Seattle and San Jose.
> It is also affecting some data centers in Europe too. but haven't seen
> flaps there, just suboptimal routing.
> >
> > On Sun, Aug 30, 2020 at 8:53 AM Drew Weaver 
> wrote:
> > Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…
> >
> >
> >
> > From: Tomas Lynch 
> > Sent: Sunday, August 30, 2020 8:45 AM
> > To: Mel Beckman 
> > Cc: Drew Weaver ; nanog@nanog.org
> > Subject: Re: Centurylink having a bad morning?
> >
> >
> >
> > BGP sessions randomly flapping or having routing issues in different
> cities since ~5AM EST
> >
> >
> >
> > On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman  wrote:
> >
> > The CL portal loads for me, and I can log in, but it is slower than
> usual. Not seeing traffic issues on our CL circuits.
> >
> > -mel via cell
> >
> >
> >
> >
> > On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG 
> wrote:
> >
> > 
> >
> > Hello,
> >
> >
> >
> > Woke up this morning to a bunch of reports of issues with connectivity
> had to shut down some Level3/CTL connections to get it to return to normal.
> >
> >
> >
> > As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
> >
> >
> >
> > Just wondering what others are seeing.
> >
> >
> >
>
>

RE: Centurylink having a bad morning?

[Drew Weaver]

I’m not defending them but I am sure it isn’t intentional.

From: NANOG  On Behalf Of 
Baldur Norddahl
Sent: Sunday, August 30, 2020 9:28 AM
To: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

How is that acceptable behaviour? I shall remember never to make a contract 
with these guys until they can prove that they won't advertise my prefixes 
after I pull them. Under any circumstances.

søn. 30. aug. 2020 15.14 skrev Joseph Jenkins 
mailto:j...@breathe-underwater.com>>:
Finally got through on their support line and spoke to level1. The only thing 
the tech could say was it was an issue with BGP route reflectors and it started 
about 3am(pacific). They were still trying to isolate the issue. I've tried 
failing over my circuits and no go, the traffic just dies as L3 won't stop 
advertising my routes.

On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
mailto:nanog@nanog.org>> wrote:
Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Ian Bowers]

AS3356 is the one I've seen all the chatter about this morning.

On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor 
wrote:

> On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote:
> > Woke up this morning to a bunch of reports of issues with connectivity
> > had to shut down some Level3/CTL connections to get it to return to
> normal.
>
>
>
> Just to confirm we're seeing this on AS3356 and not AS209, correct?
>
>
> We have links to both and shut down AS3356 which seems to have cleared
> "most" of the problems.
>
>
> --
> inoc.net!rblayzor
> XMPP: rblayzor.AT.inoc.net
> PGP:  https://pgp.inoc.net/rblayzor/
>

Does anyone actually like CenturyLink?

[Ross Tajvar]

I've never heard a single positive word about them, and I've had my fair
share of issues myself (as an indirect customer). But it seems that lots of
people put them in their transit blend. Other than lack of options, why
would anyone use them? To me, it just seems like asking for trouble...but
maybe I'm missing something?

Re: Centurylink having a bad morning?

[Robert DeVita]

Big leaf SDWan is seeing the outage between gateways.

https://status.bigleaf.net/incidents/31r4wts0jlrr



Robert DeVita
Managing Director
Mejeticks
c. 469-441-8864
e. radev...@mejeticks.com

From: NANOG  on behalf of Tom 
Beecher 
Sent: Sunday, August 30, 2020 8:14:27 AM
To: David Hubbard 
Cc: nanog@nanog.org 
Subject: Re: Centurylink having a bad morning?

They’re not reachable so who knows if they’re even working on it.

Gonna go out on a limb here and assume that a lot of phones were ringing and 
people are in fact working on whatever it is. :)

On Sun, Aug 30, 2020 at 8:34 AM David Hubbard < dhubb...@dino.hostasaurus.com 
 > wrote:

Same.  Also, as reported on outages list, what’s even worse is that they appear 
to be continuing to propagate advertisements from circuits whose sessions have 
been turned down.  I validated ours still were via a couple looking glass 
portals.  Down Detector shows nearly every major service provider impacted.

They’re not reachable so who knows if they’re even working on it.  I feel like 
they’ve been cutting heavily on the network ops side in recent years…

From: NANOG mailto:dino.hostasaurus@nanog.org> > on behalf of Drew Weaver via NANOG < 
nanog@nanog.org  >
Reply-To: Drew Weaver < drew.wea...@thenap.com  >
Date: Sunday, August 30, 2020 at 8:23 AM
To: " nanog@nanog.org  " < nanog@nanog.org 
 >
Subject: Centurylink having a bad morning?

Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/ 


Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Chris Adams]

Once upon a time, Robert Blayzor  said:
> Just to confirm we're seeing this on AS3356 and not AS209, correct?

Correct - we had problems with our 3356 connection but not our 209
connection.
-- 
Chris Adams 

RE: Centurylink having a bad morning?

[Drew Weaver]

Yikes, they really need to get their reflectors fixed so that we can dis-engage 
from them.

-Original Message-
From: NANOG  On Behalf Of Ben 
Russell
Sent: Sunday, August 30, 2020 9:14 AM
To: 'nanog@nanog.org' 
Subject: RE: Centurylink having a bad morning?

One of my legacy L3 peers is only advertising 498174 prefixes.  Very strange.



Ben Russell  



-Original Message-
From: NANOG  On Behalf Of Drew 
Weaver
Sent: Sunday, August 30, 2020 7:45 AM
To: 'Mel Beckman' 
Cc: 'nanog@nanog.org' 
Subject: RE: Centurylink having a bad morning?

Can't get to pretty much anything on the Internet from my Tmobile device... not 
really sure whats going on =)

-Original Message-
From: Mel Beckman  
Sent: Sunday, August 30, 2020 8:43 AM
To: Drew Weaver 
Cc: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

One anomaly I’ve just notice: we can’t reach 209.244.0.1 (ns1.level3.com) from 
anywhere on the Internet, including our CL connections. Traceroute fails at 
4.16.30.129 (lag-101.ear2.LosAngeles1.Level3.net) with 65% packet loss:

traceroute to 209.244.0.1 (209.244.0.1), 30 hops max, 56 byte packets
 1  sent:79 loss:100% last:0 ms avg:0 ms
 2  sent:79 loss:0% last:15.483 ms avg:28.071 ms
172.102.107.166
 3  sent:79 loss:2.5% last:140.81 ms avg:33.374 ms
ae8---0.scr02.lsan.ca.frontiernet.net (74.40.3.49)
 4  sent:78 loss:0% last:11.695 ms avg:23.637 ms
ae1---0.cbr01.lsan.ca.frontiernet.net (74.40.3.214)
 5  sent:78 loss:65.4% last:13.844 ms avg:19.393 ms
lag-101.ear2.LosAngeles1.Level3.net (4.16.30.129)
 6  sent:78 loss:100% last:0 ms avg:0 ms
 7  sent:78 loss:100% last:0 ms avg:0 ms
 8  sent:78 loss:100% last:0 ms avg:0 ms
 9  sent:78 loss:100% last:0 ms avg:0 ms
10  sent:78 loss:100% last:0 ms avg:0 ms




 -mel beckman

> On Aug 30, 2020, at 5:36 AM, Mel Beckman  wrote:
> 

Re: Centurylink having a bad morning?

[David Hubbard]

I just brought one of my sessions back up to attempt to avoid the blackholing, 
should be full feed, getting all of 850 v4 routes and 106 v6.

From: NANOG  on behalf 
of Tomas Lynch 
Date: Sunday, August 30, 2020 at 9:41 AM
To: Drew Weaver 
Cc: "nanog@nanog.org" 
Subject: Re: Centurylink having a bad morning?

Flapping in Miami, Dallas, Atlanta, Los Angeles, Seattle and San Jose. It is 
also affecting some data centers in Europe too. but haven't seen flaps there, 
just suboptimal routing.

On Sun, Aug 30, 2020 at 8:53 AM Drew Weaver 
mailto:drew.wea...@thenap.com>> wrote:
Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…

From: Tomas Lynch mailto:tomas.ly...@gmail.com>>
Sent: Sunday, August 30, 2020 8:45 AM
To: Mel Beckman mailto:m...@beckman.org>>
Cc: Drew Weaver mailto:drew.wea...@thenap.com>>; 
nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

BGP sessions randomly flapping or having routing issues in different cities 
since ~5AM EST

On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman 
mailto:m...@beckman.org>> wrote:
The CL portal loads for me, and I can log in, but it is slower than usual. Not 
seeing traffic issues on our CL circuits.
-mel via cell

On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG 
mailto:nanog@nanog.org>> wrote:
Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Joseph Jenkins]

Now if you call into CL you get a message stating their technicians are
working on an ip outage.

On Sun, Aug 30, 2020 at 6:56 AM Chase Christian  wrote:

> Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00
> Pacific:
>
> Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor 4.35.X.Y
> (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes
> Aug 30 03:05:13 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state Established event HoldTime new state Idle
> Aug 30 03:07:37 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state OpenConfirm event RecvKeepAlive new state Established
> Aug 30 03:15:38 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state Established event HoldTime new state Idle
> Aug 30 03:17:15 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state OpenConfirm event RecvKeepAlive new state Established
> Aug 30 03:19:55 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor
> 4.35.X.Y+52091 (proto) 6/7 (Cease/connection collision resolution) 0 bytes
> Aug 30 03:20:11 rtr02 Rib: %BGP-3-NOTIFICATION: received from neighbor
> 4.35.X.Y (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes
> Aug 30 03:20:11 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
> state Established event RecvNotify new state Idle
>
> And incoming traffic from AS3356 and AS209 both dropped to very low
> volumes.
>
> On Sun, Aug 30, 2020 at 5:58 AM Jason Kuehl 
> wrote:
>
>> Well, When I tried calling I got a fast busy, so that's nice.
>>
>> On Sun, Aug 30, 2020 at 8:33 AM David Hubbard <
>> dhubb...@dino.hostasaurus.com> wrote:
>>
>>> Same.  Also, as reported on outages list, what’s even worse is that they
>>> appear to be continuing to propagate advertisements from circuits whose
>>> sessions have been turned down.  I validated ours still were via a couple
>>> looking glass portals.  Down Detector shows nearly every major service
>>> provider impacted.
>>>
>>>
>>>
>>> They’re not reachable so who knows if they’re even working on it.  I
>>> feel like they’ve been cutting heavily on the network ops side in recent
>>> years…
>>>
>>>
>>>
>>> *From: *NANOG 
>>> on behalf of Drew Weaver via NANOG 
>>> *Reply-To: *Drew Weaver 
>>> *Date: *Sunday, August 30, 2020 at 8:23 AM
>>> *To: *"nanog@nanog.org" 
>>> *Subject: *Centurylink having a bad morning?
>>>
>>>
>>>
>>> Hello,
>>>
>>>
>>>
>>> Woke up this morning to a bunch of reports of issues with connectivity
>>> had to shut down some Level3/CTL connections to get it to return to normal.
>>>
>>>
>>>
>>> As of right now their support portal won’t load:
>>> https://www.centurylink.com/business/login/
>>>
>>>
>>>
>>> Just wondering what others are seeing.
>>>
>>>
>>>
>>
>>
>> --
>> Sincerely,
>>
>> Jason W Kuehl
>> Cell 920-419-8983
>> jason.w.ku...@gmail.com
>>
>

RE: Centurylink having a bad morning?

[Drew Weaver]

Something just now changed in this situation and now it seems to have gotten 
worse.

From: Jason Kuehl 
Sent: Sunday, August 30, 2020 10:00 AM
To: Drew Weaver 
Cc: R. Leigh Hennig ; nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

People are rebooting ghosting now.

[cid:image001.png@01D67EB4.7EBAE940]
https://twitter.com/ir_kujoe/status/1300066569645707265

Seeing other reports of this too.

On Sun, Aug 30, 2020 at 9:45 AM Drew Weaver 
mailto:drew.wea...@thenap.com>> wrote:
That site seems to be just for their cloud products, is there one of these for 
their actual network?

From: R. Leigh Hennig mailto:le...@regula.one>>
Sent: Sunday, August 30, 2020 8:54 AM
To: Drew Weaver mailto:drew.wea...@thenap.com>>; 
nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

Global impact with issues reported by Fastly, Cloudflare, OpenDNS.

https://status.ctl.io/

STARTED

Sun Aug 30 2020 08:13 (EDT)
Sun Aug 30 2020 12:13 (UTC)

AFFECTED SERVICES

External Cloud Network (CA3)

DATE
LATEST UPDATE

Sun Aug 30 2020 08:13 (EDT)
Sun Aug 30 2020 12:13 (UTC) Our technical teams are investigating an issue 
affecting some services in the CA3 data center. Ensuring the reliability of our 
services is our top priority. We will continue to provide status updates as 
this incident progresses. If you need further support, please contact us at 
h...@ctl.io.



Sent from ProtonMail Mobile


On Sun, Aug 30, 2020 at 8:14 AM, Drew Weaver via NANOG 
mailto:nanog@nanog.org>> wrote:
Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.





--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

Re: Centurylink having a bad morning?

[Jason Kuehl]

People are rebooting ghosting now.

[image: image.png]
https://twitter.com/ir_kujoe/status/1300066569645707265

Seeing other reports of this too.

On Sun, Aug 30, 2020 at 9:45 AM Drew Weaver  wrote:

> That site seems to be just for their cloud products, is there one of these
> for their actual network?
>
>
>
> *From:* R. Leigh Hennig 
> *Sent:* Sunday, August 30, 2020 8:54 AM
> *To:* Drew Weaver ; nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> Global impact with issues reported by Fastly, Cloudflare, OpenDNS.
>
>
>
> https://status.ctl.io/
>
>
> STARTED
>
> Sun Aug 30 2020 08:13 (EDT)
> Sun Aug 30 2020 12:13 (UTC)
> AFFECTED SERVICES
>
> External Cloud Network (CA3)
> DATE
> LATEST UPDATE
>
> Sun Aug 30 2020 08:13 (EDT)
> Sun Aug 30 2020 12:13 (UTC) Our technical teams are investigating an issue
> affecting some services in the CA3 data center. Ensuring the reliability of
> our services is our top priority. We will continue to provide status
> updates as this incident progresses. If you need further support, please
> contact us at h...@ctl.io.
>
>
>
>
>
>
>
> Sent from ProtonMail Mobile
>
>
>
>
>
> On Sun, Aug 30, 2020 at 8:14 AM, Drew Weaver via NANOG 
> wrote:
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>
>
>
>
>


-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

RE: Centurylink having a bad morning? [EXTERNAL]

[Romeo Czumbil]

Their route-servers just can’t keep up.
They are not getting our community-string  announcements at all


From: NANOG  On Behalf Of 
Drew Weaver
Sent: Sunday, August 30, 2020 8:38 AM
To: 'David Hubbard' ; 'nanog@nanog.org' 

Subject: RE: Centurylink having a bad morning? [EXTERNAL]

Yeah, I am still seeing them announce our Ips even though we’ve shut down our 
sessions with them.


From: NANOG 
mailto:nanog-bounces+drew.weaver=thenap@nanog.org>>
 On Behalf Of David Hubbard
Sent: Sunday, August 30, 2020 8:28 AM
To: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

Same.  Also, as reported on outages list, what’s even worse is that they appear 
to be continuing to propagate advertisements from circuits whose sessions have 
been turned down.  I validated ours still were via a couple looking glass 
portals.  Down Detector shows nearly every major service provider impacted.

They’re not reachable so who knows if they’re even working on it.  I feel like 
they’ve been cutting heavily on the network ops side in recent years…

From: NANOG 
mailto:nanog-bounces+dhubbard=dino.hostasaurus@nanog.org>>
 on behalf of Drew Weaver via NANOG mailto:nanog@nanog.org>>
Reply-To: Drew Weaver mailto:drew.wea...@thenap.com>>
Date: Sunday, August 30, 2020 at 8:23 AM
To: "nanog@nanog.org" 
mailto:nanog@nanog.org>>
Subject: Centurylink having a bad morning?

Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://urldefense.com/v3/__https://www.centurylink.com/business/login/__;!!LG9nLpOADg!HY5bCa8KrN15HDakrv4TyH42jTlzeKVruHiQ0XUOnXr76N501izj2Mea7P2E$
 


Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Antonios Chariton]

Reporting from Europe, any IP with them in the path is unreachable from various 
providers. I guess they wanted to try IPv6-only.. :P IPv6 is fine, working 
fine, IPv4 not at all..

Antonis 

> On 30 Aug 2020, at 14:58, Tomas Lynch  wrote:
> 
> Flapping in Miami, Dallas, Atlanta, Los Angeles, Seattle and San Jose. It is 
> also affecting some data centers in Europe too. but haven't seen flaps there, 
> just suboptimal routing.
> 
> On Sun, Aug 30, 2020 at 8:53 AM Drew Weaver  wrote:
> Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…
> 
>  
> 
> From: Tomas Lynch  
> Sent: Sunday, August 30, 2020 8:45 AM
> To: Mel Beckman 
> Cc: Drew Weaver ; nanog@nanog.org
> Subject: Re: Centurylink having a bad morning?
> 
>  
> 
> BGP sessions randomly flapping or having routing issues in different cities 
> since ~5AM EST
> 
>  
> 
> On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman  wrote:
> 
> The CL portal loads for me, and I can log in, but it is slower than usual. 
> Not seeing traffic issues on our CL circuits. 
> 
> -mel via cell
> 
> 
> 
> 
> On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG  wrote:
> 
> 
> 
> Hello,
> 
>  
> 
> Woke up this morning to a bunch of reports of issues with connectivity had to 
> shut down some Level3/CTL connections to get it to return to normal.
> 
>  
> 
> As of right now their support portal won’t load: 
> https://www.centurylink.com/business/login/
> 
>  
> 
> Just wondering what others are seeing.
> 
>  
> 

Re: Centurylink having a bad morning?

[Baldur Norddahl]

How is that acceptable behaviour? I shall remember never to make a contract
with these guys until they can prove that they won't advertise my prefixes
after I pull them. Under any circumstances.

søn. 30. aug. 2020 15.14 skrev Joseph Jenkins :

> Finally got through on their support line and spoke to level1. The only
> thing the tech could say was it was an issue with BGP route reflectors and
> it started about 3am(pacific). They were still trying to isolate the issue.
> I've tried failing over my circuits and no go, the traffic just dies as L3
> won't stop advertising my routes.
>
> On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
> wrote:
>
>> Hello,
>>
>>
>>
>> Woke up this morning to a bunch of reports of issues with connectivity
>> had to shut down some Level3/CTL connections to get it to return to normal.
>>
>>
>>
>> As of right now their support portal won’t load:
>> https://www.centurylink.com/business/login/
>>
>>
>>
>> Just wondering what others are seeing.
>>
>>
>>
>

RE: Centurylink having a bad morning?

[Drew Weaver]

I don’t think it is anywhere near back to normal but I can get to 2-3 more 
sites than I could when I got called in.

From: NANOG  On Behalf Of Andy 
Brezinsky
Sent: Sunday, August 30, 2020 8:48 AM
To: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?


Started about 5:05am central, started clearing up for me about 7:15am.   My 
route from ATT in Chicago is still going through NYC to get back to Chicago but 
at least packet loss isn't 70-100% anymore.

I also tried turning down sessions and still was seeing stale announcements on 
other LGs.

On 08/30/2020 07:27 AM, David Hubbard wrote:
Same.  Also, as reported on outages list, what’s even worse is that they appear 
to be continuing to propagate advertisements from circuits whose sessions have 
been turned down.  I validated ours still were via a couple looking glass 
portals.  Down Detector shows nearly every major service provider impacted.

They’re not reachable so who knows if they’re even working on it.  I feel like 
they’ve been cutting heavily on the network ops side in recent years…

From: NANOG 

 on behalf of Drew Weaver via NANOG 
Reply-To: Drew Weaver 
Date: Sunday, August 30, 2020 at 8:23 AM
To: "nanog@nanog.org" 

Subject: Centurylink having a bad morning?

Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Jason Kuehl]

I'm over in MA in a CL building, it's very much still broken. I shut down
the interfaces to CL and now just using Comcast.

On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky  wrote:

> Started about 5:05am central, started clearing up for me about 7:15am.
> My route from ATT in Chicago is still going through NYC to get back to
> Chicago but at least packet loss isn't 70-100% anymore.
>
> I also tried turning down sessions and still was seeing stale
> announcements on other LGs.
>
> On 08/30/2020 07:27 AM, David Hubbard wrote:
>
> Same.  Also, as reported on outages list, what’s even worse is that they
> appear to be continuing to propagate advertisements from circuits whose
> sessions have been turned down.  I validated ours still were via a couple
> looking glass portals.  Down Detector shows nearly every major service
> provider impacted.
>
>
>
> They’re not reachable so who knows if they’re even working on it.  I feel
> like they’ve been cutting heavily on the network ops side in recent years…
>
>
>
> *From: *NANOG 
>  on behalf of Drew
> Weaver via NANOG  
> *Reply-To: *Drew Weaver  
> *Date: *Sunday, August 30, 2020 at 8:23 AM
> *To: *"nanog@nanog.org"  
> 
> *Subject: *Centurylink having a bad morning?
>
>
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>
>
>

-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

Re: Centurylink having a bad morning?

[Tom Beecher]

>
> They’re not reachable so who knows if they’re even working on it.
>

Gonna go out on a limb here and assume that a lot of phones were ringing
and people are in fact working on whatever it is. :)

On Sun, Aug 30, 2020 at 8:34 AM David Hubbard 
wrote:

> Same.  Also, as reported on outages list, what’s even worse is that they
> appear to be continuing to propagate advertisements from circuits whose
> sessions have been turned down.  I validated ours still were via a couple
> looking glass portals.  Down Detector shows nearly every major service
> provider impacted.
>
>
>
> They’re not reachable so who knows if they’re even working on it.  I feel
> like they’ve been cutting heavily on the network ops side in recent years…
>
>
>
> *From: *NANOG  on
> behalf of Drew Weaver via NANOG 
> *Reply-To: *Drew Weaver 
> *Date: *Sunday, August 30, 2020 at 8:23 AM
> *To: *"nanog@nanog.org" 
> *Subject: *Centurylink having a bad morning?
>
>
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>

RE: Centurylink having a bad morning?

[Ben Russell]

One of my legacy L3 peers is only advertising 498174 prefixes.  Very strange.



Ben Russell  



-Original Message-
From: NANOG  On Behalf Of Drew 
Weaver
Sent: Sunday, August 30, 2020 7:45 AM
To: 'Mel Beckman' 
Cc: 'nanog@nanog.org' 
Subject: RE: Centurylink having a bad morning?

Can't get to pretty much anything on the Internet from my Tmobile device... not 
really sure whats going on =)

-Original Message-
From: Mel Beckman  
Sent: Sunday, August 30, 2020 8:43 AM
To: Drew Weaver 
Cc: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

One anomaly I’ve just notice: we can’t reach 209.244.0.1 (ns1.level3.com) from 
anywhere on the Internet, including our CL connections. Traceroute fails at 
4.16.30.129 (lag-101.ear2.LosAngeles1.Level3.net) with 65% packet loss:

traceroute to 209.244.0.1 (209.244.0.1), 30 hops max, 56 byte packets
 1  sent:79 loss:100% last:0 ms avg:0 ms
 2  sent:79 loss:0% last:15.483 ms avg:28.071 ms
172.102.107.166
 3  sent:79 loss:2.5% last:140.81 ms avg:33.374 ms
ae8---0.scr02.lsan.ca.frontiernet.net (74.40.3.49)
 4  sent:78 loss:0% last:11.695 ms avg:23.637 ms
ae1---0.cbr01.lsan.ca.frontiernet.net (74.40.3.214)
 5  sent:78 loss:65.4% last:13.844 ms avg:19.393 ms
lag-101.ear2.LosAngeles1.Level3.net (4.16.30.129)
 6  sent:78 loss:100% last:0 ms avg:0 ms
 7  sent:78 loss:100% last:0 ms avg:0 ms
 8  sent:78 loss:100% last:0 ms avg:0 ms
 9  sent:78 loss:100% last:0 ms avg:0 ms
10  sent:78 loss:100% last:0 ms avg:0 ms




 -mel beckman

> On Aug 30, 2020, at 5:36 AM, Mel Beckman  wrote:
> 

Re: Centurylink having a bad morning?

[Robert Blayzor]

On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote:
> Woke up this morning to a bunch of reports of issues with connectivity
> had to shut down some Level3/CTL connections to get it to return to normal.



Just to confirm we're seeing this on AS3356 and not AS209, correct?


We have links to both and shut down AS3356 which seems to have cleared
"most" of the problems.


-- 
inoc.net!rblayzor
XMPP: rblayzor.AT.inoc.net
PGP:  https://pgp.inoc.net/rblayzor/

Re: Centurylink having a bad morning?

[Chase Christian]

Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00
Pacific:

Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor 4.35.X.Y
(AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes
Aug 30 03:05:13 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
state Established event HoldTime new state Idle
Aug 30 03:07:37 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
state OpenConfirm event RecvKeepAlive new state Established
Aug 30 03:15:38 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
state Established event HoldTime new state Idle
Aug 30 03:17:15 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
state OpenConfirm event RecvKeepAlive new state Established
Aug 30 03:19:55 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor
4.35.X.Y+52091 (proto) 6/7 (Cease/connection collision resolution) 0 bytes
Aug 30 03:20:11 rtr02 Rib: %BGP-3-NOTIFICATION: received from neighbor
4.35.X.Y (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes
Aug 30 03:20:11 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
state Established event RecvNotify new state Idle

And incoming traffic from AS3356 and AS209 both dropped to very low volumes.

On Sun, Aug 30, 2020 at 5:58 AM Jason Kuehl  wrote:

> Well, When I tried calling I got a fast busy, so that's nice.
>
> On Sun, Aug 30, 2020 at 8:33 AM David Hubbard <
> dhubb...@dino.hostasaurus.com> wrote:
>
>> Same.  Also, as reported on outages list, what’s even worse is that they
>> appear to be continuing to propagate advertisements from circuits whose
>> sessions have been turned down.  I validated ours still were via a couple
>> looking glass portals.  Down Detector shows nearly every major service
>> provider impacted.
>>
>>
>>
>> They’re not reachable so who knows if they’re even working on it.  I feel
>> like they’ve been cutting heavily on the network ops side in recent years…
>>
>>
>>
>> *From: *NANOG  on
>> behalf of Drew Weaver via NANOG 
>> *Reply-To: *Drew Weaver 
>> *Date: *Sunday, August 30, 2020 at 8:23 AM
>> *To: *"nanog@nanog.org" 
>> *Subject: *Centurylink having a bad morning?
>>
>>
>>
>> Hello,
>>
>>
>>
>> Woke up this morning to a bunch of reports of issues with connectivity
>> had to shut down some Level3/CTL connections to get it to return to normal.
>>
>>
>>
>> As of right now their support portal won’t load:
>> https://www.centurylink.com/business/login/
>>
>>
>>
>> Just wondering what others are seeing.
>>
>>
>>
>
>
> --
> Sincerely,
>
> Jason W Kuehl
> Cell 920-419-8983
> jason.w.ku...@gmail.com
>

Re: Centurylink having a bad morning?

[K. Scott Helms]

We've been on hold for more than an hour trying to get an update.

We see the same behavior where they continue to announce our blocks
despite all the interfaces to them being hard down.

Scott Helms


On Sun, Aug 30, 2020 at 8:58 AM Jason Kuehl  wrote:
>
> Well, When I tried calling I got a fast busy, so that's nice.
>
> On Sun, Aug 30, 2020 at 8:33 AM David Hubbard  
> wrote:
>>
>> Same.  Also, as reported on outages list, what’s even worse is that they 
>> appear to be continuing to propagate advertisements from circuits whose 
>> sessions have been turned down.  I validated ours still were via a couple 
>> looking glass portals.  Down Detector shows nearly every major service 
>> provider impacted.
>>
>>
>>
>> They’re not reachable so who knows if they’re even working on it.  I feel 
>> like they’ve been cutting heavily on the network ops side in recent years…
>>
>>
>>
>> From: NANOG  on 
>> behalf of Drew Weaver via NANOG 
>> Reply-To: Drew Weaver 
>> Date: Sunday, August 30, 2020 at 8:23 AM
>> To: "nanog@nanog.org" 
>> Subject: Centurylink having a bad morning?
>>
>>
>>
>> Hello,
>>
>>
>>
>> Woke up this morning to a bunch of reports of issues with connectivity had 
>> to shut down some Level3/CTL connections to get it to return to normal.
>>
>>
>>
>> As of right now their support portal won’t load: 
>> https://www.centurylink.com/business/login/
>>
>>
>>
>> Just wondering what others are seeing.
>>
>>
>
>
>
> --
> Sincerely,
>
> Jason W Kuehl
> Cell 920-419-8983
> jason.w.ku...@gmail.com

RE: Centurylink having a bad morning?

[Drew Weaver]

That site seems to be just for their cloud products, is there one of these for 
their actual network?

From: R. Leigh Hennig 
Sent: Sunday, August 30, 2020 8:54 AM
To: Drew Weaver ; nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

Global impact with issues reported by Fastly, Cloudflare, OpenDNS.

https://status.ctl.io/

STARTED

Sun Aug 30 2020 08:13 (EDT)
Sun Aug 30 2020 12:13 (UTC)

AFFECTED SERVICES

External Cloud Network (CA3)

DATE
LATEST UPDATE

Sun Aug 30 2020 08:13 (EDT)
Sun Aug 30 2020 12:13 (UTC) Our technical teams are investigating an issue 
affecting some services in the CA3 data center. Ensuring the reliability of our 
services is our top priority. We will continue to provide status updates as 
this incident progresses. If you need further support, please contact us at 
h...@ctl.io.



Sent from ProtonMail Mobile


On Sun, Aug 30, 2020 at 8:14 AM, Drew Weaver via NANOG 
mailto:nanog@nanog.org>> wrote:
Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Tomas Lynch]

Flapping in Miami, Dallas, Atlanta, Los Angeles, Seattle and San Jose. It
is also affecting some data centers in Europe too. but haven't seen flaps
there, just suboptimal routing.

On Sun, Aug 30, 2020 at 8:53 AM Drew Weaver  wrote:

> Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…
>
>
>
> *From:* Tomas Lynch 
> *Sent:* Sunday, August 30, 2020 8:45 AM
> *To:* Mel Beckman 
> *Cc:* Drew Weaver ; nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> BGP sessions randomly flapping or having routing issues in different
> cities since ~5AM EST
>
>
>
> On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman  wrote:
>
> The CL portal loads for me, and I can log in, but it is slower than usual.
> Not seeing traffic issues on our CL circuits.
>
> -mel via cell
>
>
>
> On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG 
> wrote:
>
> 
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>
>

RE: Centurylink having a bad morning?

[Drew Weaver]

Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…

From: Tomas Lynch 
Sent: Sunday, August 30, 2020 8:45 AM
To: Mel Beckman 
Cc: Drew Weaver ; nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

BGP sessions randomly flapping or having routing issues in different cities 
since ~5AM EST

On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman 
mailto:m...@beckman.org>> wrote:
The CL portal loads for me, and I can log in, but it is slower than usual. Not 
seeing traffic issues on our CL circuits.
-mel via cell


On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG 
mailto:nanog@nanog.org>> wrote:

Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Mitchell Lewis]

Looks like this is causing problems with Meraki Cloud connectivity since level3 
is a transit provider for Meraki.



From: NANOG  on behalf of Mel 
Beckman 
Sent: Sunday, August 30, 2020, 08:47
To: Drew Weaver
Cc: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

I have checked all of our circuits in the LA area and are seeing no BGP issues, 
but we don’t provide transit for CL, only use them for DIA. Traffic flows 
appear normal, but early Sunday morning is normally the lowest time of the week.

 -mel beckman

> On Aug 30, 2020, at 5:29 AM, Mel Beckman  wrote:
>

Re: Centurylink having a bad morning?

[Tom Beecher]

I am seeing some odd traffic deflections in the EU via 3356, but nothing in
the US so far.

Some scattershot oddball reports landing in our NOC, but nothing
conclusive.

On Sun, Aug 30, 2020 at 8:41 AM Mel Beckman  wrote:

> The CL portal loads for me, and I can log in, but it is slower than usual.
> Not seeing traffic issues on our CL circuits.
>
> -mel via cell
>
> On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG 
> wrote:
>
> 
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>
>

Re: Centurylink having a bad morning?

[Andy Brezinsky]

Started about 5:05am central, started clearing up for me about 7:15am.   
My route from ATT in Chicago is still going through NYC to get back to 
Chicago but at least packet loss isn't 70-100% anymore.


I also tried turning down sessions and still was seeing stale 
announcements on other LGs.



On 08/30/2020 07:27 AM, David Hubbard wrote:


Same.  Also, as reported on outages list, what’s even worse is that 
they appear to be continuing to propagate advertisements from circuits 
whose sessions have been turned down.  I validated ours still were via 
a couple looking glass portals.  Down Detector shows nearly every 
major service provider impacted.


They’re not reachable so who knows if they’re even working on it.  I 
feel like they’ve been cutting heavily on the network ops side in 
recent years…


*From: *NANOG  
on behalf of Drew Weaver via NANOG 

*Reply-To: *Drew Weaver 
*Date: *Sunday, August 30, 2020 at 8:23 AM
*To: *"nanog@nanog.org" 
*Subject: *Centurylink having a bad morning?

Hello,

Woke up this morning to a bunch of reports of issues with connectivity 
had to shut down some Level3/CTL connections to get it to return to 
normal.


As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/ 



Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Joseph Jenkins]

Finally got through on their support line and spoke to level1. The only
thing the tech could say was it was an issue with BGP route reflectors and
it started about 3am(pacific). They were still trying to isolate the issue.
I've tried failing over my circuits and no go, the traffic just dies as L3
won't stop advertising my routes.

On Sun, Aug 30, 2020 at 5:21 AM Drew Weaver via NANOG 
wrote:

> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>

RE: Centurylink having a bad morning?

[Drew Weaver]

Can't get to pretty much anything on the Internet from my Tmobile device... not 
really sure whats going on =)

-Original Message-
From: Mel Beckman  
Sent: Sunday, August 30, 2020 8:43 AM
To: Drew Weaver 
Cc: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

One anomaly I’ve just notice: we can’t reach 209.244.0.1 (ns1.level3.com) from 
anywhere on the Internet, including our CL connections. Traceroute fails at 
4.16.30.129 (lag-101.ear2.LosAngeles1.Level3.net) with 65% packet loss:

traceroute to 209.244.0.1 (209.244.0.1), 30 hops max, 56 byte packets
 1  sent:79 loss:100% last:0 ms avg:0 ms
 2  sent:79 loss:0% last:15.483 ms avg:28.071 ms
172.102.107.166
 3  sent:79 loss:2.5% last:140.81 ms avg:33.374 ms
ae8---0.scr02.lsan.ca.frontiernet.net (74.40.3.49)
 4  sent:78 loss:0% last:11.695 ms avg:23.637 ms
ae1---0.cbr01.lsan.ca.frontiernet.net (74.40.3.214)
 5  sent:78 loss:65.4% last:13.844 ms avg:19.393 ms
lag-101.ear2.LosAngeles1.Level3.net (4.16.30.129)
 6  sent:78 loss:100% last:0 ms avg:0 ms
 7  sent:78 loss:100% last:0 ms avg:0 ms
 8  sent:78 loss:100% last:0 ms avg:0 ms
 9  sent:78 loss:100% last:0 ms avg:0 ms
10  sent:78 loss:100% last:0 ms avg:0 ms




 -mel beckman

> On Aug 30, 2020, at 5:36 AM, Mel Beckman  wrote:
> 

Re: Centurylink having a bad morning?

[Tomas Lynch]

BGP sessions randomly flapping or having routing issues in different cities
since ~5AM EST

On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman  wrote:

> The CL portal loads for me, and I can log in, but it is slower than usual.
> Not seeing traffic issues on our CL circuits.
>
> -mel via cell
>
> On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG 
> wrote:
>
> 
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>
>

Re: Centurylink having a bad morning?

[Mel Beckman]

One anomaly I’ve just notice: we can’t reach 209.244.0.1 (ns1.level3.com) from 
anywhere on the Internet, including our CL connections. Traceroute fails at 
4.16.30.129 (lag-101.ear2.LosAngeles1.Level3.net) with 65% packet loss:

traceroute to 209.244.0.1 (209.244.0.1), 30 hops max, 56 byte packets
 1  sent:79 loss:100% last:0 ms avg:0 ms
 2  sent:79 loss:0% last:15.483 ms avg:28.071 ms
172.102.107.166
 3  sent:79 loss:2.5% last:140.81 ms avg:33.374 ms
ae8---0.scr02.lsan.ca.frontiernet.net (74.40.3.49)
 4  sent:78 loss:0% last:11.695 ms avg:23.637 ms
ae1---0.cbr01.lsan.ca.frontiernet.net (74.40.3.214)
 5  sent:78 loss:65.4% last:13.844 ms avg:19.393 ms
lag-101.ear2.LosAngeles1.Level3.net (4.16.30.129)
 6  sent:78 loss:100% last:0 ms avg:0 ms
 7  sent:78 loss:100% last:0 ms avg:0 ms
 8  sent:78 loss:100% last:0 ms avg:0 ms
 9  sent:78 loss:100% last:0 ms avg:0 ms
10  sent:78 loss:100% last:0 ms avg:0 ms




 -mel beckman

> On Aug 30, 2020, at 5:36 AM, Mel Beckman  wrote:
> 

Re: Centurylink having a bad morning?

[Jason Kuehl]

Well, When I tried calling I got a fast busy, so that's nice.

On Sun, Aug 30, 2020 at 8:33 AM David Hubbard 
wrote:

> Same.  Also, as reported on outages list, what’s even worse is that they
> appear to be continuing to propagate advertisements from circuits whose
> sessions have been turned down.  I validated ours still were via a couple
> looking glass portals.  Down Detector shows nearly every major service
> provider impacted.
>
>
>
> They’re not reachable so who knows if they’re even working on it.  I feel
> like they’ve been cutting heavily on the network ops side in recent years…
>
>
>
> *From: *NANOG  on
> behalf of Drew Weaver via NANOG 
> *Reply-To: *Drew Weaver 
> *Date: *Sunday, August 30, 2020 at 8:23 AM
> *To: *"nanog@nanog.org" 
> *Subject: *Centurylink having a bad morning?
>
>
>
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>


-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

RE: Centurylink having a bad morning?

[Drew Weaver]

Yeah, I am still seeing them announce our Ips even though we’ve shut down our 
sessions with them.


From: NANOG  On Behalf Of David 
Hubbard
Sent: Sunday, August 30, 2020 8:28 AM
To: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

Same.  Also, as reported on outages list, what’s even worse is that they appear 
to be continuing to propagate advertisements from circuits whose sessions have 
been turned down.  I validated ours still were via a couple looking glass 
portals.  Down Detector shows nearly every major service provider impacted.

They’re not reachable so who knows if they’re even working on it.  I feel like 
they’ve been cutting heavily on the network ops side in recent years…

From: NANOG 
mailto:nanog-bounces+dhubbard=dino.hostasaurus@nanog.org>>
 on behalf of Drew Weaver via NANOG mailto:nanog@nanog.org>>
Reply-To: Drew Weaver mailto:drew.wea...@thenap.com>>
Date: Sunday, August 30, 2020 at 8:23 AM
To: "nanog@nanog.org" 
mailto:nanog@nanog.org>>
Subject: Centurylink having a bad morning?

Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Mel Beckman]

I have checked all of our circuits in the LA area and are seeing no BGP issues, 
but we don’t provide transit for CL, only use them for DIA. Traffic flows 
appear normal, but early Sunday morning is normally the lowest time of the 
week. 

 -mel beckman

> On Aug 30, 2020, at 5:29 AM, Mel Beckman  wrote:
> 

Re: Centurylink having a bad morning?

[Mel Beckman]

The CL portal loads for me, and I can log in, but it is slower than usual. Not 
seeing traffic issues on our CL circuits.

-mel via cell

On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG  wrote:


Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[David Hubbard]

Same.  Also, as reported on outages list, what’s even worse is that they appear 
to be continuing to propagate advertisements from circuits whose sessions have 
been turned down.  I validated ours still were via a couple looking glass 
portals.  Down Detector shows nearly every major service provider impacted.

They’re not reachable so who knows if they’re even working on it.  I feel like 
they’ve been cutting heavily on the network ops side in recent years…

From: NANOG  on behalf 
of Drew Weaver via NANOG 
Reply-To: Drew Weaver 
Date: Sunday, August 30, 2020 at 8:23 AM
To: "nanog@nanog.org" 
Subject: Centurylink having a bad morning?

Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[Jason Kuehl]

Its dead JIM. I also can't get in starting around 7 est

On Sun, Aug 30, 2020 at 8:19 AM Drew Weaver via NANOG 
wrote:

> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As of right now their support portal won’t load:
> https://www.centurylink.com/business/login/
>
>
>
> Just wondering what others are seeing.
>
>
>


-- 
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.ku...@gmail.com

Centurylink having a bad morning?

[Drew Weaver via NANOG]

Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won't load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Ipv6 help

[JORDI PALET MARTINEZ via NANOG]

In many situations it will make sense to keep the CPE provided by the ISP in a 
configuration equivalent to a "bridge" (I know is not a bridge, I'm trying to 
use a single word to describe it), so it runs things like the NAT for IPv4 and 
the CLAT for IPv6, even DHCP, or RA, etc. It all depends on what you have 
behind. Many customers will just have one or two routers acting just as APs in 
a single bridged domain, others may have different subnets, etc.

This keeps the customer happy if they need to have a better WiFi, or whatever.

If you have just a dozen of deployments per month, I think the choice to buy a 
good Chinese hardware that already comes with OpenWRT, or your preferred retail 
CPE that allows to be reflashed with OpenWRT, is handy. That system will work 
for most of the customers, and if some customer want a "bigger" system, just 
provide the low cost CPE as a "bridge" to its real network.
 
 

El 30/8/20 3:05, "NANOG en nombre de Brandon Martin" 
 escribió:

On 8/26/20 12:48 PM, JORDI PALET MARTINEZ via NANOG wrote:
> I work and I'm in touch with many CPE vendors since long time ago ... 
many are on the way (I can remember about 12 on top of my head right now, but 
because contracts, can't name them). It takes time. However, in many cases, 
they just do for specific customers or specific models. I know other people 
that contacted the same vendors and they told them "we could do it for the 
model you use as well". In some cases, they require a minimum volume per year 
(less than what you could expect. I've seen cases that start with just 500 
units per month).
> 
> But this only works if you contact them. The CPE vendors business model 
seems to be very "ISP" direct. I think the retail marked models, unfortunately, 
will take a bit more time.
> 
> A hint about some vendors: You may take a look at the co-authors in the 
RFC.

The whole point here is not that some vendors can support these features 
in a semi-custom firmware image that's specific to a particular ISP 
deployment - I know that's a possibility and have vendors willing to 
work with me on a much smaller volume than even what you've indicated - 
but rather what about customers who want to "upgrade" their router or, 
for whatever reason (and there are some very valid ones) want to provide 
their own.

Right now, it's essentially impossible for them to walk into a local 
retailer and walk out with a model that they know for sure will work 
with my network if I'm requiring e.g. 464XLAT for basic functionality. 
Even if they buy online and dive fairly deep into model documentation, 
it is still hard to come up with a model that they know will work, and 
the lack of documentation will limit their choice of models 
unnecessarily (i.e. there are probably some options that support the 
necessary functionality but don't advertise it in the slightest).

If someone wants me to provide them with a router, then of course I'll 
hand them one that I know works on my network.  That's easy since I have 
control over the selection of it (and have presumably done some testing) 
even if I don't have my own provider-customized firmware.

I'll point out that 500 new services (taking a new router) per month is 
not a particularly small provider.  It's not large, no, but it's also a 
lot bigger than many deployments are at least when they're new, and 
these are questions you have to essentially answer "up front" in many cases.
-- 
Brandon Martin



**
IPv4 is over
Are you ready for the new Internet ?
http://www.theipv6company.com
The IPv6 Company

This electronic message contains information which may be privileged or 
confidential. The information is intended to be for the exclusive use of the 
individual(s) named above and further non-explicilty authorized disclosure, 
copying, distribution or use of the contents of this information, even if 
partially, including attached files, is strictly prohibited and will be 
considered a criminal offense. If you are not the intended recipient be aware 
that any disclosure, copying, distribution or use of the contents of this 
information, even if partially, including attached files, is strictly 
prohibited, will be considered a criminal offense, so you must reply to the 
original sender to inform about this communication and delete it.