Re: Centurylink having a bad morning?

On 5/Sep/20 18:46, Randy Bush via NANOG wrote: > fracking dmark illegal header hacking :( > > apologies No worries :-). Mark.

Re: Centurylink having a bad morning?

On 5/Sep/20 17:58, Randy Bush via NANOG wrote: > spoken like a tier two Even "Tier 1's" aren't present everywhere :-)... Although I'm not sure whether having presence in every city means you are a Tier-anything either... Mark.

Re: Centurylink having a bad morning?

*From: *"Mark Tinka" *To: *nanog@nanog.org *Sent: *Thursday, September 3, 2020 1:35:46 PM *Subject: *Re: Centurylink having a bad morning? On 31/Aug/20 17:57, Bryan Holloway wrote: > Not everyone will peer with you, notably, AS3356 (unle

Re: Centurylink having a bad morning?

> [ off list ] fracking dmark illegal header hacking :( apologies

Re: Centurylink having a bad morning?

[ off list ] >> Oh, yes! Let's not start another "what's a tier one" war! > > Oh no, let's :-). > > We get over here in Africa as well. Local operators either calling > themselves Tier 1, or being called a Tier 1. Nonsensical. > > Years back, our Marketing team asked me to comment on the use

Re: Centurylink having a bad morning?

From: "Mark Tinka via NANOG" To: "Tomas Lynch" Cc: "NANOG" Sent: Saturday, September 5, 2020 5:26:21 AM Subject: Re: Centurylink having a bad morning? On 4/Sep/20 23:41, Tomas Lynch wrote: Oh, yes! Let's not start another "what's a tier one&

Re: Centurylink having a bad morning?

1:35:46 PM Subject: Re: Centurylink having a bad morning? On 31/Aug/20 17:57, Bryan Holloway wrote: > Not everyone will peer with you, notably, AS3356 (unless you're big > enough, which few can say.) I think Tomas meant more diverse peering, not peering with CL. Mark.

Re: Centurylink having a bad morning?

On 4/Sep/20 23:41, Tomas Lynch wrote: > > Oh, yes! Let's not start another "what's a tier one" war! Oh no, let's :-). We get over here in Africa as well. Local operators either calling themselves Tier 1, or being called a Tier 1. Nonsensical. Years back, our Marketing team asked me to

Re: Centurylink having a bad morning?

On Thu, Sep 3, 2020 at 2:37 PM Mark Tinka wrote: > > > On 31/Aug/20 17:57, Bryan Holloway wrote: > > > Not everyone will peer with you, notably, AS3356 (unless you're big > > enough, which few can say.) > > I think Tomas meant more diverse peering, not peering with CL. > Oh, yes! Let's not

Re: Centurylink having a bad morning?

On 31/Aug/20 17:57, Bryan Holloway wrote: > Not everyone will peer with you, notably, AS3356 (unless you're big > enough, which few can say.) I think Tomas meant more diverse peering, not peering with CL. Mark.

Re: Centurylink having a bad morning?

On Wed, Sep 2, 2020 at 11:02, Warren Kumari wrote: >> well, what you REALLY need is one of these: >> https://www.cru-inc.com/products/wiebetech/hotplug_field_kit_product/ >> > > Yeah, no... actually, hell no! > > That setup scares me, and I'm surprised that it can be sold at all, > even with

Re: Centurylink having a bad morning?

On 2/Sep/20 05:53, Alain Hebert wrote: > >     Beat installing a Cisco 12k solo with 2x4's to align the mounting > holes... I still have those in a rack somewhere, trapping air, if you want to test that :-)... Mark.

Re: Centurylink having a bad morning?

On 31/Aug/20 16:33, Tomas Lynch wrote: > Maybe we are idealizing these so-called tier-1 carriers and we, > tier-ns, should treat them as what they really are: another AS. Accept > that they are going to fail and do our best to mitigate the impact on > our own networks, i.e. more peering.

Re: Centurylink having a bad morning?

- > > *De :* Shawn L via NANOG > > *Envoyé :* mercredi 2 septembre 2020 13:15 > > *À :* nanog > > *Objet :* Re: Centurylink having a bad morning? > > > > We once moved a 3u server 30 miles between data centers this way. Plug > > redu

Re: Centurylink having a bad morning?

That is what the 5G router is for... ons. 2. sep. 2020 19.47 skrev Michael Hallgren : > While conserving connectivity?  > > > -- > *De :* Shawn L via NANOG > *Envoyé :* mercredi 2 septembre 2020 13:15 > *À :* nanog > *Objet :* Re: Centuryl

Re: Centurylink having a bad morning?

While conserving connectivity?  De : Shawn L via NANOG Envoyé : mercredi 2 septembre 2020 13:15 À : nanog Objet : Re: Centurylink having a bad morning? We once moved a 3u server 30 miles between data centers this way.  Plug redundant psu into a ups and 2

Re: Centurylink having a bad morning?

On Wed, Sep 2, 2020 at 12:00 AM Christopher Morrow wrote: > > On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert wrote: > > > > As a coincidence... I was *thinking* of moving a 90TB SAN (with > > mechanical's) to another rack that way... skateboard, long fibers and long > > power cords =D > >

Re: Centurylink having a bad morning?

If the client pays me a shit ton of money to make sure the server won't turn off, and they pay for the hardware to make it happen. I;d think about it. It's a like a colo move on hardmode. Its extremely stupid, and I would advise not doing it. Hell even when I migrated e911 server, we had a 20

Re: Centurylink having a bad morning?

On 9/2/20 1:49 PM, Nick Hilliard wrote: Shawn L via NANOG wrote on 02/09/2020 12:15: We once moved a 3u server 30 miles between data centers this way. Plug redundant psu into a ups and 2 people carried it out and put them in a vehicle. hopefully none of these server moves that people have

Re: Centurylink having a bad morning?

Shawn L via NANOG wrote on 02/09/2020 12:15: We once moved a 3u server 30 miles between data centers this way. Plug redundant psu into a ups and 2 people carried it out and put them in a vehicle. hopefully none of these server moves that people have been talking about involved spinning disks.

Re: Centurylink having a bad morning?

We once moved a 3u server 30 miles between data centers this way. Plug redundant psu into a ups and 2 people carried it out and put them in a vehicle. Sent from my iPhone > On Sep 1, 2020, at 11:58 PM, Christopher Morrow > wrote: > > On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert wrote:

Re: Centurylink having a bad morning?

On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert wrote: > > As a coincidence... I was *thinking* of moving a 90TB SAN (with > mechanical's) to another rack that way... skateboard, long fibers and long > power cords =D > well, what you REALLY need is one of these:

Re: Centurylink having a bad morning?

    As a coincidence...  I was *thinking* of moving a 90TB SAN (with mechanical's) to another rack that way...  skateboard, long fibers and long power cords =D     Beat installing a Cisco 12k solo with 2x4's to align the mounting holes... - Alain Hebert

Re: Centurylink having a bad morning?

design of BGP is such >>> that this should not happen easily if at all. They did something unwise. >>> Did they make a route reflector based on a database or something? >>> >>> Regards, >>> >>> Baldur >>> >>> On Sun, Aug 30,

Re: Centurylink having a bad morning?

On Mon, Aug 31, 2020 at 11:28 PM Bjørn Mork wrote: > Well, many of us are paying for redundant power supplies or redundant > REs, even if that doesn't make any difference when the chassis is on > fire. I guess most people know that, and still buy those redundant > components. > I buy it so I

Re: Centurylink having a bad morning?

Eric Kuhnke writes: > There's a number of enterprise end user type customers of 3356 that have > on-premises server rooms/hosting for their stuff. And they spend a lot of > money every month for a 'redundant' metro ethernet circuit that takes > diverse fiber paths from their business park office

Re: Centurylink having a bad morning?

omer at an outlet who has a system > that will do that. Luckily we do not currently have a contract and now they > will have to convince me it is safe for me to make a contract with them. If > that is impossible I guess I won't be getting a contract with them. > >>> > >>&

Re: Centurylink having a bad morning?

rylink a bit puzzling. >>> It could have just been poorly written. >>> >>> >>> >>> >>> >>> From: NANOG On Behalf Of >>> Tom Beecher >>> Sent: Monday, August 31, 2020 9:26 AM >>> To: Hank Nussbacher &g

Re: Centurylink having a bad morning?

re that don’t touch Centurylink a bit puzzling. It >> could have just been poorly written. >> >> >> >> >> >> From: NANOG On Behalf Of >> Tom Beecher >> Sent: Monday, August 31, 2020 9:26 AM >> To: Hank Nussbacher >> Cc: NANOG

Re: Centurylink having a bad morning?

ossible. They need to make a good >> report telling exactly what went wrong and how they changed the design, so >> something like this can not happen again. The basic design of BGP is such >> that this should not happen easily if at all. They did something unwise. >> Did they

Re: Centurylink having a bad morning?

From: NANOG On Behalf Of >> Tom Beecher >> Sent: Monday, August 31, 2020 9:26 AM >> To: Hank Nussbacher >> Cc: NANOG >> Subject: Re: Centurylink having a bad morning? >> >> >> >> https://blog.cloudflare.com/analysis-of-todays-centurylink

Re: Centurylink having a bad morning?

NOG *On Behalf > Of *Tom Beecher > *Sent:* Monday, August 31, 2020 9:26 AM > *To:* Hank Nussbacher > *Cc:* NANOG > *Subject:* Re: Centurylink having a bad morning? > > > > https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/ > > > > I defi

RE: Centurylink having a bad morning?

. It could have just been poorly written. From: NANOG On Behalf Of Tom Beecher Sent: Monday, August 31, 2020 9:26 AM To: Hank Nussbacher Cc: NANOG Subject: Re: Centurylink having a bad morning? https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/ I definitely found Mr

Re: Centurylink having a bad morning?

intentional. *From:* NANOG mailto:thenap@nanog.org>> *On Behalf Of *Baldur Norddahl *Sent:* Sunday, August 30, 2020 9:28 AM *To:* nanog@nanog.org <mailto:nanog@nanog.org> *Subject:* Re

Re: Centurylink having a bad morning?

possible. They need to make a good >>>> report telling exactly what went wrong and how they changed the design, so >>>> something like this can not happen again. The basic design of BGP is such >>>> that this should not happen easily if at all. They did something unwi

Re: Centurylink having a bad morning?

flector based on a database or something? >>> >>> Regards, >>> >>> Baldur >>> >>> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho >>> wrote: >>> >>>> Exactly. And asking that they somehow prove this won't happen again

Re: Centurylink having a bad morning?

to:drew.wea...@thenap.com>> wrote: I’m not defending them but I am sure it isn’t intentional. From: NANOG mailto:thenap@nanog.org>> On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Centurylink having a

Re: Centurylink having a bad morning?

se or something? >> >> Regards, >> >> Baldur >> >> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho >> wrote: >> >>> Exactly. And asking that they somehow prove this won't happen again is >>> impossible. >>> >>> - Mike Boli

Re: Centurylink having a bad morning?

p@nanog.org>> On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: Centurylink having a bad morning? How is that acceptable behaviour? I shall remember never to make a contract with these guys until they can pr

Re: Centurylink having a bad morning?

gt; I’m not defending them but I am sure it isn’t intentional. >>> >>> >>> >>> *From:* NANOG *On >>> Behalf Of *Baldur Norddahl >>> *Sent:* Sunday, August 30, 2020 9:28 AM >>> *To:* nanog@nanog.org >>> *Subje

Re: Centurylink having a bad morning?

intentional.   From: NANOG thenap@nanog.org> On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org Subject: Re: Centuryli

Re: Centurylink having a bad morning?

On Sun, 30 Aug 2020 at 20:00, Baldur Norddahl wrote: > Not really the point. BGP is designed such that if I take down the link, the > prefixes MUST be withdrawn within reasonable time. The self healing aspect of > the internet entirely depends on this. Clearly they have some kind of system >

Re: Centurylink having a bad morning?

Drew Weaver wrote: >> >>> I’m not defending them but I am sure it isn’t intentional. >>> >>> >>> >>> *From:* NANOG *On >>> Behalf Of *Baldur Norddahl >>> *Sent:* Sunday, August 30, 2020 9:28 AM >>> *To:* nanog@nanog.org

Re: Centurylink having a bad morning? [EXTERNAL]

On 30/08/2020 18:22, Joseph Jenkins wrote: Well at least it looks like the issue is starting to resolve  and stuff is coming back up. On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes

Re: Centurylink having a bad morning?

sn’t intentional. >>> >>> >>> >>> From: NANOG On Behalf Of >>> Baldur Norddahl >>> Sent: Sunday, August 30, 2020 9:28 AM >>> To: nanog@nanog.org >>> Subject: Re: Centurylink having a bad morning? >>> >>>

Re: Centurylink having a bad morning?

2020, 8:10 AM Drew Weaver wrote: > >> I’m not defending them but I am sure it isn’t intentional. >> >> >> >> *From:* NANOG *On >> Behalf Of *Baldur Norddahl >> *Sent:* Sunday, August 30, 2020 9:28 AM >> *To:* nanog@nanog.org >> *Subject:* Re: Cen

Re: Centurylink having a bad morning?

Yep Regards Ray Ludendorff On Aug 30, 2020, at 16:16, Jason Kuehl wrote:  I'm over in MA in a CL building, it's very much still broken. I shut down the interfaces to CL and now just using Comcast. On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky mailto:a...@mbrez.com>> wrote: Started about

Re: Centurylink having a bad morning?

On Sun, Aug 30, 2020 at 5:21 PM Chris Adams wrote: > Once upon a time, Baldur Norddahl said: > > How is that acceptable behaviour? I shall remember never to make a > contract > > with these guys until they can prove that they won't advertise my > prefixes > > after I pull them. Under any

Re: Centurylink having a bad morning?

ot defending them but I am sure it isn’t intentional. >> >> >> >> >> >> >> >> *From:* NANOG >> >> *On Behalf Of *Baldur Norddahl >> >> >> *Sent:* Sunday, August 30, 2020 9:28 AM >> >> >> *To:* nan

Re: Centurylink having a bad morning?

I've been burning before. I'll wait at least an hour before turning my links back on. On Sun, Aug 30, 2020 at 11:31 AM Job Snijders wrote: > I believe from this moment forward things are converging back to normal. > > Kind regards, > > Job > -- Sincerely, Jason W Kuehl Cell 920-419-8983

Re: Centurylink having a bad morning? [EXTERNAL]

Well at least it looks like the issue is starting to resolve and stuff is coming back up. On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes < mattli...@rivervalleyinternet.net> wrote: > Is this what happens when your entire network is database driven? >

Re: Centurylink having a bad morning?

; *From:* NANOG *On Behalf > Of *Baldur Norddahl > *Sent:* Sunday, August 30, 2020 9:28 AM > *To:* nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > How is that acceptable behaviour? I shall remember never to make a > contract with these guys until t

Re: Centurylink having a bad morning?

t:* Sunday, August 30, 2020 9:28 AM > *To:* nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > How is that acceptable behaviour? I shall remember never to make a > contract with these guys until they can prove that they won't advertise my > prefix

Re: Centurylink having a bad morning?

AS3356 is the Level3 internet. On Sun, Aug 30, 2020 at 8:09 AM Ian Bowers wrote: > AS3356 is the one I've seen all the chatter about this morning. > > On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor > wrote: > >> On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote: >> > Woke up this morning to a

RE: Centurylink having a bad morning?

Weaver Sent: Sunday, August 30, 2020 10:07 AM To: 'Ben Russell' ; 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? Yikes, they really need to get their reflectors fixed so that we can dis-engage from them. -Original Message- From: NANOG On Behalf Of Ben Russell Sent

Re: Centurylink having a bad morning?

I believe from this moment forward things are converging back to normal. Kind regards, Job

Re: Centurylink having a bad morning?

10:00 AM > *To:* Drew Weaver > *Cc:* R. Leigh Hennig ; nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > People are rebooting ghosting now. > > > > https://twitter.com/ir_kujoe/status/1300066569645707265 > > > > Seeing other

Re: Centurylink having a bad morning?

hburn… > > > > > > > > From: Tomas Lynch > > Sent: Sunday, August 30, 2020 8:45 AM > > To: Mel Beckman > > Cc: Drew Weaver ; nanog@nanog.org > > Subject: Re: Centurylink having a bad morning? > > > > > > > > BGP sessions randoml

Re: Centurylink having a bad morning?

Once upon a time, Baldur Norddahl said: > How is that acceptable behaviour? I shall remember never to make a contract > with these guys until they can prove that they won't advertise my prefixes > after I pull them. Under any circumstances. Umm, then I guess you won't sign a contract with

Re: Centurylink having a bad morning? [EXTERNAL]

Is this what happens when your entire network is database driven?

Re: Centurylink having a bad morning?

or Ashburn… > > > > > > > > From: Tomas Lynch > > Sent: Sunday, August 30, 2020 8:45 AM > > To: Mel Beckman > > Cc: Drew Weaver ; nanog@nanog.org > > Subject: Re: Centurylink having a bad morning? > > > > > > > > BGP sessi

RE: Centurylink having a bad morning?

I’m not defending them but I am sure it isn’t intentional. From: NANOG On Behalf Of Baldur Norddahl Sent: Sunday, August 30, 2020 9:28 AM To: nanog@nanog.org Subject: Re: Centurylink having a bad morning? How is that acceptable behaviour? I shall remember never to make a contract

Re: Centurylink having a bad morning?

AS3356 is the one I've seen all the chatter about this morning. On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor wrote: > On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote: > > Woke up this morning to a bunch of reports of issues with connectivity > > had to shut down some Level3/CTL connections

Re: Centurylink having a bad morning?

:14:27 AM To: David Hubbard Cc: nanog@nanog.org Subject: Re: Centurylink having a bad morning? They’re not reachable so who knows if they’re even working on it. Gonna go out on a limb here and assume that a lot of phones were ringing and people are in fact working on whatever it is. :) On Sun

Re: Centurylink having a bad morning?

Once upon a time, Robert Blayzor said: > Just to confirm we're seeing this on AS3356 and not AS209, correct? Correct - we had problems with our 3356 connection but not our 209 connection. -- Chris Adams

RE: Centurylink having a bad morning?

Yikes, they really need to get their reflectors fixed so that we can dis-engage from them. -Original Message- From: NANOG On Behalf Of Ben Russell Sent: Sunday, August 30, 2020 9:14 AM To: 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? One of my legacy L3 peers

Re: Centurylink having a bad morning?

I just brought one of my sessions back up to attempt to avoid the blackholing, should be full feed, getting all of 850 v4 routes and 106 v6. From: NANOG on behalf of Tomas Lynch Date: Sunday, August 30, 2020 at 9:41 AM To: Drew Weaver Cc: "nanog@nanog.org" Subject: Re: Centuryl

Re: Centurylink having a bad morning?

Now if you call into CL you get a message stating their technicians are working on an ip outage. On Sun, Aug 30, 2020 at 6:56 AM Chase Christian wrote: > Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00 > Pacific: > > Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent

RE: Centurylink having a bad morning?

Something just now changed in this situation and now it seems to have gotten worse. From: Jason Kuehl Sent: Sunday, August 30, 2020 10:00 AM To: Drew Weaver Cc: R. Leigh Hennig ; nanog@nanog.org Subject: Re: Centurylink having a bad morning? People are rebooting ghosting now. [cid:image001

Re: Centurylink having a bad morning?

actual network? > > > > *From:* R. Leigh Hennig > *Sent:* Sunday, August 30, 2020 8:54 AM > *To:* Drew Weaver ; nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > Global impact with issues reported by Fastly, Cloudflare, OpenDNS. > > >

RE: Centurylink having a bad morning? [EXTERNAL]

Their route-servers just can’t keep up. They are not getting our community-string announcements at all From: NANOG On Behalf Of Drew Weaver Sent: Sunday, August 30, 2020 8:38 AM To: 'David Hubbard' ; 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? [EXTERNAL] Yeah, I am

Re: Centurylink having a bad morning?

gt; > From: Tomas Lynch > Sent: Sunday, August 30, 2020 8:45 AM > To: Mel Beckman > Cc: Drew Weaver ; nanog@nanog.org > Subject: Re: Centurylink having a bad morning? > > > > BGP sessions randomly flapping or having routing issues in different cities > since ~

Re: Centurylink having a bad morning?

How is that acceptable behaviour? I shall remember never to make a contract with these guys until they can prove that they won't advertise my prefixes after I pull them. Under any circumstances. søn. 30. aug. 2020 15.14 skrev Joseph Jenkins : > Finally got through on their support line and spoke

RE: Centurylink having a bad morning?

I don’t think it is anywhere near back to normal but I can get to 2-3 more sites than I could when I got called in. From: NANOG On Behalf Of Andy Brezinsky Sent: Sunday, August 30, 2020 8:48 AM To: nanog@nanog.org Subject: Re: Centurylink having a bad morning? Started about 5:05am central

Re: Centurylink having a bad morning?

I'm over in MA in a CL building, it's very much still broken. I shut down the interfaces to CL and now just using Comcast. On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky wrote: > Started about 5:05am central, started clearing up for me about 7:15am. > My route from ATT in Chicago is still going

Re: Centurylink having a bad morning?

> > They’re not reachable so who knows if they’re even working on it. > Gonna go out on a limb here and assume that a lot of phones were ringing and people are in fact working on whatever it is. :) On Sun, Aug 30, 2020 at 8:34 AM David Hubbard wrote: > Same. Also, as reported on outages list,

RE: Centurylink having a bad morning?

One of my legacy L3 peers is only advertising 498174 prefixes. Very strange. Ben Russell  -Original Message- From: NANOG On Behalf Of Drew Weaver Sent: Sunday, August 30, 2020 7:45 AM To: 'Mel Beckman' Cc: 'nanog@nanog.org' Subject: RE: Centurylink having a bad morning? Can't

Re: Centurylink having a bad morning?

On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote: > Woke up this morning to a bunch of reports of issues with connectivity > had to shut down some Level3/CTL connections to get it to return to normal. Just to confirm we're seeing this on AS3356 and not AS209, correct? We have links to both and

Re: Centurylink having a bad morning?

Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00 Pacific: Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor 4.35.X.Y (AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes Aug 30 03:05:13 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old state

Re: Centurylink having a bad morning?

We've been on hold for more than an hour trying to get an update. We see the same behavior where they continue to announce our blocks despite all the interfaces to them being hard down. Scott Helms On Sun, Aug 30, 2020 at 8:58 AM Jason Kuehl wrote: > > Well, When I tried calling I got a fast

RE: Centurylink having a bad morning?

That site seems to be just for their cloud products, is there one of these for their actual network? From: R. Leigh Hennig Sent: Sunday, August 30, 2020 8:54 AM To: Drew Weaver ; nanog@nanog.org Subject: Re: Centurylink having a bad morning? Global impact with issues reported by Fastly

Re: Centurylink having a bad morning?

tti or Ashburn… > > > > *From:* Tomas Lynch > *Sent:* Sunday, August 30, 2020 8:45 AM > *To:* Mel Beckman > *Cc:* Drew Weaver ; nanog@nanog.org > *Subject:* Re: Centurylink having a bad morning? > > > > BGP sessions randomly flapping or having routing issues in different

RE: Centurylink having a bad morning?

Saw the flapping in Cleveland but not in Cincinnatti or Ashburn… From: Tomas Lynch Sent: Sunday, August 30, 2020 8:45 AM To: Mel Beckman Cc: Drew Weaver ; nanog@nanog.org Subject: Re: Centurylink having a bad morning? BGP sessions randomly flapping or having routing issues in different cities

Re: Centurylink having a bad morning?

Looks like this is causing problems with Meraki Cloud connectivity since level3 is a transit provider for Meraki. From: NANOG on behalf of Mel Beckman Sent: Sunday, August 30, 2020, 08:47 To: Drew Weaver Cc: nanog@nanog.org Subject: Re: Centurylink having

Re: Centurylink having a bad morning?

I am seeing some odd traffic deflections in the EU via 3356, but nothing in the US so far. Some scattershot oddball reports landing in our NOC, but nothing conclusive. On Sun, Aug 30, 2020 at 8:41 AM Mel Beckman wrote: > The CL portal loads for me, and I can log in, but it is slower than

Re: Centurylink having a bad morning?

Started about 5:05am central, started clearing up for me about 7:15am.   My route from ATT in Chicago is still going through NYC to get back to Chicago but at least packet loss isn't 70-100% anymore. I also tried turning down sessions and still was seeing stale announcements on other LGs.

Re: Centurylink having a bad morning?

Finally got through on their support line and spoke to level1. The only thing the tech could say was it was an issue with BGP route reflectors and it started about 3am(pacific). They were still trying to isolate the issue. I've tried failing over my circuits and no go, the traffic just dies as L3

RE: Centurylink having a bad morning?

Can't get to pretty much anything on the Internet from my Tmobile device... not really sure whats going on =) -Original Message- From: Mel Beckman Sent: Sunday, August 30, 2020 8:43 AM To: Drew Weaver Cc: nanog@nanog.org Subject: Re: Centurylink having a bad morning? One anomaly I’ve

Re: Centurylink having a bad morning?

BGP sessions randomly flapping or having routing issues in different cities since ~5AM EST On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman wrote: > The CL portal loads for me, and I can log in, but it is slower than usual. > Not seeing traffic issues on our CL circuits. > > -mel via cell > > On Aug

Re: Centurylink having a bad morning?

One anomaly I’ve just notice: we can’t reach 209.244.0.1 (ns1.level3.com) from anywhere on the Internet, including our CL connections. Traceroute fails at 4.16.30.129 (lag-101.ear2.LosAngeles1.Level3.net) with 65% packet loss: traceroute to 209.244.0.1 (209.244.0.1), 30 hops max, 56 byte

Re: Centurylink having a bad morning?

Well, When I tried calling I got a fast busy, so that's nice. On Sun, Aug 30, 2020 at 8:33 AM David Hubbard wrote: > Same. Also, as reported on outages list, what’s even worse is that they > appear to be continuing to propagate advertisements from circuits whose > sessions have been turned

RE: Centurylink having a bad morning?

Yeah, I am still seeing them announce our Ips even though we’ve shut down our sessions with them. From: NANOG On Behalf Of David Hubbard Sent: Sunday, August 30, 2020 8:28 AM To: nanog@nanog.org Subject: Re: Centurylink having a bad morning? Same. Also, as reported on outages list, what’s

Re: Centurylink having a bad morning?

I have checked all of our circuits in the LA area and are seeing no BGP issues, but we don’t provide transit for CL, only use them for DIA. Traffic flows appear normal, but early Sunday morning is normally the lowest time of the week. -mel beckman > On Aug 30, 2020, at 5:29 AM, Mel Beckman

Re: Centurylink having a bad morning?

The CL portal loads for me, and I can log in, but it is slower than usual. Not seeing traffic issues on our CL circuits. -mel via cell On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG wrote:  Hello, Woke up this morning to a bunch of reports of issues with connectivity had to shut down

Re: Centurylink having a bad morning?

Same. Also, as reported on outages list, what’s even worse is that they appear to be continuing to propagate advertisements from circuits whose sessions have been turned down. I validated ours still were via a couple looking glass portals. Down Detector shows nearly every major service

Re: Centurylink having a bad morning?

Its dead JIM. I also can't get in starting around 7 est On Sun, Aug 30, 2020 at 8:19 AM Drew Weaver via NANOG wrote: > Hello, > > > > Woke up this morning to a bunch of reports of issues with connectivity had > to shut down some Level3/CTL connections to get it to return to normal. > > > > As