On 5/Sep/20 18:46, Randy Bush via NANOG wrote:
> fracking dmark illegal header hacking :(
>
> apologies
No worries :-).
Mark.
On 5/Sep/20 17:58, Randy Bush via NANOG wrote:
> spoken like a tier two
Even "Tier 1's" aren't present everywhere :-)...
Although I'm not sure whether having presence in every city means you
are a Tier-anything either...
Mark.
*From: *"Mark Tinka"
*To: *nanog@nanog.org
*Sent: *Thursday, September 3, 2020 1:35:46 PM
*Subject: *Re: Centurylink having a bad morning?
On 31/Aug/20 17:57, Bryan Holloway wrote:
> Not everyone will peer with you, notably, AS3356 (unle
> [ off list ]
fracking dmark illegal header hacking :(
apologies
[ off list ]
>> Oh, yes! Let's not start another "what's a tier one" war!
>
> Oh no, let's :-).
>
> We get over here in Africa as well. Local operators either calling
> themselves Tier 1, or being called a Tier 1. Nonsensical.
>
> Years back, our Marketing team asked me to comment on the use
From: "Mark Tinka via NANOG"
To: "Tomas Lynch"
Cc: "NANOG"
Sent: Saturday, September 5, 2020 5:26:21 AM
Subject: Re: Centurylink having a bad morning?
On 4/Sep/20 23:41, Tomas Lynch wrote:
Oh, yes! Let's not start another "what's a tier one&
1:35:46 PM
Subject: Re: Centurylink having a bad morning?
On 31/Aug/20 17:57, Bryan Holloway wrote:
> Not everyone will peer with you, notably, AS3356 (unless you're big
> enough, which few can say.)
I think Tomas meant more diverse peering, not peering with CL.
Mark.
On 4/Sep/20 23:41, Tomas Lynch wrote:
>
> Oh, yes! Let's not start another "what's a tier one" war!
Oh no, let's :-).
We get over here in Africa as well. Local operators either calling
themselves Tier 1, or being called a Tier 1. Nonsensical.
Years back, our Marketing team asked me to
On Thu, Sep 3, 2020 at 2:37 PM Mark Tinka wrote:
>
>
> On 31/Aug/20 17:57, Bryan Holloway wrote:
>
> > Not everyone will peer with you, notably, AS3356 (unless you're big
> > enough, which few can say.)
>
> I think Tomas meant more diverse peering, not peering with CL.
>
Oh, yes! Let's not
On 31/Aug/20 17:57, Bryan Holloway wrote:
> Not everyone will peer with you, notably, AS3356 (unless you're big
> enough, which few can say.)
I think Tomas meant more diverse peering, not peering with CL.
Mark.
On Wed, Sep 2, 2020 at 11:02, Warren Kumari wrote:
>> well, what you REALLY need is one of these:
>> https://www.cru-inc.com/products/wiebetech/hotplug_field_kit_product/
>>
>
> Yeah, no... actually, hell no!
>
> That setup scares me, and I'm surprised that it can be sold at all,
> even with
On 2/Sep/20 05:53, Alain Hebert wrote:
>
> Beat installing a Cisco 12k solo with 2x4's to align the mounting
> holes...
I still have those in a rack somewhere, trapping air, if you want to
test that :-)...
Mark.
On 31/Aug/20 16:33, Tomas Lynch wrote:
> Maybe we are idealizing these so-called tier-1 carriers and we,
> tier-ns, should treat them as what they really are: another AS. Accept
> that they are going to fail and do our best to mitigate the impact on
> our own networks, i.e. more peering.
-
> > *De :* Shawn L via NANOG
> > *Envoyé :* mercredi 2 septembre 2020 13:15
> > *À :* nanog
> > *Objet :* Re: Centurylink having a bad morning?
> >
> > We once moved a 3u server 30 miles between data centers this way. Plug
> > redu
That is what the 5G router is for...
ons. 2. sep. 2020 19.47 skrev Michael Hallgren :
> While conserving connectivity?
>
>
> --
> *De :* Shawn L via NANOG
> *Envoyé :* mercredi 2 septembre 2020 13:15
> *À :* nanog
> *Objet :* Re: Centuryl
While conserving connectivity?
De : Shawn L via NANOG
Envoyé : mercredi 2 septembre 2020 13:15
À : nanog
Objet : Re: Centurylink having a bad morning?
We once moved a 3u server 30 miles between data centers this way. Plug
redundant psu into a ups and 2
On Wed, Sep 2, 2020 at 12:00 AM Christopher Morrow
wrote:
>
> On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert wrote:
> >
> > As a coincidence... I was *thinking* of moving a 90TB SAN (with
> > mechanical's) to another rack that way... skateboard, long fibers and long
> > power cords =D
> >
If the client pays me a shit ton of money to make sure the server
won't turn off, and they pay for the hardware to make it happen. I;d think
about it. It's a like a colo move on hardmode.
Its extremely stupid, and I would advise not doing it.
Hell even when I migrated e911 server, we had a 20
On 9/2/20 1:49 PM, Nick Hilliard wrote:
Shawn L via NANOG wrote on 02/09/2020 12:15:
We once moved a 3u server 30 miles between data centers this way.
Plug redundant psu into a ups and 2 people carried it out and put
them in a vehicle.
hopefully none of these server moves that people have
Shawn L via NANOG wrote on 02/09/2020 12:15:
We once moved a 3u server 30 miles between data centers this way.
Plug redundant psu into a ups and 2 people carried it out and put
them in a vehicle.
hopefully none of these server moves that people have been talking about
involved spinning disks.
We once moved a 3u server 30 miles between data centers this way. Plug
redundant psu into a ups and 2 people carried it out and put them in a vehicle.
Sent from my iPhone
> On Sep 1, 2020, at 11:58 PM, Christopher Morrow
> wrote:
>
> On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert wrote:
On Tue, Sep 1, 2020 at 11:53 PM Alain Hebert wrote:
>
> As a coincidence... I was *thinking* of moving a 90TB SAN (with
> mechanical's) to another rack that way... skateboard, long fibers and long
> power cords =D
>
well, what you REALLY need is one of these:
As a coincidence... I was *thinking* of moving a 90TB SAN (with
mechanical's) to another rack that way... skateboard, long fibers and
long power cords =D
Beat installing a Cisco 12k solo with 2x4's to align the mounting
holes...
-
Alain Hebert
design of BGP is such
>>> that this should not happen easily if at all. They did something unwise.
>>> Did they make a route reflector based on a database or something?
>>>
>>> Regards,
>>>
>>> Baldur
>>>
>>> On Sun, Aug 30,
On Mon, Aug 31, 2020 at 11:28 PM Bjørn Mork wrote:
> Well, many of us are paying for redundant power supplies or redundant
> REs, even if that doesn't make any difference when the chassis is on
> fire. I guess most people know that, and still buy those redundant
> components.
>
I buy it so I
Eric Kuhnke writes:
> There's a number of enterprise end user type customers of 3356 that have
> on-premises server rooms/hosting for their stuff. And they spend a lot of
> money every month for a 'redundant' metro ethernet circuit that takes
> diverse fiber paths from their business park office
omer at an outlet who has a system
> that will do that. Luckily we do not currently have a contract and now they
> will have to convince me it is safe for me to make a contract with them. If
> that is impossible I guess I won't be getting a contract with them.
> >>>
> >>&
rylink a bit puzzling.
>>> It could have just been poorly written.
>>>
>>>
>>>
>>>
>>>
>>> From: NANOG On Behalf Of
>>> Tom Beecher
>>> Sent: Monday, August 31, 2020 9:26 AM
>>> To: Hank Nussbacher
&g
re that don’t touch Centurylink a bit puzzling. It
>> could have just been poorly written.
>>
>>
>>
>>
>>
>> From: NANOG On Behalf Of
>> Tom Beecher
>> Sent: Monday, August 31, 2020 9:26 AM
>> To: Hank Nussbacher
>> Cc: NANOG
ossible. They need to make a good
>> report telling exactly what went wrong and how they changed the design, so
>> something like this can not happen again. The basic design of BGP is such
>> that this should not happen easily if at all. They did something unwise.
>> Did they
From: NANOG On Behalf Of
>> Tom Beecher
>> Sent: Monday, August 31, 2020 9:26 AM
>> To: Hank Nussbacher
>> Cc: NANOG
>> Subject: Re: Centurylink having a bad morning?
>>
>>
>>
>> https://blog.cloudflare.com/analysis-of-todays-centurylink
NOG *On Behalf
> Of *Tom Beecher
> *Sent:* Monday, August 31, 2020 9:26 AM
> *To:* Hank Nussbacher
> *Cc:* NANOG
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/
>
>
>
> I defi
. It could have just
been poorly written.
From: NANOG On Behalf Of Tom
Beecher
Sent: Monday, August 31, 2020 9:26 AM
To: Hank Nussbacher
Cc: NANOG
Subject: Re: Centurylink having a bad morning?
https://blog.cloudflare.com/analysis-of-todays-centurylink-level-3-outage/
I definitely found Mr
intentional.
*From:* NANOG
mailto:thenap@nanog.org>> *On Behalf Of *Baldur
Norddahl
*Sent:* Sunday, August 30, 2020 9:28 AM
*To:* nanog@nanog.org <mailto:nanog@nanog.org>
*Subject:* Re
possible. They need to make a good
>>>> report telling exactly what went wrong and how they changed the design, so
>>>> something like this can not happen again. The basic design of BGP is such
>>>> that this should not happen easily if at all. They did something unwi
flector based on a database or something?
>>>
>>> Regards,
>>>
>>> Baldur
>>>
>>> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho
>>> wrote:
>>>
>>>> Exactly. And asking that they somehow prove this won't happen again
to:drew.wea...@thenap.com>> wrote:
I’m not defending them but I am sure it isn’t intentional.
From: NANOG
mailto:thenap@nanog.org>>
On Behalf Of Baldur Norddahl
Sent: Sunday, August 30, 2020 9:28 AM
To: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: Centurylink having a
se or something?
>>
>> Regards,
>>
>> Baldur
>>
>> On Sun, Aug 30, 2020 at 5:13 PM Mike Bolitho
>> wrote:
>>
>>> Exactly. And asking that they somehow prove this won't happen again is
>>> impossible.
>>>
>>> - Mike Boli
p@nanog.org>>
On Behalf Of Baldur Norddahl
Sent: Sunday, August 30, 2020 9:28 AM
To: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: Centurylink having a bad morning?
How is that acceptable behaviour? I shall remember never to make a contract
with these guys until they can pr
gt; I’m not defending them but I am sure it isn’t intentional.
>>>
>>>
>>>
>>> *From:* NANOG *On
>>> Behalf Of *Baldur Norddahl
>>> *Sent:* Sunday, August 30, 2020 9:28 AM
>>> *To:* nanog@nanog.org
>>> *Subje
intentional.
From: NANOG
thenap@nanog.org>
On Behalf Of Baldur Norddahl
Sent: Sunday, August 30, 2020 9:28 AM
To: nanog@nanog.org
Subject: Re: Centuryli
On Sun, 30 Aug 2020 at 20:00, Baldur Norddahl wrote:
> Not really the point. BGP is designed such that if I take down the link, the
> prefixes MUST be withdrawn within reasonable time. The self healing aspect of
> the internet entirely depends on this. Clearly they have some kind of system
>
Drew Weaver wrote:
>>
>>> I’m not defending them but I am sure it isn’t intentional.
>>>
>>>
>>>
>>> *From:* NANOG *On
>>> Behalf Of *Baldur Norddahl
>>> *Sent:* Sunday, August 30, 2020 9:28 AM
>>> *To:* nanog@nanog.org
On 30/08/2020 18:22, Joseph Jenkins
wrote:
Well at least it looks like the issue is starting
to resolve and stuff is coming back up.
On Sun, Aug 30, 2020 at 8:21
AM Matt Hoppes
sn’t intentional.
>>>
>>>
>>>
>>> From: NANOG On Behalf Of
>>> Baldur Norddahl
>>> Sent: Sunday, August 30, 2020 9:28 AM
>>> To: nanog@nanog.org
>>> Subject: Re: Centurylink having a bad morning?
>>>
>>>
2020, 8:10 AM Drew Weaver wrote:
>
>> I’m not defending them but I am sure it isn’t intentional.
>>
>>
>>
>> *From:* NANOG *On
>> Behalf Of *Baldur Norddahl
>> *Sent:* Sunday, August 30, 2020 9:28 AM
>> *To:* nanog@nanog.org
>> *Subject:* Re: Cen
Yep
Regards
Ray Ludendorff
On Aug 30, 2020, at 16:16, Jason Kuehl wrote:
I'm over in MA in a CL building, it's very much still broken. I shut down the
interfaces to CL and now just using Comcast.
On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky
mailto:a...@mbrez.com>> wrote:
Started about
On Sun, Aug 30, 2020 at 5:21 PM Chris Adams wrote:
> Once upon a time, Baldur Norddahl said:
> > How is that acceptable behaviour? I shall remember never to make a
> contract
> > with these guys until they can prove that they won't advertise my
> prefixes
> > after I pull them. Under any
ot defending them but I am sure it isn’t intentional.
>>
>>
>>
>>
>>
>>
>>
>> *From:* NANOG
>>
>> *On Behalf Of *Baldur Norddahl
>>
>>
>> *Sent:* Sunday, August 30, 2020 9:28 AM
>>
>>
>> *To:* nan
I've been burning before. I'll wait at least an hour before turning my
links back on.
On Sun, Aug 30, 2020 at 11:31 AM Job Snijders wrote:
> I believe from this moment forward things are converging back to normal.
>
> Kind regards,
>
> Job
>
--
Sincerely,
Jason W Kuehl
Cell 920-419-8983
Well at least it looks like the issue is starting to resolve and stuff is
coming back up.
On Sun, Aug 30, 2020 at 8:21 AM Matt Hoppes <
mattli...@rivervalleyinternet.net> wrote:
> Is this what happens when your entire network is database driven?
>
; *From:* NANOG *On Behalf
> Of *Baldur Norddahl
> *Sent:* Sunday, August 30, 2020 9:28 AM
> *To:* nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> How is that acceptable behaviour? I shall remember never to make a
> contract with these guys until t
t:* Sunday, August 30, 2020 9:28 AM
> *To:* nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> How is that acceptable behaviour? I shall remember never to make a
> contract with these guys until they can prove that they won't advertise my
> prefix
AS3356 is the Level3 internet.
On Sun, Aug 30, 2020 at 8:09 AM Ian Bowers wrote:
> AS3356 is the one I've seen all the chatter about this morning.
>
> On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor
> wrote:
>
>> On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote:
>> > Woke up this morning to a
Weaver
Sent: Sunday, August 30, 2020 10:07 AM
To: 'Ben Russell' ; 'nanog@nanog.org'
Subject: RE: Centurylink having a bad morning?
Yikes, they really need to get their reflectors fixed so that we can dis-engage
from them.
-Original Message-
From: NANOG On Behalf Of Ben
Russell
Sent
I believe from this moment forward things are converging back to normal.
Kind regards,
Job
10:00 AM
> *To:* Drew Weaver
> *Cc:* R. Leigh Hennig ; nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> People are rebooting ghosting now.
>
>
>
> https://twitter.com/ir_kujoe/status/1300066569645707265
>
>
>
> Seeing other
hburn…
> >
> >
> >
> > From: Tomas Lynch
> > Sent: Sunday, August 30, 2020 8:45 AM
> > To: Mel Beckman
> > Cc: Drew Weaver ; nanog@nanog.org
> > Subject: Re: Centurylink having a bad morning?
> >
> >
> >
> > BGP sessions randoml
Once upon a time, Baldur Norddahl said:
> How is that acceptable behaviour? I shall remember never to make a contract
> with these guys until they can prove that they won't advertise my prefixes
> after I pull them. Under any circumstances.
Umm, then I guess you won't sign a contract with
Is this what happens when your entire network is database driven?
or Ashburn…
> >
> >
> >
> > From: Tomas Lynch
> > Sent: Sunday, August 30, 2020 8:45 AM
> > To: Mel Beckman
> > Cc: Drew Weaver ; nanog@nanog.org
> > Subject: Re: Centurylink having a bad morning?
> >
> >
> >
> > BGP sessi
I’m not defending them but I am sure it isn’t intentional.
From: NANOG On Behalf Of
Baldur Norddahl
Sent: Sunday, August 30, 2020 9:28 AM
To: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?
How is that acceptable behaviour? I shall remember never to make a contract
AS3356 is the one I've seen all the chatter about this morning.
On Sun, Aug 30, 2020 at 10:03 AM Robert Blayzor
wrote:
> On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote:
> > Woke up this morning to a bunch of reports of issues with connectivity
> > had to shut down some Level3/CTL connections
:14:27 AM
To: David Hubbard
Cc: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?
They’re not reachable so who knows if they’re even working on it.
Gonna go out on a limb here and assume that a lot of phones were ringing and
people are in fact working on whatever it is. :)
On Sun
Once upon a time, Robert Blayzor said:
> Just to confirm we're seeing this on AS3356 and not AS209, correct?
Correct - we had problems with our 3356 connection but not our 209
connection.
--
Chris Adams
Yikes, they really need to get their reflectors fixed so that we can dis-engage
from them.
-Original Message-
From: NANOG On Behalf Of Ben
Russell
Sent: Sunday, August 30, 2020 9:14 AM
To: 'nanog@nanog.org'
Subject: RE: Centurylink having a bad morning?
One of my legacy L3 peers
I just brought one of my sessions back up to attempt to avoid the blackholing,
should be full feed, getting all of 850 v4 routes and 106 v6.
From: NANOG on behalf
of Tomas Lynch
Date: Sunday, August 30, 2020 at 9:41 AM
To: Drew Weaver
Cc: "nanog@nanog.org"
Subject: Re: Centuryl
Now if you call into CL you get a message stating their technicians are
working on an ip outage.
On Sun, Aug 30, 2020 at 6:56 AM Chase Christian wrote:
> Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00
> Pacific:
>
> Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent
Something just now changed in this situation and now it seems to have gotten
worse.
From: Jason Kuehl
Sent: Sunday, August 30, 2020 10:00 AM
To: Drew Weaver
Cc: R. Leigh Hennig ; nanog@nanog.org
Subject: Re: Centurylink having a bad morning?
People are rebooting ghosting now.
[cid:image001
actual network?
>
>
>
> *From:* R. Leigh Hennig
> *Sent:* Sunday, August 30, 2020 8:54 AM
> *To:* Drew Weaver ; nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> Global impact with issues reported by Fastly, Cloudflare, OpenDNS.
>
>
>
Their route-servers just can’t keep up.
They are not getting our community-string announcements at all
From: NANOG On Behalf Of
Drew Weaver
Sent: Sunday, August 30, 2020 8:38 AM
To: 'David Hubbard' ; 'nanog@nanog.org'
Subject: RE: Centurylink having a bad morning? [EXTERNAL]
Yeah, I am
gt;
> From: Tomas Lynch
> Sent: Sunday, August 30, 2020 8:45 AM
> To: Mel Beckman
> Cc: Drew Weaver ; nanog@nanog.org
> Subject: Re: Centurylink having a bad morning?
>
>
>
> BGP sessions randomly flapping or having routing issues in different cities
> since ~
How is that acceptable behaviour? I shall remember never to make a contract
with these guys until they can prove that they won't advertise my prefixes
after I pull them. Under any circumstances.
søn. 30. aug. 2020 15.14 skrev Joseph Jenkins :
> Finally got through on their support line and spoke
I don’t think it is anywhere near back to normal but I can get to 2-3 more
sites than I could when I got called in.
From: NANOG On Behalf Of Andy
Brezinsky
Sent: Sunday, August 30, 2020 8:48 AM
To: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?
Started about 5:05am central
I'm over in MA in a CL building, it's very much still broken. I shut down
the interfaces to CL and now just using Comcast.
On Sun, Aug 30, 2020 at 9:20 AM Andy Brezinsky wrote:
> Started about 5:05am central, started clearing up for me about 7:15am.
> My route from ATT in Chicago is still going
>
> They’re not reachable so who knows if they’re even working on it.
>
Gonna go out on a limb here and assume that a lot of phones were ringing
and people are in fact working on whatever it is. :)
On Sun, Aug 30, 2020 at 8:34 AM David Hubbard
wrote:
> Same. Also, as reported on outages list,
One of my legacy L3 peers is only advertising 498174 prefixes. Very strange.
Ben Russell
-Original Message-
From: NANOG On Behalf Of Drew
Weaver
Sent: Sunday, August 30, 2020 7:45 AM
To: 'Mel Beckman'
Cc: 'nanog@nanog.org'
Subject: RE: Centurylink having a bad morning?
Can't
On 8/30/20 8:14 AM, Drew Weaver via NANOG wrote:
> Woke up this morning to a bunch of reports of issues with connectivity
> had to shut down some Level3/CTL connections to get it to return to normal.
Just to confirm we're seeing this on AS3356 and not AS209, correct?
We have links to both and
Multiple BGP sessions with Level3 (DIA) started flapping at approx 03:00
Pacific:
Aug 30 03:05:13 rtr02 Rib: %BGP-3-NOTIFICATION: sent to neighbor 4.35.X.Y
(AS 3356) 4/0 (Hold Timer Expired Error/Unspecified) 0 bytes
Aug 30 03:05:13 rtr02 Rib: %BGP-5-ADJCHANGE: peer 4.35.X.Y (AS 3356) old
state
We've been on hold for more than an hour trying to get an update.
We see the same behavior where they continue to announce our blocks
despite all the interfaces to them being hard down.
Scott Helms
On Sun, Aug 30, 2020 at 8:58 AM Jason Kuehl wrote:
>
> Well, When I tried calling I got a fast
That site seems to be just for their cloud products, is there one of these for
their actual network?
From: R. Leigh Hennig
Sent: Sunday, August 30, 2020 8:54 AM
To: Drew Weaver ; nanog@nanog.org
Subject: Re: Centurylink having a bad morning?
Global impact with issues reported by Fastly
tti or Ashburn…
>
>
>
> *From:* Tomas Lynch
> *Sent:* Sunday, August 30, 2020 8:45 AM
> *To:* Mel Beckman
> *Cc:* Drew Weaver ; nanog@nanog.org
> *Subject:* Re: Centurylink having a bad morning?
>
>
>
> BGP sessions randomly flapping or having routing issues in different
Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…
From: Tomas Lynch
Sent: Sunday, August 30, 2020 8:45 AM
To: Mel Beckman
Cc: Drew Weaver ; nanog@nanog.org
Subject: Re: Centurylink having a bad morning?
BGP sessions randomly flapping or having routing issues in different cities
Looks like this is causing problems with Meraki Cloud connectivity since level3
is a transit provider for Meraki.
From: NANOG on behalf of Mel
Beckman
Sent: Sunday, August 30, 2020, 08:47
To: Drew Weaver
Cc: nanog@nanog.org
Subject: Re: Centurylink having
I am seeing some odd traffic deflections in the EU via 3356, but nothing in
the US so far.
Some scattershot oddball reports landing in our NOC, but nothing
conclusive.
On Sun, Aug 30, 2020 at 8:41 AM Mel Beckman wrote:
> The CL portal loads for me, and I can log in, but it is slower than
Started about 5:05am central, started clearing up for me about 7:15am.
My route from ATT in Chicago is still going through NYC to get back to
Chicago but at least packet loss isn't 70-100% anymore.
I also tried turning down sessions and still was seeing stale
announcements on other LGs.
Finally got through on their support line and spoke to level1. The only
thing the tech could say was it was an issue with BGP route reflectors and
it started about 3am(pacific). They were still trying to isolate the issue.
I've tried failing over my circuits and no go, the traffic just dies as L3
Can't get to pretty much anything on the Internet from my Tmobile device... not
really sure whats going on =)
-Original Message-
From: Mel Beckman
Sent: Sunday, August 30, 2020 8:43 AM
To: Drew Weaver
Cc: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?
One anomaly I’ve
BGP sessions randomly flapping or having routing issues in different cities
since ~5AM EST
On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman wrote:
> The CL portal loads for me, and I can log in, but it is slower than usual.
> Not seeing traffic issues on our CL circuits.
>
> -mel via cell
>
> On Aug
One anomaly I’ve just notice: we can’t reach 209.244.0.1 (ns1.level3.com) from
anywhere on the Internet, including our CL connections. Traceroute fails at
4.16.30.129 (lag-101.ear2.LosAngeles1.Level3.net) with 65% packet loss:
traceroute to 209.244.0.1 (209.244.0.1), 30 hops max, 56 byte
Well, When I tried calling I got a fast busy, so that's nice.
On Sun, Aug 30, 2020 at 8:33 AM David Hubbard
wrote:
> Same. Also, as reported on outages list, what’s even worse is that they
> appear to be continuing to propagate advertisements from circuits whose
> sessions have been turned
Yeah, I am still seeing them announce our Ips even though we’ve shut down our
sessions with them.
From: NANOG On Behalf Of David
Hubbard
Sent: Sunday, August 30, 2020 8:28 AM
To: nanog@nanog.org
Subject: Re: Centurylink having a bad morning?
Same. Also, as reported on outages list, what’s
I have checked all of our circuits in the LA area and are seeing no BGP issues,
but we don’t provide transit for CL, only use them for DIA. Traffic flows
appear normal, but early Sunday morning is normally the lowest time of the
week.
-mel beckman
> On Aug 30, 2020, at 5:29 AM, Mel Beckman
The CL portal loads for me, and I can log in, but it is slower than usual. Not
seeing traffic issues on our CL circuits.
-mel via cell
On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG wrote:
Hello,
Woke up this morning to a bunch of reports of issues with connectivity had to
shut down
Same. Also, as reported on outages list, what’s even worse is that they appear
to be continuing to propagate advertisements from circuits whose sessions have
been turned down. I validated ours still were via a couple looking glass
portals. Down Detector shows nearly every major service
Its dead JIM. I also can't get in starting around 7 est
On Sun, Aug 30, 2020 at 8:19 AM Drew Weaver via NANOG
wrote:
> Hello,
>
>
>
> Woke up this morning to a bunch of reports of issues with connectivity had
> to shut down some Level3/CTL connections to get it to return to normal.
>
>
>
> As
96 matches
Mail list logo