It gets better ... https://twitter.com/g_bonfiglio/status/1300086336121176064
The day Telia (and NTT, and others) de-peered Level3. "AS1299 have temporarily disabled all IPv4 peering with CenturyLink AS3356 on their request." Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 [email protected]<mailto:[email protected]> www.atlantic.net<http://www.atlantic.net/> From: Outages <[email protected]> On Behalf Of Tino Montemor via Outages Sent: Sunday, August 30, 2020 11:06 AM To: [email protected] Subject: Re: [outages] Outages Digest, Vol 147, Issue 10 CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Just want to reflect (pun?) what others are saying, Centurylink wasn't letting go of our routes. As of about 2 minutes ago it seems out AS Prepend has finally began to propagate _____ Tino Montemor | Skechers USA, Inc. O: 310-406-0132 M: 909-721-6673 ________________________________ From: Outages <[email protected]<mailto:[email protected]>> on behalf of [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Sent: Sunday, August 30, 2020 6:32:35 AM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Outages Digest, Vol 147, Issue 10 Send Outages mailing list submissions to [email protected]<mailto:[email protected]> To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= or, via email, send a message with subject or body 'help' to [email protected]<mailto:[email protected]> You can reach the person managing the list at [email protected]<mailto:[email protected]> When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Re: CenturyLink peering issues? (David Hubbard) 2. Re: 3356 does not WITHDRAW bgp routes (randal k) 3. Re: 3356 does not WITHDRAW bgp routes (David Hubbard) 4. Re: CenturyLink peering issues? (Chris Adams) 5. Re: 3356 does not WITHDRAW bgp routes (randal k) 6. Re: CenturyLink peering issues? (Stephen Flynn) ---------------------------------------------------------------------- Message: 1 Date: Sun, 30 Aug 2020 12:22:27 +0000 From: David Hubbard <[email protected]<mailto:[email protected]>> To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [outages] CenturyLink peering issues? Message-ID: <[email protected]<mailto:[email protected]>> Content-Type: text/plain; charset="utf-8" Ugh; seeing same thing. Have had sessions turned down for over two hours and looking glasses are still showing 3356 propagating the advertisements. ?On 8/30/20, 8:11 AM, "Outages on behalf of Chris Adams via Outages" <[email protected] on behalf of [email protected]<mailto:[email protected]%20on%20behalf%20of%[email protected]>> wrote: Once upon a time, Stephen Flynn via Outages <[email protected]<mailto:[email protected]>> said: > Odd part --- I disconnected my Level3 circuit at the ORL-FL facility so that I could fully failover to my other carrier link. > Level3 is still advertising my routes, even though my link and BGP session is down. I can confirm this - I shut down IPv4 BGP with Level3 in Chicago, but checking route-views and such, they're still advertising our routes (but don't know how to get to us once packets hit their network).. AS7007 all over again? -- Chris Adams <[email protected]<mailto:[email protected]>> _______________________________________________ Outages mailing list [email protected]<mailto:[email protected]> https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= ------------------------------ Message: 2 Date: Sun, 30 Aug 2020 06:33:41 -0600 From: randal k <[email protected]<mailto:[email protected]>> To: [email protected]<mailto:[email protected]> Subject: Re: [outages] 3356 does not WITHDRAW bgp routes Message-ID: <CANeLk7RmHnXa=EieXpahpmM966bL8==7qsywr89yrd0fuz9...@mail.gmail.com<mailto:CANeLk7RmHnXa=EieXpahpmM966bL8==7qsywr89yrd0fuz9...@mail.gmail.com>> Content-Type: text/plain; charset="UTF-8" Seeing the same thing - shut a peer, still seeing those routes via 3356 across multiple route-servers & looking glasses. So, do we disconnect 3356 and suffer the blackhole in hopes that it will eventually withdraw those routes, or leave it on to prevent blackholing but suffer massive packet loss to other carriers? Wow. On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages <[email protected]<mailto:[email protected]>> wrote: > > As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale > bgp routes, can be confirmed with AT&T's route server at (telnet > route-server.ip.att.net). > > Stale routes from 1 hour + are still announced by 3356. > > > This is causing blackholing. > _______________________________________________ > Outages mailing list > [email protected]<mailto:[email protected]> > https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= ------------------------------ Message: 3 Date: Sun, 30 Aug 2020 12:47:57 +0000 From: David Hubbard <[email protected]<mailto:[email protected]>> To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [outages] 3356 does not WITHDRAW bgp routes Message-ID: <[email protected]<mailto:[email protected]>> Content-Type: text/plain; charset="utf-8" I tried bringing two circuits back up hoping to not have blackholing, one never left idle, the other received <2000 routes, which I know from prior outages to mean that the entire region (Tampa Bay) has been BGP isolated from the rest of their network. Good times... ?On 8/30/20, 8:43 AM, "Outages on behalf of randal k via Outages" <[email protected] on behalf of [email protected]<mailto:[email protected]%20on%20behalf%20of%[email protected]>> wrote: Seeing the same thing - shut a peer, still seeing those routes via 3356 across multiple route-servers & looking glasses. So, do we disconnect 3356 and suffer the blackhole in hopes that it will eventually withdraw those routes, or leave it on to prevent blackholing but suffer massive packet loss to other carriers? Wow. On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages <[email protected]<mailto:[email protected]>> wrote: > > As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale > bgp routes, can be confirmed with AT&T's route server at (telnet > route-server.ip.att.net). > > Stale routes from 1 hour + are still announced by 3356. > > > This is causing blackholing. > _______________________________________________ > Outages mailing list > [email protected]<mailto:[email protected]> > https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= _______________________________________________ Outages mailing list [email protected]<mailto:[email protected]> https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= ------------------------------ Message: 4 Date: Sun, 30 Aug 2020 07:48:14 -0500 From: Chris Adams <[email protected]<mailto:[email protected]>> To: [email protected]<mailto:[email protected]> Subject: Re: [outages] CenturyLink peering issues? Message-ID: <[email protected]<mailto:[email protected]>> Content-Type: text/plain; charset=us-ascii Once upon a time, randal k <[email protected]<mailto:[email protected]>> said: > It's weird you say that - I have been attempting to use Level3's well > known communities to attempt to prepend, no-export etc and have been > having limited luck ... and those communities even appear in > route-views, so I know they're being sent! I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes. But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws? -- Chris Adams <[email protected]<mailto:[email protected]>> ------------------------------ Message: 5 Date: Sun, 30 Aug 2020 06:52:16 -0600 From: randal k <[email protected]<mailto:[email protected]>> To: [email protected]<mailto:[email protected]> Subject: Re: [outages] 3356 does not WITHDRAW bgp routes Message-ID: <CANeLk7RcTGWkQp4SYJQO=j=c04dtpysgrxtxfk9ovi_qkgy...@mail.gmail.com<mailto:CANeLk7RcTGWkQp4SYJQO=j=c04dtpysgrxtxfk9ovi_qkgy...@mail.gmail.com>> Content-Type: text/plain; charset="UTF-8" Watching closely, after applying ^3356$ on our inbound routes, I can see that they are bouncing sessions and slowly adding in prefixes -- I have 17x 3356-originated routes in PA, and 893x in CO. And they have reset the PA BGP session numerous times. On Sun, Aug 30, 2020 at 6:33 AM randal k <[email protected]<mailto:[email protected]>> wrote: > > Seeing the same thing - shut a peer, still seeing those routes via > 3356 across multiple route-servers & looking glasses. > > So, do we disconnect 3356 and suffer the blackhole in hopes that it > will eventually withdraw those routes, or leave it on to prevent > blackholing but suffer massive packet loss to other carriers? Wow. > > On Sun, Aug 30, 2020 at 6:28 AM Lukas Tribus via Outages > <[email protected]<mailto:[email protected]>> wrote: > > > > As previously mentioned by Stephen Flynn, 3356 does not WITHDRAW stale > > bgp routes, can be confirmed with AT&T's route server at (telnet > > route-server.ip.att.net). > > > > Stale routes from 1 hour + are still announced by 3356. > > > > > > This is causing blackholing. > > _______________________________________________ > > Outages mailing list > > [email protected]<mailto:[email protected]> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= ------------------------------ Message: 6 Date: Sun, 30 Aug 2020 13:32:31 +0000 From: Stephen Flynn <[email protected]<mailto:[email protected]>> To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [outages] CenturyLink peering issues? Message-ID: <bn6pr19mb10269d437047598c160a6724eb...@bn6pr19mb1026.namprd19.prod.outlook.com<mailto:bn6pr19mb10269d437047598c160a6724eb...@bn6pr19mb1026.namprd19.prod.outlook.com>> Content-Type: text/plain; charset="us-ascii" Just attempted a prepend (x4) on my Level3 advertisements --- only noticed a change within the NTT network tables. Other large carriers received no advertisement changes (Telia, Hurricane Electric, AT&T) I then noticed that my BGP session was constantly flapping. I've now shutdown my circuit again. Level3 is still announcing my address space. Thank you Level3! Regards, Stephen Flynn Atlantic.Net Direct: (321) 206-1390 [email protected]<mailto:[email protected]> www.atlantic.net<http://www.atlantic.net> -----Original Message----- From: Outages <[email protected]<mailto:[email protected]>> On Behalf Of Chris Adams via Outages Sent: Sunday, August 30, 2020 8:48 AM To: [email protected]<mailto:[email protected]> Subject: Re: [outages] CenturyLink peering issues? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Once upon a time, randal k <[email protected]<mailto:[email protected]>> said: > It's weird you say that - I have been attempting to use Level3's well > known communities to attempt to prepend, no-export etc and have been > having limited luck ... and those communities even appear in > route-views, so I know they're being sent! I brought my session back up, not accepting any routes, and prepending ours (just in case the change propagated)... it took a while, but eventually I do see the prepended routes. But for a provider that said they shut down their link to Level3, I still see their routes... possibly (slowly) propagating changes but not actual withdraws? -- Chris Adams <[email protected]<mailto:[email protected]>> _______________________________________________ Outages mailing list [email protected]<mailto:[email protected]> https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= ------------------------------ Subject: Digest Footer _______________________________________________ Outages mailing list [email protected]<mailto:[email protected]> https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwICAg&c=bFAhm8rB5vXbGyJn-mWQSZrkEgY9uWkKpuo-JUqRuG4&r=Z3AiGMt06fETQSnc5l2T7vSIOF-bixsu4CbUYvAlOxM&m=Lr0Sp0x0UoGt1nlwfLsZu9_oFAIXiSqjH50CkdMWtVc&s=JPfcY3xr11kFtDNgRuUxEfMQsBNdk3iRqqwiw04UU60&e= ------------------------------ End of Outages Digest, Vol 147, Issue 10 ****************************************
_______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
