Re: [outages] Optus/Akamai
Those are actually one and the same if you think about it, but yes, you are correct… To make matters worse, it’s not particularly deterministic which routes are dropped. However, I’ll make the argument that losing some random assortment of destinations is almost always going to be better than losing your entire ability to control your peering routers. YMMV. Owen > On Nov 14, 2023, at 11:37, Jon Lewis via Outages wrote: > > In my experience with such things, it's not a matter of "some routes will be > dropped", but "traffic to certain destinations will blackhole unless there's > a covering route in FIB." > > On Thu, 9 Nov 2023, Ross Tajvar via Outages wrote: > >> Surely it's better to drop some routes than to drop the whole session. >> On Thu, Nov 9, 2023, 5:26 PM Rubens Kuhl via Outages >> wrote: >> On Thu, Nov 9, 2023 at 7:22 PM Job Snijders via Outages >> wrote: >> > >> > On Thu, 9 Nov 2023 at 23:18, Steve Meuse via Outages >> wrote: >> >> >> >> Are people using max-prefix for iBGP sessions? >> >> >> >> That seems.unwise. >> > >> > >> > >> > Yes, I find it hard to imagine what risks would be mitigated by >> applying max-prefix limits to IBGP sessions. >> >> TCAM limits, perhaps ? >> >> Rubens >> ___ >> Outages mailing list >> Outages@outages.org >> https://puck.nether.net/mailman/listinfo/outages >> > > -- > Jon Lewis, MCP :) | I route > Blue Stream Fiber, Sr. Neteng | therefore you are > _ http://www.lewis.org/~jlewis/pgp for PGP public > key > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
In my experience with such things, it's not a matter of "some routes will be dropped", but "traffic to certain destinations will blackhole unless there's a covering route in FIB." On Thu, 9 Nov 2023, Ross Tajvar via Outages wrote: Surely it's better to drop some routes than to drop the whole session. On Thu, Nov 9, 2023, 5:26 PM Rubens Kuhl via Outages wrote: On Thu, Nov 9, 2023 at 7:22 PM Job Snijders via Outages wrote: > > On Thu, 9 Nov 2023 at 23:18, Steve Meuse via Outages wrote: >> >> Are people using max-prefix for iBGP sessions? >> >> That seems.unwise. > > > > Yes, I find it hard to imagine what risks would be mitigated by applying max-prefix limits to IBGP sessions. TCAM limits, perhaps ? Rubens ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- Jon Lewis, MCP :) | I route Blue Stream Fiber, Sr. Neteng | therefore you are _ http://www.lewis.org/~jlewis/pgp for PGP public key Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
- On Nov 9, 2023, at 2:31 PM, Ross Tajvar via Outages outages@outages.org wrote: Hi, > Surely it's better to drop some routes than to drop the whole session. Not necessarily. If part of your IBGP feed includes 10/8 (common in datacenters), you may end up in a situation where it's better to disable a leaking peer. Imagine a Trident 3 box in a spine layer, with 32 northbound peers and 32 southbound peers. Northbound you'll receive a few hundred routes, including 10/8. Southbound, you'll receive a few hundred host subnets and perhaps some any-cast /32s. You'll aggregate the host subnets northbound. If one of those northbound peers leaks a full table, you run the risk of losing some of your host subnets, or perhaps important any-cast routes. I would prefer to lose the offending peer over risking host routes, primarily because shutting down the peer is deterministic while you have no control over which routes are lost. Thanks, Sabri ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
Surely it's better to drop some routes than to drop the whole session. On Thu, Nov 9, 2023, 5:26 PM Rubens Kuhl via Outages wrote: > On Thu, Nov 9, 2023 at 7:22 PM Job Snijders via Outages > wrote: > > > > On Thu, 9 Nov 2023 at 23:18, Steve Meuse via Outages < > outages@outages.org> wrote: > >> > >> Are people using max-prefix for iBGP sessions? > >> > >> That seems.unwise. > > > > > > > > Yes, I find it hard to imagine what risks would be mitigated by applying > max-prefix limits to IBGP sessions. > > TCAM limits, perhaps ? > > > Rubens > ___ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
On Thu, Nov 9, 2023 at 7:22 PM Job Snijders via Outages wrote: > > On Thu, 9 Nov 2023 at 23:18, Steve Meuse via Outages > wrote: >> >> Are people using max-prefix for iBGP sessions? >> >> That seems.unwise. > > > > Yes, I find it hard to imagine what risks would be mitigated by applying > max-prefix limits to IBGP sessions. TCAM limits, perhaps ? Rubens ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
On Thu, 9 Nov 2023 at 23:18, Steve Meuse via Outages wrote: > Are people using max-prefix for iBGP sessions? > > That seems.unwise. > Yes, I find it hard to imagine what risks would be mitigated by applying max-prefix limits to IBGP sessions. Kind regards, Job > ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
Are people using max-prefix for iBGP sessions?
That seems.unwise.
-Steve
On Thu, Nov 9, 2023 at 1:24 AM Gert Doering via Outages
wrote:
> Hi,
>
> On Wed, Nov 08, 2023 at 03:25:13PM -0600, Aaron Wendel via Outages wrote:
> > That makes no sense. How would tripping the max prefix on a single peer
> cause
> > a major outage?
>
> If you have
>
> client --> border router -> route reflector -> all other BGP speakers
>
> and the "RR -> BGP speakers" sessions get tripped due to "client sending
> in too many new routes", then your whole network will fall apart until
> you can shutdown that initial BGP session (or re-provision the other
> sessions, which might not work due to "there is no connectivity to
> the management systems, because, BGP is down").
>
> *Iff* this happens, and you do not have working OOB access including
> being able to do local config changes on the routers ("all configs are
> done by the automatization, no local access possible"), such a problem will
> be extremely messy to recover. Especially figuring out *what* happened,
> if you have no visibility because the routers have lost the route to your
> syslog servers
>
> gert
> --
> "If was one thing all people took for granted, was conviction that if you
> feed honest figures into a computer, honest figures come out. Never
> doubted
> it myself till I met a computer with a sense of humor."
> Robert A. Heinlein, The Moon is a Harsh
> Mistress
>
> Gert Doering - Munich, Germany
> g...@greenie.muc.de
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
--
-Steve
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
Hi,
On Wed, Nov 08, 2023 at 03:25:13PM -0600, Aaron Wendel via Outages wrote:
> That makes no sense. How would tripping the max prefix on a single peer cause
> a major outage?
If you have
client --> border router -> route reflector -> all other BGP speakers
and the "RR -> BGP speakers" sessions get tripped due to "client sending
in too many new routes", then your whole network will fall apart until
you can shutdown that initial BGP session (or re-provision the other
sessions, which might not work due to "there is no connectivity to
the management systems, because, BGP is down").
*Iff* this happens, and you do not have working OOB access including
being able to do local config changes on the routers ("all configs are
done by the automatization, no local access possible"), such a problem will
be extremely messy to recover. Especially figuring out *what* happened,
if you have no visibility because the routers have lost the route to your
syslog servers
gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress
Gert Doering - Munich, Germany g...@greenie.muc.de
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
That makes no sense. How would tripping the max prefix on a single peer cause a major outage? On 11/8/2023 3:13 PM, DaZZa via Outages wrote: There was definitely a major outage with Optus yesterday - not just internet services, but mobile and landline. roughly 10 hour outage for effectively their entire consumer network (weirdly, I have an enterprise service which was not effected by the outage). Definitely got hit by it - our entire corporate mobile fleet was out the whole day. Nobody has given a RFO yet - but I have my doubts it was just BGP issues. That's speculation at this point. D On Thu, 9 Nov 2023 at 01:21, Jay Ashworth via Outages wrote: Off of Facebook of all places, I hear this report that Optus had a major outage in Australia last night, which is believed to have been a BGP screw up involving Akamai: "Nope: it was Akamai increasing their advertised prefix count from their cdn nodes inside the Optus network, and Optus’s configured max-prefix settings on their core bgp sessions from their route reflectors weren’t quite high enough and their whole network went down. Someone didn’t quite think through the value for internal max-prefix rules.." Anybody confirm, deny, got hit by it? Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity. ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages -- Aaron Wendel Chief Technical Officer Wholesale Internet, Inc. (AS 32097) (816)550-9030 http://www.wholesaleinternet.com ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
There was definitely a major outage with Optus yesterday - not just internet services, but mobile and landline. roughly 10 hour outage for effectively their entire consumer network (weirdly, I have an enterprise service which was not effected by the outage). Definitely got hit by it - our entire corporate mobile fleet was out the whole day. Nobody has given a RFO yet - but I have my doubts it was just BGP issues. That's speculation at this point. D On Thu, 9 Nov 2023 at 01:21, Jay Ashworth via Outages wrote: > > Off of Facebook of all places, I hear this report that Optus had a major > outage in Australia last night, which is believed to have been a BGP screw up > involving Akamai: > > "Nope: it was Akamai increasing their advertised prefix count from their cdn > nodes inside the Optus network, and Optus’s configured max-prefix settings on > their core bgp sessions from their route reflectors weren’t quite high enough > and their whole network went down. Someone didn’t quite think through the > value for internal max-prefix rules.." > > Anybody confirm, deny, got hit by it? > > Cheers, > -- jra > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > ___ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages -- veg·e·tar·i·an: Ancient tribal slang for the village idiot who can't hunt, fish or ride ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
Disclaimer: I am employed by Akamai I’m not aware of any changes that could have triggered this, if someone has information they can share you can reach out to me directly or to our NOCC. I can be reached at ja...@akamai.com if you have something private to share and my mobile number is easily findable as well. - Jared > On Nov 8, 2023, at 9:20 AM, Jay Ashworth via Outages > wrote: > > Off of Facebook of all places, I hear this report that Optus had a major > outage in Australia last night, which is believed to have been a BGP screw up > involving Akamai: > > "Nope: it was Akamai increasing their advertised prefix count from their cdn > nodes inside the Optus network, and Optus’s configured max-prefix settings on > their core bgp sessions from their route reflectors weren’t quite high enough > and their whole network went down. Someone didn’t quite think through the > value for internal max-prefix rules.." > > Anybody confirm, deny, got hit by it? > > Cheers, > -- jra > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > ___ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Re: [outages] Optus/Akamai
Something def went very wrong. Completely out for ~6h with ongoing reports of some mobile cells still down, fixed line not working etc. https://ioda.inetintel.cc.gatech.edu/asn/4804?from=1699351533=1699481133 From: Outages on behalf of Jay Ashworth via Outages Sent: 08 November 2023 15:20 To: outages@outages.org Subject: [outages] Optus/Akamai EXTERNAL SENDER. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username or password. Off of Facebook of all places, I hear this report that Optus had a major outage in Australia last night, which is believed to have been a BGP screw up involving Akamai: "Nope: it was Akamai increasing their advertised prefix count from their cdn nodes inside the Optus network, and Optus’s configured max-prefix settings on their core bgp sessions from their route reflectors weren’t quite high enough and their whole network went down. Someone didn’t quite think through the value for internal max-prefix rules.." Anybody confirm, deny, got hit by it? Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity. ___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
[outages] Optus/Akamai
Off of Facebook of all places, I hear this report that Optus had a major outage in Australia last night, which is believed to have been a BGP screw up involving Akamai: "Nope: it was Akamai increasing their advertised prefix count from their cdn nodes inside the Optus network, and Optus’s configured max-prefix settings on their core bgp sessions from their route reflectors weren’t quite high enough and their whole network went down. Someone didn’t quite think through the value for internal max-prefix rules.." Anybody confirm, deny, got hit by it? Cheers, -- jra -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
