Re: anyone else seeing very long AS paths?
Typo in one part so sending to make it accurate. The workaround is to implement CmdBold bgp maxas-limit X NoCmdBold on the device that after prepending would need to send an update with over 255 AS hops. Since IOS limits the inbound prepending value to 10 the most that could be added iss 11 AS hops (10 on ingress, 10 on egress, and 1 for normal eBGP AS hop addition). Therefore, a conservative value to configure would be 200 to prevent this condition. It should be 21 hops (10 in a route-map on ingress, 10 in a route-map on egress, and 1 normal eBGP AS hop addition). Thanks to John Stuppi for pointing it out. Rodney On Thu, Feb 19, 2009 at 03:15:02PM -0500, Rodney Dunn wrote: We are working on a document for Cisco.com but in the interim here is the bug that will fix the issue of a Cisco IOS device sending an incorrectly formatted BGP update when as a result of prepending it goes over 255 AS hops. Note: The Title and Release-note on bug toolkit may be a bit different as I just updated it to be more accurate. Of all the scenarios I've looked at (thanks to those that responded offline) there wasn't a condition found where this could happen without AS path prepending being used. Please respond offline or let's move the discussion over to cisco-nsp at this point. CSCsx73770 Invalid BGP formatted update causes peer reset with AS prepending BSymptom:/B A Cisco IOS device that receives a BGP update message and as a result of AS prepending needs to send an update downstream that would have over 255 AS hops will send an invalid formatted update. This update when received by a downstream BGP speaker triggers a NOTIFICATION back to the sender which results in the BGP session being reset. BConditions:/B This problem is seen when a Cisco IOS device receives a BGP update and due to a combination of either inbound, outbound, or both AS prepending it needs to send an update downstream that has more than 255 AS hops. BWorkaround:/B The workaround is to implement CmdBold bgp maxas-limit X NoCmdBold on the device that after prepending would need to send an update with over 255 AS hops. Since IOS limits the inbound prepending value to 10 the most that could be added iss 11 AS hops (10 on ingress, 10 on egress, and 1 for normal eBGP AS hop addition). Therefore, a conservative value to configure would be 200 to prevent this condition. Full support of Section 5.1.2 of RFC4271 is being tracked under CSCsx75937 Add BGP support of AS paths longer than 255 per Section 5.1.2 of RFC4271 Thanks to those that worked offline with us to verify the field results reported. Rodney On Tue, Feb 17, 2009 at 05:27:01PM -0500, Rodney Dunn wrote: If you want to take this offline send it unicast or we could move it to cisco-nsp. What scenarios are you seeing that appear broken other than when a notification is sent when a 255 hop update is received? That's the one I'm working on right now. Rodney On Tue, Feb 17, 2009 at 05:31:49PM -0500, German Martinez wrote: On Tue Feb 17, 2009, Rodney Dunn wrote: Hello Rodney, It will be great if you can share with us your findings. It seems like we are hitting different bugs in different platforms. Thanks German Ivan, It is confusing but from what I have tested you have it correct. The confusing part comes from multiple issues. a) The documentation about the default maxas limit being 75 appears to be incorrect. I'll get that fixed. b) Prior to CSCee30718 there was a hard limit of 255. After that fix AS sets of more than 255 should work. c) CSCeh13489 implemented the maxas command to mark it as invalid and not send. There does appear to be an issue when you cross the 255 boundary and the next hop router sends a notification back. I've got it recreated in the lab and we are working to clearly understand why that is. I'll post an update once we have more. The way to prevent it is the upstream device that crosses the 255 boundary on sending needs to use the maxas limit command to keep it less than 255. It doesn't work on the device that receives the update with the AS path larger than 255. Rodney On Tue, Feb 17, 2009 at 08:58:48PM +0100, Ivan Pepelnjak wrote: We were dropping ALL prefixes and the eBGP session was still resetting. Upstream or downstream? 1) bgp maxas-limit 75 had no effect mitigating this problem on the IOS we were using. That is: it was previously verified to be working just fine to drop paths longer than 75, but once we started receiving paths 255 then BGP started resetting. I was able to receive BGP paths longer than 255 on IOS release 12.2SRC. The paths were generated by Quagga BGP daemon.
Re: anyone else seeing very long AS paths?
We are working on a document for Cisco.com but in the interim here is the bug that will fix the issue of a Cisco IOS device sending an incorrectly formatted BGP update when as a result of prepending it goes over 255 AS hops. Note: The Title and Release-note on bug toolkit may be a bit different as I just updated it to be more accurate. Of all the scenarios I've looked at (thanks to those that responded offline) there wasn't a condition found where this could happen without AS path prepending being used. Please respond offline or let's move the discussion over to cisco-nsp at this point. CSCsx73770 Invalid BGP formatted update causes peer reset with AS prepending BSymptom:/B A Cisco IOS device that receives a BGP update message and as a result of AS prepending needs to send an update downstream that would have over 255 AS hops will send an invalid formatted update. This update when received by a downstream BGP speaker triggers a NOTIFICATION back to the sender which results in the BGP session being reset. BConditions:/B This problem is seen when a Cisco IOS device receives a BGP update and due to a combination of either inbound, outbound, or both AS prepending it needs to send an update downstream that has more than 255 AS hops. BWorkaround:/B The workaround is to implement CmdBold bgp maxas-limit X NoCmdBold on the device that after prepending would need to send an update with over 255 AS hops. Since IOS limits the inbound prepending value to 10 the most that could be added iss 11 AS hops (10 on ingress, 10 on egress, and 1 for normal eBGP AS hop addition). Therefore, a conservative value to configure would be 200 to prevent this condition. Full support of Section 5.1.2 of RFC4271 is being tracked under CSCsx75937 Add BGP support of AS paths longer than 255 per Section 5.1.2 of RFC4271 Thanks to those that worked offline with us to verify the field results reported. Rodney On Tue, Feb 17, 2009 at 05:27:01PM -0500, Rodney Dunn wrote: If you want to take this offline send it unicast or we could move it to cisco-nsp. What scenarios are you seeing that appear broken other than when a notification is sent when a 255 hop update is received? That's the one I'm working on right now. Rodney On Tue, Feb 17, 2009 at 05:31:49PM -0500, German Martinez wrote: On Tue Feb 17, 2009, Rodney Dunn wrote: Hello Rodney, It will be great if you can share with us your findings. It seems like we are hitting different bugs in different platforms. Thanks German Ivan, It is confusing but from what I have tested you have it correct. The confusing part comes from multiple issues. a) The documentation about the default maxas limit being 75 appears to be incorrect. I'll get that fixed. b) Prior to CSCee30718 there was a hard limit of 255. After that fix AS sets of more than 255 should work. c) CSCeh13489 implemented the maxas command to mark it as invalid and not send. There does appear to be an issue when you cross the 255 boundary and the next hop router sends a notification back. I've got it recreated in the lab and we are working to clearly understand why that is. I'll post an update once we have more. The way to prevent it is the upstream device that crosses the 255 boundary on sending needs to use the maxas limit command to keep it less than 255. It doesn't work on the device that receives the update with the AS path larger than 255. Rodney On Tue, Feb 17, 2009 at 08:58:48PM +0100, Ivan Pepelnjak wrote: We were dropping ALL prefixes and the eBGP session was still resetting. Upstream or downstream? 1) bgp maxas-limit 75 had no effect mitigating this problem on the IOS we were using. That is: it was previously verified to be working just fine to drop paths longer than 75, but once we started receiving paths 255 then BGP started resetting. I was able to receive BGP paths longer than 255 on IOS release 12.2SRC. The paths were generated by Quagga BGP daemon. 12.2SRC causes the downstream session to break when the installed AS-path length is close to 255 and you use downstream AS-path prepending. In your case, I'm assuming you were hit with an older bug (probably at the 128 AS-path length boundary). It would be very hard to generate just the right AS-path length to unintentionally break your upstream EBGP session (as I said before, it's a nice targeted attack if you know your downstream topology). If your IOS is vulnerable to the older bugs that break inbound processing of AS paths longer than 128, there's nothing you can do on your end. The internal BGP checks reject the inbound update before the inbound filters (or bgp maxas-limit) can touch it and reset the upstream BGP session. Hope this
Re: anyone else seeing very long AS paths?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, I just received reply from Sloan-Park, that they have shutdown that customer yesterday 6:40pm CET and the customer has been requested to clean-up his config. BR Jens Jason Kalai Arasu schrieb: I encountered it yesterday from AS47868. -Original Message- From: Paul Ferguson [mailto:fergdawgs...@gmail.com] Sent: Tuesday, February 17, 2009 01:02 PM To: nanog@nanog.org Subject: Re: anyone else seeing very long AS paths? On Mon, Feb 16, 2009 at 8:51 PM, Michael Ulitskiy mulits...@acedsl.com wrote: It hit my routers at 11:26:40, EST. Michael On Monday 16 February 2009 07:26:23 pm Adam Greene wrote: Anyone have an estimate as to when these long announcements began? Seems like the first reports appeared just before noon, UTC-05. We noticed a significant dip in Internet traffic to AS11579 for a few minutes last night (19:00 UTC-05) which we've been trying to hunt down the cause of. At first glance, the two events seem unrelated. Anyone else see anything similar? Just as a follow-up -- and in case anyone hasn't read these yet: http://www.renesys.com/blog/2009/02/the-flap-heard-around-the-worl.shtml http://asert.arbornetworks.com/2009/02/ahh-the-ease-of-introducing-globa l-r outing-instability/ - ferg - -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/ - -- === Jens Ott Leiter Network Management Tel: +49 22 33 - 612 - 3501 Fax: +49 22 33 - 612 - 53501 E-Mail: j@plusserver.de GPG-Fingerprint: 808A EADF C476 FABE 2366 8402 31FD 328C C2CA 7D7A PlusServer AG Daimlerstraße 9-11 50354 Hürth Germany HRB 58428 / Amtsgericht Köln, USt-ID DE216 740 823 Vorstand: Jochen Berger, Frank Gross, Jan Osthues, Thomas Strohe Aufsichtsratsvorsitz: Claudius Schmalschläger === -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmafQAACgkQMf0yjMLKfXowUACgi4F/j+eGkFfL+2G01r/Ohb0Q XgIAoI4jH6WrkngSOUlDK5lBUZZ3wuEE =/66k -END PGP SIGNATURE-
Re: anyone else seeing very long AS paths?
* Hank Nussbacher: They will keep trying and until a vast majority of ISPs implement maxas, this will keep happening. Or enthusiastic prepending will be used more often to override local preference. Hard to tell. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
Re: anyone else seeing very long AS paths?
On Tue, Feb 17, 2009 at 08:07:36AM +0200, Hank Nussbacher wrote: A regular UN of attempts to do this previously: 24532 - PT. Inet Global Indo, Indonesia 43179 - Team Consulting AS, Bosnia and Herzegovina 48262 - Noblecom Ltd., Bulgaria 6488 - Arizona Macintosh Users Group, USA 39625 - Omni-Araneo, Poland 33838 - BetaNET sp. z o.o, Poland 47868 - SUPRO, spol. s r.o., Czech Republic They will keep trying and until a vast majority of ISPs implement maxas, this will keep happening. Or until people who are still running multi-year old cisco code actually upgrade? This seems to primarily impact: 1) Old cisco code 2) PC based bgp daemons Both of which likely just need to be upgraded. I actually suspect that a lot of people who dropped their bgp sessions did not notice something happened, and still will not upgrade their code. I searched the archives, some variations of this have happened since 2001. There's been a few PSIRT and other issues since then, I suspect these people don't even know they have a bgp speaking device anymore. - Jared -- Jared Mauch | pgp key available via finger from ja...@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: anyone else seeing very long AS paths?
Jared Mauch wrote: On Tue, Feb 17, 2009 at 08:07:36AM +0200, Hank Nussbacher wrote: They will keep trying and until a vast majority of ISPs implement maxas, this will keep happening. Or until people who are still running multi-year old cisco code actually upgrade? This seems to primarily impact: 1) Old cisco code 2) PC based bgp daemons Both of which likely just need to be upgraded. I actually suspect that a lot of people who dropped their bgp sessions did not notice something happened, and still will not upgrade their codeI suspect these people don't even know they have a bgp speaking device anymore. On the other hand, the fact that various entities have gone out of their way to advertise that they're running old hardware/out-of-date software has been noted elsewhere. I'd strongly suggest, if you're reading NANOG, that you update, before someone less pleasant and friendly than myself finds you. Please. -- Remember, if it's in the news, don't worry about it. The very definition of news is something that almost never happens. When something is so common that it's no longer news -- car crashes, domestic violence -- that's when you should worry about it. (Bruce Schneier)
Re: anyone else seeing very long AS paths?
On Tue, Feb 17, 2009, Etaoin Shrdlu wrote:
On the other hand, the fact that various entities have gone out of their
way to advertise that they're running old hardware/out-of-date software
has been noted elsewhere. I'd strongly suggest, if you're reading NANOG,
that you update, before someone less pleasant and friendly than myself
finds you. Please.
What, and the other, make sure you hard limit the max AS path length from
customers and peers, in case of ${LINK_TO_THIS_NANOG_THREAD} ?
Adrian
Re: anyone else seeing very long AS paths?
My bgp speaking devices are a couple of 7200s running 12.2(40). Not the newest IOS out there, but it's been doing the job just fine up until yesterday. Yesterday, when that malformed announcement hit my routers they didn't crash, but they did reset bgp sessions (even though I didn't accept the route) and they kept doing so until I got my upstream to filter it out. According to cisco bug toolkit CSCdr54230 should be fixed in 12.2, so obviously it's not enough. Does anybody know what IOS version has fix this problem, 'cause I couldn't find this info at CCO? Thanks, Michael On Tuesday 17 February 2009 10:21:07 am Etaoin Shrdlu wrote: Jared Mauch wrote: On Tue, Feb 17, 2009 at 08:07:36AM +0200, Hank Nussbacher wrote: They will keep trying and until a vast majority of ISPs implement maxas, this will keep happening. Or until people who are still running multi-year old cisco code actually upgrade? This seems to primarily impact: 1) Old cisco code 2) PC based bgp daemons Both of which likely just need to be upgraded. I actually suspect that a lot of people who dropped their bgp sessions did not notice something happened, and still will not upgrade their codeI suspect these people don't even know they have a bgp speaking device anymore. On the other hand, the fact that various entities have gone out of their way to advertise that they're running old hardware/out-of-date software has been noted elsewhere. I'd strongly suggest, if you're reading NANOG, that you update, before someone less pleasant and friendly than myself finds you. Please.
Re: anyone else seeing very long AS paths?
On Tue, 17 Feb 2009, Jared Mauch wrote: Or until people who are still running multi-year old cisco code actually upgrade? This seems to primarily impact: 1) Old cisco code 2) PC based bgp daemons Both of which likely just need to be upgraded. I actually suspect that a lot of people who dropped their bgp sessions did not notice something happened, and still will not upgrade their code. I searched the archives, some variations of this have happened since 2001. There's been a few PSIRT and other issues since then, I suspect these people don't even know they have a bgp speaking device anymore. While at it - perhaps others wish to join this bugid so as to enhance IOS: CSCso47162 Bug Details BGP-6-ASPATH message should print offending prefix(es) None Symptoms Syslog message below doesn't print info about offending prefix(es) %BGP-6-ASPATH: Invalid AS path [chars] received from [int]: [chars] Further Problem Description Examples of such a message : %BGP-6-ASPATH: Long AS path 64501 64501 65000 65000 received from x.x.x.x: Morethan configured MAXAS-LIMIT %BGP-6-ASPATH: Invalid AS path (64721) 64700 64720 65400 65231 received from x.x.x.x: Non confederation peer I opened it in March 2008 and the more people who bug Cisco to implement this sev 6 request - the better off we will all be in the future. -Hank
Re: anyone else seeing very long AS paths?
On Tue Feb 17, 2009, Michael Ulitskiy wrote: Hello, CSCee30718 – it removes the default value of bgp max-as from the router. The solution is introduced in CSCeh13489 BGP shouldn't propogate an update w excessive AS Path 255 Symptoms: A router may reset its Border Gateway Protocol (BGP) session. Conditions: This symptom is observed when a Cisco router that peers with other routers receives an Autonomous System (AS) path with a length that is equal to or greater than 255. Workaround: Configure the bgp maxas limit command in such as way that the maximum length of the AS path is a value below 255. When the router receives an update with an excessive AS path value, the prefix is rejected and recorded the event in the log. This workaround has been suggested previously by Hank. Anyone knows about any possible CPU impacts in case that you implement bgp maxas? Thanks German My bgp speaking devices are a couple of 7200s running 12.2(40). Not the newest IOS out there, but it's been doing the job just fine up until yesterday. Yesterday, when that malformed announcement hit my routers they didn't crash, but they did reset bgp sessions (even though I didn't accept the route) and they kept doing so until I got my upstream to filter it out. According to cisco bug toolkit CSCdr54230 should be fixed in 12.2, so obviously it's not enough. Does anybody know what IOS version has fix this problem, 'cause I couldn't find this info at CCO? Thanks, Michael On Tuesday 17 February 2009 10:21:07 am Etaoin Shrdlu wrote: Jared Mauch wrote: On Tue, Feb 17, 2009 at 08:07:36AM +0200, Hank Nussbacher wrote: They will keep trying and until a vast majority of ISPs implement maxas, this will keep happening. Or until people who are still running multi-year old cisco code actually upgrade? This seems to primarily impact: 1) Old cisco code 2) PC based bgp daemons Both of which likely just need to be upgraded. I actually suspect that a lot of people who dropped their bgp sessions did not notice something happened, and still will not upgrade their codeI suspect these people don't even know they have a bgp speaking device anymore. On the other hand, the fact that various entities have gone out of their way to advertise that they're running old hardware/out-of-date software has been noted elsewhere. I'd strongly suggest, if you're reading NANOG, that you update, before someone less pleasant and friendly than myself finds you. Please. pgptsqV6JzgBN.pgp Description: PGP signature
Re: anyone else seeing very long AS paths?
German Martinez wrote: Workaround: Configure the bgp maxas limit command in such as way that the maximum length of the AS path is a value below 255. When the router receives an update with an excessive AS path value, the prefix is rejected and recorded the event in the log. This workaround has been suggested previously by Hank. Anyone knows about any possible CPU impacts in case that you implement bgp maxas? bgp max-as will NOT protect you from this exploit (but if you are not vulnerable it should prevent you from propogating it). As far as I can tell the ONLY defense for a vulnerable IOS is to not run BGP. Dropping every received route with a filter on 0/0 does not mitigate the attack - as soon as that bogus as-path is received the BGP session resets, even if the route is never actually installed (and as far as I can tell the only real effect of the bgp maxas-limit 75 is to cause all paths with more than 75 ASN to not be installed in the routing table).
Re: anyone else seeing very long AS paths?
On Tue Feb 17, 2009, Mike Lewinski wrote: bgp max-as will NOT protect you from this exploit (but if you are not vulnerable it should prevent you from propogating it). Are you trying to say that the receiving bgp speaker will drop the session no matter what but it won't forward the update? Here is what I have found on Cisco's website bgp maxas-limit To configure Border Gateway Protocol (BGP) to discard routes that have a number of as-path segments that exceed the specified value, use the bgp maxas-limit command in router configuration mode. To return the router to default operation, use the no form of this command. Usage Guidelines The bgp maxas-limit command is used to limit the number of as-path segments that are permitted in inbound routes. If a route is received with an as-path segment that exceeds the configured limit, the BGP routing process will discard the route. I heard about people running this command that were not impacted As far as I can tell the ONLY defense for a vulnerable IOS is to not run BGP. Dropping every received route with a filter on 0/0 does not mitigate the attack - as soon as that bogus as-path is received the BGP session resets, even if the route is never actually installed (and as far as I can tell the only real effect of the bgp maxas-limit 75 is to cause all paths with more than 75 ASN to not be installed in the routing table). pgphPXgUDIi6Q.pgp Description: PGP signature
Re: anyone else seeing very long AS paths?
German Martinez wrote: On Tue Feb 17, 2009, Mike Lewinski wrote: bgp max-as will NOT protect you from this exploit (but if you are not vulnerable it should prevent you from propogating it). Are you trying to say that the receiving bgp speaker will drop the session no matter what but it won't forward the update? There are reports that some versions of IOS will drop a peer upon receiving the long AS, even with a bgp max-as command. I can only presume that there are some IOS versions that determine the update is invalid prior to the max-as command determining we are not keeping the route. The whole is the update valid? vs do I want this in my routing table? Jack
RE: anyone else seeing very long AS paths?
According to publicly available bug toolkit, CSCee30718 did not touch the maxas limit. The hard-coded maxas-limit in recent IOS releases is 254 (not 75 as suggested in a previous e-mail). Classic IOS (I did not test XE, XR or NX) can handle inbound updates with AS path lengths above 255, but fails miserably when it has to send an oversized update (producing invalid BGP UPDATE message), resulting in a flapping BGP session (anyone who wants to test this behavior and report/fix this bug can get all the files needed to reproduce it). The hard-coded maxas-limit prevents this behavior (254 + my AS = 255), but if you use AS-path prepending on outbound update, you're fried. The __ONLY__ way to be on the safe side is to configure bgp maxas-limit, otherwise someone who knows you're doing AS-path prepending on major peering sessions (and no inbound AS-path filtering) can selectively target your peering points. I've summarized everything I've discovered in various stress tests today (well, not the targeted attack :) in this article: http://wiki.nil.com/Limit_the_maximum_BGP_AS-path_length Feel free to improve it:) Ivan http://blog.ioshints.info -Original Message- From: German Martinez [mailto:gmart...@ajax.opentransit.net] Sent: Tuesday, February 17, 2009 7:55 PM To: Michael Ulitskiy Cc: nanog@nanog.org Subject: Re: anyone else seeing very long AS paths? On Tue Feb 17, 2009, Michael Ulitskiy wrote: Hello, CSCee30718 - it removes the default value of bgp max-as from the router. The solution is introduced in CSCeh13489 BGP shouldn't propogate an update w excessive AS Path 255 Symptoms: A router may reset its Border Gateway Protocol (BGP) session. Conditions: This symptom is observed when a Cisco router that peers with other routers receives an Autonomous System (AS) path with a length that is equal to or greater than 255. Workaround: Configure the bgp maxas limit command in such as way that the maximum length of the AS path is a value below 255. When the router receives an update with an excessive AS path value, the prefix is rejected and recorded the event in the log. This workaround has been suggested previously by Hank. Anyone knows about any possible CPU impacts in case that you implement bgp maxas? Thanks German My bgp speaking devices are a couple of 7200s running 12.2(40). Not the newest IOS out there, but it's been doing the job just fine up until yesterday. Yesterday, when that malformed announcement hit my routers they didn't crash, but they did reset bgp sessions (even though I didn't accept the route) and they kept doing so until I got my upstream to filter it out. According to cisco bug toolkit CSCdr54230 should be fixed in 12.2, so obviously it's not enough. Does anybody know what IOS version has fix this problem, 'cause I couldn't find this info at CCO? Thanks, Michael On Tuesday 17 February 2009 10:21:07 am Etaoin Shrdlu wrote: Jared Mauch wrote: On Tue, Feb 17, 2009 at 08:07:36AM +0200, Hank Nussbacher wrote: They will keep trying and until a vast majority of ISPs implement maxas, this will keep happening. Or until people who are still running multi-year old cisco code actually upgrade? This seems to primarily impact: 1) Old cisco code 2) PC based bgp daemons Both of which likely just need to be upgraded. I actually suspect that a lot of people who dropped their bgp sessions did not notice something happened, and still will not upgrade their codeI suspect these people don't even know they have a bgp speaking device anymore. On the other hand, the fact that various entities have gone out of their way to advertise that they're running old hardware/out-of-date software has been noted elsewhere. I'd strongly suggest, if you're reading NANOG, that you update, before someone less pleasant and friendly than myself finds you. Please.
Re: anyone else seeing very long AS paths?
Ivan Pepelnjak wrote: Classic IOS (I did not test XE, XR or NX) can handle inbound updates with AS path lengths above 255, but fails miserably when it has to send an oversized update (producing invalid BGP UPDATE message), resulting in a flapping BGP session (anyone who wants to test this behavior and report/fix this bug can get all the files needed to reproduce it). Just to reconfirm. The issue arrives with sending an update, not receiving? So if an ISP does not have a limit and their IOS cannot handle this, they will send an invalid BGP UPDATE to the downstream peers causing them to reset regardless of their max as-path settings? Just trying to reconfirm, so I can inform my customers if there was anything they could do to prevent it, or if it is actually their provider that instigated the peer reset by sending invalid updates. -Jack
Re: anyone else seeing very long AS paths?
Jack Bates wrote: Just to reconfirm. The issue arrives with sending an update, not receiving? So if an ISP does not have a limit and their IOS cannot handle this, they will send an invalid BGP UPDATE to the downstream peers causing them to reset regardless of their max as-path settings? Just trying to reconfirm, so I can inform my customers if there was anything they could do to prevent it, or if it is actually their provider that instigated the peer reset by sending invalid updates. We were dropping ALL prefixes and the eBGP session was still resetting. I used this filter: ip prefix-list drop seq 1 deny 0.0.0.0/0 ge 32 router bgp neighbor x.x.x.x prefix-list drop in I confirmed via show ip bgp sum that there were 0 prefixes received. Of course show ip bgp nei x.x.x.x received-routes still showed the routes coming in, they just weren't being installed into the tables (or redistributed to any iBGP neighbors). So, to reiterate: 1) bgp maxas-limit 75 had no effect mitigating this problem on the IOS we were using. That is: it was previously verified to be working just fine to drop paths longer than 75, but once we started receiving paths 255 then BGP started resetting. 2) prefix-list drop in ensured that ALL eBGP learned routes were dropped before being installed or re-advertised. eBGP still reset itself every three minutes or so, which was about the length of time it took for the session to restore itself and get to the offending route. I believe that upgraded IOS is the ONLY possibly fix in some cases.
RE: anyone else seeing very long AS paths?
As far as I understand the issues :) There are two limits: the first one @ 128 AS numbers (where BGP switches to the 'extended length' of BGP attribute), the other one @ 256 AS numbers (where BGP has to use two AS_SEQUENCE segments). Old IOS releases break on the first boundary when processing INBOUND update. bgp maxas-limit cannot save you, as the router drops UPSTREAM BGP session before the update is fully processed. New IOS releases accept all INBOUND updates. Prefixes with AS-path length above 254 are never valid (the long printout contains maxas-limit status). bgp maxas-limit works on inbound updates and thus drops whatever you feel is oversized. New IOS release fail when sending OUTBOUND updates. If you've configured bgp maxas-limit, you're safe. If not, your DOWNSTREAM BGP session will be dropped by your neighbor receiving invalid BGP update. If your customers are using old IOS, there was nothing they could do to prevent the BGP session reset (apart from upgrading, obviously :). Who was the actual culprit depends on the AS-path length ... See above. Does this make sense? I know it's complex :) Ivan -Original Message- From: Jack Bates [mailto:jba...@brightok.net] Sent: Tuesday, February 17, 2009 8:31 PM To: Ivan Pepelnjak Cc: nanog@nanog.org Subject: Re: anyone else seeing very long AS paths? Ivan Pepelnjak wrote: Classic IOS (I did not test XE, XR or NX) can handle inbound updates with AS path lengths above 255, but fails miserably when it has to send an oversized update (producing invalid BGP UPDATE message), resulting in a flapping BGP session (anyone who wants to test this behavior and report/fix this bug can get all the files needed to reproduce it). Just to reconfirm. The issue arrives with sending an update, not receiving? So if an ISP does not have a limit and their IOS cannot handle this, they will send an invalid BGP UPDATE to the downstream peers causing them to reset regardless of their max as-path settings? Just trying to reconfirm, so I can inform my customers if there was anything they could do to prevent it, or if it is actually their provider that instigated the peer reset by sending invalid updates. -Jack
RE: anyone else seeing very long AS paths?
We were dropping ALL prefixes and the eBGP session was still resetting. Upstream or downstream? 1) bgp maxas-limit 75 had no effect mitigating this problem on the IOS we were using. That is: it was previously verified to be working just fine to drop paths longer than 75, but once we started receiving paths 255 then BGP started resetting. I was able to receive BGP paths longer than 255 on IOS release 12.2SRC. The paths were generated by Quagga BGP daemon. 12.2SRC causes the downstream session to break when the installed AS-path length is close to 255 and you use downstream AS-path prepending. In your case, I'm assuming you were hit with an older bug (probably at the 128 AS-path length boundary). It would be very hard to generate just the right AS-path length to unintentionally break your upstream EBGP session (as I said before, it's a nice targeted attack if you know your downstream topology). If your IOS is vulnerable to the older bugs that break inbound processing of AS paths longer than 128, there's nothing you can do on your end. The internal BGP checks reject the inbound update before the inbound filters (or bgp maxas-limit) can touch it and reset the upstream BGP session. Hope this helps Ivan Disclaimer: as I don't have internal access to Cisco, all of the above is a result of lab tests.
Re: anyone else seeing very long AS paths?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Feb 17, 2009, at 1:50 PM, Ivan Pepelnjak wrote: As far as I understand the issues :) There are two limits: the first one @ 128 AS numbers (where BGP switches to the 'extended length' of BGP attribute), the other one @ 256 AS numbers (where BGP has to use two AS_SEQUENCE segments). Old IOS releases break on the first boundary when processing INBOUND update. bgp maxas-limit cannot save you, as the router drops UPSTREAM BGP session before the update is fully processed. New IOS releases accept all INBOUND updates. Prefixes with AS-path length above 254 are never valid (the long printout contains maxas-limit status). bgp maxas-limit works on inbound updates and thus drops whatever you feel is oversized. New IOS release fail when sending OUTBOUND updates. If you've configured bgp maxas-limit, you're safe. If not, your DOWNSTREAM BGP session will be dropped by your neighbor receiving invalid BGP update. If your customers are using old IOS, there was nothing they could do to prevent the BGP session reset (apart from upgrading, obviously :). Who was the actual culprit depends on the AS-path length ... See above. Does this make sense? I know it's complex :) Ivan What is not yet clear is, what are the definitions of Old IOS release and New IOS release? There has been talk of a bug referred to as CSCdr54230. I have seen statements on another list that this was fixed in 12.1(4) and 12.0(10)S3, but yet this problem was experienced on such releases as 12.2(40). Has there been any definitive word yet on what it takes to qualify as a new IOS release? Steve - -- - --- Steven Saner ssa...@hubris.net Director of Network Operations Hubris Communications -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkmbGI8ACgkQvgCxUpg3pZOfgQCeOCnoDIwX/IMF+wfnM8md2SiM LSEAoIptOHmO7yPhAGdVZ8+dlhCiOI8k =WD0q -END PGP SIGNATURE-
Re: anyone else seeing very long AS paths?
On Tue Feb 17, 2009, Ivan Pepelnjak wrote: According to publicly available bug toolkit, CSCee30718 did not touch the maxas limit. I will double check this with Cisco pgpeuQs06hcKd.pgp Description: PGP signature
Re: anyone else seeing very long AS paths?
Ivan, It is confusing but from what I have tested you have it correct. The confusing part comes from multiple issues. a) The documentation about the default maxas limit being 75 appears to be incorrect. I'll get that fixed. b) Prior to CSCee30718 there was a hard limit of 255. After that fix AS sets of more than 255 should work. c) CSCeh13489 implemented the maxas command to mark it as invalid and not send. There does appear to be an issue when you cross the 255 boundary and the next hop router sends a notification back. I've got it recreated in the lab and we are working to clearly understand why that is. I'll post an update once we have more. The way to prevent it is the upstream device that crosses the 255 boundary on sending needs to use the maxas limit command to keep it less than 255. It doesn't work on the device that receives the update with the AS path larger than 255. Rodney On Tue, Feb 17, 2009 at 08:58:48PM +0100, Ivan Pepelnjak wrote: We were dropping ALL prefixes and the eBGP session was still resetting. Upstream or downstream? 1) bgp maxas-limit 75 had no effect mitigating this problem on the IOS we were using. That is: it was previously verified to be working just fine to drop paths longer than 75, but once we started receiving paths 255 then BGP started resetting. I was able to receive BGP paths longer than 255 on IOS release 12.2SRC. The paths were generated by Quagga BGP daemon. 12.2SRC causes the downstream session to break when the installed AS-path length is close to 255 and you use downstream AS-path prepending. In your case, I'm assuming you were hit with an older bug (probably at the 128 AS-path length boundary). It would be very hard to generate just the right AS-path length to unintentionally break your upstream EBGP session (as I said before, it's a nice targeted attack if you know your downstream topology). If your IOS is vulnerable to the older bugs that break inbound processing of AS paths longer than 128, there's nothing you can do on your end. The internal BGP checks reject the inbound update before the inbound filters (or bgp maxas-limit) can touch it and reset the upstream BGP session. Hope this helps Ivan Disclaimer: as I don't have internal access to Cisco, all of the above is a result of lab tests.
Re: anyone else seeing very long AS paths?
Steven Saner wrote: What is not yet clear is, what are the definitions of Old IOS release and New IOS release? There has been talk of a bug referred to as CSCdr54230. I have seen statements on another list that this was fixed in 12.1(4) and 12.0(10)S3, but yet this problem was experienced on such releases as 12.2(40). Has there been any definitive word yet on what it takes to qualify as a new IOS release? I believe we are looking at multiple bugs with similar effects at different boundaries. Cisco, per earlier in the thread, will be having lots of coffee tonight. :) Jack
Re: anyone else seeing very long AS paths?
On Tue, 17 Feb 2009, Mike Lewinski wrote: German Martinez wrote: bgp max-as will NOT protect you from this exploit (but if you are not vulnerable it should prevent you from propogating it). I can confirm this statement... (unfortunately) L.
Re: anyone else seeing very long AS paths?
On Tue Feb 17, 2009, Rodney Dunn wrote: Hello Rodney, It will be great if you can share with us your findings. It seems like we are hitting different bugs in different platforms. Thanks German Ivan, It is confusing but from what I have tested you have it correct. The confusing part comes from multiple issues. a) The documentation about the default maxas limit being 75 appears to be incorrect. I'll get that fixed. b) Prior to CSCee30718 there was a hard limit of 255. After that fix AS sets of more than 255 should work. c) CSCeh13489 implemented the maxas command to mark it as invalid and not send. There does appear to be an issue when you cross the 255 boundary and the next hop router sends a notification back. I've got it recreated in the lab and we are working to clearly understand why that is. I'll post an update once we have more. The way to prevent it is the upstream device that crosses the 255 boundary on sending needs to use the maxas limit command to keep it less than 255. It doesn't work on the device that receives the update with the AS path larger than 255. Rodney On Tue, Feb 17, 2009 at 08:58:48PM +0100, Ivan Pepelnjak wrote: We were dropping ALL prefixes and the eBGP session was still resetting. Upstream or downstream? 1) bgp maxas-limit 75 had no effect mitigating this problem on the IOS we were using. That is: it was previously verified to be working just fine to drop paths longer than 75, but once we started receiving paths 255 then BGP started resetting. I was able to receive BGP paths longer than 255 on IOS release 12.2SRC. The paths were generated by Quagga BGP daemon. 12.2SRC causes the downstream session to break when the installed AS-path length is close to 255 and you use downstream AS-path prepending. In your case, I'm assuming you were hit with an older bug (probably at the 128 AS-path length boundary). It would be very hard to generate just the right AS-path length to unintentionally break your upstream EBGP session (as I said before, it's a nice targeted attack if you know your downstream topology). If your IOS is vulnerable to the older bugs that break inbound processing of AS paths longer than 128, there's nothing you can do on your end. The internal BGP checks reject the inbound update before the inbound filters (or bgp maxas-limit) can touch it and reset the upstream BGP session. Hope this helps Ivan Disclaimer: as I don't have internal access to Cisco, all of the above is a result of lab tests. pgpKi14UX012c.pgp Description: PGP signature
Re: anyone else seeing very long AS paths?
If you want to take this offline send it unicast or we could move it to cisco-nsp. What scenarios are you seeing that appear broken other than when a notification is sent when a 255 hop update is received? That's the one I'm working on right now. Rodney On Tue, Feb 17, 2009 at 05:31:49PM -0500, German Martinez wrote: On Tue Feb 17, 2009, Rodney Dunn wrote: Hello Rodney, It will be great if you can share with us your findings. It seems like we are hitting different bugs in different platforms. Thanks German Ivan, It is confusing but from what I have tested you have it correct. The confusing part comes from multiple issues. a) The documentation about the default maxas limit being 75 appears to be incorrect. I'll get that fixed. b) Prior to CSCee30718 there was a hard limit of 255. After that fix AS sets of more than 255 should work. c) CSCeh13489 implemented the maxas command to mark it as invalid and not send. There does appear to be an issue when you cross the 255 boundary and the next hop router sends a notification back. I've got it recreated in the lab and we are working to clearly understand why that is. I'll post an update once we have more. The way to prevent it is the upstream device that crosses the 255 boundary on sending needs to use the maxas limit command to keep it less than 255. It doesn't work on the device that receives the update with the AS path larger than 255. Rodney On Tue, Feb 17, 2009 at 08:58:48PM +0100, Ivan Pepelnjak wrote: We were dropping ALL prefixes and the eBGP session was still resetting. Upstream or downstream? 1) bgp maxas-limit 75 had no effect mitigating this problem on the IOS we were using. That is: it was previously verified to be working just fine to drop paths longer than 75, but once we started receiving paths 255 then BGP started resetting. I was able to receive BGP paths longer than 255 on IOS release 12.2SRC. The paths were generated by Quagga BGP daemon. 12.2SRC causes the downstream session to break when the installed AS-path length is close to 255 and you use downstream AS-path prepending. In your case, I'm assuming you were hit with an older bug (probably at the 128 AS-path length boundary). It would be very hard to generate just the right AS-path length to unintentionally break your upstream EBGP session (as I said before, it's a nice targeted attack if you know your downstream topology). If your IOS is vulnerable to the older bugs that break inbound processing of AS paths longer than 128, there's nothing you can do on your end. The internal BGP checks reject the inbound update before the inbound filters (or bgp maxas-limit) can touch it and reset the upstream BGP session. Hope this helps Ivan Disclaimer: as I don't have internal access to Cisco, all of the above is a result of lab tests.
anyone else seeing very long AS paths?
I am seeing them from 39625 and 47868. -Matt
Re: anyone else seeing very long AS paths?
I'm seeing it too Feb 16 16:44:36.065 GMT: BGP: x.x.x.x Bad attributes Feb 16 16:45:43.389 GMT: %BGP-6-ASPATH: Long AS path 6461 1299 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 received from x.x.x.x: Has more than 255 AS Feb 16 16:45:43.389 GMT: %BGP-5-ADJCHANGE: neighbor x.x.x.x Down BGP Notification sent Feb 16 16:45:43.389 GMT: %BGP-3-NOTIFICATION: sent to neighbor x.x.x.x 3/11 (invalid or corrupt AS path) 516 bytes 50020200 02FF193D 051371B9 BAFCBAFC BA Feb 16 16:45:43.389 GMT: BGP: x.x.x.x Bad attributes On 16/02/2009 16:55, Matt Liotta mlio...@r337.com wrote: I am seeing them from 39625 and 47868. -Matt James Ponter UK Operations Manager NaviSite jpon...@navisite.com +44 (0) 207 510 2504 (Office) +44 (0) 798 938 1039 (Cell) Reduce the Cost of IT with Managed Hosting and Application Services from NaviSite. Visit www.NaviSite.com http://www.navisite.com/ today. This e-mail is the property of NaviSite, Inc. It is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential, or otherwise protected from disclosure. Distribution or copying of this e-mail, or the information contained herein, to anyone other than the intended recipient is prohibited.
RE: anyone else seeing very long AS paths?
I just see it from 47868 and I just filtered it so it would stop blowing up BGP sessions to customers. In our case we are only seeing the prefix from level3 which prompted me to create a route map to block it: ip as-path access-list 500 permit _47868_ route-map as3356-in deny 1 match as-path 500 John van Oppen Spectrum Networks LLC Direct: 206.973.8302 Main: 206.973.8300 Website: http://spectrumnetworks.us -Original Message- From: Matt Liotta [mailto:mlio...@r337.com] Sent: Monday, February 16, 2009 8:55 AM To: nanog@nanog.org Subject: anyone else seeing very long AS paths? I am seeing them from 39625 and 47868. -Matt
Re: anyone else seeing very long AS paths?
Hi, Yep, we see them too. Nasty because there are lots of networks flapping as the long as-paths are tickling old bug CSCdr54230, so even networks not affected by the bug will be getting lots of extra updates. Anyone with contacts at 47868 ? Any upstreams onlist that want to bin them ? Andy
Re: anyone else seeing very long AS paths?
Sprint has already expunged 47868 from their offerings but Paetec nee McLeod is still bouncing sessions to us. It is Monday, isn't it? On Mon, Feb 16, 2009 at 11:10 AM, Andy Davidson a...@nosignal.org wrote: Hi, Yep, we see them too. Nasty because there are lots of networks flapping as the long as-paths are tickling old bug CSCdr54230, so even networks not affected by the bug will be getting lots of extra updates. Anyone with contacts at 47868 ? Any upstreams onlist that want to bin them ? Andy -- mailto:n...@layer3arts.com // GoogleTalk: nrauhau...@gmail.com IM: nealrauhauser
RE: anyone else seeing very long AS paths?
Yep we saw the same, every customer with old IOS had their sessions die to us at the same time... That always makes for an interesting time when watching the NMS system... We are seeing a much more sane path now: agg2-sea-Ashow ip bgp 94.125.216.0/21 BGP routing table entry for 94.125.216.0/21, version 25944571 Paths: (1 available, best #1) Multipath: eBGP Advertised to update-groups: 2 3 5 6 7 8 9 3561 3356 29113 47868 208.76.153.96 (metric 5102) from 208.76.153.96 (208.76.153.96) Origin IGP, metric 0, localpref 50, valid, internal, best Community: 11404:1000 11404:1040 agg2-sea-A John van Oppen Spectrum Networks LLC Direct: 206.973.8302 Main: 206.973.8300 Website: http://spectrumnetworks.us -Original Message- From: Andy Davidson [mailto:a...@nosignal.org] Sent: Monday, February 16, 2009 9:10 AM To: John van Oppen Cc: Matt Liotta; nanog@nanog.org Subject: Re: anyone else seeing very long AS paths? Hi, Yep, we see them too. Nasty because there are lots of networks flapping as the long as-paths are tickling old bug CSCdr54230, so even networks not affected by the bug will be getting lots of extra updates. Anyone with contacts at 47868 ? Any upstreams onlist that want to bin them ? Andy
RE: anyone else seeing very long AS paths?
Looks good here too tel...@edge1.chi.ubiquityservers.com(config)#show ip bgp 94.125.216.0/21 Number of BGP Routes matching display condition : 2 Status codes: s suppressed, d damped, h history, * valid, best, i internal Origin codes: i - IGP, e - EGP, ? - incomplete NetworkNext HopMetric LocPrf Weight Path * 94.125.216.0/2172.37.148.177 3 1000 25973 29113 47868 i * 94.125.216.0/2165.47.180.113 3 1000 2828 3257 29113 47868 i Regards, Jake Mertel Nobis Technology Group, L.L.C. Web: http://www.nobistech.net/ Phone: (312) 281-5101 ext. 401 Fax: (808) 356-0417 Mail: 201 West Olive Street Second Floor, Suite 2B Bloomington, IL 61701 -Original Message- From: John van Oppen [mailto:j...@vanoppen.com] Sent: Monday, February 16, 2009 11:25 AM To: Andy Davidson Cc: nanog@nanog.org Subject: RE: anyone else seeing very long AS paths? Yep we saw the same, every customer with old IOS had their sessions die to us at the same time... That always makes for an interesting time when watching the NMS system... We are seeing a much more sane path now: agg2-sea-Ashow ip bgp 94.125.216.0/21 BGP routing table entry for 94.125.216.0/21, version 25944571 Paths: (1 available, best #1) Multipath: eBGP Advertised to update-groups: 2 3 5 6 7 8 9 3561 3356 29113 47868 208.76.153.96 (metric 5102) from 208.76.153.96 (208.76.153.96) Origin IGP, metric 0, localpref 50, valid, internal, best Community: 11404:1000 11404:1040 agg2-sea-A John van Oppen Spectrum Networks LLC Direct: 206.973.8302 Main: 206.973.8300 Website: http://spectrumnetworks.us -Original Message- From: Andy Davidson [mailto:a...@nosignal.org] Sent: Monday, February 16, 2009 9:10 AM To: John van Oppen Cc: Matt Liotta; nanog@nanog.org Subject: Re: anyone else seeing very long AS paths? Hi, Yep, we see them too. Nasty because there are lots of networks flapping as the long as-paths are tickling old bug CSCdr54230, so even networks not affected by the bug will be getting lots of extra updates. Anyone with contacts at 47868 ? Any upstreams onlist that want to bin them ? Andy
RE: anyone else seeing very long AS paths?
On Mon, 16 Feb 2009, John van Oppen wrote: Yep we saw the same, every customer with old IOS had their sessions die to us at the same time... That always makes for an interesting time when watching the NMS system... Is there a reason you don't use something like bgp maxas-limit NN on your transit sessions? We saw this too, but it stopped at our transit routers. There was actually another a few days ago. Feb 13 18:46:07: %BGP-6-ASPATH: Long AS path 4323 1299 12887 12741 39412 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625... Feb 16 11:24:53: %BGP-6-ASPATH: Long AS path 4323 3257 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868... -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
RE: anyone else seeing very long AS paths?
bgp maxas-limit has a default value of 75 if you don't include it explicitly in the config so in this case it wouldn't have made much of a difference. L. On Mon, 16 Feb 2009, Jon Lewis wrote: On Mon, 16 Feb 2009, John van Oppen wrote: Yep we saw the same, every customer with old IOS had their sessions die to us at the same time... That always makes for an interesting time when watching the NMS system... Is there a reason you don't use something like bgp maxas-limit NN on your transit sessions? We saw this too, but it stopped at our transit routers. There was actually another a few days ago. Feb 13 18:46:07: %BGP-6-ASPATH: Long AS path 4323 1299 12887 12741 39412 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625... Feb 16 11:24:53: %BGP-6-ASPATH: Long AS path 4323 3257 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868... -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
RE: anyone else seeing very long AS paths?
Why do you say that? I counted 78 instances of 47868 in this long as-path, and our maxas-limit settings did trigger and reject these. On Mon, 16 Feb 2009, Leland E. Vandervort wrote: bgp maxas-limit has a default value of 75 if you don't include it explicitly in the config so in this case it wouldn't have made much of a difference. L. On Mon, 16 Feb 2009, Jon Lewis wrote: On Mon, 16 Feb 2009, John van Oppen wrote: Yep we saw the same, every customer with old IOS had their sessions die to us at the same time... That always makes for an interesting time when watching the NMS system... Is there a reason you don't use something like bgp maxas-limit NN on your transit sessions? We saw this too, but it stopped at our transit routers. There was actually another a few days ago. Feb 13 18:46:07: %BGP-6-ASPATH: Long AS path 4323 1299 12887 12741 39412 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625... Feb 16 11:24:53: %BGP-6-ASPATH: Long AS path 4323 3257 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868... -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_ -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
RE: anyone else seeing very long AS paths?
On Mon, 16 Feb 2009, John van Oppen wrote: I am also a bit leery of setting it much lower than the defaults due to the possibility of filtering something my customers will care about... I am not sure what the best strategy is but what really bit a couple of our customers was their old IOSes that tore the sessions down. I note that most of our customers speaking BGP had no issue just three out of about 25. What do people think is a reasonable maximum as-path length to enforce at ones edge? I've been using 75 for years and have never seen it trigger on 'legitimate' routes. -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: anyone else seeing very long AS paths?
On Feb 16, 2009, at 12:57 PM, John van Oppen wrote: I am also a bit leery of setting it much lower than the defaults due to the possibility of filtering something my customers will care about... I am not sure what the best strategy is but what really bit a couple of our customers was their old IOSes that tore the sessions down. I note that most of our customers speaking BGP had no issue just three out of about 25. What do people think is a reasonable maximum as-path length to enforce at ones edge? Would you want your upstream to set an arbitrary limit on these announcements for you, or should the few wayward souls finally upgrade their code? If your upstream were to set a limit (64, 96, 128, 192, 255) what would you expect that to be and how should it be disclosed? my opinion is that if you're going to operate in an active environment (eg: bgp) where messages are constantly being sent, you need to be an active participant in managing your risk. If you're not, perhaps you don't really need BGP since you can't afford the 'operational costs' of managing that asset. - Jared
RE: anyone else seeing very long AS paths?
http://www.cisco.com/en/US/docs/ios/12_2/iproute/command/reference/1rfbgp1.html#wp1254976 Defaults The default value in Cisco IOS software for the number argument is 75. On Mon, 16 Feb 2009, Jon Lewis wrote: Why do you say that? I counted 78 instances of 47868 in this long as-path, and our maxas-limit settings did trigger and reject these. On Mon, 16 Feb 2009, Leland E. Vandervort wrote: bgp maxas-limit has a default value of 75 if you don't include it explicitly in the config so in this case it wouldn't have made much of a difference. L. On Mon, 16 Feb 2009, Jon Lewis wrote: On Mon, 16 Feb 2009, John van Oppen wrote: Yep we saw the same, every customer with old IOS had their sessions die to us at the same time... That always makes for an interesting time when watching the NMS system... Is there a reason you don't use something like bgp maxas-limit NN on your transit sessions? We saw this too, but it stopped at our transit routers. There was actually another a few days ago. Feb 13 18:46:07: %BGP-6-ASPATH: Long AS path 4323 1299 12887 12741 39412 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625... Feb 16 11:24:53: %BGP-6-ASPATH: Long AS path 4323 3257 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868... -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_ -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: anyone else seeing very long AS paths?
Hi, Anyone onlist knows the similar command (bgp maxas-limit NN) on Foundry XMR platform ? regards, --- Nuno Vieira nfsi telecom, lda. nuno.vie...@nfsi.pt Tel. (+351) 21 949 2300 - Fax (+351) 21 949 2301 http://www.nfsi.pt/ - Jon Lewis jle...@lewis.org wrote: On Mon, 16 Feb 2009, John van Oppen wrote: Yep we saw the same, every customer with old IOS had their sessions die to us at the same time... That always makes for an interesting time when watching the NMS system... Is there a reason you don't use something like bgp maxas-limit NN on your transit sessions? We saw this too, but it stopped at our transit routers. There was actually another a few days ago. Feb 13 18:46:07: %BGP-6-ASPATH: Long AS path 4323 1299 12887 12741 39412 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625... Feb 16 11:24:53: %BGP-6-ASPATH: Long AS path 4323 3257 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868... -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: anyone else seeing very long AS paths?
Anyone have an estimate as to when these long announcements began? Seems like the first reports appeared just before noon, UTC-05. We noticed a significant dip in Internet traffic to AS11579 for a few minutes last night (19:00 UTC-05) which we've been trying to hunt down the cause of. At first glance, the two events seem unrelated. Anyone else see anything similar? - Jon Lewis jle...@lewis.org wrote: On Mon, 16 Feb 2009, John van Oppen wrote: Yep we saw the same, every customer with old IOS had their sessions die to us at the same time... That always makes for an interesting time when watching the NMS system.. Is there a reason you don't use something like bgp maxas-limit NN on your transit sessions? We saw this too, but it stopped at our transit routers. There was actually another a few days ago. Feb 13 18:46:07: %BGP-6-ASPATH: Long AS path 4323 1299 12887 12741 39412 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625 39625... Feb 16 11:24:53: %BGP-6-ASPATH: Long AS path 4323 3257 29113 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868 47868... -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: anyone else seeing very long AS paths?
It hit my routers at 11:26:40, EST. Michael On Monday 16 February 2009 07:26:23 pm Adam Greene wrote: Anyone have an estimate as to when these long announcements began? Seems like the first reports appeared just before noon, UTC-05. We noticed a significant dip in Internet traffic to AS11579 for a few minutes last night (19:00 UTC-05) which we've been trying to hunt down the cause of. At first glance, the two events seem unrelated. Anyone else see anything similar?
Re: anyone else seeing very long AS paths?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, Feb 16, 2009 at 8:51 PM, Michael Ulitskiy mulits...@acedsl.com wrote: It hit my routers at 11:26:40, EST. Michael On Monday 16 February 2009 07:26:23 pm Adam Greene wrote: Anyone have an estimate as to when these long announcements began? Seems like the first reports appeared just before noon, UTC-05. We noticed a significant dip in Internet traffic to AS11579 for a few minutes last night (19:00 UTC-05) which we've been trying to hunt down the cause of. At first glance, the two events seem unrelated. Anyone else see anything similar? Just as a follow-up -- and in case anyone hasn't read these yet: http://www.renesys.com/blog/2009/02/the-flap-heard-around-the-worl.shtml http://asert.arbornetworks.com/2009/02/ahh-the-ease-of-introducing-global-r outing-instability/ - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFJmkSgq1pz9mNUZTMRAvSrAKDx5/Z5vECDGbxul3rU+vJfr4kiYQCfSNbR um+X7vSLClA3fTkBqszSkWM= =Gm4l -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/
RE: anyone else seeing very long AS paths?
I encountered it yesterday from AS47868. -Original Message- From: Paul Ferguson [mailto:fergdawgs...@gmail.com] Sent: Tuesday, February 17, 2009 01:02 PM To: nanog@nanog.org Subject: Re: anyone else seeing very long AS paths? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, Feb 16, 2009 at 8:51 PM, Michael Ulitskiy mulits...@acedsl.com wrote: It hit my routers at 11:26:40, EST. Michael On Monday 16 February 2009 07:26:23 pm Adam Greene wrote: Anyone have an estimate as to when these long announcements began? Seems like the first reports appeared just before noon, UTC-05. We noticed a significant dip in Internet traffic to AS11579 for a few minutes last night (19:00 UTC-05) which we've been trying to hunt down the cause of. At first glance, the two events seem unrelated. Anyone else see anything similar? Just as a follow-up -- and in case anyone hasn't read these yet: http://www.renesys.com/blog/2009/02/the-flap-heard-around-the-worl.shtml http://asert.arbornetworks.com/2009/02/ahh-the-ease-of-introducing-globa l-r outing-instability/ - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFJmkSgq1pz9mNUZTMRAvSrAKDx5/Z5vECDGbxul3rU+vJfr4kiYQCfSNbR um+X7vSLClA3fTkBqszSkWM= =Gm4l -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/
RE: anyone else seeing very long AS paths?
On Mon, 16 Feb 2009, Jon Lewis wrote: Is there a reason you don't use something like bgp maxas-limit NN on your transit sessions? I've been using maxas=50 for years now (see the 2005 thread: http://www.atm.tut.fi/list-archive/nanog-2005/msg04753.html) I also knew this would happen one day and have tried to get various lists engaged but the consensus was live with it. Here is just my 3 month logs: Nov 30 18:51:32: %BGP-6-ASPATH: Long AS path 8551 3491 7473 45147 18351 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 24532 2 Dec 17 20:47:33: %BGP-6-ASPATH: Long AS path 8551 174 5391 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 43179 received from 128.139.247.2: More than configured MAXAS-LIMIT Dec 24 01:52:20: %BGP-6-ASPATH: Long AS path 8551 174 34224 42555 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 received from 128.139.247.2: More than configured MAXAS-LIMIT Dec 24 02:02:27: %BGP-6-ASPATH: Long AS path 8551 174 34224 42555 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 received from 128.139.247.2: More than configured MAXAS-LIMIT Dec 27 12:31:57: %BGP-6-ASPATH: Long AS path 8551 174 34224 42555 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 received from 128.139.247.2: More than configured MAXAS-LIMIT Dec 27 12:33:16: %BGP-6-ASPATH: Long AS path 8551 3491 174 34224 42555 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 received from 128.139.247.2: More than configured MAXAS-LIMIT Dec 27 12:46:49: %BGP-6-ASPATH: Long AS path 8551 174 34224 42555 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 48262 received from 128.139.247.2: More than configured MAXAS-LIMIT Jan 7 19:26:45: %BGP-6-ASPATH: Long AS path 8551 12989 19181 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 received from 128.139.247.2: More than configured MAXAS-LIMIT Jan 7 19:30:27: %BGP-6-ASPATH: Long AS path 8551 12989 19181 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 6488 received from 128.139.247.2: More than configured MAXAS-LIMIT Jan 7 19:38:06: %BGP-6-ASPATH: Long AS
