Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Rob, Eloy Paris from the Cisco PSIRT here. Please see below (inline) for some comments regarding the issue you brought up in your email to the cisco-nsp and nanog mailing lists this past Jan. 16th: On Fri Jan 16 07:57:52 2009, Rob Shakir wrote: Strict RFC 4893 (4-byte ASN support) BGP4 implementations are vulnerable to a session reset by distant (not directly connected) ASes. This vulnerability is a feature of the standard, and unless immediate action is taken an increasingly significant number of networks will be open to attack. Accidental triggering of this vulnerability has already been seen in the wild, although the limited number of RFC 4893 deployments has limited its effect. Summary: It is possible to cause BGP sessions to remotely reset by injecting invalid data into the AS4_PATH attribute provided to store 4-byte ASN paths. Since AS4_PATH is an optional transitive attribute, the invalid data will be transited through many intermediate ASes which will not examine the content. To be vulnerable, an operator does not have to be actively using 4-byte AS support. This problem was first reported by Andy Davidson on NANOG in December 2008 [0], furthermore we have been able to demonstrate that a device running Cisco IOS release 12.0(32)S12 behaves as per this description. Details: [...] Cisco Bug CSCsx10140 was filed for Cisco IOS. Cisco IOS behaves exactly as you described - upon receipt of AS_CONFED_SEQUENCE data in the AS4_PATH attribute IOS will send a NOTIFICATION message to the peer, which causes a termination of the BGP session. After the fix for this bug IOS will ignore AS_CONFED_SEQUENCE data in the AS4_PATH attribute of received BGP UPDATE messages and continue to process the UPDATE. This is the new behavior that the revised RFC 4893 will require. CSCsx18598 was filed for Cisco IOS XR. Cisco IOS XR doesn't reset the session but accepts and forwards the invalid AS4_PATH data, so this bug was filed to change this behavior. CSCsx23179 was filed for Cisco NX-OS (for the Nexus switches.) Cisco NX-OS behaves like IOS (it will reset the BGP session when it sees AS_CONFED_SEQUENCE data in the AS4_PATH attribute), and this bug was filed to change this and have the BGP implementation in Cisco NX-OS follow the revised RFC 4893. The Release Notes for each bug may have some additional information. These are available via the Bug Toolkit on cisco.com (http://tools.cisco.com/Support/BugToolKit) To date, the only version of Cisco IOS that supports 4-byte AS numbers is 12.0(32)S12, released in late December. A fix to the 12.0(32)Sxx branch has been committed so the next 12.0(32)S-based release will have the fix. 12.0(32)SY8 is coming out soon, and it will also have support for 4-byte AS numbers, as well as the fix for the problem. Thanks for bringing attention to this issue and for working with us, specifically with the Cisco TAC, to get to the bottom of it and test the proposed fix. Cheers, - -- Eloy Paris Cisco PSIRT -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkmR9OoACgkQagjTfAtNY9jv5ACgg3fKuuWKv38h8F8d8QHBML5J CTsAnAnGMB/fBIQhk5z4E922JlhHVU5A =FSOP -END PGP SIGNATURE-
Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
Hi, Further to the initial research sent to NANOG, after discussions with a number of operators, we have compiled some recommendations on the handling of invalid AS4_PATH attributes. Any feedback on these recommendations is appreciated: As discussed on the IETF IDR list last month, there are concerns relating to the treatment of AS_CONFED_SET/SEQUENCE in AS4_PATH as described in RFC4893 [0]. Since the last post to that thread the situation has been made more urgent with the release of Cisco IOS 12.0(32)S12, which responds to malformed AS4_PATH attributes by sending a NOTIFICATION to the neighbour, and tearing down the BGP adjacency. This behaviour seems to be required by RFC4721 section 6.3, as there is no alternative error handling defined in RFC4893. As posted last Friday [1], and discussed on the IDR list, this strict implementation introduces a new attack vector by which a BGP session can be torn down due to a an attribute populated by a distant BGP neighbour. These malformed attributes have already been seen in the wild as a result of a error in Juniper's implementation of RFC4893. Following discussions with a number of operators, we have attempted to generate some recommendations relating to the behaviour that would be operationally most useful when treating the invalid data in the AS4_PATH optional transitive attribute. There are two cases to consider when an invalid AS4_PATH is received: (1) A path to the prefix is not already known from that neighbour. (2) A path to the prefix has already been learnt from that neighbour; In case (1) we recommend that the BGP speaker should discard the UPDATE and log the fact. The log entry should include the received AS_PATH and AS4_PATH to aid in debugging. In case (2) we recommend that the BGP speaker should treat the UPDATE as a withdrawal of existing path to the prefix. As per case (1) a log entry should be raised to indicate that this has occurred. It is quite possible that in both cases this behaviour may result in the BGP speaker no longer having a valid path to the destination. We foresee that this lack of a prefix in a BGP speaker's routing table may cause some operational load initially, however, we feel that this is acceptable, considering the alternate behaviours. Should a prefix be injected into the global table with an invalid AS4_PATH, and should the newly advertised (invalid) path be selected by all upstreams available to a given ASN then this ASN will lose reachability to the prefix. Whilst this can be abused we do not see this as more serious than the existing possibility of malicious injection and blackholing of a prefix by a 3rd party. As long as the rejection of paths due to invalid AS4_PATHs is clearly reported to the administrator the source of the problem can be clearly identified. We consider that attempting to extract a valid AS4 or AS_PATH from the invalid UPDATE is a mistake since this allows the propagation of invalid BGP data. In addition, incorrect implementation of this comparatively complex mechanism by a vendor may result in loops. By explicitly not installing prefixes with invalid AS_PATH or AS4_PATH into the routing table, the possibility of loops caused by these invalid paths is avoided. The defined behaviour in RFC4893 and RFC4271 has significantly harmful effects and it seems only by virtue of the fact that the implementations of many vendors do not strictly comply with the RFCs that this problem has not had the same impact for every vendor. At the current time, however, one cannot deploy a 4-byte capable Cisco IOS device, or an OpenBGP (current stable release) router into the global table, without risking teardown of a every session via which a global table is learnt. Further discussion of this issue would be much appreciated, as a common and consistent approach to rectifying the problem will benefit network operators far more than individual vendor implementing their own solution. Should a consensus be reached an update to the RFC is required in order to ensure that future implementations do not exhibit this harmful behaviour. Kind regards, Andy Davidson (NetSumo), andy.david...@netsumo.com Jonathan Oddy (HostWay), jonathan.o...@hostway.co.uk Rob Shakir (GX Networks), r...@eng.gxn.net [0]: http://www.ietf.org/mail-archive/web/idr/current/msg03368.html [1]: http://www.merit.edu/mail.archives/nanog/msg14345.html Many thanks to David Freedman (Claranet) for assistance in developing the recommendations in this document.
Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
On Mon, Jan 19, 2009 at 03:58:17PM +, Jonathan Oddy wrote: As mentioned in both [1] and [2] this is especially critical as at present Cisco IOS will tear down sessions when receiving an AS4_PATH containing an AS_CONFED_SET/SEQUENCE. Hi, Whilst this is behaviour is RFC compliant, as previously described, it is sub-optimal operationally. I have raised this issue with Cisco TAC, and CSCsx10140 has been opened to track this problem. I would encourage those network operators who may be planning to deploy AS4-support and use Cisco equipment to open a SR with Cisco, tracking this bug, to try to ensure that both the IOS behaviour, and RFC are changed. Many thanks, Rob -- Rob Shakir r...@eng.gxn.net Network Development EngineerGX Networks/Vialtus Solutions ddi: +44208 587 6077mob: +44797 155 4098 pgp: 0xc07e6deb nic-hdl: RJS-RIPE This email is subject to: http//www.vialtus.com/disclaimer.html
Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
have been able to demonstrate that a device running Cisco IOS release 12.0(32)S12 behaves as per this description. Has anyone looked into IOS XR behaviour, if it's the same as 12.0(32)S12? -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
On Tue, Jan 20, 2009 at 01:01:03PM +0100, Mikael Abrahamsson wrote: have been able to demonstrate that a device running Cisco IOS release 12.0(32)S12 behaves as per this description. Has anyone looked into IOS XR behaviour, if it's the same as 12.0(32)S12? Mikael, Pierfrancesco Caci was kind enough to provide me with some output from an XR box. It appears that IOS XR behaves in the same manner as Force10, and JunOS, whereby the session is not torn down, and the path is installed, albeit with a munged AS_PATH. The output below is for the prefix from 196629 which we originally analysed: Path #1: Received by speaker 0 3356 35320 3.21 23456 Given that XR box is an AS4-speaker, one would not expect to see 23456 in the AS_PATH, the prescence of this AS seems to be a symptom of the bug (and again occurs on Juniper/Force10). Kind regards, Rob -- Rob Shakir r...@eng.gxn.net Network Development EngineerGX Networks/Vialtus Solutions ddi: +44208 587 6077mob: +44797 155 4098 pgp: 0xc07e6deb nic-hdl: RJS-RIPE This email is subject to: http//www.vialtus.com/disclaimer.html
Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I was indeed aware of the OpenBGPD discussion and patch, and I'm glad it has been worked around in what I believe to be a sensible way, however I disagree with the comment in the code that states that the standard does not specify how to handle this situation. I believe that RFC 4271* and 4893** currently require a teardown of the session in this case and indeed the person who committed the fix to OpenBGPD seems to agree in their commit message (although still kept the code comment.) This really needs to lead to more debate on the correct way to handle this situation and an updated standard, before implementers decide to fix this in their own different ways. The discussion on the IETF IDR mailing list[1] was promising, but looks to have died off before reaching a conclusion. There was mention of stripping the AS_CONFED_SET/SEQUENCE from the AS4_PATH, however several people pointed out this approach is not without issues. Dropping the UPDATE entirely is also discussed, but can lead to loops. Personally I favour treating receipt of an UPDATE with a malformed attribute as a withdrawal, although this was only briefly mentioned and its implications were never discussed in any detail... The reason for us publishing this report was to alert people to the fact that this problem is definitely in the wild, there are broken AS4_PATHs being announced, and, critically, Cisco's IOS releases to support RFC4893 are vulnerable to having their sessions reset as a result of their standards compliant implementation. At present our advice has to be that upgrading to an IOS version with RFC4893 support is extremely dangerous, and should be avoided at all costs (where this leaves Cisco shops who have been given 32 bit AS numbers by their RIR is somewhat unpleasant to consider.) It must be emphasized that this is due to no fault on Cisco's part, but rather a feature of the standard that must be corrected as soon as possible. [1] http://www.ietf.org/mail-archive/web/idr/current/msg03368.html * From RFC4271: Section 6: ~ When any of the conditions described here are detected, a ~ NOTIFICATION message, with the indicated Error Code, Error Subcode, ~ and Data fields, is sent, and the BGP connection is closed (unless it ~ is explicitly stated that no NOTIFICATION message is to be sent and ~ the BGP connection is not to be closed). If no Error Subcode is ~ specified, then a zero MUST be used. Section 6.3: ~ If an optional attribute is recognized, then the value of this ~ attribute MUST be checked. If an error is detected, the attribute ~ MUST be discarded, and the Error Subcode MUST be set to Optional ~ Attribute Error. The Data field MUST contain the attribute (type, ~ length, and value). ** From RFC4893: Section 3: ~ To prevent the possible propagation of confederation path segments ~ outside of a confederation, the path segment types AS_CONFED_SEQUENCE ~ and AS_CONFED_SET [RFC3065] are declared invalid for the AS4_PATH ~ attribute. - -- Jonathan Oddy Hostway UK -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJdFjqWGqmTqbbikoRAmNuAJoCPqNUTYOW9lFUQXFfLAFgA/bIcQCeODVz Wo1MjYgtdDw1SmWhmHdzcWM= =AGvq -END PGP SIGNATURE-
Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
Jonathan Oddy wrote: dangerous, and should be avoided at all costs (where this leaves Cisco shops who have been given 32 bit AS numbers by their RIR is somewhat unpleasant to consider.) It must be emphasized that this is due to no Suddenly makes one wonder if it would have been easier to take back any ASN's which weren't justified versus butchering the protocol. Jack
Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
After some lab work we have established that the source of the invalid
AS4_PATHs discussed in [1] is likely a non compliant implementation of
RFC4893 (AS4) in some versions of Juniper JunOS.
We have observed the following behaviour with both JunOS 9.3R1.7 and
9.1R2.10, and suspect it may be present in all other JunOS versions
since they introduced AS4 support in 9.1R1. Unfortunately we have
limited resources so have not been able to test with other versions.
When a mix of pre and post 9.1R1 JunOS devices are deployed within a
network utilising confederations the AS4_PATH (if present) is used by
the AS4 supporting devices to hold an AS_CONFED_SET/SEQUENCE. This
behaviour is explicitly forbidden by RFC4893 [3]. If the egress router
from the AS utilising confederations is not AS4-aware the confederation
information is never removed from the AS4_PATH, and is passed onto the
neighbouring networks with the repercussions discussed in [1].
As mentioned in both [1] and [2] this is especially critical as at
present Cisco IOS will tear down sessions when receiving an AS4_PATH
containing an AS_CONFED_SET/SEQUENCE.
Lab setup:
AS1.0 - obgp1 (OpenBGPD)
AS64512 {
~AS65001 - juniper1 (JunOS 9.1 or 9.3) (32 bit ASN support)
~AS65002 - juniper2 (JunOS 8.4) (no 32 bit ASN support)
}
AS64513 - obgp2 (OpenBGPD)
Where AS1.0 is an AS with a 32bit AS number, AS64512 is a Juniper
network using confederations and with mixed AS4 support, and AS64513 is
another network (doesn't matter what it supports.)
On announcing a prefix from obgp1 we observe the following in the UPDATE
from juniper1 to juniper2:
AS_PATH: (65001) 23456
AS4_PATH: (65001) 65536
And at obgp2:
AS_PATH: 64512 23456
AS4_PATH: (65001) 65536
This shows juniper1, which is AS4-aware, adding an AS_CONFED_SET to both
the AS_PATH and AS4_PATH before announcing the prefix to juniper2. As
juniper2 is not AS4-aware it does not strip the AS_CONFED_SET from the
AS4_PATH before announcing it to obgp2, resulting in an invalid AS4_PATH
attribute in the UPDATE to obgp2.
Conclusions:
~ * If you use JunOS and make use of confederations you should ensure
that your entire network either supports AS4 (9.1R1 or later) or doesn't
(pre 9.1.)
~ * While the Juniper implementation is clearly non-compliant with the
standard, and should be corrected, the number of versions in which this
bug is probably present means that these versions will never be
completely eliminated from use.
~ * The flaw in the standard can still be misused maliciously.
We do not see that going forward it will be possible to completely
eliminate the possibility of an AS_CONFED_SET appearing in an AS4_PATH.
We believe that this problem requires a consistent response from the
vendors, and that to facilitate such a response the standard must be
revised. Even if vendors do implement their own workarounds the standard
needs to be revised to ensure that future implementers don't fall into
this trap.
Regards,
~Andy Davidson, NetSumo (andy.david...@netsumo.com),
~Jonathan Oddy, Hostway UK (jonathan.o...@hostway.co.uk),
~Rob Shakir, GX Networks (r...@eng.gxn.net)
[1] http://www.merit.edu/mail.archives/nanog/msg14345.html
[2] http://www.merit.edu/mail.archives/nanog/msg14388.html
[3] From RFC4893 section 3:
~ To prevent the possible propagation of confederation path segments
~ outside of a confederation, the path segment types AS_CONFED_SEQUENCE
~ and AS_CONFED_SET [RFC3065] are declared invalid for the AS4_PATH
~ attribute.
Thanks to Dan Goscomb (Goscomb Tech) for loan of a J2320 for the lab.
Thanks to Will Hargrave (LONAP) for assistance with this document.
- --
Jonathan Oddy
Hostway UK
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFJdKMZWGqmTqbbikoRAuDFAJ9WTlvAE/5KogtgShiBmXJo238kHQCfdSjG
s3p8pIfX7JmPKC84/yxE67w=
=53KL
-END PGP SIGNATURE-
