Re: BGP Route Reflector - Route Server, Router, etc
On Thu 2017-Jan-12 22:59:21 +, James Bensley wrote: On 12 January 2017 at 20:32, Justin Krejci wrote: . I have not found many resources discussing using a non-router box as a route reflector (ie a device not necessarily in the forwarding path of the through traffic). I am thinking things like OpenBGPd and BIRD could make a good route reflector though they are most often discussed in the context of IXPs (ie eBGP sessions). The CSR1000v (IOS-XE),IOS-XRv and vMX are production ready. People are deploying these in production and its increasing in popularity. Any thoughts on vRR vs. vMX for this use case? I see Mark called out vRR as having morphed into vMX, but AFAIK vRR is just vMX minus the forwarding plane, is targeted as an out-of-path reflector, and coexists with vMX as a different deployment option rather than having been replaced by it. I would assume that vRR should come in a few bucks lower than the vMX as a result, but I've only previously gotten quotes on vRR not vMX. Mark Tinka gave a good preso at a recent Nanog: https://www.nanog.org/sites/default/files/2_Tinka_21st_Century_iBGP_Route_Reflection.pdf https://www.youtube.com/watch?v=wLEjOj2fyp8&list=PLO8DR5ZGla8hcpeEDSBNPE5OrZf70iXZg&index=21 Cheers, James. -- Hugo Slabbert | email, xmpp/jabber: h...@slabnet.com pgp key: B178313E | also on Signal signature.asc Description: Digital signature
Re: BGP Route Reflector - Route Server, Router, etc
Dear Justin, You could take a look at this presentation from Mark Tinka during last NANOG : https://m.youtube.com/watch?v=wLEjOj2fyp8 HTH. Y. > Le 12 janv. 2017 à 23:41, Łukasz Bromirski a écrit : > > >> On 12 Jan 2017, at 21:32, Justin Krejci wrote: >> >> Nanog, >> […] > > You did some homework. In essence, there’s no immediate problem with running > Quagga or OpenBGPd as > RR apart from lack of different knobs and not-so-stellar > performance/scalability. BIRD is grounds up built > to act as high-performance BGP daemon, and it’s actually used as RR in live > deployments, not only at IXes. > >> I am wondering if people can point me in the direction to some good resource >> material on how to select a good BGP route reflector design. Should I just >> dust off some 7206VXR routers to act as route reflectors? Use a few existing >> live routers and just add the responsibility of being route reflectors, is >> there a performance hit? Install and run BIRD on new server hardware? Buy >> some newer purpose built routers (Cisco, Juniper, Brocade, etc) to act as >> route reflectors and add them to the iBGP topology? GNS3 running IOS on >> server hardware? Something else? How many reflectors should be implemented? >> Two? Four? > > Disclaimer: I work at Cisco. > > If You have some 7200VXRs that have 1 or 2GBs of RAM, that may be the best > option (IF you have them). > Loaded with 12.2S/15S software they may actually be the most cost-effective > solution and at the same > time support things like AddPath, BGP error handling, etc - when time comes > to use such features. > If that’s a NPE400 based chassis or something even older - leave it for > lab/etc as You need rather > performant CPU. > > So, if that’s not the option, try to work with the BIRD, CSR 1000v (IOS-XE on > VM) or ASR 1001X/HX > (currently, the most scaleable and fastest BGP route reflector out there, but > one that will cost $$$). > > Two RRs provide ample redundancy to run even very large deployments (1000+ > clients), so unless you’re > trying to hit higher numbers or plan to play fancy games with one pair of RRs > for IPv4/IPv6 unicast > and other pair for different AFs, four may be an overkill to maintain, > synchronize and monitor. > > Don’t go with GNS3, running compiled at runtime emulation is wrong idea for > any production deployment, > not to mention rights/licenses to do it. > > — > Łukasz Bromirski
RE: BGP Route Reflector - Route Server, Router, etc
Justin, Fusion has run Route Reflection for some time as we didn't want to play full mesh. Our current design is that we have 2 routers that are designated as the route reflectors, and every other router maintains RR sessions with those. There is no real additional overhead as the BGP transactions and messaging are handled at the management plane of most routers, not the routing plane. Ours are directly on our Brocade MLXe. As we are scaling past the initial build, we are running into certain minor routing hiccups dealing with remote routers sometime preferring routes from the reflectors vs their closer routes. We found this to be a function of ingesting transit and default routes directly into route reflector routers and that those routes tended to get preferred in the table than routes from other routers in the network. Our answer to this and what we are deploying this year is that we are picking a site per time zone to be a Route Reflector, which will give us 4 RRs in the states, but we will not use sites that are Transit ingest sites. This way we are more balancing the BGP table across the entire network. Also, I believe we will move to default-free status this year with this same move. Happy to discuss more indepth offlist if you'd like. -- James Breeden The Fusion Network j...@gotfusion.net fusion 844.548.1421 direct 512.360. cell 512.304.0745 www.gotfusion.net facebook.com/gotfusion -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Justin Krejci Sent: Thursday, January 12, 2017 2:33 PM To: NANOG [nanog@nanog.org] Subject: BGP Route Reflector - Route Server, Router, etc Nanog, I am working on some network designs and am adding some additional routers to a BGP network. I'd like to build a plan of changing all of the existing routers over from full iBGP mesh to something more scalable (ie route reflection). Fortunately, I am also going to be able to decommission some older routers from the network and so shrinking the existing iBGP full mesh is something I am all too happy to spend time and energy on. For the purpose of this thread though, I am not really interested in the route reflector vs confederation discussion. In doing some research[1][2][3][4][5] I see a lot of discussions, config examples, etc on using route reflectors but most suggest picking a router, or more appropriately a set of routers, to become route reflectors within an ASN. I have not found many resources discussing using a non-router box as a route reflector (ie a device not necessarily in the forwarding path of the through traffic). I am thinking things like OpenBGPd and BIRD could make a good route reflector though they are most often discussed in the context of IXPs (ie eBGP sessions). I am wondering if people can point me in the direction to some good resource material on how to select a good BGP route reflector design. Should I just dust off some 7206VXR routers to act as route reflectors? Use a few existing live routers and just add the responsibility of being route reflectors, is there a performance hit? Install and run BIRD on new server hardware? Buy some newer purpose built routers (Cisco, Juniper, Brocade, etc) to act as route reflectors and add them to the iBGP topology? GNS3 running IOS on server hardware? Something else? How many reflectors should be implemented? Two? Four? What are the pros and cons of one design over another? On list or private off list replies would be great; I'd welcome real world experiences (especially any big gotchas or caveats people learned the hard way) as well as just links to previous discussions, PDFs, slideshows, etc. Heck even a good book suggestion that covers this topic would be appreciated. [1] - iBGP-to-RR migration slideshow: http://meetings.ripe.net/ripe-42/presentations/ripe42-eof-bgp/sld015.html [2] - General RR design issues: http://www.netcraftsmen.com/bgp-route-reflector-design-issues/ [3] - Video intro to RR from Cisco: http://www.cisco.com/c/dam/en_us/training-events/le31/le46/cln/qlm/CCIP/bgp/introducing-route-reflectors-2/player.html [4] - Quagga and BIRD as RR example: https://bsdrp.net/documentation/examples/bgp_route_reflector_and_confederation_using_quagga_and_bird [5] - Countless hours on youtube: https://www.youtube.com/results?search_query=bgp+route+reflector Lots more data is out there of course as that is part of my problem. Thanks! Justin
Apple Caching Server question
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
In message , Fernando Gont writes: > El 12/1/2017 16:32, "Saku Ytti" escribi=C3=B3: > > On 12 January 2017 at 17:02, Fernando Gont wrote: > > That's the point: If you don't allow fragments, but your peer honors > > ICMPv6 PTB<1280, then dropping fragments creates the attack vector. > > Thanks. I think I got it now. Best I can offer is that B could try to > verify the embedded original packet? Hopefully attacker won't have > access to that information. An if attacker has access to that > information, they may as well do TCP RST, right? > > Didn't we have same issues in IPv4 with ICMP unreachable and frag > neeeded, DF set? And vendors implemented more verification if the ICMP > message should be accepted. > > > Yes and no. > > 1) there was no way in v4 to trigger use of fragmentation for an arbitrary > flow. > > 2) in v4 you were guaranteed to get the IP+TCP header in the ICMP payload. > In ipv6, you aren't (think ipv6 EHs) So drop the packet if you don't get to the end of the extension headers in the ICMPv6 payload. Has anyone, except in testing, seen a extension header chain that was not fully containable in the ICMPv6 payload? Mark > Thanks, > Fernando -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
In message , Fernando Gont writes: > El 12/1/2017 16:28, "Mark Andrews" escribi=C3=B3: > > > In message <11ff128d-2fba-7c26-4a9c-5611433d8...@si6networks.com>, Fernando > > Gont writes: > > > Hi, Saku, > > > > > > On 01/12/2017 11:43 AM, Saku Ytti wrote: > > > > On 12 January 2017 at 13:19, Fernando Gont > > wrote: > > > > > > > > Hey, > > > > > > > >> I'm curious about whether folks are normally filtering ICMPv6 PTB<1280 > > > >> and/or IPv6 fragments targeted to BGP routers (off-list datapoints are > > > >> welcome). > > > > > > > > Generally may be understood differently by different people. If > > > > generally is defined as single most typical behaviour/configuration, > > > > then generally people don't protect their infrastructure in any way at > > > > all, but fully rely vendor doing something reasonable. > > > > > > > > I would argue BCP is to have 'strict' CoPP. Where you specifically > > > > allow what you must then have ultimate rule to deny everything. If you > > > > have such CoPP, then this attack won't work, as you clearly didn't > > > > allow any fragments at all (as you didn't expect to receive BGP > > > > fragments from your neighbours). > > > > > > That's the point: If you don't allow fragments, but your peer honors > > > ICMPv6 PTB<1280, then dropping fragments creates the attack vector. > > > > And fragments are a *normal* part of IP for both IPv4 and IPv6. > > This obsession with dropping all fragments (and yes it is a obsession) > > is breaking the internet. > > Vendors got the frag reassembly code wrong so many times , that I > understand the folk that decides to drop them if deemed unnecessary. Most of them literally decades ago. 20+ years ago while you waited for you vendor to fix the bug it made some sense as most of your boxes were vulnerable. It was a new threat back then. It doesn't make sense today. Packet bigger than 1500 are a part of todays internet. Have a look a the stats for dropped fragments. They aren't for the most part attack traffic. Its legitmate reply traffic that has been requested. > > Even if you don't want to allow all fragments through you can allow > > fragments between the two endpoints of a "active" connection. > > > > > At times folks want to get rid of fragments directed to them, rather than > > those going *through* them. > > > > > > You > > can apply port filters to the offset 0 fragments. If that fragment > > doesn't have enough headers to be able to filter then drop it. If > > your firewall is incapable of doing this then find a better firewall > > as the current one is a piece of garbage and should be in the recycle > > bin. > > > Which DoS is the bigger issue? Firewalls dropping fragments or > > reassembly buffers being exhausted? > > > > If there is no way for an attacker to trigger the use of fragmentation, and > > you don't need fragments (e.g. only tcp-based services), from a security > > pov you're certainly better off dropping frags that are thrown at you. Not > > that I like it, but > > Thanks, > Fernando > -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
RE: BGP Route Reflector - Route Server, Router, etc
> I am thinking things like OpenBGPd and BIRD could make a good route reflector > though they are most often discussed in the context of IXPs (ie eBGP > sessions). We use openbgpd - well, the native OpenBSD equivalent - for route-reflection in a couple of places, as well as a full bgp feed for at least one site, using (old) Poweredge 1950 Gen2's. They were on-hand, so the price was right. It's not caused us any grief to date. That said, neither have our 7204VXR's which do the same thing in some areas. Needless to say, we don't use the reflectors to actually move the bits, but have at least on one occasion measured ~88,000pp/s out of one of the 1950's that takes a full feed, before interrupts were starting to look worrisome on old non-smp safe code. But switches with bgp or ospf support are cheap provided you're not feeding them with a full table. Convergence times haven't been a problem for us, but we're only hovering around 1500 routes at the moment. Having something you can tcpdump on is nice for the few situations that call for it, pf is always extremely handy, re-distributing to/from ospfd is trivial (also in OpenBSD base). As long as you can find hardware with memory enough to scale to your number of routes, it's been a perfectly valid and sound option for us. My 5 cents. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Justin Krejci Sent: January-12-17 12:33 PM To: NANOG [nanog@nanog.org] Subject: BGP Route Reflector - Route Server, Router, etc Nanog, I am working on some network designs and am adding some additional routers to a BGP network. I'd like to build a plan of changing all of the existing routers over from full iBGP mesh to something more scalable (ie route reflection). Fortunately, I am also going to be able to decommission some older routers from the network and so shrinking the existing iBGP full mesh is something I am all too happy to spend time and energy on. For the purpose of this thread though, I am not really interested in the route reflector vs confederation discussion. In doing some research[1][2][3][4][5] I see a lot of discussions, config examples, etc on using route reflectors but most suggest picking a router, or more appropriately a set of routers, to become route reflectors within an ASN. I have not found many resources discussing using a non-router box as a route reflector (ie a device not necessarily in the forwarding path of the through traffic). I am thinking things like OpenBGPd and BIRD could make a good route reflector though they are most often discussed in the context of IXPs (ie eBGP sessions). I am wondering if people can point me in the direction to some good resource material on how to select a good BGP route reflector design. Should I just dust off some 7206VXR routers to act as route reflectors? Use a few existing live routers and just add the responsibility of being route reflectors, is there a performance hit? Install and run BIRD on new server hardware? Buy some newer purpose built routers (Cisco, Juniper, Brocade, etc) to act as route reflectors and add them to the iBGP topology? GNS3 running IOS on server hardware? Something else? How many reflectors should be implemented? Two? Four? What are the pros and cons of one design over another? On list or private off list replies would be great; I'd welcome real world experiences (especially any big gotchas or caveats people learned the hard way) as well as just links to previous discussions, PDFs, slideshows, etc. Heck even a good book suggestion that covers this topic would be appreciated. [1] - iBGP-to-RR migration slideshow: http://meetings.ripe.net/ripe-42/presentations/ripe42-eof-bgp/sld015.html [2] - General RR design issues: http://www.netcraftsmen.com/bgp-route-reflector-design-issues/ [3] - Video intro to RR from Cisco: http://www.cisco.com/c/dam/en_us/training-events/le31/le46/cln/qlm/CCIP/bgp/introducing-route-reflectors-2/player.html [4] - Quagga and BIRD as RR example: https://bsdrp.net/documentation/examples/bgp_route_reflector_and_confederation_using_quagga_and_bird [5] - Countless hours on youtube: https://www.youtube.com/results?search_query=bgp+route+reflector Lots more data is out there of course as that is part of my problem. Thanks! Justin
Re: BGP Route Reflector - Route Server, Router, etc
On 12 January 2017 at 20:32, Justin Krejci wrote: > . I have not found many resources discussing using a non-router box as a > route reflector (ie a device not necessarily in the forwarding path of the > through traffic). I am thinking things like OpenBGPd and BIRD could make a > good route reflector though they are most often discussed in the context of > IXPs (ie eBGP sessions). The CSR1000v (IOS-XE),IOS-XRv and vMX are production ready. People are deploying these in production and its increasing in popularity. Mark Tinka gave a good preso at a recent Nanog: https://www.nanog.org/sites/default/files/2_Tinka_21st_Century_iBGP_Route_Reflection.pdf https://www.youtube.com/watch?v=wLEjOj2fyp8&list=PLO8DR5ZGla8hcpeEDSBNPE5OrZf70iXZg&index=21 Cheers, James.
Re: BGP Route Reflector - Route Server, Router, etc
Your knowledge of OpenBGPd's scalability issues may be a bit dated. 1) I'm not sure many would have run into it anyway. 2) A patch was submitted and I believe is in a stable release now. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Łukasz Bromirski" To: "Justin Krejci" Cc: "NANOG [nanog@nanog.org]" Sent: Thursday, January 12, 2017 4:41:20 PM Subject: Re: BGP Route Reflector - Route Server, Router, etc > On 12 Jan 2017, at 21:32, Justin Krejci wrote: > > Nanog, > […] You did some homework. In essence, there’s no immediate problem with running Quagga or OpenBGPd as RR apart from lack of different knobs and not-so-stellar performance/scalability. BIRD is grounds up built to act as high-performance BGP daemon, and it’s actually used as RR in live deployments, not only at IXes. > I am wondering if people can point me in the direction to some good resource > material on how to select a good BGP route reflector design. Should I just > dust off some 7206VXR routers to act as route reflectors? Use a few existing > live routers and just add the responsibility of being route reflectors, is > there a performance hit? Install and run BIRD on new server hardware? Buy > some newer purpose built routers (Cisco, Juniper, Brocade, etc) to act as > route reflectors and add them to the iBGP topology? GNS3 running IOS on > server hardware? Something else? How many reflectors should be implemented? > Two? Four? Disclaimer: I work at Cisco. If You have some 7200VXRs that have 1 or 2GBs of RAM, that may be the best option (IF you have them). Loaded with 12.2S/15S software they may actually be the most cost-effective solution and at the same time support things like AddPath, BGP error handling, etc - when time comes to use such features. If that’s a NPE400 based chassis or something even older - leave it for lab/etc as You need rather performant CPU. So, if that’s not the option, try to work with the BIRD, CSR 1000v (IOS-XE on VM) or ASR 1001X/HX (currently, the most scaleable and fastest BGP route reflector out there, but one that will cost $$$). Two RRs provide ample redundancy to run even very large deployments (1000+ clients), so unless you’re trying to hit higher numbers or plan to play fancy games with one pair of RRs for IPv4/IPv6 unicast and other pair for different AFs, four may be an overkill to maintain, synchronize and monitor. Don’t go with GNS3, running compiled at runtime emulation is wrong idea for any production deployment, not to mention rights/licenses to do it. — Łukasz Bromirski
Re: BGP Route Reflector - Route Server, Router, etc
> On 12 Jan 2017, at 21:32, Justin Krejci wrote: > > Nanog, > […] You did some homework. In essence, there’s no immediate problem with running Quagga or OpenBGPd as RR apart from lack of different knobs and not-so-stellar performance/scalability. BIRD is grounds up built to act as high-performance BGP daemon, and it’s actually used as RR in live deployments, not only at IXes. > I am wondering if people can point me in the direction to some good resource > material on how to select a good BGP route reflector design. Should I just > dust off some 7206VXR routers to act as route reflectors? Use a few existing > live routers and just add the responsibility of being route reflectors, is > there a performance hit? Install and run BIRD on new server hardware? Buy > some newer purpose built routers (Cisco, Juniper, Brocade, etc) to act as > route reflectors and add them to the iBGP topology? GNS3 running IOS on > server hardware? Something else? How many reflectors should be implemented? > Two? Four? Disclaimer: I work at Cisco. If You have some 7200VXRs that have 1 or 2GBs of RAM, that may be the best option (IF you have them). Loaded with 12.2S/15S software they may actually be the most cost-effective solution and at the same time support things like AddPath, BGP error handling, etc - when time comes to use such features. If that’s a NPE400 based chassis or something even older - leave it for lab/etc as You need rather performant CPU. So, if that’s not the option, try to work with the BIRD, CSR 1000v (IOS-XE on VM) or ASR 1001X/HX (currently, the most scaleable and fastest BGP route reflector out there, but one that will cost $$$). Two RRs provide ample redundancy to run even very large deployments (1000+ clients), so unless you’re trying to hit higher numbers or plan to play fancy games with one pair of RRs for IPv4/IPv6 unicast and other pair for different AFs, four may be an overkill to maintain, synchronize and monitor. Don’t go with GNS3, running compiled at runtime emulation is wrong idea for any production deployment, not to mention rights/licenses to do it. — Łukasz Bromirski
BGP Route Reflector - Route Server, Router, etc
Nanog, I am working on some network designs and am adding some additional routers to a BGP network. I'd like to build a plan of changing all of the existing routers over from full iBGP mesh to something more scalable (ie route reflection). Fortunately, I am also going to be able to decommission some older routers from the network and so shrinking the existing iBGP full mesh is something I am all too happy to spend time and energy on. For the purpose of this thread though, I am not really interested in the route reflector vs confederation discussion. In doing some research[1][2][3][4][5] I see a lot of discussions, config examples, etc on using route reflectors but most suggest picking a router, or more appropriately a set of routers, to become route reflectors within an ASN. I have not found many resources discussing using a non-router box as a route reflector (ie a device not necessarily in the forwarding path of the through traffic). I am thinking things like OpenBGPd and BIRD could make a good route reflector though they are most often discussed in the context of IXPs (ie eBGP sessions). I am wondering if people can point me in the direction to some good resource material on how to select a good BGP route reflector design. Should I just dust off some 7206VXR routers to act as route reflectors? Use a few existing live routers and just add the responsibility of being route reflectors, is there a performance hit? Install and run BIRD on new server hardware? Buy some newer purpose built routers (Cisco, Juniper, Brocade, etc) to act as route reflectors and add them to the iBGP topology? GNS3 running IOS on server hardware? Something else? How many reflectors should be implemented? Two? Four? What are the pros and cons of one design over another? On list or private off list replies would be great; I'd welcome real world experiences (especially any big gotchas or caveats people learned the hard way) as well as just links to previous discussions, PDFs, slideshows, etc. Heck even a good book suggestion that covers this topic would be appreciated. [1] - iBGP-to-RR migration slideshow: http://meetings.ripe.net/ripe-42/presentations/ripe42-eof-bgp/sld015.html [2] - General RR design issues: http://www.netcraftsmen.com/bgp-route-reflector-design-issues/ [3] - Video intro to RR from Cisco: http://www.cisco.com/c/dam/en_us/training-events/le31/le46/cln/qlm/CCIP/bgp/introducing-route-reflectors-2/player.html [4] - Quagga and BIRD as RR example: https://bsdrp.net/documentation/examples/bgp_route_reflector_and_confederation_using_quagga_and_bird [5] - Countless hours on youtube: https://www.youtube.com/results?search_query=bgp+route+reflector Lots more data is out there of course as that is part of my problem. Thanks! Justin
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
El 12/1/2017 16:32, "Saku Ytti" escribió: On 12 January 2017 at 17:02, Fernando Gont wrote: > That's the point: If you don't allow fragments, but your peer honors > ICMPv6 PTB<1280, then dropping fragments creates the attack vector. Thanks. I think I got it now. Best I can offer is that B could try to verify the embedded original packet? Hopefully attacker won't have access to that information. An if attacker has access to that information, they may as well do TCP RST, right? Didn't we have same issues in IPv4 with ICMP unreachable and frag neeeded, DF set? And vendors implemented more verification if the ICMP message should be accepted. Yes and no. 1) there was no way in v4 to trigger use of fragmentation for an arbitrary flow. 2) in v4 you were guaranteed to get the IP+TCP header in the ICMP payload. In ipv6, you aren't (think ipv6 EHs) Thanks, Fernando
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
El 12/1/2017 16:28, "Mark Andrews" escribió: In message <11ff128d-2fba-7c26-4a9c-5611433d8...@si6networks.com>, Fernando Gon t writes: > Hi, Saku, > > On 01/12/2017 11:43 AM, Saku Ytti wrote: > > On 12 January 2017 at 13:19, Fernando Gont wrote: > > > > Hey, > > > >> I'm curious about whether folks are normally filtering ICMPv6 PTB<1280 > >> and/or IPv6 fragments targeted to BGP routers (off-list datapoints are > >> welcome). > > > > Generally may be understood differently by different people. If > > generally is defined as single most typical behaviour/configuration, > > then generally people don't protect their infrastructure in any way at > > all, but fully rely vendor doing something reasonable. > > > > I would argue BCP is to have 'strict' CoPP. Where you specifically > > allow what you must then have ultimate rule to deny everything. If you > > have such CoPP, then this attack won't work, as you clearly didn't > > allow any fragments at all (as you didn't expect to receive BGP > > fragments from your neighbours). > > That's the point: If you don't allow fragments, but your peer honors > ICMPv6 PTB<1280, then dropping fragments creates the attack vector. And fragments are a *normal* part of IP for both IPv4 and IPv6. This obsession with dropping all fragments (and yes it is a obsession) is breaking the internet. Vendors got the frag reassembly code wrong so many times , that I understand the folk that decides to drop them if deemed unnecessary. Even if you don't want to allow all fragments through you can allow fragments between the two endpoints of a "active" connection. At times folks want to get rid of fragments directed to them, rather than those going *through* them. You can apply port filters to the offset 0 fragments. If that fragment doesn't have enough headers to be able to filter then drop it. If your firewall is incapable of doing this then find a better firewall as the current one is a piece of garbage and should be in the recycle bin. Which DoS is the bigger issue? Firewalls dropping fragments or reassembly buffers being exhausted? If there is no way for an attacker to trigger the use of fragmentation, and you don't need fragments (e.g. only tcp-based services), from a security pov you're certainly better off dropping frags that are thrown at you. Not that I like it, but Thanks, Fernando
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
On 12 January 2017 at 21:53, Fernando Gont wrote: > besides, becaude of ipv6 ehs, you're not really guaranteed to receive e.g. > the tcp header in the embedded payload (the embedded payload could easily be > fixed ipv6 header + ehs). If the CoPP drops what has not been explicitly allowed, then packets with EH should be dropped. Particularly if you're not allowing 'tcp' but you're allowing 'next-header tcp'. I.e. the real BGP session (that attacker is trying to disrupt) would not have EH, on account that it would not come up if it had. But I agree if you do need and do use EH, things can get really dirty really fast, fundamentally no one implements standard compliant IPv6, if we're insisting that even pathological EH chains should work (which is fair viewpoint, while not one that I share). Maybe embedded flow-label could used to add confidence ICMP message is for packet we've actually sent? Make flow-label hash, a'la syn cookie. Spooffed ICMP message to disrupt existing TCP isn't novel. Coincidentally one service I use stopped working yesterday, but just for me, after short debugging, there was route cache entry on the server for my client ip which needed to be flushed, perhaps ended up there due to rogue ICMP redirect. -- ++ytti
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
Many (most?) Implementations don't even check the embedded port numbers...do tye attacker does not even need to guess the client port. besides, becaude of ipv6 ehs, you're not really guaranteed to receive e.g. the tcp header in the embedded payload (the embedded payload could easily be fixed ipv6 header + ehs). Cheers, Fernando El 12/1/2017 16:32, "Saku Ytti" escribió: > On 12 January 2017 at 17:02, Fernando Gont wrote: > > That's the point: If you don't allow fragments, but your peer honors > > ICMPv6 PTB<1280, then dropping fragments creates the attack vector. > > Thanks. I think I got it now. Best I can offer is that B could try to > verify the embedded original packet? Hopefully attacker won't have > access to that information. An if attacker has access to that > information, they may as well do TCP RST, right? > > Didn't we have same issues in IPv4 with ICMP unreachable and frag > neeeded, DF set? And vendors implemented more verification if the ICMP > message should be accepted. > > -- > ++ytti >
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
On 12 January 2017 at 17:02, Fernando Gont wrote: > That's the point: If you don't allow fragments, but your peer honors > ICMPv6 PTB<1280, then dropping fragments creates the attack vector. Thanks. I think I got it now. Best I can offer is that B could try to verify the embedded original packet? Hopefully attacker won't have access to that information. An if attacker has access to that information, they may as well do TCP RST, right? Didn't we have same issues in IPv4 with ICMP unreachable and frag neeeded, DF set? And vendors implemented more verification if the ICMP message should be accepted. -- ++ytti
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
In message <11ff128d-2fba-7c26-4a9c-5611433d8...@si6networks.com>, Fernando Gon t writes: > Hi, Saku, > > On 01/12/2017 11:43 AM, Saku Ytti wrote: > > On 12 January 2017 at 13:19, Fernando Gont wrote: > > > > Hey, > > > >> I'm curious about whether folks are normally filtering ICMPv6 PTB<1280 > >> and/or IPv6 fragments targeted to BGP routers (off-list datapoints are > >> welcome). > > > > Generally may be understood differently by different people. If > > generally is defined as single most typical behaviour/configuration, > > then generally people don't protect their infrastructure in any way at > > all, but fully rely vendor doing something reasonable. > > > > I would argue BCP is to have 'strict' CoPP. Where you specifically > > allow what you must then have ultimate rule to deny everything. If you > > have such CoPP, then this attack won't work, as you clearly didn't > > allow any fragments at all (as you didn't expect to receive BGP > > fragments from your neighbours). > > That's the point: If you don't allow fragments, but your peer honors > ICMPv6 PTB<1280, then dropping fragments creates the attack vector. And fragments are a *normal* part of IP for both IPv4 and IPv6. This obsession with dropping all fragments (and yes it is a obsession) is breaking the internet. Even if you don't want to allow all fragments through you can allow fragments between the two endpoints of a "active" connection. You can apply port filters to the offset 0 fragments. If that fragment doesn't have enough headers to be able to filter then drop it. If your firewall is incapable of doing this then find a better firewall as the current one is a piece of garbage and should be in the recycle bin. Which DoS is the bigger issue? Firewalls dropping fragments or reassembly buffers being exhausted? Yes, firewalls dropping fragments is a denial of service attack. The initial TCP exchange does not contain fragments. Most UDP protocols don't start with a packet that will need to be fragmented. For other protocols YMMV. Mark > -- > Fernando Gont > SI6 Networks > e-mail: fg...@si6networks.com > PGP Fingerprint: 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492 > > > > -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
Hi, Saku, On 01/12/2017 11:43 AM, Saku Ytti wrote: > On 12 January 2017 at 13:19, Fernando Gont wrote: > > Hey, > >> I'm curious about whether folks are normally filtering ICMPv6 PTB<1280 >> and/or IPv6 fragments targeted to BGP routers (off-list datapoints are >> welcome). > > Generally may be understood differently by different people. If > generally is defined as single most typical behaviour/configuration, > then generally people don't protect their infrastructure in any way at > all, but fully rely vendor doing something reasonable. > > I would argue BCP is to have 'strict' CoPP. Where you specifically > allow what you must then have ultimate rule to deny everything. If you > have such CoPP, then this attack won't work, as you clearly didn't > allow any fragments at all (as you didn't expect to receive BGP > fragments from your neighbours). That's the point: If you don't allow fragments, but your peer honors ICMPv6 PTB<1280, then dropping fragments creates the attack vector. -- Fernando Gont SI6 Networks e-mail: fg...@si6networks.com PGP Fingerprint: 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
Re: ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
On 12 January 2017 at 13:19, Fernando Gont wrote: Hey, > I'm curious about whether folks are normally filtering ICMPv6 PTB<1280 > and/or IPv6 fragments targeted to BGP routers (off-list datapoints are > welcome). Generally may be understood differently by different people. If generally is defined as single most typical behaviour/configuration, then generally people don't protect their infrastructure in any way at all, but fully rely vendor doing something reasonable. I would argue BCP is to have 'strict' CoPP. Where you specifically allow what you must then have ultimate rule to deny everything. If you have such CoPP, then this attack won't work, as you clearly didn't allow any fragments at all (as you didn't expect to receive BGP fragments from your neighbours). -- ++ytti
Re: Bandwidth Savings
At the end of the day it’s a cost/benefit analysis, reading the OP’s message it’s clear that the IX or transit isn’t really the problem, but rather how much traffic traverses to the point of transit/IX. I think until we see some numbers both in terms of traffic and in terms of price to say Miami, Willemstad, etc. we can’t easily determine a good solution. Regards, Marty Strong -- Cloudflare - AS13335 Network Engineer ma...@cloudflare.com +44 7584 906 055 smartflare (Skype) https://www.peeringdb.com/asn/13335 > On 12 Jan 2017, at 13:38, Mike Hammett wrote: > > I remember there were a couple different owners down there, but Columbus > swallowed them up, then C&W got Columbus then Liberty got C&W. > > Not knowing the international cable market, is this like AT&T, Comcast or > Verizon (the largest US last mile operators) being your only option? > > > > > - > Mike Hammett > Intelligent Computing Solutions > > Midwest Internet Exchange > > The Brothers WISP > > - Original Message - > > From: "Eric Kuhnke" > To: "nanog@nanog.org list" > Sent: Wednesday, January 11, 2017 3:23:58 PM > Subject: Re: Bandwidth Savings > > The challenges are almost certainly economics related, at the lack of > competition and high costs for layer 1/2 transport from his Caribbean > island to Miami. Via whatever submarine cables exist that are controlled by > larger ILEC type entities/telcos. Or satellite (whether geostationary > transponder capacity or o3b). > > Depending on what island we're talking about, the $/month for a single > 1GbE or 10GbE layer 2 transport service from $ISLAND to Miami will be very > high compared to what a network operator in the US 48 states is accustomed > to paying. > > On Wed, Jan 11, 2017 at 12:55 PM, Richard Hicks > wrote: > >> >> I don't know the the Caribbean Internet Exchanges market. Are any worth >> peering at versus buying additional L2 bandwidth to Miami? >> >> https://cw.ams-ix.net/ >> http://www.ocix.net/ocix/ >> >> Rick >> >> On Tue, Jan 10, 2017 at 8:08 PM, Keenan Singh >> wrote: >> >>> Hi Guys >>> >>> We are an ISP in the Caribbean, and are faced with extremely high >> Bandwidth >>> costs, compared to the US, we currently use Peer App for Caching however >>> with most services now moving to HTTPS the cache is proving to be less >> and >>> less effective. We are currently looking at any way we can save on >>> Bandwidth or to be more Efficient with the Bandwidth we currently have. >> We >>> do have a Layer 2 Circuit between the Island and Miami, I am seeing there >>> are WAN Accelerators where they would put a Server on either end and sort >>> of Compress and decompress the Traffic before it goes over the Layer 2, I >>> have never used this before, has any one here used anything like this, >> what >>> results would I be able to expect for ISP Traffic? >>> >>> If not any ideas on Bandwidth Savings, or being more Efficient with want >> we >>> currently. >>> >>> Many thanks for any Help >>> >>> Keenan >>> >> >
Re: Bandwidth Savings
I remember there were a couple different owners down there, but Columbus swallowed them up, then C&W got Columbus then Liberty got C&W. Not knowing the international cable market, is this like AT&T, Comcast or Verizon (the largest US last mile operators) being your only option? - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Eric Kuhnke" To: "nanog@nanog.org list" Sent: Wednesday, January 11, 2017 3:23:58 PM Subject: Re: Bandwidth Savings The challenges are almost certainly economics related, at the lack of competition and high costs for layer 1/2 transport from his Caribbean island to Miami. Via whatever submarine cables exist that are controlled by larger ILEC type entities/telcos. Or satellite (whether geostationary transponder capacity or o3b). Depending on what island we're talking about, the $/month for a single 1GbE or 10GbE layer 2 transport service from $ISLAND to Miami will be very high compared to what a network operator in the US 48 states is accustomed to paying. On Wed, Jan 11, 2017 at 12:55 PM, Richard Hicks wrote: > > I don't know the the Caribbean Internet Exchanges market. Are any worth > peering at versus buying additional L2 bandwidth to Miami? > > https://cw.ams-ix.net/ > http://www.ocix.net/ocix/ > > Rick > > On Tue, Jan 10, 2017 at 8:08 PM, Keenan Singh > wrote: > > > Hi Guys > > > > We are an ISP in the Caribbean, and are faced with extremely high > Bandwidth > > costs, compared to the US, we currently use Peer App for Caching however > > with most services now moving to HTTPS the cache is proving to be less > and > > less effective. We are currently looking at any way we can save on > > Bandwidth or to be more Efficient with the Bandwidth we currently have. > We > > do have a Layer 2 Circuit between the Island and Miami, I am seeing there > > are WAN Accelerators where they would put a Server on either end and sort > > of Compress and decompress the Traffic before it goes over the Layer 2, I > > have never used this before, has any one here used anything like this, > what > > results would I be able to expect for ISP Traffic? > > > > If not any ideas on Bandwidth Savings, or being more Efficient with want > we > > currently. > > > > Many thanks for any Help > > > > Keenan > > >
Re: Bandwidth Savings
Peering is great when you can get to the IX inexpensively. I assume that glass that goes underwater has a significant increase in cost and therefore the cost savings of peering would be minuscule in comparison to the cost of the rest of the connectivity. - Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP - Original Message - From: "Richard Hicks" To: "Keenan Singh" Cc: nanog@nanog.org Sent: Wednesday, January 11, 2017 2:55:05 PM Subject: Re: Bandwidth Savings I don't know the the Caribbean Internet Exchanges market. Are any worth peering at versus buying additional L2 bandwidth to Miami? https://cw.ams-ix.net/ http://www.ocix.net/ocix/ Rick On Tue, Jan 10, 2017 at 8:08 PM, Keenan Singh wrote: > Hi Guys > > We are an ISP in the Caribbean, and are faced with extremely high Bandwidth > costs, compared to the US, we currently use Peer App for Caching however > with most services now moving to HTTPS the cache is proving to be less and > less effective. We are currently looking at any way we can save on > Bandwidth or to be more Efficient with the Bandwidth we currently have. We > do have a Layer 2 Circuit between the Island and Miami, I am seeing there > are WAN Accelerators where they would put a Server on either end and sort > of Compress and decompress the Traffic before it goes over the Layer 2, I > have never used this before, has any one here used anything like this, what > results would I be able to expect for ISP Traffic? > > If not any ideas on Bandwidth Savings, or being more Efficient with want we > currently. > > Many thanks for any Help > > Keenan >
Re: Bandwidth Savings
On Tue, Jan 10, 2017 at 11:08:45PM -0500, Keenan Singh wrote: > We are currently looking at any way we can save on > Bandwidth or to be more Efficient with the Bandwidth we currently have. Measure what you're doing in as much detail as you can. Slice-and-dice it by source, destination, time-of-day, protocol, day-of-week, etc. and plot the results. In most cases, the problems will make themselves leap off the page. (We could all probably predict what they are right now, but real live measurements are still a good idea.) Once you've identified the problems, you'll be much closer to enumerating possible solutions (and you can avoid wasting time on the issues that aren't worth addressing). ---rsk
ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)
Folks, I'm curious about whether folks are normally filtering ICMPv6 PTB<1280 and/or IPv6 fragments targeted to BGP routers (off-list datapoints are welcome). In any case, you mind find it worth reading to check if you're affected (from Section 2 of recently-published RFC8021): cut here The security implications of IP fragmentation have been discussed at length in [RFC6274] and [RFC7739]. An attacker can leverage the generation of IPv6 atomic fragments to trigger the use of fragmentation in an arbitrary IPv6 flow (in scenarios in which actual fragmentation of packets is not needed) and can subsequently perform any type of fragmentation-based attack against legacy IPv6 nodes that do not implement [RFC6946]. That is, employing fragmentation where not actually needed allows for fragmentation-based attack vectors to be employed, unnecessarily. We note that, unfortunately, even nodes that already implement [RFC6946] can be subject to DoS attacks as a result of the generation of IPv6 atomic fragments. Let us assume that Host A is communicating with Host B and that, as a result of the widespread dropping of IPv6 packets that contain extension headers (including fragmentation) [RFC7872], some intermediate node filters fragments between Host B and Host A. If an attacker sends a forged ICMPv6 PTB error message to Host B, reporting an MTU smaller than 1280, this will trigger the generation of IPv6 atomic fragments from that moment on (as required by [RFC2460]). When Host B starts sending IPv6 atomic fragments (in response to the received ICMPv6 PTB error message), these packets will be dropped, since we previously noted that IPv6 packets with extension headers were being dropped between Host B and Host A. Thus, this situation will result in a DoS scenario. Another possible scenario is that in which two BGP peers are employing IPv6 transport and they implement Access Control Lists (ACLs) to drop IPv6 fragments (to avoid control-plane attacks). If the aforementioned BGP peers drop IPv6 fragments but still honor received ICMPv6 PTB error messages, an attacker could easily attack the corresponding peering session by simply sending an ICMPv6 PTB message with a reported MTU smaller than 1280 bytes. Once the attack packet has been sent, the aforementioned routers will themselves be the ones dropping their own traffic. cut here Is this something waiting to be exploited? Am I missing something? Thanks, -- Fernando Gont SI6 Networks e-mail: fg...@si6networks.com PGP Fingerprint: 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
