RE: best practice for advertising peering fabric routes
However, a good engineer would know there are drawbacks to next-hop-self, in particular it slows convergence in a number of situations. There are networks where fast convergence is more important than route scaling, and thus the traditional design of BGP next-hops being edge interfaces, and edge interfaces in the IGP performs better. Well it's not true anymore BGP PIC edge and core converges under 50ms. fast external failover and local repair where available long before -but yes that's applicable only for MPLS. adam -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Wednesday, January 15, 2014 3:18 PM To: Dobbins, Roland Cc: NANOG list Subject: Re: best practice for advertising peering fabric routes On Jan 15, 2014, at 12:02 AM, Dobbins, Roland rdobb...@arbor.net wrote: Again, folks, this isn't theoretical. When the particular attacks cited in this thread were taking place, I was astonished that the IXP infrastructure routes were even being advertised outside of the IXP network, because of these very issues. I know a lot of people push next-hop-self, and if you're a large ISP with thousands of BGP customers is pretty much required to scale. However, a good engineer would know there are drawbacks to next-hop-self, in particular it slows convergence in a number of situations. There are networks where fast convergence is more important than route scaling, and thus the traditional design of BGP next-hops being edge interfaces, and edge interfaces in the IGP performs better. By attempting to force IX participants to not put the route in IGP, those IX participants are collectively deciding on a slower converging network for everyone. I don't like a world where connecting to an exchange point forces a particular network design on participants. IXPs are not the problem when it comes to breaking PMTU-D. The problem is largely with enterprise networks, and with 'security' vendors who've propagated the myth that simply blocking all ICMP somehow increases 'security'. That's some circular reasoning. Networks won't 9K peer at exchange points for a number of reasons, including PMTU-D discovery issues. Since there are virtual no 9K peering at exchange points, PMTU-D is a non-issue. Maybe if IXP design didn't break PMTU-D it would help attract more 9K peers, or there might even be a future where 9K peering was required? This whole problem smacks to me of exchange points that are too big to fail. Since some of these exchanges are so big, everyone else must bend to their needs. I think the world would be a better place if some of these were broken up into smaller exchanges and they imposed less restrictions on their participants. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
RE: best practice for advertising peering fabric routes
But hey, I get why ISP's don't want to offer 9K MTU clean paths end to end. Customers could then buy a VPN appliance and manage their own VPN's with no vendor lock-in. MPLS VPN revenues would tumble, and customers would move more fluidly between providers. That's terrible if you're an ISP. No it's exactly why some carriers do their best to provide 9K+ MTU to most of their POPs so that they can provide L2 services to ISPs and other carriers that require 9K MTU for their BB links to capitalize on these new emerging markets. Customers locked in to a single provider (MPLS VPN) can rely on certain class of service (predictable delay, jitter and packet loss) properties you can't get out of a pure internet connection from NY to Tokyo. adam -Original Message- From: Leo Bicknell [mailto:bickn...@ufp.org] Sent: Wednesday, January 15, 2014 4:31 PM To: Dobbins, Roland Cc: NANOG list Subject: Re: best practice for advertising peering fabric routes On Jan 15, 2014, at 8:49 AM, Dobbins, Roland rdobb...@arbor.net wrote: Not really. What I'm saying is that since PMTU-D is already broken on so many endpoint networks - i.e., where traffic originates and where it terminates - that any issues arising from PMTU-D irregularities in IXP networks are trivial by comparison. I think we're looking at two different aspects of the same issue. I believe you're coming at it from a 'for all users of the Internet, what's the chance they have connectivity that does not break PMTU-D.' That's an important group to study, particularly for those DSL users still left with 1500 byte MTU's. And you're right, for those users IXP's are the least of their worries, mostly it's content-side poor networking, like load balancers and firewalls that don't work correctly. I am approaching it from a different perspective, 'where is PMTU-D broken for people who want to use 1500-9K frames end to end?' I'm the network guy who wants to buy transit in the US, and transit in Germany and run a tunnel of 1500 byte packets end to end, necessitating a ~1540 byte packet. Finding transit providers who will configure jumbo frames is trivial these days, and most backbones are jumbo frame clean (at least to 4470, but many to 9K). There's probably about a 25% chance private peelings are also jumbo clean. Pretty much the only thing broken for this use case is IXP's. Only a few have a second VLAN for 9K peerings, and most participants don't use it for a host of reasons, including PMTU-D problems. I'm an oddball. I think MPLS VPN's are a terrible idea for the consumer, locking them into a single provider in the vast majority of cases. Consumers would be better served by having a tunnel box (IPSec maybe?) at their edge and running there own tunnel over IP provider-independently, if they could get 1500B MTU at the edge, and move those packets end to end. While I've always thought that, in the post-Snowden world I think I seem a little less crazy, rather than relying on your provider to keep your VPN traffic secret, customers should be encrypting it in a device they own. But hey, I get why ISP's don't want to offer 9K MTU clean paths end to end. Customers could then buy a VPN appliance and manage their own VPN's with no vendor lock-in. MPLS VPN revenues would tumble, and customers would move more fluidly between providers. That's terrible if you're an ISP. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Re: best practice for advertising peering fabric routes
* Patrick W. Gilmore: NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. Any ideas why DE-CIX doesn't enforce this? One advantage is that IXP participants can perform emergency maintenance if they have isolated their IXP router from their own network.
Re: best practice for advertising peering fabric routes
Hello Leo, On Wed, 15 Jan 2014 08:18:13 -0600 Leo Bicknell bickn...@ufp.org wrote: This whole problem smacks to me of exchange points that are too big to fail. Since some of these exchanges are so big, everyone else must bend to their needs. I think the world would be a better place if some of these were broken up into smaller exchanges and they imposed less restrictions on their participants. You forgot to add and would break down on a weekly basis. The restrictions that IXPs impose on their customers have nothing to do with the size of their peering LAN, but everything with offering a reliable service to these same customers. Kind regards, Martin signature.asc Description: PGP signature
Re: best practice for advertising peering fabric routes
On Wednesday, January 15, 2014 09:57:32 AM Michael Hallgren wrote: I don't think you need route-reflection in a 5 node iBGP. I'm for doing it now and not worrying about it later. Also, don't originate your routes from your peering router Mark. signature.asc Description: This is a digitally signed message part.
Re: best practice for advertising peering fabric routes
On Jan 15, 2014, at 12:02 AM, Dobbins, Roland rdobb...@arbor.net wrote: Again, folks, this isn't theoretical. When the particular attacks cited in this thread were taking place, I was astonished that the IXP infrastructure routes were even being advertised outside of the IXP network, because of these very issues. I know a lot of people push next-hop-self, and if you're a large ISP with thousands of BGP customers is pretty much required to scale. However, a good engineer would know there are drawbacks to next-hop-self, in particular it slows convergence in a number of situations. There are networks where fast convergence is more important than route scaling, and thus the traditional design of BGP next-hops being edge interfaces, and edge interfaces in the IGP performs better. By attempting to force IX participants to not put the route in IGP, those IX participants are collectively deciding on a slower converging network for everyone. I don't like a world where connecting to an exchange point forces a particular network design on participants. IXPs are not the problem when it comes to breaking PMTU-D. The problem is largely with enterprise networks, and with 'security' vendors who've propagated the myth that simply blocking all ICMP somehow increases 'security'. That's some circular reasoning. Networks won't 9K peer at exchange points for a number of reasons, including PMTU-D discovery issues. Since there are virtual no 9K peering at exchange points, PMTU-D is a non-issue. Maybe if IXP design didn't break PMTU-D it would help attract more 9K peers, or there might even be a future where 9K peering was required? This whole problem smacks to me of exchange points that are too big to fail. Since some of these exchanges are so big, everyone else must bend to their needs. I think the world would be a better place if some of these were broken up into smaller exchanges and they imposed less restrictions on their participants. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
On Jan 15, 2014, at 9:18 PM, Leo Bicknell bickn...@ufp.org wrote: However, a good engineer would know there are drawbacks to next-hop-self, in particular it slows convergence in a number of situations. There are networks where fast convergence is more important than route scaling, and thus the traditional design of BGP next-hops being edge interfaces, and edge interfaces in the IGP performs better. A good engineer also knows that there are huge drawbacks to having a peer's network infrastructure DDoSed, routes flapping, core bandwidth consumed by tens and hundreds of gb/sec of attack traffic, et. al., too. ; By attempting to force IX participants to not put the route in IGP, those IX participants are collectively deciding on a slower converging network for everyone. I don't like a world where connecting to an exchange point forces a particular network design on participants. Concur. But that's the world we live in, unfortunately. It's just another example of the huge, concentric nature of the collateral damage arising from DDoS attacks, both from the attacks themselves, and from the compromises folks have to make in order to increase resilience against such attacks. That's some circular reasoning. Not really. What I'm saying is that since PMTU-D is already broken on so many endpoint networks - i.e., where traffic originates and where it terminates - that any issues arising from PMTU-D irregularities in IXP networks are trivial by comparison. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
On (2014-01-15 08:18 -0600), Leo Bicknell wrote: I know a lot of people push next-hop-self, and if you're a large ISP with thousands of BGP customers is pretty much required to scale. It's actually the polar opposite. If you are small, there are no compelling reasons to put IXP in IGP. If you are large, you may wish to have different IGP metric to two egress points in same peering router. In which case you should at very least have IP ACL in IXP interface which only allows LAN2LAN. -- ++ytti
Re: best practice for advertising peering fabric routes
On Jan 15, 2014, at 8:49 AM, Dobbins, Roland rdobb...@arbor.net wrote: Not really. What I'm saying is that since PMTU-D is already broken on so many endpoint networks - i.e., where traffic originates and where it terminates - that any issues arising from PMTU-D irregularities in IXP networks are trivial by comparison. I think we're looking at two different aspects of the same issue. I believe you're coming at it from a 'for all users of the Internet, what's the chance they have connectivity that does not break PMTU-D.' That's an important group to study, particularly for those DSL users still left with 1500 byte MTU's. And you're right, for those users IXP's are the least of their worries, mostly it's content-side poor networking, like load balancers and firewalls that don't work correctly. I am approaching it from a different perspective, 'where is PMTU-D broken for people who want to use 1500-9K frames end to end?' I'm the network guy who wants to buy transit in the US, and transit in Germany and run a tunnel of 1500 byte packets end to end, necessitating a ~1540 byte packet. Finding transit providers who will configure jumbo frames is trivial these days, and most backbones are jumbo frame clean (at least to 4470, but many to 9K). There's probably about a 25% chance private peelings are also jumbo clean. Pretty much the only thing broken for this use case is IXP's. Only a few have a second VLAN for 9K peerings, and most participants don't use it for a host of reasons, including PMTU-D problems. I'm an oddball. I think MPLS VPN's are a terrible idea for the consumer, locking them into a single provider in the vast majority of cases. Consumers would be better served by having a tunnel box (IPSec maybe?) at their edge and running there own tunnel over IP provider-independently, if they could get 1500B MTU at the edge, and move those packets end to end. While I've always thought that, in the post-Snowden world I think I seem a little less crazy, rather than relying on your provider to keep your VPN traffic secret, customers should be encrypting it in a device they own. But hey, I get why ISP's don't want to offer 9K MTU clean paths end to end. Customers could then buy a VPN appliance and manage their own VPN's with no vendor lock-in. MPLS VPN revenues would tumble, and customers would move more fluidly between providers. That's terrible if you're an ISP. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
On Jan 15, 2014, at 10:31 PM, Leo Bicknell bickn...@ufp.org wrote: I am approaching it from a different perspective, 'where is PMTU-D broken for people who want to use 1500-9K frames end to end?' I understand that perspective, absolutely. But what I'm saying is that that whether or not they want to use jumbo frames for Internet traffic, it doesn't matter, because PMTU-D is likely to be broken either at the place where the traffic is initiated, the place where the traffic is received, or both - so any nonsense in the middle, especially on IXP networks in particular, isn't really a significant issue in and of itself. If we could get things optimized and remediated to the point where potential PMTU-D breakage in IXP networks were a significant issue of iteself, the Internet would be much improved. But I don't see any likelihood of that happening anytime soon. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
On Tue, Jan 14, 2014 at 10:11 PM, Patrick W. Gilmore patr...@ianai.net wrote: NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. Hi Patrick, I have to disagree with you. If it appears in a traceroute to somewhere else, I'd like to be able to ping and traceroute directly to it. When I can't, that impairs my ability to troubleshoot the all too common can't-get-there-from-here problems. The more you hide the infrastructure, the more intractable problems become for your customers. The IXP LAN should be reachable from every device on the ASes which connect to it, not just the immediate router. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: best practice for advertising peering fabric routes
On Jan 15, 2014, at 9:37 AM, Dobbins, Roland rdobb...@arbor.net wrote: But what I'm saying is that that whether or not they want to use jumbo frames for Internet traffic, it doesn't matter, because PMTU-D is likely to be broken either at the place where the traffic is initiated, the place where the traffic is received, or both - so any nonsense in the middle, especially on IXP networks in particular, isn't really a significant issue in and of itself. Your assertion does not match my deployment experience. When I have deployed endpoints that have working PMTU-D, I have 99.999% success with the ISP's in the middle having working PMTU-D. It even works fine for 9K providers connected to 1500B exchange points, because the packet-too-big typically originates from the input side of the router (the backbone link to the IXP router). Indeed, the only place I've seen it broken is where the ISP 9K peers at an exchange, and the far end ISP runs a 9K backbone (like 4470), so the far end IXP-router does the packet-to-big, and originates it from the exchange LAN, which because it's no longer in the table fails to past uRPF. (Business class) ISP's don't break PMTU-D, end users break it with the equipment they connect. So a smart user connecting equipment that is properly configured should be able to expect it to work properly. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
On Jan 15, 2014, at 10:44 , William Herrin b...@herrin.us wrote: On Tue, Jan 14, 2014 at 10:11 PM, Patrick W. Gilmore patr...@ianai.net wrote: NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. Hi Patrick, I have to disagree with you. If it appears in a traceroute to somewhere else, I'd like to be able to ping and traceroute directly to it. When I can't, that impairs my ability to troubleshoot the all too common can't-get-there-from-here problems. The more you hide the infrastructure, the more intractable problems become for your customers. The IXP LAN should be reachable from every device on the ASes which connect to it, not just the immediate router. We disagree. Plus, you really can't type ping on the router connected to the IXP? _If_ you can guarantee your network has zero bots, abusable [DNS|NTP|etc.] servers, all your downstreams are perfectly clean, etc., etc., then maybe I could see you carrying it in your IGP. As I know 100% of ISPs (to at least one decimal place) cannot make such a guarantee, then doing so puts the IXP and all other members - whether peers of yours or not - at risk. Putting others at risk because you are lazy or because it makes your life easier is .. I believe I called it bad manners before. But let's take the philosophical out of this. The prefix in question is owned by the IXP. I said in an earlier post that if you carry a prefix I own, did not announce to you, and make it very clear I specifically do not want you to carry, I will ask you to stop or face possible disconnection. You may claim convergence (a bit of BS), troubleshooting (non-issue, IMO), or even but I wnt to!!1!1! (whatever). Doesn't matter. That's not your prefix, you were not given it and told not to carry it, so Do Not Carry It. Ask your IXP if they mind whether you carry the prefix. See what they say. -- TTFN, patrick
Re: best practice for advertising peering fabric routes
On Jan 15, 2014, at 10:52 PM, Leo Bicknell bickn...@ufp.org wrote: (Business class) ISP's don't break PMTU-D, end users break it with the equipment they connect. Concur 100%. That's my point. So a smart user connecting equipment that is properly configured should be able to expect it to work properly. In my deployment experience, many (most?) end-user organization break PMTU-D to/through their LANs outside of their IDCs, much less to the Internet, for themselves, and for everyone who wishes to communicate with them across the Internet. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton signature.asc Description: Message signed with OpenPGP using GPGMail
RE: best practice for advertising peering fabric routes
UUnet once advertised the /24 for MAE-East to me (well, Net99), and because I also had it in my IGP, my network was using UUnet's backbone for west-to-east coast traffic for a couple of days until I noticed and fixed it (with next-hop-self). I agree 100% with Patrick and others on this point. No good can come from propagating IXP address space any further than is absolutely necessary. Best not to propagate it at all. Dave -Original Message- From: Patrick W. Gilmore [mailto:patr...@ianai.net] Sent: Wednesday, January 15, 2014 8:57 AM To: NANOG list Subject: Re: best practice for advertising peering fabric routes On Jan 15, 2014, at 10:44 , William Herrin b...@herrin.us wrote: On Tue, Jan 14, 2014 at 10:11 PM, Patrick W. Gilmore patr...@ianai.net wrote: NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. Hi Patrick, I have to disagree with you. If it appears in a traceroute to somewhere else, I'd like to be able to ping and traceroute directly to it. When I can't, that impairs my ability to troubleshoot the all too common can't-get-there-from-here problems. The more you hide the infrastructure, the more intractable problems become for your customers. The IXP LAN should be reachable from every device on the ASes which connect to it, not just the immediate router. We disagree. Plus, you really can't type ping on the router connected to the IXP? _If_ you can guarantee your network has zero bots, abusable [DNS|NTP|etc.] servers, all your downstreams are perfectly clean, etc., etc., then maybe I could see you carrying it in your IGP. As I know 100% of ISPs (to at least one decimal place) cannot make such a guarantee, then doing so puts the IXP and all other members - whether peers of yours or not - at risk. Putting others at risk because you are lazy or because it makes your life easier is .. I believe I called it bad manners before. But let's take the philosophical out of this. The prefix in question is owned by the IXP. I said in an earlier post that if you carry a prefix I own, did not announce to you, and make it very clear I specifically do not want you to carry, I will ask you to stop or face possible disconnection. You may claim convergence (a bit of BS), troubleshooting (non-issue, IMO), or even but I wnt to!!1!1! (whatever). Doesn't matter. That's not your prefix, you were not given it and told not to carry it, so Do Not Carry It. Ask your IXP if they mind whether you carry the prefix. See what they say. -- TTFN, patrick
Re: best practice for advertising peering fabric routes
On Wed, Jan 15, 2014 at 10:57 AM, Patrick W. Gilmore patr...@ianai.net wrote: On Jan 15, 2014, at 10:44 , William Herrin b...@herrin.us wrote: I have to disagree with you. If it appears in a traceroute to somewhere else, I'd like to be able to ping and traceroute directly to it. When I can't, that impairs my ability to troubleshoot the all too common can't-get-there-from-here problems. The more you hide the infrastructure, the more intractable problems become for your customers. The IXP LAN should be reachable from every device on the ASes which connect to it, not just the immediate router. We disagree. Plus, you really can't type ping on the router connected to the IXP? Not when I'm the downstream customer, no. It's jolly good that *you* can test, but before the rest of us can get through the layers of support which insulate you, we have to be able to convincingly test too. As I know 100% of ISPs (to at least one decimal place) cannot make such a guarantee, then doing so puts the IXP and all other members - whether peers of yours or not - at risk. Putting others at risk because you are lazy or because it makes your life easier is .. I believe I called it bad manners before. That makes no sense. The IXP is at no more or less risk from your customers than any other connection you have for Internet carriage. Risk which you are responsible for managing either way. I said in an earlier post that if you carry a prefix I own, did not announce to you, and make it very clear I specifically do not want you to carry, I will ask you to stop or face possible disconnection. [...] That's not your prefix, you were not given it and told not to carry it, so Do Not Carry It. Well yes, of course. If you participate in an IXP you follow the rules of the IXP. I respectfully question the wisdom of such a rule and the IXPs I deal with only ask that you not announce the IXP prefix externally. But it's not OK to unilaterally break the IXP's rules, however poorly conceived. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: best practice for advertising peering fabric routes
On 1/14/14, 8:41 PM, Patrick W. Gilmore wrote: I repeat: NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device except those directly attached to that LAN. Period. So ... RFC1918 addresses for the IXP fabric, then? (Half kidding, but still ) Jim Shankland
Re: best practice for advertising peering fabric routes
On 2014-01-15, at 12:04, Jim Shankland na...@shankland.org wrote: On 1/14/14, 8:41 PM, Patrick W. Gilmore wrote: I repeat: NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device except those directly attached to that LAN. Period. So ... RFC1918 addresses for the IXP fabric, then? I've heard apparently non-drunk people suggest IPv6 link-local addresses as BGP endpoints across exchanges, too. (Half kidding, but still ) RFC 6752. One observation on this thread: some networks have customers who react badly to unusual things seen in traceroute. Sometimes the margin on an individual customer is low enough that one support call displaces any profit you were going to make off them this month. It's understandable to me that such network operators would choose to carry IXP routes internally in order to avoid that potential support burden. I don't pretend to have any universal good/bad answer to the original question, though. I don't think the world is that simple. Joe signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
* na...@shankland.org (Jim Shankland) [Wed 15 Jan 2014, 18:04 CET]: So ... RFC1918 addresses for the IXP fabric, then? (Half kidding, but still ) They need to be globally unique. -- Niels. -- It's amazing what people will do to get their name on the internet, which is odd, because all you really need is a Blogspot account. -- roy edroso, alicublog.blogspot.com
Re: best practice for advertising peering fabric routes
* patr...@ianai.net (Patrick W. Gilmore) [Wed 15 Jan 2014, 04:36 CET]: [..] NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. This is correct, and protects both your (ISP) infrastructure and the IXP's. All major European IXPs revisited their policy after the giant DDoS attack on CloudFlare, and the above was pretty much the outcome. -- Niels. -- It's amazing what people will do to get their name on the internet, which is odd, because all you really need is a Blogspot account. -- roy edroso, alicublog.blogspot.com
Re: best practice for advertising peering fabric routes
On Wed, Jan 15, 2014 at 12:54 PM, Niels Bakker niels=na...@bakker.net wrote: * na...@shankland.org (Jim Shankland) [Wed 15 Jan 2014, 18:04 CET]: So ... RFC1918 addresses for the IXP fabric, then? (Half kidding, but still ) They need to be globally unique. do they? :) also... there is/was an exchange in south america (columbia maybe? it's been a while since I saw this in configs) that used 192.168.0.0/16 space for their exchange.
Re: best practice for advertising peering fabric routes
On Wed, Jan 15, 2014 at 12:54 PM, Niels Bakker niels=na...@bakker.net wrote: * na...@shankland.org (Jim Shankland) [Wed 15 Jan 2014, 18:04 CET]: So ... RFC1918 addresses for the IXP fabric, then? (Half kidding, but still ) They need to be globally unique. Hi Niels, Actually, they don't. To meet the basic definition of working, they just have to be able to originate ICMP destination unreachable packets with a reasonable expectation that the recipient will receive those packets. Global uniqueness is not required for that. However, RFC1918 addresses don't meet the requirement for a different reason: they're routinely dropped at AS borders, thus don't have an expectation of reaching the external destination. Of course working, monitorable and testable are three different things. If my NMS can't reach the IXP's addresses, my view of the IXP is impaired. And the Internet is broken is not a trouble report that leads to a successful outcome with customer support... it helps to be able to pin things down with some specificity. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004
Re: best practice for advertising peering fabric routes
On Wed, Jan 15, 2014 at 1:26 PM, William Herrin b...@herrin.us wrote: On Wed, Jan 15, 2014 at 12:54 PM, Niels Bakker niels=na...@bakker.net wrote: * na...@shankland.org (Jim Shankland) [Wed 15 Jan 2014, 18:04 CET]: So ... RFC1918 addresses for the IXP fabric, then? (Half kidding, but still ) They need to be globally unique. Hi Niels, Actually, they don't. To meet the basic definition of working, they just have to be able to originate ICMP destination unreachable packets with a reasonable expectation that the recipient will receive those packets. Global uniqueness is not required for that. However, RFC1918 addresses don't meet the requirement for a different reason: they're routinely dropped at AS borders, thus don't have an expectation of reaching the external destination. Of course working, monitorable and testable are three different things. If my NMS can't reach the IXP's addresses, my view of the IXP is impaired. And the Internet is broken is not a trouble report that leads to a successful outcome with customer support... it helps to be able to pin things down with some specificity. Regards, Bill Herrin Using RFC1918 would incur the assumption that one will need to use a unique router or routing instance for every exchange connected to since exchanges are likely to have overlapping space at that point (RFC1918 IXP registry anyone?). I don't think it'd be a good idea to go down that path.. Also mentioned in a past nanog was the idea of potentially getting someone like team cymru to setup all exchange prefixes in a special bogon list and you could null route on your edge all those prefixes.. I inquired to team cymru about this back when originally discussed but never got anywhere with them. -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004 -- [stillwa...@gmail.com ~]$ cat .signature cat: .signature: No such file or directory [stillwa...@gmail.com ~]$
Re: best practice for advertising peering fabric routes
On Jan 15, 2014, at 10:26 AM, William Herrin b...@herrin.us wrote: Of course working, monitorable and testable are three different things. If my NMS can't reach the IXP's addresses, my view of the IXP is impaired. And the Internet is broken is not a trouble report that leads to a successful outcome with customer support... it helps to be able to pin things down with some specificity. This approach concerns me for a number of reasons. First, having your NMS ping your upstream’s IXP peers probably doesn’t scale. If I’m a peer of a reasonably large provider, I’m pretty sure I don’t want all their customers hammering my management plane. Even if you’re the only one doing it, you also don’t know if I’m rate-limiting pings for that or any other reason. Second, what information do you get that you didn’t already have? If you saw the IP in a traceroute then you know it exists, is alive, is in the path, and a rough estimation of the latency. Pinging it may even give you negative information. Platforms vary and all, but in my experience pinging a router, especially a potentially busy one peering at an IXP, shows notably worse performance than “real” traffic experiences (admittedly somewhat true of TTL Expired responses, but less so in my experience). Now you’re potentially seeing high latency and packet loss which in reality might not even be there at all. Third, you don’t know that your ping to the peering IP is even taking the same path as the packets addressed to the real destination. MTR for example looks nice, but it would probably be more accurate if it simply ran the traceroute over and over instead of pinging each hop directly. You would also detect path changes for the real destination that pinging intermediate hops wouldn’t show you. While I appreciate the desire to be able to do as much of your own detective work as possible, I can also see where you’re now shifting workload onto someone else’s support organization when they’re not necessarily the problem either (“Hey, my NMS says your peering router is causing latency and packet loss, fix it!”). I’m also not saying there isn’t a troubleshooting gap caused by this. I’m just not sure being able to ping the IXP hop solves that problem either. Semi-related tangent: Working in an IXP setting I have seen weird corner cases cause issues in conjunction with the IXP subnet existing in BGP. Say someone’s got proxy ARP enabled on their router (sadly, more common than it should be, and not just from noobs at startups). Now say your IXP is growing and you expand the subnet. No matter how much you harp on the customers to make the change, they don’t all do it at once. Someone announces the new, larger subnet in BGP. Now when anyone ARPs for IPs in the new part of the range, proxy ARP guy (still on the smaller subnet) says “hey I have a route for that, send it here”. That was fun to troubleshoot. :) -c
Re: best practice for advertising peering fabric routes
* c...@bloomcounty.org (Clay Fiske) [Wed 15 Jan 2014, 20:34 CET]: Semi-related tangent: Working in an IXP setting I have seen weird corner cases cause issues in conjunction with the IXP subnet existing in BGP. Say someone’s got proxy ARP enabled on their router (sadly, more common than it should be, and not just from noobs at startups). Now say your IXP is growing and you expand the subnet. No matter how much you harp on the customers to make the change, they don’t all do it at once. Someone announces the new, larger subnet in BGP. Now when anyone ARPs for IPs in the new part of the range, proxy ARP guy (still on the smaller subnet) says “hey I have a route for that, send it here”. That was fun to troubleshoot. :) Proper run IXPs pay engineers to hunt down people with Proxy ARP enabled on their peering interfaces. -- Niels. -- It's amazing what people will do to get their name on the internet, which is odd, because all you really need is a Blogspot account. -- roy edroso, alicublog.blogspot.com
Re: best practice for advertising peering fabric routes
* b...@herrin.us (William Herrin) [Wed 15 Jan 2014, 19:27 CET]: On Wed, Jan 15, 2014 at 12:54 PM, Niels Bakker niels=na...@bakker.net wrote: * na...@shankland.org (Jim Shankland) [Wed 15 Jan 2014, 18:04 CET]: So ... RFC1918 addresses for the IXP fabric, then? (Half kidding, but still ) They need to be globally unique. Actually, they don't. To meet the basic definition of working, they just have to be able to originate ICMP destination unreachable packets with a reasonable expectation that the recipient will receive those packets. Global uniqueness is not required for that. However, RFC1918 addresses don't meet the requirement for a different reason: they're routinely dropped at AS borders, thus don't have an expectation of reaching the external destination. They need to be globally unique because otherwise a connected network might be using them already internally, thus keeping them from connecting - or as another followup mail stated, force everything into their own VRFs, and that may still collide. This was rehashed a few years ago on the RIPE AP-WG mailing list, IIRC. -- Niels. -- It's amazing what people will do to get their name on the internet, which is odd, because all you really need is a Blogspot account. -- roy edroso, alicublog.blogspot.com
Proxy ARP detection (was re: best practice for advertising peering fabric routes)
On Jan 15, 2014, at 12:46 PM, Niels Bakker niels=na...@bakker.net wrote: * c...@bloomcounty.org (Clay Fiske) [Wed 15 Jan 2014, 20:34 CET]: Semi-related tangent: Working in an IXP setting I have seen weird corner cases cause issues in conjunction with the IXP subnet existing in BGP. Say someone’s got proxy ARP enabled on their router (sadly, more common than it should be, and not just from noobs at startups). Now say your IXP is growing and you expand the subnet. No matter how much you harp on the customers to make the change, they don’t all do it at once. Someone announces the new, larger subnet in BGP. Now when anyone ARPs for IPs in the new part of the range, proxy ARP guy (still on the smaller subnet) says “hey I have a route for that, send it here”. That was fun to troubleshoot. :) Proper run IXPs pay engineers to hunt down people with Proxy ARP enabled on their peering interfaces. Yes, yes, I expected a smug reply like this. I just didn’t expect it to take so long. But how can I detect proxy ARP when detecting proxy ARP was patented in 1996? http://www.google.com/patents/US5708654 Seriously though, it’s not so simple. You only get replies if the IP you ARP for is in the offender’s route table (or they have a default route). I’ve seen different routers respond depending on which non-local IP was ARPed for. And while using something like 8.8.8.8 might be an obvious choice, I don’t care to hose up everyone’s connectivity to it just to find local proxy ARP offenders on my network. -c
Re: Proxy ARP detection (was re: best practice for advertising peering fabric routes)
On 1/15/2014 6:31 PM, Clay Fiske wrote: Yes, yes, I expected a smug reply like this. I just didn’t expect it to take so long. But how can I detect proxy ARP when detecting proxy ARP was patented in 1996? http://www.google.com/patents/US5708654 Seriously though, it’s not so simple. You only get replies if the IP you ARP for is in the offender’s route table (or they have a default route). I’ve seen different routers respond depending on which non-local IP was ARPed for. And while using something like 8.8.8.8 might be an obvious choice, I don’t care to hose up everyone’s connectivity to it just to find local proxy ARP offenders on my network. -c Shouldn't ARP inspection be a common feature?
Re: Proxy ARP detection (was re: best practice for advertising peering fabric routes)
On Wed, Jan 15, 2014 at 10:49 PM, ML m...@kenweb.org wrote: Shouldn't ARP inspection be a common feature? Dynamic ARP inspection is mostly useful only when the trusted ports receive their MAC to IP address mapping from a trusted DHCP server, and the trusted mapping is established using DHCP snooping. Or else, you have a manually entered entries in the secure ARP database of MAC to IP mappings. Which most operators would be resistant to dealing with, because of all the extra work. -It's not as if the switches know what the valid subnets are and suppress ARP requests for outside networks. Therefore, in most cases; ARP inspection won't be used, except for DHCP clients. Arp inspection goes hand-in-hand with increasing resistance against a Man in the Middle attack from a compromised workstation on a LAN, using ARP hijacking to capture traffic or distribute malware to a neighboring workstation. In most cases, DHCP-based configuration will not be used for routers (the very devices that might inadvertently have proxy-arp) -- -JH
Re: best practice for advertising peering fabric routes
On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
Re: best practice for advertising peering fabric routes
On Tue, Jan 14, 2014 at 9:09 PM, Cb B cb.li...@gmail.com wrote: On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. good plan. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric 4. all peering-fabric routes get next-hop-self on your peering router before going into ibgp... all the rest of your network sees your local loopback as nexthop and things just work. Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
Re: best practice for advertising peering fabric routes
Pardon the top post, but I really don't have anything to comment below other than to agree with Chris and say rfc5963 is broken. NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. -- TTFN, patrick On Jan 14, 2014, at 21:22 , Christopher Morrow morrowc.li...@gmail.com wrote: On Tue, Jan 14, 2014 at 9:09 PM, Cb B cb.li...@gmail.com wrote: On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. good plan. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric 4. all peering-fabric routes get next-hop-self on your peering router before going into ibgp... all the rest of your network sees your local loopback as nexthop and things just work. Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
Re: best practice for advertising peering fabric routes
On Jan 14, 2014 7:13 PM, Patrick W. Gilmore patr...@ianai.net wrote: Pardon the top post, but I really don't have anything to comment below other than to agree with Chris and say rfc5963 is broken. NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. +1. Rfc5963 needs to update that guidance. Set next hop self loopback0 and done CB -- TTFN, patrick On Jan 14, 2014, at 21:22 , Christopher Morrow morrowc.li...@gmail.com wrote: On Tue, Jan 14, 2014 at 9:09 PM, Cb B cb.li...@gmail.com wrote: On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. good plan. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric 4. all peering-fabric routes get next-hop-self on your peering router before going into ibgp... all the rest of your network sees your local loopback as nexthop and things just work. Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
Re: best practice for advertising peering fabric routes
On Jan 14, 2014, at 7:55 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. There's a two part problem lurking. Problem #1 is how you handle your internal routing. Most of the big boys will next-hop-self in iBGP all external routes. However depending on the size and configuration of your network there may be advantages to not using next-hop-self, or just putting it in your IGP. Basically, you should be doing the same thing you do for a /30 from a peer or transit provider in your network. There is one thing special about an exchange point though, for security reasons you probably want to add it to your never accept routing filter from peers/customers/transit providers. You don't need someone injecting a couple of more specifics to mess with your routing. Problem #2 is your customers. If you have customers that may operate default free, and they use one of the traceroute tools that not only finds the route, but then continues to probe it (like MTR, or Visual Traceroute) there can be an issue. The initial traceroute probe may return an IP on the exchange of your peer's router, but then when they subsequently source ICMP Ping to that IP there will be no route in their network, and it will simply never respond. Some call this a feature, some call this a problem. There is also an extremely rare problem where the far end of the peering exchange steps down MTU, and thus PMTU discovery is invoked, but your customers use Unicast RPF. Since the exchange LAN isn't in their table, Unicast RPF may drop the PMTU packet-too-big message, causing a timeout. If your customers have a default to you, all is well. However if they have a default to someone else, and take a table from you to selectively override the same problem can occur for any routes they select through you that also traverse the exchange. IMHO the best fix for #2 is that the exchange have an ASN, and announce the exchange LAN from that ASN, typically via the route server. You should then peer with the route server to pick up that network. That makes the announcement consistent, and makes it clear who operates that network, and your customers can then access it. Many exchanges do not do this, and then the next best solution might be to originate it from your ASN and announce it to your customers only, with no-export set on the way out. Various people will no doubt chime in and tell you the last two suggestions are either excellent wonderful and the worst idea ever. Safe to say I know of networks doing both and the world has not ended. YMMV, some assembly required, batteries not included, actual conditions may affect product performance, do not taunt the happy fun ball, and consult a doctor if your network is up for more than four hours. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
On Jan 14, 2014, at 22:20 , Leo Bicknell bickn...@ufp.org wrote: On Jan 14, 2014, at 7:55 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. There's a two part problem lurking. Problem #1 is how you handle your internal routing. Most of the big boys will next-hop-self in iBGP all external routes. However depending on the size and configuration of your network there may be advantages to not using next-hop-self, or just putting it in your IGP. Basically, you should be doing the same thing you do for a /30 from a peer or transit provider in your network. There is one thing special about an exchange point though, for security reasons you probably want to add it to your never accept routing filter from peers/customers/transit providers. You don't need someone injecting a couple of more specifics to mess with your routing. Problem #2 is your customers. If you have customers that may operate default free, and they use one of the traceroute tools that not only finds the route, but then continues to probe it (like MTR, or Visual Traceroute) there can be an issue. The initial traceroute probe may return an IP on the exchange of your peer's router, but then when they subsequently source ICMP Ping to that IP there will be no route in their network, and it will simply never respond. Some call this a feature, some call this a problem. There is also an extremely rare problem where the far end of the peering exchange steps down MTU, and thus PMTU discovery is invoked, but your customers use Unicast RPF. Since the exchange LAN isn't in their table, Unicast RPF may drop the PMTU packet-too-big message, causing a timeout. If your customers have a default to you, all is well. However if they have a default to someone else, and take a table from you to selectively override the same problem can occur for any routes they select through you that also traverse the exchange. IMHO the best fix for #2 is that the exchange have an ASN, and announce the exchange LAN from that ASN, typically via the route server. You should then peer with the route server to pick up that network. That makes the announcement consistent, and makes it clear who operates that network, and your customers can then access it. Many exchanges do not do this, and then the next best solution might be to originate it from your ASN and announce it to your customers only, with no-export set on the way out. Various people will no doubt chime in and tell you the last two suggestions are either excellent wonderful and the worst idea ever. Safe to say I know of networks doing both and the world has not ended. YMMV, some assembly required, batteries not included, actual conditions may affect product performance, do not taunt the happy fun ball, and consult a doctor if your network is up for more than four hours. I've known Leo for .. well, let's just say a long time. And I have great respect for his networking abilities. But I fall into the second camp. As someone who owns operates an IXP, and is on the board of a couple more, and helped start even more, I'm going to stick to my guns here. As for knowing networks that do both, blah, blah, blah. I know lots of networks that allow spam, don't configure BCP38, have abusable name or NTP servers, etc. and the world has not come to an end. Doesn't mean you should. Lame excuse, Leo, and beneath you to even go there. NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. If for no better reason, how about because it is not your prefix, and chances are the IXP does not want you to use the prefix. In fact, I challenge you to find a major IXP route server which is announcing the IXP block. But because this is a teaching list, let's go through the problems Leo mentions. Anyone who steps down MTU on an IXP is far too broken to worry about your customer having RFP and not getting PMTU. Again, I challenge you to find someone doing this today, their network would be close to unusable. As for traceroute Seriously? You want to increase breakage on the Internet because it might cause 3 stars in a traceroute? Puh-LEEEZE. Sorry, neither of those pass the sniff test, IMHO. So Just Don't Do It. Setting next-hop-self is not just for big guys, the crappiest, tiniest router that can do peering at an IXP has the same ability. Use it. Stop putting me and every one of your peers in danger because you are lazy. -- TTFN, patrick signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
On Jan 14, 2014, at 9:35 PM, Patrick W. Gilmore patr...@ianai.net wrote: So Just Don't Do It. Setting next-hop-self is not just for big guys, the crappiest, tiniest router that can do peering at an IXP has the same ability. Use it. Stop putting me and every one of your peers in danger because you are lazy. I'm going to have to disagree here with Patrick, because this is security through obscurity, and that doesn't work well. For some history about why people like Patrick take the position he did, read: http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet Exchange points got attacked, so people yanked them from the routing table hoping to prevent attacks. If you're on this list it should take you all of about 3 seconds to realize the attackers could do a traceroute, and attack the IP one hop on the far side of the exchange for a few dozen providers and still cause all sorts of havoc, or do any of another half dozen things I won't mention to cause problems. The effect would be nearly, if not perfectly identical, since that traffic still has to cross the exchange. I'll point out the MTU step-down issue is real, and it's part of why we can't have 9K MTU exchanges be the default on the Internet, which would really make things better for a significant number of users. I think Patrick is a bit quick to dismiss some of the potential issues. Every link on every router is subject to attack. Exchange point LAN's really aren't special in that regard. If anything the only thing that makes them slightly special is that they may in fact be more oversubscribed than most links. Where a backbone might have a router with 20x10GE, so attackers could try and drive 190GE out a 10GE in theory; an exchange point may have 100 people with 20x10GE coming in. An alternate view that mega-exchange points are massively oversubscribed potential single points of failure, and perhaps network operators should consider that. While a DDOS taking an exchange down for half a day is bad, imagine if there was a more sinister attack, taking out the physical infrastructure of an exchange. That can't be fixed with a routing advertisement. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
On Jan 14, 2014, at 23:03 , Leo Bicknell bickn...@ufp.org wrote: On Jan 14, 2014, at 9:35 PM, Patrick W. Gilmore patr...@ianai.net wrote: So Just Don't Do It. Setting next-hop-self is not just for big guys, the crappiest, tiniest router that can do peering at an IXP has the same ability. Use it. Stop putting me and every one of your peers in danger because you are lazy. I'm going to have to disagree here with Patrick, because this is security through obscurity, and that doesn't work well. Leo, each of your points below is incorrect. I'm happy to discuss off-list if you'd like. For some history about why people like Patrick take the position he did, read: http://blog.cloudflare.com/the-ddos-that-almost-broke-the-internet Exchange points got attacked, so people yanked them from the routing table hoping to prevent attacks. If you're on this list it should take you all of about 3 seconds to realize the attackers could do a traceroute, and attack the IP one hop on the far side of the exchange for a few dozen providers and still cause all sorts of havoc, or do any of another half dozen things I won't mention to cause problems. The effect would be nearly, if not perfectly identical, since that traffic still has to cross the exchange. Let's take just the incident mentioned in the blog post above (which is pretty broken itself, but hey, who said the CEO of CDN had to know anything about networking... ? :). To where would the attacker traceroute -to-? Somewhere inside Cloudflare? Other LINX members? Remember, most of the attack was sourced from networks which were not attached to the LINX. If the source network or the source network's upstreams are not LINX members, there is probably _no_ path that goes through LINX. Even if they are members, lots of networks have alternative paths (other IXPs, private interconnections, etc.). For instance, sources in Germany may well flow over DE-CIX even if there is a peering session at LINX. Etc. There is no single or set of IP addresses that will guarantee even a majority of packets traverse a specific IXP except the IXP LAN. Also, the attack was reflected DNS, so the attacker couldn't actually perform the traceroutes you suggest from each source as he did not control the sources. He _might_ be able to find _some_ of the paths with a lot of sleuthing through route traceroute servers, but that would make things massively more difficult, as well as massively cut the number of servers he can abuse to the same effect. Both of which are huge wins for the good guys. Pulling the IXP prefix has a enormous benefits and essentially no downside. I know literally hundreds of ISPs large small who do not carry the IXP prefix, and none have seen any significant issues (most have seen zero, a few get asked about 3 stars, but as I said before, puh-lze). I'm a bit surprised you even tried to bring this up. I know you well enough to know you would have realized all of the above if you had though about it for a while (or just asked). I'll point out the MTU step-down issue is real, and it's part of why we can't have 9K MTU exchanges be the default on the Internet, which would really make things better for a significant number of users. I think Patrick is a bit quick to dismiss some of the potential issues. MTU step-down is a real issue, and it's real enough whether IXP LANs are in the DFZ or not. Let's solve the overarching problem before doing something which has real, proven harm and leaves the root cause in place. Besides, the two VLAN method already exists in multiple places and it hasn't helped adoption of 9K packets. Unless you are talking about letting some people attach with 1500 MTU and others with 9000 MTU? 'Cause if that's what you meant, then I'm just going to call you loony and ask what you're smoking. Every link on every router is subject to attack. Exchange point LAN's really aren't special in that regard. If anything the only thing that makes them slightly special is that they may in fact be more oversubscribed than most links. Where a backbone might have a router with 20x10GE, so attackers could try and drive 190GE out a 10GE in theory; an exchange point may have 100 people with 20x10GE coming in. An alternate view that mega-exchange points are massively oversubscribed potential single points of failure, and perhaps network operators should consider that. While a DDOS taking an exchange down for half a day is bad, imagine if there was a more sinister attack, taking out the physical infrastructure of an exchange. That can't be fixed with a routing advertisement. IXPs are more special because they are shared. Other links are between you one other network not hundreds of other networks, some of whom have no relationship with you. If you don't like the rules of IXPs, don't join one. But hooking up to one and deciding I'm going to carry this prefix even when told not
Re: best practice for advertising peering fabric routes
On Jan 15, 2014, at 11:41 AM, Patrick W. Gilmore patr...@ianai.net wrote: I repeat: NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device except those directly attached to that LAN. Period. +1 Again, folks, this isn't theoretical. When the particular attacks cited in this thread were taking place, I was astonished that the IXP infrastructure routes were even being advertised outside of the IXP network, because of these very issues. IXPs are not the problem when it comes to breaking PMTU-D. The problem is largely with enterprise networks, and with 'security' vendors who've propagated the myth that simply blocking all ICMP somehow increases 'security'. --- Roland Dobbins rdobb...@arbor.net // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton signature.asc Description: Message signed with OpenPGP using GPGMail
Re: best practice for advertising peering fabric routes
Thank you - I will heed the warning. I want to be a good community member and make sure we're maintaining the agreed-upon practices (I'll re-read/review my agreement with the IXP) So if that is the case, I have to rely on the peering fabric to just return traffic, since the rest of my network (save the directly connected router) will not know about those routes outbound? And what about my customers who are counting on me routing their office traffic through my network into the peering fabric to their properties? (I have one specifically who is eventually looking for that capability) Do I have to provide them some sort of VPN to make that happen across my network to the peering fabric router? From: Patrick W. Gilmore patr...@ianai.net To: NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 7:11 PM Subject: Re: best practice for advertising peering fabric routes Pardon the top post, but I really don't have anything to comment below other than to agree with Chris and say rfc5963 is broken. NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. -- TTFN, patrick On Jan 14, 2014, at 21:22 , Christopher Morrow morrowc.li...@gmail.com wrote: On Tue, Jan 14, 2014 at 9:09 PM, Cb B cb.li...@gmail.com wrote: On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. good plan. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric 4. all peering-fabric routes get next-hop-self on your peering router before going into ibgp... all the rest of your network sees your local loopback as nexthop and things just work. Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
Re: best practice for advertising peering fabric routes
Never mind, I just carefully re-read the point. Right, I'll filter the prefix(es) of the IXP LAN(s) that I'm connected to and not let THAT get out, no reason to advertise it since no traffic ever goes to it. That still has me asking to how best to advertise the rest of the public prefixes coming from the other fabric members. From: Eric A Louie elo...@yahoo.com To: Patrick W. Gilmore patr...@ianai.net; NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 10:22 PM Subject: Re: best practice for advertising peering fabric routes Thank you - I will heed the warning. I want to be a good community member and make sure we're maintaining the agreed-upon practices (I'll re-read/review my agreement with the IXP) So if that is the case, I have to rely on the peering fabric to just return traffic, since the rest of my network (save the directly connected router) will not know about those routes outbound? And what about my customers who are counting on me routing their office traffic through my network into the peering fabric to their properties? (I have one specifically who is eventually looking for that capability) Do I have to provide them some sort of VPN to make that happen across my network to the peering fabric router? From: Patrick W. Gilmore patr...@ianai.net To: NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 7:11 PM Subject: Re: best practice for advertising peering fabric routes Pardon the top post, but I really don't have anything to comment below other than to agree with Chris and say rfc5963 is broken. NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. -- TTFN, patrick On Jan 14, 2014, at 21:22 , Christopher Morrow morrowc.li...@gmail.com wrote: On Tue, Jan 14, 2014 at 9:09 PM, Cb B cb.li...@gmail.com wrote: On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. good plan. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric 4. all peering-fabric routes get next-hop-self on your peering router before going into ibgp... all the rest of your network sees your local loopback as nexthop and things just work. Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
Re: best practice for advertising peering fabric routes
On Wed, Jan 15, 2014 at 1:22 AM, Eric A Louie elo...@yahoo.com wrote: Thank you - I will heed the warning. I want to be a good community member and make sure we're maintaining the agreed-upon practices (I'll re-read/review my agreement with the IXP) So if that is the case, I have to rely on the peering fabric to just return traffic, since the rest of my network (save the directly connected router) will not know about those routes outbound? And what about my customers who are counting on me routing their office traffic through my network into the peering fabric to their properties? (I have one specifically who is eventually looking for that capability) Do I have to provide them some sort of VPN to make that happen across my network to the peering fabric router? perhaps I'm confused, but you have sort of this situation: ixp-participants - ixp - your-router - your-network - your-customer you get routes for ixp-participants from 'ixp' you send to the 'ixp' (and on to 'ixp-participants') routes for 'your-customer' and 'your-network' right? then so long as you send 'your-customer' the routes you learn from 'ixp' (which you set 'next-hop-self' on in ibgp from 'your-router' to 'your-network' (in the ibgp-mesh that you will setup) ... everything just works. All routers behind 'your-router' in 'your-netowrk' see 'ixp-participants' with a next-hop of 'your-router' who still knows 'send to ixp!' for the route(s) in question. From: Patrick W. Gilmore patr...@ianai.net To: NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 7:11 PM Subject: Re: best practice for advertising peering fabric routes Pardon the top post, but I really don't have anything to comment below other than to agree with Chris and say rfc5963 is broken. NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. -- TTFN, patrick On Jan 14, 2014, at 21:22 , Christopher Morrow morrowc.li...@gmail.com wrote: On Tue, Jan 14, 2014 at 9:09 PM, Cb B cb.li...@gmail.com wrote: On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. good plan. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric 4. all peering-fabric routes get next-hop-self on your peering router before going into ibgp... all the rest of your network sees your local loopback as nexthop and things just work. Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
Re: best practice for advertising peering fabric routes
On Wed, Jan 15, 2014 at 1:36 AM, Eric A Louie elo...@yahoo.com wrote: Never mind, I just carefully re-read the point. Right, I'll filter the prefix(es) of the IXP LAN(s) that I'm connected to and not let THAT get out, no reason to advertise it since no traffic ever goes to it. That still has me asking to how best to advertise the rest of the public prefixes coming from the other fabric members. on your ibgp peers on 'your-router' you'd have something like: match community community-added-for-all-ixp-participant-routes set next-hop-self http://www.cisco.com/en/US/tech/tk365/technologies_q_and_a_item09186a00800949e8.shtml#eleven for one vendors view of the situation... and there is a link to: http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800c95bb.shtml that's worth a read. http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a00800c95bb.shtml From: Eric A Louie elo...@yahoo.com To: Patrick W. Gilmore patr...@ianai.net; NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 10:22 PM Subject: Re: best practice for advertising peering fabric routes Thank you - I will heed the warning. I want to be a good community member and make sure we're maintaining the agreed-upon practices (I'll re-read/review my agreement with the IXP) So if that is the case, I have to rely on the peering fabric to just return traffic, since the rest of my network (save the directly connected router) will not know about those routes outbound? And what about my customers who are counting on me routing their office traffic through my network into the peering fabric to their properties? (I have one specifically who is eventually looking for that capability) Do I have to provide them some sort of VPN to make that happen across my network to the peering fabric router? From: Patrick W. Gilmore patr...@ianai.net To: NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 7:11 PM Subject: Re: best practice for advertising peering fabric routes Pardon the top post, but I really don't have anything to comment below other than to agree with Chris and say rfc5963 is broken. NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. -- TTFN, patrick On Jan 14, 2014, at 21:22 , Christopher Morrow morrowc.li...@gmail.com wrote: On Tue, Jan 14, 2014 at 9:09 PM, Cb B cb.li...@gmail.com wrote: On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. good plan. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric 4. all peering-fabric routes get next-hop-self on your peering router before going into ibgp... all the rest of your network sees your local loopback as nexthop and things just work. Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
Re: best practice for advertising peering fabric routes
Ok, so the right way to do it is in iBGP. That pretty much answers the question - don't redistribute those ixp-participant prefixes into my IGP. I have a lot of iBGP homework to do, to make it work with the 5 POPs that are all taking full route feeds. I tried once and couldn't get the BGP tables working correctly with a full mesh of the 5 routers, so it looks like time to try it again, this time with a route reflector. From: Christopher Morrow morrowc.li...@gmail.com To: Eric A Louie elo...@yahoo.com Cc: Patrick W. Gilmore patr...@ianai.net; NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 10:37 PM Subject: Re: best practice for advertising peering fabric routes On Wed, Jan 15, 2014 at 1:22 AM, Eric A Louie elo...@yahoo.com wrote: Thank you - I will heed the warning. I want to be a good community member and make sure we're maintaining the agreed-upon practices (I'll re-read/review my agreement with the IXP) So if that is the case, I have to rely on the peering fabric to just return traffic, since the rest of my network (save the directly connected router) will not know about those routes outbound? And what about my customers who are counting on me routing their office traffic through my network into the peering fabric to their properties? (I have one specifically who is eventually looking for that capability) Do I have to provide them some sort of VPN to make that happen across my network to the peering fabric router? perhaps I'm confused, but you have sort of this situation: ixp-participants - ixp - your-router - your-network - your-customer you get routes for ixp-participants from 'ixp' you send to the 'ixp' (and on to 'ixp-participants') routes for 'your-customer' and 'your-network' right? then so long as you send 'your-customer' the routes you learn from 'ixp' (which you set 'next-hop-self' on in ibgp from 'your-router' to 'your-network' (in the ibgp-mesh that you will setup) ... everything just works. All routers behind 'your-router' in 'your-netowrk' see 'ixp-participants' with a next-hop of 'your-router' who still knows 'send to ixp!' for the route(s) in question. From: Patrick W. Gilmore patr...@ianai.net To: NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 7:11 PM Subject: Re: best practice for advertising peering fabric routes Pardon the top post, but I really don't have anything to comment below other than to agree with Chris and say rfc5963 is broken. NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. -- TTFN, patrick On Jan 14, 2014, at 21:22 , Christopher Morrow morrowc.li...@gmail.com wrote: On Tue, Jan 14, 2014 at 9:09 PM, Cb B cb.li...@gmail.com wrote: On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. good plan. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric 4. all peering-fabric routes get next-hop-self on your peering router before going into ibgp... all the rest of your network sees your local loopback as nexthop and things just work. Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
Re: best practice for advertising peering fabric routes
Le 15/01/2014 07:59, Eric A Louie a écrit : Ok, so the right way to do it is in iBGP. That pretty much answers the question - don't redistribute those ixp-participant prefixes into my IGP. Yes, using next-hop self (rather than importing IXP routes) as pointed out earlier in this thread. I have a lot of iBGP homework to do, to make it work with the 5 POPs that are all taking full route feeds. I tried once and couldn't get the BGP tables working correctly with a full mesh of the 5 routers, so it looks like time to try it again, this time with a route reflector. I don't think you need route-reflection in a 5 node iBGP. What do you mean by couldn't get the BGP tables working correctly? Cheers, mh From: Christopher Morrow morrowc.li...@gmail.com To: Eric A Louie elo...@yahoo.com Cc: Patrick W. Gilmore patr...@ianai.net; NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 10:37 PM Subject: Re: best practice for advertising peering fabric routes On Wed, Jan 15, 2014 at 1:22 AM, Eric A Louie elo...@yahoo.com wrote: Thank you - I will heed the warning. I want to be a good community member and make sure we're maintaining the agreed-upon practices (I'll re-read/review my agreement with the IXP) So if that is the case, I have to rely on the peering fabric to just return traffic, since the rest of my network (save the directly connected router) will not know about those routes outbound? And what about my customers who are counting on me routing their office traffic through my network into the peering fabric to their properties? (I have one specifically who is eventually looking for that capability) Do I have to provide them some sort of VPN to make that happen across my network to the peering fabric router? perhaps I'm confused, but you have sort of this situation: ixp-participants - ixp - your-router - your-network - your-customer you get routes for ixp-participants from 'ixp' you send to the 'ixp' (and on to 'ixp-participants') routes for 'your-customer' and 'your-network' right? then so long as you send 'your-customer' the routes you learn from 'ixp' (which you set 'next-hop-self' on in ibgp from 'your-router' to 'your-network' (in the ibgp-mesh that you will setup) ... everything just works. All routers behind 'your-router' in 'your-netowrk' see 'ixp-participants' with a next-hop of 'your-router' who still knows 'send to ixp!' for the route(s) in question. From: Patrick W. Gilmore patr...@ianai.net To: NANOG list nanog@nanog.org Sent: Tuesday, January 14, 2014 7:11 PM Subject: Re: best practice for advertising peering fabric routes Pardon the top post, but I really don't have anything to comment below other than to agree with Chris and say rfc5963 is broken. NEVER EVER EVER put an IX prefix into BGP, IGP, or even static route. An IXP LAN should not be reachable from any device not directly attached to that LAN. Period. Doing so endangers your peers the IX itself. It is on the order of not implementing BCP38, except no one has the (lame, ridiculous, idiotic, and pure cost-shifting BS) excuse that they can't do this. -- TTFN, patrick On Jan 14, 2014, at 21:22 , Christopher Morrow morrowc.li...@gmail.com wrote: On Tue, Jan 14, 2014 at 9:09 PM, Cb B cb.li...@gmail.com wrote: On Jan 14, 2014 6:01 PM, Eric A Louie elo...@yahoo.com wrote: I have a connection to a peering fabric and I'm not distributing the peering fabric routes into my network. good plan. I see three options 1. redistribute into my igp (OSPF) 2. configure ibgp and route them within that infrastructure. All the default routes go out through the POPs so iBGP would see packets destined for the peering fabric and route it that-a-way 3. leave it as is, and let the outbound traffic go out my upstreams and the inbound traffic come back through the peering fabric 4. all peering-fabric routes get next-hop-self on your peering router before going into ibgp... all the rest of your network sees your local loopback as nexthop and things just work. Advantages and disadvantages, pros and cons? Recommendations? Experiences, good and bad? I have 5 POPs, 2 OSPF areas, and have not brought iBGP up between the POPs yet. That's another issue completely from a planning perspective. thanks Eric http://tools.ietf.org/html/rfc5963 I like no-export
