Re: Multi-homing with multiple ASNs
On 11/24/14 8:58 AM, Dave Crocker wrote: On 11/23/2014 11:20 AM, joel jaeggli wrote: Their grasp of load-balancing seems a bit shallow also. Are there discussion/guidance papers that one can point to, to improve the depth of understanding, or at least get better configuration choices? (Those are independent points of improvement...) Bassim Halabi's book is getting a bit long in the tooth, but it was my jumping-off point for my own forays into this space. http://www.amazon.com/Internet-Routing-Architectures-2nd-Halabi/dp/157870233X/ref=sr_1_1?ie=UTF8qid=1417390786sr=8-1keywords=halabi+routing The nanog tutorials have been assiduous about updating the bgp materials https://www.nanog.org/resources/tutorials So there are several iterations of the practical materials. joel d/ signature.asc Description: OpenPGP digital signature
RE: Multi-homing with multiple ASNs
Thanks to everyone for your input on our less than desirable BGP situation. I do want to make sure I add that the state network we are a part of serves everything from elementary schools, to universities. to the traffic cameras on the interstate.Many of these are in rural locations and in the past each state entity had created their own network including two separate state university networks.The state vendor managed network was created to save money and provide higher level services than just an ISP. Among other things it serves as the private WAN for some state agencies.As our internet redundancy and bandwidth demands have increased we have outgrown the need for the high touch services offered by the state network but we must participate in order to maintain WAN access to other state universities. Thanks again for the feedback. Curtis Curtis Parish Senior Network Engineer Middle Tennessee State University -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of joel jaeggli Sent: Sunday, November 23, 2014 1:21 PM To: mark.ti...@seacom.mu; nanog@nanog.org Subject: Re: Multi-homing with multiple ASNs On 11/21/14 1:07 AM, Mark Tinka wrote: On Friday, November 21, 2014 12:00:47 AM Curtis L. Parish wrote: We have recently added a second ISP (third if you count I2). Our first ISP is actually a private state network that peers with two Tier 1 providers. We own an AS number and our IP space but at the last minute learned our state network is advertising our network using two different ASNs (neither ours) so they can load balance their connections.If you hit the right looking glass server you can see our network advertised by three different ASNs.We were told by the new ISP that this is a problem but the state network says it is not. Looking for opinions and words of wisdom on this split advertising issue. Why aren't you originating your own prefixes and ASN by yourselves, since you own both? The practical problem here is that the control of prefix origination is distributed. so if there is a need to withdraw it from the state network or advertise it no export for some reason (e.g. performance problem maintenance etc) you likely can't. Their grasp of load-balancing seems a bit shallow also. Mark.
Re: Multi-homing with multiple ASNs
On 11/23/2014 11:20 AM, joel jaeggli wrote: Their grasp of load-balancing seems a bit shallow also. Are there discussion/guidance papers that one can point to, to improve the depth of understanding, or at least get better configuration choices? (Those are independent points of improvement...) d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net
Re: Multi-homing with multiple ASNs
On 11/21/14 1:07 AM, Mark Tinka wrote: On Friday, November 21, 2014 12:00:47 AM Curtis L. Parish wrote: We have recently added a second ISP (third if you count I2). Our first ISP is actually a private state network that peers with two Tier 1 providers. We own an AS number and our IP space but at the last minute learned our state network is advertising our network using two different ASNs (neither ours) so they can load balance their connections.If you hit the right looking glass server you can see our network advertised by three different ASNs.We were told by the new ISP that this is a problem but the state network says it is not. Looking for opinions and words of wisdom on this split advertising issue. Why aren't you originating your own prefixes and ASN by yourselves, since you own both? The practical problem here is that the control of prefix origination is distributed. so if there is a need to withdraw it from the state network or advertise it no export for some reason (e.g. performance problem maintenance etc) you likely can't. Their grasp of load-balancing seems a bit shallow also. Mark. signature.asc Description: OpenPGP digital signature
Re: Multi-homing with multiple ASNs
On Fri, Nov 21, 2014 at 9:49 AM, Curtis L. Parish curtis.par...@mtsu.edu wrote: We advertise our ASN into the state network with more specific routes that we advertise via ISP2 via our ASN.This is done because the state (vendor managed) network runs stateful firewalls and we have to force other multi-home entities on the state network to use our state connection instead of ISP2. Our network has been removed from the state firewall due to previous problems with asymmetric routing with our I2 circuit. Hi Curtis, As you've already noted, the presence of a stateful firewall beyond your BGP border is inimical to BGP multihoming. Traffic between two multihomed networks must never cross a stateful firewall that is outside both networks' borders. Practically speaking, there will asymmetry, path flapping, per-packet load balancing and other quirks at locations outside your control. The Internet DFZ is a chaotic system. Over time you won't be able to make the packets reliably transit the firewall. It sounds like this is a learning experience for both you and the folks at the state network. If you have a friendly relationship with them, now would be a good time to visit and talk about what are likely to be significant changes to their network architecture to make multihomed users feasible. Preferably with a the help of a local consultant who has BGP expertise. If that doesn't sound like it would be a productive conversation then I suggest you consider three different options: 1. Return to the state network alone, 2. Replace your state network connection with another commercial ISP, 3. Add an additional commercial ISP for the sake of your Internet access needs, drop the BGP advertisements with the state network and then implement resources which should only transit the state network using IP addresses assigned by the state network rather than your BGP addresses. Here is a question. I know that having one network advertised by multiple ASNs is unconventional and thus it will probably be harder to get help troubleshooting routing problems when they arise.Do you see a situation where our network might be caught in a loop or black hole due to asymmetric routing and conflicting advertisements? Yes. And frequently. You have this thing balanced on the head of a pin. Regards, Bill Herrin -- William Herrin her...@dirtside.com b...@herrin.us Owner, Dirtside Systems . Web: http://www.dirtside.com/ May I solve your unusual networking challenges?
Re: Multi-homing with multiple ASNs
On Fri, Nov 21, 2014 at 8:49 AM, Curtis L. Parish curtis.par...@mtsu.edu wrote: I believe the state will modify their advertisements to add our ASN to the path but changes to advertising via the state network has to go through a design and change management process and then be scheduled into maintenance windows.Any attempts to balance the traffic via prepending will take weeks. [snip] In other words, you are in effect not in control of the advertisement of your prefix, therefore you practically don't actually have an autonomous system, you have the number technically, but not the administrative division that is intended to exist. An appropriate amount of time to push out any change needed to an announcement should be no more than 1 business day, but less than 2 hours in an emergency, to add extra impending or pull an announcement. I would call a change management process that requires any longer unacceptable, or not reflecting the reality of the importance of well-maintained optimal properly functioning network connectivity. You have what seems to be something very fragile, and you have very low configuration agility, since you cannot change your announcements as needed out through the state as you need them to. A stateful firewall, has no correct place outside the border of a multihomed network; by definition, to have a stateful firewall, there must be a single point of failure (on the stateful firewall element) at least for each unique load-balancing tuple. So I would call (in this case), the origination of your prefix by multiple ASes a bad thing. The protocol allows this, but the other constraints related to the situation are serious impediments that make the solidity multihoming seem improper or potentially precarious, in terms of the true originating AS' ability to function as an AS and manage their network -- -JH
Re: Multi-homing with multiple ASNs
Agreed. You could still recieve their routes and no/export your as but I wouldn't go beyond the firewall. Jason Bothe, Manager of Networking Rice University o +1 713 348 5500 m +1 713 703 3552 ja...@rice.edu On Nov 23, 2014, at 17:57, William Herrin b...@herrin.us wrote: On Fri, Nov 21, 2014 at 9:49 AM, Curtis L. Parish curtis.par...@mtsu.edu wrote: We advertise our ASN into the state network with more specific routes that we advertise via ISP2 via our ASN.This is done because the state (vendor managed) network runs stateful firewalls and we have to force other multi-home entities on the state network to use our state connection instead of ISP2. Our network has been removed from the state firewall due to previous problems with asymmetric routing with our I2 circuit. Hi Curtis, As you've already noted, the presence of a stateful firewall beyond your BGP border is inimical to BGP multihoming. Traffic between two multihomed networks must never cross a stateful firewall that is outside both networks' borders. Practically speaking, there will asymmetry, path flapping, per-packet load balancing and other quirks at locations outside your control. The Internet DFZ is a chaotic system. Over time you won't be able to make the packets reliably transit the firewall. It sounds like this is a learning experience for both you and the folks at the state network. If you have a friendly relationship with them, now would be a good time to visit and talk about what are likely to be significant changes to their network architecture to make multihomed users feasible. Preferably with a the help of a local consultant who has BGP expertise. If that doesn't sound like it would be a productive conversation then I suggest you consider three different options: 1. Return to the state network alone, 2. Replace your state network connection with another commercial ISP, 3. Add an additional commercial ISP for the sake of your Internet access needs, drop the BGP advertisements with the state network and then implement resources which should only transit the state network using IP addresses assigned by the state network rather than your BGP addresses. Here is a question. I know that having one network advertised by multiple ASNs is unconventional and thus it will probably be harder to get help troubleshooting routing problems when they arise.Do you see a situation where our network might be caught in a loop or black hole due to asymmetric routing and conflicting advertisements? Yes. And frequently. You have this thing balanced on the head of a pin. Regards, Bill Herrin -- William Herrin her...@dirtside.com b...@herrin.us Owner, Dirtside Systems . Web: http://www.dirtside.com/ May I solve your unusual networking challenges?
Re: Multi-homing with multiple ASNs
On Friday, November 21, 2014 12:00:47 AM Curtis L. Parish wrote: We have recently added a second ISP (third if you count I2). Our first ISP is actually a private state network that peers with two Tier 1 providers. We own an AS number and our IP space but at the last minute learned our state network is advertising our network using two different ASNs (neither ours) so they can load balance their connections.If you hit the right looking glass server you can see our network advertised by three different ASNs.We were told by the new ISP that this is a problem but the state network says it is not. Looking for opinions and words of wisdom on this split advertising issue. Why aren't you originating your own prefixes and ASN by yourselves, since you own both? Mark. signature.asc Description: This is a digitally signed message part.
Re: Multi-homing with multiple ASNs
On Fri, 21 Nov 2014 11:07:49 +0200, Mark Tinka mark.ti...@seacom.mu said: We own an AS number and our IP space but at the last minute learned our state network is advertising our network using two different ASNs (neither ours) This will work, as in the BGP path selection algorithm will work as designed in this situation. But it also means that the routing policy is out of your control which is kind of the point of having an ASN! It also makes it harder to track down who is operationally responsible for that address space since it appears to the outside world to be in two (or three! different places). I'd say don't do this unless you really have no choice. Why aren't you originating your own prefixes and ASN by yourselves, since you own both? Good question. We (AS60241) almost ended up doing similarly for a while. Because of a close association with the universities in Scotland, we discussed the possibility of transit via JANET. This turned out to be difficult because they run a whole bunch of private ASNs internally -- unlike in North America where universities typically have their own real one. So it would have been us - private stuff - AS786 and for some reason that I forget they were unable to remove private ASNs from the path. The best that might have been possible would be to have had them announce our networks with synchronisation on, which would have meant the outside world would have seen them originating in both AS786 and AS60241. Icky. We (mutually) decided against this. Just to say that there are strange, but not completely unreasonable circumstances in which this can happen... -w pgpohqpKewJg4.pgp Description: PGP signature
Re: Multi-homing with multiple ASNs
On Thu, Nov 20, 2014 at 5:00 PM, Curtis L. Parish curtis.par...@mtsu.edu wrote: We have recently added a second ISP (third if you count I2). Our first ISP is actually a private state network that peers with two Tier 1 providers. We own an AS number and our IP space but at the last minute learned our state network is advertising our network using two different ASNs (neither ours) so they can load balance their connections.If you hit the right looking glass server you can see our network advertised by three different ASNs.We were told by the new ISP that this is a problem but the state network says it is not. Howdy, If you drop your connection to the state network, do the routes with their AS numbers drop out of the looking glasses? If not, then there's a problem. If you depreference your connection to the state network by prepending your AS number, do comparable prepends appear at the looking glasses or does the state network continue to give its advertisement of your address space top billing? If the state network's behavior strips your ability to load balance your network then there's a problem. Conventionally, the state network should be adding its AS number after yours, not stripping your AS number. More often than not, this convention is also the technically correct course of action. Regards, Bill Herrin -- William Herrin her...@dirtside.com b...@herrin.us Owner, Dirtside Systems . Web: http://www.dirtside.com/ May I solve your unusual networking challenges?
RE: Multi-homing with multiple ASNs
Thanks for all the responses. I will answer a few questions that have come on and off list. (Sorry for length) We advertise our ASN into the state network with more specific routes that we advertise via ISP2 via our ASN.This is done because the state (vendor managed) network runs stateful firewalls and we have to force other multi-home entities on the state network to use our state connection instead of ISP2. Our network has been removed from the state firewall due to previous problems with asymmetric routing with our I2 circuit.I am told the state network does drop our network from their advertisements when our network is unreachable. That has not been explained or tested. What we did not realize until about a week before turning up ISP2 was the state was consolidating all state networks to use two of the vendor’s ASNs when it peers with their two ISPs. Our ASN is not part of the path.We had no choice but to turn up ISP2 due to bandwidth reasons. Miraculously we achieved almost a 50/50 balance of traffic.Bandwidth will be increased on ISP2 as demand grows so we will need the ability to prepend on the state network to make ISP2 look more desirable. I believe the state will modify their advertisements to add our ASN to the path but changes to advertising via the state network has to go through a design and change management process and then be scheduled into maintenance windows. Any attempts to balance the traffic via prepending will take weeks.As long as the traffic stays balanced we are OK.When replaying BGP route changes I normally see our network only advertised out one of state ASNs but occasionally I see it with two so traffic balance may be impacted depending on which ISP the state is egressing. Here is a question. I know that having one network advertised by multiple ASNs is unconventional and thus it will probably be harder to get help troubleshooting routing problems when they arise.Do you see a situation where our network might be caught in a loop or black hole due to asymmetric routing and conflicting advertisements? Thanks again. New to the list but have already learned much by reading the archives. Curtis Curtis Parish Senior Network Engineer Middle Tennessee State University Subject: Re: Multi-homing with multiple ASNs Howdy, If you drop your connection to the state network, do the routes with their AS numbers drop out of the looking glasses? If not, then there's a problem. If you depreference your connection to the state network by prepending your AS number, do comparable prepends appear at the looking glasses or does the state network continue to give its advertisement of your address space top billing? If the state network's behavior strips your ability to load balance your network then there's a problem. Conventionally, the state network should be adding its AS number after yours, not stripping your AS number. More often than not, this convention is also the technically correct course of action.
Multi-homing with multiple ASNs
Greetings, We have recently added a second ISP (third if you count I2). Our first ISP is actually a private state network that peers with two Tier 1 providers. We own an AS number and our IP space but at the last minute learned our state network is advertising our network using two different ASNs (neither ours) so they can load balance their connections.If you hit the right looking glass server you can see our network advertised by three different ASNs.We were told by the new ISP that this is a problem but the state network says it is not. Looking for opinions and words of wisdom on this split advertising issue. Thanks curtis Curtis Parish Senior Network Engineer Middle Tennessee State University
