Re: Anycast but for egress
On 7/27/21 10:54, Vimal wrote: > (Unsure if this is the right forum to ask this question, but here goes:) > > From what I understand, IP Anycast can be used to steer traffic into a > server that's close to the client. > > I am curious if anyone here has/encountered a setup where they use > anycast IP on their gateways... to have a predictable egress IP for > their traffic, regardless of where they are located? Stateless outbound load-balancing setups exist. Example you have two or more nat44 / nat64 / cgnat boxes behind a common ecmp path with the same destination IP(s). this is normally so that you have more boxes that accumulate state rather than being bound to a single one. > > For example, a search engine crawler could in principle have the same > IP advertised all over the world, but it looks like they don't... I > wonder why? So this is a somewhat different problem... There's no assurance that when you initiate a connection that the return path will return to the same instance of your anycast announcement when the server on the other side replies. Effectively the initiating party needs a unicast address or you need some out of band path to get an errant packet back to the correct host. > -- > Vimal >
Re: Anycast but for egress
On Thu, Jul 29, 2021 at 4:58 PM Joe Maimon wrote: > > > Vimal wrote: > > (Unsure if this is the right forum to ask this question, but here goes:) > > > > From what I understand, IP Anycast can be used to steer traffic into a > > server that's close to the client. > > > > I am curious if anyone here has/encountered a setup where they use > > anycast IP on their gateways... to have a predictable egress IP for > > their traffic, regardless of where they are located? > > > > For example, a search engine crawler could in principle have the same > > IP advertised all over the world, but it looks like they don't... I > > wonder why? > > > > -- > > Vimal > > > Its definitely possible, but would need a layer of software (kernel > mode) on all the anycast holders synchronizing state to ensure > asymmetric replies/connections get forwarded/shifted to the correct host. > > is it actually that hard? isn't it more like: "use an outbound path local to that inbound path cone which NAT's (or proxy's or...) to a small set of staticlly assigned addresses" Provided you don't re-use the outbound addresses on different deployments this should 'just work'[tm] 'anycast but outbound' is really: "get me local nat pools for my service by locality" I think this is, bascially, what every enterprise network in the world does, effectively. If the goals are worth that kind of effort is another question. And > performance is likely to be "tricky". > >
Re: Anycast but for egress
Vimal wrote: (Unsure if this is the right forum to ask this question, but here goes:) From what I understand, IP Anycast can be used to steer traffic into a server that's close to the client. I am curious if anyone here has/encountered a setup where they use anycast IP on their gateways... to have a predictable egress IP for their traffic, regardless of where they are located? For example, a search engine crawler could in principle have the same IP advertised all over the world, but it looks like they don't... I wonder why? -- Vimal Its definitely possible, but would need a layer of software (kernel mode) on all the anycast holders synchronizing state to ensure asymmetric replies/connections get forwarded/shifted to the correct host. If the goals are worth that kind of effort is another question. And performance is likely to be "tricky".
Re: Anycast but for egress
Great point. We don't need geo-diversity for websites with the IP address issue, so we could design for that case specially on a one-off basis. For throughput it shouldn't be an issue where we're located, but we often find websites serving different content based on the source IP of the traffic. So, having a presence closer to the user is useful. But then again, this is a different concern that's orthogonal to the original question, because geo-ip doesn't make much sense with an anycast IP. For those websites that need a stable IP for NACLs *and* serve different content based on source IP, we have to use the predictable 3-5 IPs per site suggestion of yours. On Wed, Jul 28, 2021 at 11:27 AM Glenn McGurrin via NANOG wrote: > I'd had a similar thought/question, though keeping the geo diversity, > you manage the crawlers, and are making contact individually with these > sites from what you have stated (and so don't need a one size fit's all > list for public posting), so why not have a restricted subset of the > crawlers handle sites with these issues (which subset may be unique per > site, which makes maintaining even load balancing not overly complex > /limiting, especially as you are using nat anyway, so multiple servers > can be behind each ip and that number can vary). That let's you have > geo diversity (or even multi cloud diversity) for every site, but each > site that needs this IP whitelisting only needs 3-5 IP's at any site, > but yet you can distribute load over a much larger overall set of > machines and nat gateways. > > As I understand it even CDN's that anycast TCP (externally or internally > [load balancing via routers and multi path]) do similar by spreading > load over multiple IP's at the DNS layer first. > > As the transition to IPv6 happens you may have it easier as getting a > large enough allocation to allow for splitting it out into multiple > subnets advertised from different locations without providers dropping > the route as too long a prefix is much easier on the v6 side, so you > could give one /36 or /40 or even /44 out to whitelist but have /48's at > each location. For sites with ipv6 support that may help now, but it > won't help all sites for quite some time, though the number that support > v6 is slowly getting better. For the foreseeable future you still need > to handle the v4 side one way or another though. > > On 7/28/2021 10:21 AM, William Herrin wrote: > > On Wed, Jul 28, 2021 at 6:04 AM Vimal wrote: > >> My intention is to run a web-crawling service on a public cloud. This > service > >> is geographically distributed, and therefore will run in multiple > regions > >> around the world inside AWS... this means there will be multiple AWS > VPCs, > >> each with their own NAT gateway, and traffic destined to websites > >> that we crawl will appear to come from this NAT gateway's IP address. > > > > Hello, > > > > AWS does not provide the ability to attach anycasted IP addresses to a > > NAT gateway, regardless of whether it would work, so that's the end of > > your quest. > > > >> The reason I want a predictable IP is to communicate this IP to website > >> owners so they can allow access from these IPs into their networks. > >> I chose IP as an example; it can also be a subnet, but what I don't > want to > >> provide is a list of 100 different IP addresses without any > predictability. > > > > If you bring your own IP addresses, you can attach a separate /24s of > > them to your VPCs in each region, providing you with a single > > predictable range of source addresses. You will find it difficult and > > expensive to acquire that many IP addresses from the regional > > registries for the purpose you describe. > > > > > > Silly question but: for a web crawler, why do you care whether it has > > the limited geographically distribution that a cloud service provides? > > It's a parallel batch task. It doesn't exactly matter whether you have > > minimum latency. > > > > Regards, > > Bill Herrin > > > > > > > -- Vimal
Re: Anycast but for egress
I'd had a similar thought/question, though keeping the geo diversity, you manage the crawlers, and are making contact individually with these sites from what you have stated (and so don't need a one size fit's all list for public posting), so why not have a restricted subset of the crawlers handle sites with these issues (which subset may be unique per site, which makes maintaining even load balancing not overly complex /limiting, especially as you are using nat anyway, so multiple servers can be behind each ip and that number can vary). That let's you have geo diversity (or even multi cloud diversity) for every site, but each site that needs this IP whitelisting only needs 3-5 IP's at any site, but yet you can distribute load over a much larger overall set of machines and nat gateways. As I understand it even CDN's that anycast TCP (externally or internally [load balancing via routers and multi path]) do similar by spreading load over multiple IP's at the DNS layer first. As the transition to IPv6 happens you may have it easier as getting a large enough allocation to allow for splitting it out into multiple subnets advertised from different locations without providers dropping the route as too long a prefix is much easier on the v6 side, so you could give one /36 or /40 or even /44 out to whitelist but have /48's at each location. For sites with ipv6 support that may help now, but it won't help all sites for quite some time, though the number that support v6 is slowly getting better. For the foreseeable future you still need to handle the v4 side one way or another though. On 7/28/2021 10:21 AM, William Herrin wrote: On Wed, Jul 28, 2021 at 6:04 AM Vimal wrote: My intention is to run a web-crawling service on a public cloud. This service is geographically distributed, and therefore will run in multiple regions around the world inside AWS... this means there will be multiple AWS VPCs, each with their own NAT gateway, and traffic destined to websites that we crawl will appear to come from this NAT gateway's IP address. Hello, AWS does not provide the ability to attach anycasted IP addresses to a NAT gateway, regardless of whether it would work, so that's the end of your quest. The reason I want a predictable IP is to communicate this IP to website owners so they can allow access from these IPs into their networks. I chose IP as an example; it can also be a subnet, but what I don't want to provide is a list of 100 different IP addresses without any predictability. If you bring your own IP addresses, you can attach a separate /24s of them to your VPCs in each region, providing you with a single predictable range of source addresses. You will find it difficult and expensive to acquire that many IP addresses from the regional registries for the purpose you describe. Silly question but: for a web crawler, why do you care whether it has the limited geographically distribution that a cloud service provides? It's a parallel batch task. It doesn't exactly matter whether you have minimum latency. Regards, Bill Herrin
Re: Anycast but for egress
On 7/28/21 17:09, Bill Woodcock wrote: I was about to say something about us having equal success over 105 or so countries, when I came to the realization that inviting quantitative comparisons of manhood with Mark is the very definition of folly. :-) Well, we are nowhere close to the 105 countries PCH boasts. That's a whole other level of scale :-). Impressed! Anyway, yeah, the folks who were scared of anycast in the 1990s were running from shadows, not basing it on experience or data. In the real world, the number of stateful flows affected by route changes is dwarfed by those disrupted by other causes, and is immeasurably small. And when they do crop up on the radar, it’s almost always someone’s equal-cost-multi-path gone wrong, rather than an actual shift. So, not an issue at all in the real world, just in the imaginations of folks who thought TCP was a complex thing reserved for the specific use-cases that they’d already conceived of in the 1980s. Took a while to get beyond their protestations, but here we are in the 21st century. Planck's principle holds. Science progresses one funeral at a time. 100%. Mark.
Re: Anycast but for egress
> On Jul 28, 2021, at 3:21 AM, Mark Tinka wrote: > On 7/28/21 01:16, Daniel Corbe wrote: > >>> This is interesting... I wonder whether Anycast will still have some >>> failure modes and break TCP connections if routing (configuration) were to >>> change? I checked the PDF linked by Bill Woodcock... while the methodology >>> is the same from 20y ago, would the data still be the same (order of >>> magnitude)? :) > > We are Anycast'ing DNS (authoritative and recursive), NTP and TACACS+. All > works well, across 11 or so countries. I was about to say something about us having equal success over 105 or so countries, when I came to the realization that inviting quantitative comparisons of manhood with Mark is the very definition of folly. :-) Anyway, yeah, the folks who were scared of anycast in the 1990s were running from shadows, not basing it on experience or data. In the real world, the number of stateful flows affected by route changes is dwarfed by those disrupted by other causes, and is immeasurably small. And when they do crop up on the radar, it’s almost always someone’s equal-cost-multi-path gone wrong, rather than an actual shift. So, not an issue at all in the real world, just in the imaginations of folks who thought TCP was a complex thing reserved for the specific use-cases that they’d already conceived of in the 1980s. Took a while to get beyond their protestations, but here we are in the 21st century. Planck's principle holds. Science progresses one funeral at a time. -Bill signature.asc Description: Message signed with OpenPGP
Re: Anycast but for egress
> On Jul 27, 2021, at 6:15 PM, Vimal wrote: > > AWS Global Accelerator gives anycast IPs that's good for ingress, but my > original question was about having predictable egress IPs. > > It looks like having a few EIPs/a contiguous network block is the way to go. Yes. Predictable and unchanging (but each unique per location) static IP addresses is what you’re looking for. It would be a huge convenience to others if you could specify a single contiguous CIDR block for others to “permit” in their access control lists, but alas that would be very difficult as well… Since BGP announcements generally need to be aggregated up to at least a /24 or a /48 (though people are less strict on the v6 side), each group of hosts numbered from the same block of that size would need to have internally contiguous convex routing, meaning that it would have to be interconnected by its own network (albeit that could be tunnels) and accept inbound traffic at any point on the surface of that network, backhauling it to the appropriate location. So if you wanted to be able to identify a single CIDR block with eight locations in it, you’d either need to specify a /24 that was 97% wasted, and was fully internally interconnected (i.e. no efficiencies in localizing traffic), or you’d need to advertise eight /24s, which would aggregate up to a single /21, which was 99.6% wasted. So, you can see why the combination of scarce IPv4 addresses, scarce BGP routing slots, and content routing tricks often don’t play well together. -Bill signature.asc Description: Message signed with OpenPGP
Re: Anycast but for egress
On Wed, Jul 28, 2021 at 6:04 AM Vimal wrote: > My intention is to run a web-crawling service on a public cloud. This service > is geographically distributed, and therefore will run in multiple regions > around the world inside AWS... this means there will be multiple AWS VPCs, > each with their own NAT gateway, and traffic destined to websites > that we crawl will appear to come from this NAT gateway's IP address. Hello, AWS does not provide the ability to attach anycasted IP addresses to a NAT gateway, regardless of whether it would work, so that's the end of your quest. > The reason I want a predictable IP is to communicate this IP to website > owners so they can allow access from these IPs into their networks. > I chose IP as an example; it can also be a subnet, but what I don't want to > provide is a list of 100 different IP addresses without any predictability. If you bring your own IP addresses, you can attach a separate /24s of them to your VPCs in each region, providing you with a single predictable range of source addresses. You will find it difficult and expensive to acquire that many IP addresses from the regional registries for the purpose you describe. Silly question but: for a web crawler, why do you care whether it has the limited geographically distribution that a cloud service provides? It's a parallel batch task. It doesn't exactly matter whether you have minimum latency. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: Anycast but for egress
we, verio, did anycast tcp streaming (hour long) of the tony awards in about '96. solid. randy --- ra...@psg.com `gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com` signatures are back, thanks to dmarc header butchery
Re: Anycast but for egress
l they are on the internet. Also it would trigger IDS/IPS systems > all over the place, having gobs and gobs of connections coming from a > single IP. > > > That's setting aside the technical issues involved; routing is often > asymmetric, i.e. the return packet takes a different path than the inbound > packet. So it would, as Owen implied, be nearly impossible to ensure the > reply packets got back to the correct TCP stack. As an example, I'm > multi-homed and use path-prepending, so if a packet claiming to be from > 8.8.8.8 arrived on one of my commercial links, I would send the reply out > the cheapest link, which in my case is a flat-rate R network (that has a > path to Google), thus ensuring the reply does not get to the originating > anycast node. > > > When my clients make connections outbound to anycast addresses, the > destination is more-or-less stable, and the replies come back to the > client's unique IP, so anycast works in that direction. The guarantees are > not present in the reverse direction. > > > The logical extremity of this is that it would be nearly impossible for > two anycast addresses to establish a TCP connection to each other. (In > general. There will be lots of local cases where it does happen to work, > by coincidence.) > > > You'll find that even anycast nodes do not make connections outbound using > their anycast address, pretty much for these reasons. > > > -Adam > > > Adam Thompson > > Consultant, Infrastructure Services > > > > 100 - 135 Innovation Drive > > Winnipeg, MB, R3T 6A8 > > (204) 977-6824 or 1-800-430-6404 (MB only) > > athomp...@merlin.mb.ca > > www.merlin.mb.ca > > From: NANOG on behalf of > Vimal > > Sent: July 27, 2021 12:54 > > To: nanog@nanog.org > > Subject: Anycast but for egress > > > (Unsure if this is the right forum to ask this question, but here goes:) > > > From what I understand, IP Anycast can be used to steer traffic into a > server that's close to the client. > > > I am curious if anyone here has/encountered a setup where they use anycast > IP on their gateways... to have a predictable egress IP for their traffic, > regardless of where they are located? > > > For example, a search engine crawler could in principle have the same IP > advertised all over the world, but it looks like they don't... I wonder > why? > > > -- > > Vimal > > > > > -- > > Vimal > > > If I were in your shoes, I’d pick a VPS provider that assigns external, > globally routable IPs to their customers. Linode, Vultr, Digital Ocean, > etc. > > You may be able to do something on AWS with Elastic IPs but I don’t know > enough about Amazon’s infrastructure to give you a qualified answer. > > > > -- Vimal
Re: Anycast but for egress
early impossible to ensure the > reply packets got back to the correct TCP stack. As an example, I'm > multi-homed and use path-prepending, so if a packet claiming to be from > 8.8.8.8 arrived on one of my commercial links, I would send the reply out > the cheapest link, which in my case is a flat-rate R network (that has a > path to Google), thus ensuring the reply does not get to the originating > anycast node. > > > > When my clients make connections outbound to anycast addresses, the > destination is more-or-less stable, and the replies come back to the > client's unique IP, so anycast works in that direction. The guarantees are > not present in the reverse direction. > > > > The logical extremity of this is that it would be nearly impossible for > two anycast addresses to establish a TCP connection to each other. (In > general. There will be lots of local cases where it does happen to work, > by coincidence.) > > > > You'll find that even anycast nodes do not make connections outbound > using their anycast address, pretty much for these reasons. > > > > -Adam > > > > Adam Thompson > > Consultant, Infrastructure Services > > > > 100 - 135 Innovation Drive > > Winnipeg, MB, R3T 6A8 > > (204) 977-6824 or 1-800-430-6404 (MB only) > > athomp...@merlin.mb.ca > > www.merlin.mb.ca > > From: NANOG on behalf > of Vimal > > Sent: July 27, 2021 12:54 > > To: nanog@nanog.org > > Subject: Anycast but for egress > > > > (Unsure if this is the right forum to ask this question, but here goes:) > > > > From what I understand, IP Anycast can be used to steer traffic into a > server that's close to the client. > > > > I am curious if anyone here has/encountered a setup where they use > anycast IP on their gateways... to have a predictable egress IP for their > traffic, regardless of where they are located? > > > > For example, a search engine crawler could in principle have the same IP > advertised all over the world, but it looks like they don't... I wonder > why? > > > > -- > > Vimal > > > > > > > > -- > > Vimal > > If I were in your shoes, I’d pick a VPS provider that assigns external, > globally routable IPs to their customers. Linode, Vultr, Digital Ocean, > etc. > > You may be able to do something on AWS with Elastic IPs but I don’t know > enough about Amazon’s infrastructure to give you a qualified answer. > > > > -- Vimal
Re: Anycast but for egress
cast addresses to establish a TCP connection to each other. (In > general. There will be lots of local cases where it does happen to work, > by coincidence.) > > You'll find that even anycast nodes do not make connections outbound using > their anycast address, pretty much for these reasons. > > -Adam > > *Adam Thompson* > Consultant, Infrastructure Services > [image: 1593169877849] > 100 - 135 Innovation Drive > Winnipeg, MB, R3T 6A8 > (204) 977-6824 or 1-800-430-6404 (MB only) > athomp...@merlin.mb.ca > www.merlin.mb.ca > -- > *From:* NANOG on behalf > of Vimal > *Sent:* July 27, 2021 12:54 > *To:* nanog@nanog.org > *Subject:* Anycast but for egress > > (Unsure if this is the right forum to ask this question, but here goes:) > > From what I understand, IP Anycast can be used to steer traffic into a > server that's close to the client. > > I am curious if anyone here has/encountered a setup where they use anycast > IP on their gateways... to have a predictable egress IP for their traffic, > regardless of where they are located? > > For example, a search engine crawler could in principle have the same IP > advertised all over the world, but it looks like they don't... I wonder > why? > > -- > Vimal > > -- Vimal
Re: Anycast but for egress
On 7/28/21 01:16, Daniel Corbe wrote: This is interesting... I wonder whether Anycast will still have some failure modes and break TCP connections if routing (configuration) were to change? I checked the PDF linked by Bill Woodcock... while the methodology is the same from 20y ago, would the data still be the same (order of magnitude)? :) In our small experience, not at all. We are Anycast'ing DNS (authoritative and recursive), NTP and TACACS+. All works well, across 11 or so countries. Mark.
Re: Anycast but for egress
Here is what I think would happen if you were to try this setup. Let's assume you deployed in eu-west-2 (London) and eu-central-1 (Frankfurt). You would find that you could successfully connect to a number of networks but also that some of them would work from the "wrong" site. Eg. you would have clients in London that require you to use the Frankfurt instance to connect to. You would also find a lot of networks that you could not connect to from either site. And you would have some instability where some networks at random switch between these states. For example you could have a client that one day works from London, the next day it is Frankfurt and then someday it won't be working at all. As you add more sites, you also add more ways for the traffic to end up in the wrong places. You could have something that works with just the two sites above but then when you add eu-west-1 (Ireland) you could suddenly not connect to them from any of the three sites. There IS a possible solution which is to tunnel the traffic back to the correct site. This still requires you to use unique IP addresses for each site, but all addresses could be in the same subnet. You would have IP a.b.c.1 to be London, a.b.c.2 Frankfurt, a.b.c.3 Ireland. Then if London receives traffic to a.b.c.2 you would have a tunnel to send the traffic to Frankfurt. Regards, Baldur On Wed, Jul 28, 2021 at 11:07 AM Baldur Norddahl wrote: > > >> > On Jul 27, 2021, at 17:20, Vimal wrote: >> > Yes, this makes sense as the destination can be anywhere around the >> world, and that routing is asymmetric as others mentioned. However, if the >> destination service is "close" (in the routing metric sense) to the >> initiating host, anycast return IP ought to work well, right? I understand >> this is a very important caveat and impractical to implement correctly in >> the real world. >> > > No, there is no such thing as "close". You could have a direct peering > with some ISP and have them still deliver the responses on the other side > of earth. You do not control the routing of other networks and can not be > sure what they will do. > > For larger networks you may also have multiple peering points. Say you > have a peering with them in city A and city B. How do you know which of > their IP ranges are closer to A or B? You don't. And the same goes for > them, they have no idea if you prefer A or B. Therefore you could select A > and they may reply to B. They may even load balance between A and B if you > are really unlucky. > > Routing is asymmetric. That means you have absolutely no idea where the > replies end up going. Often it will not be what you think is "close". > > I do not run anycast, but I understand that the usual way of dealing with > these issues is to do as little as possible with anycast before redirecting > to a unicast address. For example you could have just your DNS on anycast > and each site would reply with unique unicast addresses. Since DNS is just > a single pair of UDP request/response, with the first packet originating > from a unicast client, this works well. > > Regards, > > Baldur > > >
Re: Anycast but for egress
> > > On Jul 27, 2021, at 17:20, Vimal wrote: > > Yes, this makes sense as the destination can be anywhere around the > world, and that routing is asymmetric as others mentioned. However, if the > destination service is "close" (in the routing metric sense) to the > initiating host, anycast return IP ought to work well, right? I understand > this is a very important caveat and impractical to implement correctly in > the real world. > No, there is no such thing as "close". You could have a direct peering with some ISP and have them still deliver the responses on the other side of earth. You do not control the routing of other networks and can not be sure what they will do. For larger networks you may also have multiple peering points. Say you have a peering with them in city A and city B. How do you know which of their IP ranges are closer to A or B? You don't. And the same goes for them, they have no idea if you prefer A or B. Therefore you could select A and they may reply to B. They may even load balance between A and B if you are really unlucky. Routing is asymmetric. That means you have absolutely no idea where the replies end up going. Often it will not be what you think is "close". I do not run anycast, but I understand that the usual way of dealing with these issues is to do as little as possible with anycast before redirecting to a unicast address. For example you could have just your DNS on anycast and each site would reply with unique unicast addresses. Since DNS is just a single pair of UDP request/response, with the first packet originating from a unicast client, this works well. Regards, Baldur
Re: Anycast but for egress
. Also it would trigger IDS/IPS systems all over >> the place, having gobs and gobs of connections coming from a single IP. >> >> That's setting aside the technical issues involved; routing is often >> asymmetric, i.e. the return packet takes a different path than the inbound >> packet. So it would, as Owen implied, be nearly impossible to ensure the >> reply packets got back to the correct TCP stack. As an example, I'm >> multi-homed and use path-prepending, so if a packet claiming to be from >> 8.8.8.8 arrived on one of my commercial links, I would send the reply out >> the cheapest link, which in my case is a flat-rate R network (that has a >> path to Google), thus ensuring the reply does not get to the originating >> anycast node. >> >> When my clients make connections outbound to anycast addresses, the >> destination is more-or-less stable, and the replies come back to the >> client's unique IP, so anycast works in that direction. The guarantees are >> not present in the reverse direction. >> >> The logical extremity of this is that it would be nearly impossible for two >> anycast addresses to establish a TCP connection to each other. (In general. >> There will be lots of local cases where it does happen to work, by >> coincidence.) >> >> You'll find that even anycast nodes do not make connections outbound using >> their anycast address, pretty much for these reasons. >> >> -Adam >> >> Adam Thompson >> Consultant, Infrastructure Services >> >> 100 - 135 Innovation Drive >> Winnipeg, MB, R3T 6A8 >> (204) 977-6824 or 1-800-430-6404 (MB only) >> athomp...@merlin.mb.ca >> www.merlin.mb.ca >> From: NANOG on behalf of >> Vimal >> Sent: July 27, 2021 12:54 >> To: nanog@nanog.org >> Subject: Anycast but for egress >> >> (Unsure if this is the right forum to ask this question, but here goes:) >> >> From what I understand, IP Anycast can be used to steer traffic into a >> server that's close to the client. >> >> I am curious if anyone here has/encountered a setup where they use anycast >> IP on their gateways... to have a predictable egress IP for their traffic, >> regardless of where they are located? >> >> For example, a search engine crawler could in principle have the same IP >> advertised all over the world, but it looks like they don't... I wonder why? >> >> -- >> Vimal >> >> >> >> -- >> Vimal > > If I were in your shoes, I’d pick a VPS provider that assigns external, > globally routable IPs to their customers. Linode, Vultr, Digital Ocean, etc. > > > You may be able to do something on AWS with Elastic IPs but I don’t know > enough about Amazon’s infrastructure to give you a qualified answer. > > >
Re: Anycast but for egress
(that has a path > to Google), thus ensuring the reply does not get to the originating anycast > node. > > When my clients make connections outbound to anycast addresses, the > destination is more-or-less stable, and the replies come back to the client's > unique IP, so anycast works in that direction. The guarantees are not > present in the reverse direction. > > The logical extremity of this is that it would be nearly impossible for two > anycast addresses to establish a TCP connection to each other. (In general. > There will be lots of local cases where it does happen to work, by > coincidence.) > > You'll find that even anycast nodes do not make connections outbound using > their anycast address, pretty much for these reasons. > > -Adam > > Adam Thompson > Consultant, Infrastructure Services > > 100 - 135 Innovation Drive > Winnipeg, MB, R3T 6A8 > (204) 977-6824 or 1-800-430-6404 (MB only) > athomp...@merlin.mb.ca > www.merlin.mb.ca > From: NANOG on behalf of > Vimal > Sent: July 27, 2021 12:54 > To: nanog@nanog.org > Subject: Anycast but for egress > > (Unsure if this is the right forum to ask this question, but here goes:) > > From what I understand, IP Anycast can be used to steer traffic into a server > that's close to the client. > > I am curious if anyone here has/encountered a setup where they use anycast IP > on their gateways... to have a predictable egress IP for their traffic, > regardless of where they are located? > > For example, a search engine crawler could in principle have the same IP > advertised all over the world, but it looks like they don't... I wonder why? > > -- > Vimal > > > > -- > Vimal If I were in your shoes, I’d pick a VPS provider that assigns external, globally routable IPs to their customers. Linode, Vultr, Digital Ocean, etc. You may be able to do something on AWS with Elastic IPs but I don’t know enough about Amazon’s infrastructure to give you a qualified answer.
Re: Anycast but for egress
On 7/27/21 20:48, Bill Woodcock wrote: In practice, that means that services are bound to a common shared address (an “anycast service address”) as those services are deployed on servers in different locations. The service address is advertised into the BGP routing infrastructure. Clients send packets to the service address, and the BGP routing infrastructure routes each packet on the shortest path to its destination, without knowing that there are multiple destinations. We use our IGP (IS-IS) for our Anycast services. We find it to be very basic, and as such, very predictable. Of course, if you don't run your own AS for the services you provide, or operate your own backbone, this is probably not a viable option. Mark.
Re: Anycast but for egress
Without any sarcasm: to make it harder to block. If, say, Google, always crawled your site from 8.8.1.2 (random made-up example) then you would see a not-insignificant number of hosts and networks null-routing that IP. I have no idea why someone would do so, but I've seen it done many times. Mostly by people who don't understand how un-special they are on the internet. Also it would trigger IDS/IPS systems all over the place, having gobs and gobs of connections coming from a single IP. That's setting aside the technical issues involved; routing is often asymmetric, i.e. the return packet takes a different path than the inbound packet. So it would, as Owen implied, be nearly impossible to ensure the reply packets got back to the correct TCP stack. As an example, I'm multi-homed and use path-prepending, so if a packet claiming to be from 8.8.8.8 arrived on one of my commercial links, I would send the reply out the cheapest link, which in my case is a flat-rate R network (that has a path to Google), thus ensuring the reply does not get to the originating anycast node. When my clients make connections outbound to anycast addresses, the destination is more-or-less stable, and the replies come back to the client's unique IP, so anycast works in that direction. The guarantees are not present in the reverse direction. The logical extremity of this is that it would be nearly impossible for two anycast addresses to establish a TCP connection to each other. (In general. There will be lots of local cases where it does happen to work, by coincidence.) You'll find that even anycast nodes do not make connections outbound using their anycast address, pretty much for these reasons. -Adam Adam Thompson Consultant, Infrastructure Services [1593169877849] 100 - 135 Innovation Drive Winnipeg, MB, R3T 6A8 (204) 977-6824 or 1-800-430-6404 (MB only) athomp...@merlin.mb.ca<mailto:athomp...@merlin.mb.ca> www.merlin.mb.ca<http://www.merlin.mb.ca/> From: NANOG on behalf of Vimal Sent: July 27, 2021 12:54 To: nanog@nanog.org Subject: Anycast but for egress (Unsure if this is the right forum to ask this question, but here goes:) >From what I understand, IP Anycast can be used to steer traffic into a server >that's close to the client. I am curious if anyone here has/encountered a setup where they use anycast IP on their gateways... to have a predictable egress IP for their traffic, regardless of where they are located? For example, a search engine crawler could in principle have the same IP advertised all over the world, but it looks like they don't... I wonder why? -- Vimal
Re: Anycast but for egress
Matt Harris|Infrastructure Lead 816-256-5446|Direct Looking for help? Helpdesk|Email Support We build customized end-to-end technology solutions powered by NetFire Cloud. On Tue, Jul 27, 2021 at 1:29 PM Vimal wrote: > (Unsure if this is the right forum to ask this question, but here goes:) > > From what I understand, IP Anycast can be used to steer traffic into a > server that's close to the client. > > I am curious if anyone here has/encountered a setup where they use anycast > IP on their gateways... to have a predictable egress IP for their traffic, > regardless of where they are located? > > For example, a search engine crawler could in principle have the same IP > advertised all over the world, but it looks like they don't... I wonder > why? > Hi Vimal, I'm not sure I see what the benefit would be here. Maybe if you explain what exactly you're trying to accomplish, someone can point you in the right direction? What you're describing here isn't really analogous to how anycast works. Anycast isn't really a protocol in and of itself, it's simply the act of advertising IP space externally in a diverse way and then routing that traffic to the nearest capable host once it's inside your network, rather than to a single very specific host that exists in one specific place. - mdh
Re: Anycast but for egress
> On Jul 27, 2021, at 10:54 AM, Vimal wrote: > > (Unsure if this is the right forum to ask this question Sure, why not… There isn’t anywhere more appropriate, really. > From what I understand, IP Anycast can be used to steer traffic into a server > that's close to the client. That’s the net effect, as it’s normally used. But anycast is really very simple, and has no concept of client/server… An IP address is assigned to multiple devices or processes, in locations which the routing topology views as diverse. In practice, that means that services are bound to a common shared address (an “anycast service address”) as those services are deployed on servers in different locations. The service address is advertised into the BGP routing infrastructure. Clients send packets to the service address, and the BGP routing infrastructure routes each packet on the shortest path to its destination, without knowing that there are multiple destinations. > I am curious if anyone here has/encountered a setup where they use anycast IP > on their gateways... to have a predictable egress IP for their traffic, > regardless of where they are located? > > For example, a search engine crawler could in principle have the same IP > advertised all over the world, but it looks like they don't... I wonder why? I think you’re going to need to construct a clearer and more precise explanation of what you’re imagining, because my reading of these two lines is that they’re saying different things; I don’t see the connection between them that you see. That said, a few reactions: Anycast is often thought to _reduce_ predictability, since it offers multiple exclusive possible termination points for each packet, whereas unicast, multicast or broadcast would each have predictable outcomes by comparison: a specific node would receive the packet, a specific set of nodes would receive the packet, or all (in-scope broadcast domain) nodes would receive the packet. If you’re asking whether it would make sense for border routers, which have access to full-table transit, to advertise that accessibility as an anycasted service, that’s what the special “default route” 0.0.0.0/0 is. Many people configure full-transit BGP routers to redistribute a 0.0.0.0/0 default route into their IGP, their internal routing protocol (albeit that may well be iBGP, nowadays) in order to accommodate routers which haven’t the resources to hold or use full routes. A search engine crawler depends upon a unicast return path in order to establish a TCP session with the web sites it’s crawling, and see the return traffic from them. If a search engine crawler shared an anycast service address with other instances of itself in other locations, the outbound queries would head to web sites (which might be unicast or might be anycast, doesn’t matter), which would then try to reply. If the source address of the query is an anycast service address, the reply will go to the nearest instance of that shared address, rather than to the specific instance which originated the query. It’s for this reason that one normally assigns unique unicast addresses to network-facing interfaces which will originate packets, and anycast addresses to internal loopback interfaces, to which services are bound… The server can receive packets addressed to the anycast shared address, but will originate packets using its unique address. Here’s a tutorial from twenty years ago (when this was all less than fifteen years old!) that explains in some detail… Things haven’t really changed since then: https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf -Bill signature.asc Description: Message signed with OpenPGP
Re: Anycast but for egress
> On Jul 27, 2021, at 12:54, Vimal wrote: > > (Unsure if this is the right forum to ask this question, but here goes:) > > From what I understand, IP Anycast can be used to steer traffic into a server > that's close to the client. > > I am curious if anyone here has/encountered a setup where they use anycast IP > on their gateways... to have a predictable egress IP for their traffic, > regardless of where they are located? > > For example, a search engine crawler could in principle have the same IP > advertised all over the world, but it looks like they don't... I wonder why? > > -- > Vimal > Unless you’re twisting knobs, egress traffic should already exit your network at the closest possible egress point to its origin. Is your intention to carry the traffic for longer than that?
Re: Anycast but for egress
> On Jul 27, 2021, at 10:54 , Vimal wrote: > > (Unsure if this is the right forum to ask this question, but here goes:) > > From what I understand, IP Anycast can be used to steer traffic into a server > that's close to the client. > > I am curious if anyone here has/encountered a setup where they use anycast IP > on their gateways... to have a predictable egress IP for their traffic, > regardless of where they are located? > For example, a search engine crawler could in principle have the same IP > advertised all over the world, but it looks like they don't... I wonder why? Because there’s no good/reliable way to get the replies back to the correct initiating host. Owen
Anycast but for egress
(Unsure if this is the right forum to ask this question, but here goes:) >From what I understand, IP Anycast can be used to steer traffic into a server that's close to the client. I am curious if anyone here has/encountered a setup where they use anycast IP on their gateways... to have a predictable egress IP for their traffic, regardless of where they are located? For example, a search engine crawler could in principle have the same IP advertised all over the world, but it looks like they don't... I wonder why? -- Vimal
