Re: Looking for anycast DNS services..

[Carlos Kamtha]

Thanks to you all for the speedy response!

-C


Thanks!

On Fri, Jun 21, 2024 at 05:01:38PM -0400, Carlos Kamtha wrote:
> 
> 
> 
> I found what I was looking for. 
> -C
> 
> Cheers!
> 
> 
> On Thu, Jun 20, 2024 at 09:44:57AM -0400, Sean Barton wrote:
> > Carlos, 
> > 
> > Was this helpful? 
> > 
> > Thank you, 
> > 
> > Sean
> > 
> > Sean Barton 
> > Chief Strategy Officer 
> > 513-460-5609
> > www.pch.net
> > https://www.linkedin.com/in/seanbarton/
> > 
> > 
> > CONFIDENTIALITY NOTICE: The information and all attachments contained in 
> > this electronic communication are privileged and confidential information 
> > and intended only for the use of the intended recipients.  If the reader of 
> > this message is not an intended recipient, you are hereby notified that any 
> > review, use, dissemination, distribution or copying of this communication 
> > is strictly prohibited.  If you have received this communication in error, 
> > please notify us immediately of the error by return email and please 
> > permanently remove any copies of this message from your system and do not 
> > retain any copies, whether in electronic or physical form or otherwise.  
> > Thank You.
> > 
> > -Original Message-
> > From: Sean Barton  
> > Sent: Thursday, June 13, 2024 6:47 PM
> > To: 'kam...@ak-labs.net' 
> > Subject: RE: Looking for anycast DNS services..
> > 
> > Carlos, 
> > 
> > I have attached our anycast brochure for your review. Would you please tell 
> > me how big are your zones?
> > 
> > Thank you,
> > 
> > Sean
> > 
> > Sean Barton 
> > Chief Strategy Officer 
> > 513-460-5609
> > www.pch.net
> > https://www.linkedin.com/in/seanbarton/
> > 
> > 
> > CONFIDENTIALITY NOTICE: The information and all attachments contained in 
> > this electronic communication are privileged and confidential information 
> > and intended only for the use of the intended recipients.  If the reader of 
> > this message is not an intended recipient, you are hereby notified that any 
> > review, use, dissemination, distribution or copying of this communication 
> > is strictly prohibited.  If you have received this communication in error, 
> > please notify us immediately of the error by return email and please 
> > permanently remove any copies of this message from your system and do not 
> > retain any copies, whether in electronic or physical form or otherwise.  
> > Thank You.
> > 
> > > From: Carlos Kamtha 
> > > Date: June 13, 2024 at 11:25:25 PM GMT+2
> > > To: nanog@nanog.org
> > > Subject: Looking for anycast DNS services..
> > > 
> > > Hello.
> > > 
> > > Looking for upstream provider where I can locate DNS servers with global 
> > > anycast service.
> > > 
> > > We have our own CIDR to announce and would prefer physical presence 
> > > starting with South Asia and Europe.
> > > 
> > > Commemts and suggestions welcome.
> > > 
> > > -C
> > 

Re: Looking for anycast DNS services..

[Elmar K. Bins]

Hey Christian,

li...@packetflux.com (Forrest Christian (List Account)) wrote:

> Many, if not most, modern hosting providers will give you a bgp session.
>  I've used vultr in the past but it's not nearly as hard to find a provider
> which accepts bgp anymore.  It seems like every time I'm looking for a new
> provider that's on the list of features.

Well... there's a difference between what's on paper and what actually works;
also, many hosting providers charge quite a bit for the privilege, while others
offer it included and with great service to boot.

During my "shopping spree" half a year ago, I found vultr and ifog to be among
the best, to be affordable, and also having decent global coverage. They also
(both) have really good support. Additional shout-out to first root, if you want
to host in Düsseldorf ;-)

Interested in more useful providers (aka "working, affordable, usable").

Elmar.

PS: I went through the list at https://bgp.services - thank you for providing
that service, even if some on the list turned out more complicated/useless
than expected ;-)

Re: Looking for anycast DNS services..

[Tore Anderson]

* Carlos Kamtha

Looking for upstream provider where I can locate DNS servers with global 
anycast service.

We have our own CIDR to announce and would prefer physical presence starting 
with South Asia and Europe.

Commemts and suggestions welcome.


Something like Netnod DNSNODE? We're using them, so far they've been 
rock solid.


https://www.netnod.se/dns/netnod-dns-services

Tore

Re: Looking for anycast DNS services..

[Forrest Christian (List Account)]

Many, if not most, modern hosting providers will give you a bgp session.
 I've used vultr in the past but it's not nearly as hard to find a provider
which accepts bgp anymore.  It seems like every time I'm looking for a new
provider that's on the list of features.

On Thu, Jun 13, 2024, 2:24 PM Carlos Kamtha  wrote:

> Hello.
>
> Looking for upstream provider where I can locate DNS servers with global
> anycast service.
>
> We have our own CIDR to announce and would prefer physical presence
> starting with South Asia and Europe.
>
> Commemts and suggestions welcome.
>
> -C
>

Looking for anycast DNS services..

[Carlos Kamtha]

Hello.

Looking for upstream provider where I can locate DNS servers with global 
anycast service.

We have our own CIDR to announce and would prefer physical presence starting 
with South Asia and Europe.

Commemts and suggestions welcome.

-C

Re: TFTP over anycast

[Saku Ytti]

On Sat, 6 Apr 2024 at 12:00, Bill Woodcock  wrote:

> That’s been the normal way of doing it for some 35 years now.  iBGP 
> advertise, or don’t advertise, the service address, which is attached to the 
> loopback, depending whether you’re ready to service traffic.

If we are talking about eBGP, then pulling routes makes sense. If we
are talking about iBGP and controlled environment, you should never
pull anycast routes, because eventually you will have failure mode,
where the check mechanism itself is broken, and you'll pull all
routes.
If instead of pulling the routes, you make them inferior, you are
covered for the failure mode of check itself being broken.


-- 
  ++ytti

Re: TFTP over anycast

[Bill Woodcock]

> On Apr 6, 2024, at 10:30, Ray Bellis  wrote:
> On 27/02/2024 18:47, William Herrin wrote:
>> Then I'd write a script to monitor the local tftp server and stop frr if it 
>> detects any problems with the tftp server.
> There are other ways to achieve this without actually stopping the routing 
> daemon.
> We have DNS servers where the anycast service address is added to a loopback 
> interface (lo1) and only advertised when present.

That’s been the normal way of doing it for some 35 years now.  iBGP advertise, 
or don’t advertise, the service address, which is attached to the loopback, 
depending whether you’re ready to service traffic.

-Bill

Re: TFTP over anycast

[Ray Bellis]

On 27/02/2024 18:47, William Herrin wrote:


Then I'd write a script to monitor the local tftp server and stop frr
if it detects any problems with the tftp server.
There are other ways to achieve this without actually stopping the 
routing daemon.


We have DNS servers where the anycast service address is added to a 
loopback interface (lo1) and only advertised when present.


The monitoring script drops and adds the service address to the 
loopback, without otherwise touching the routing daemon.  We use Bird 
rather than FRR, though.


Ray

Re: TFTP over anycast

[Dan Sneddon]

Javier,I have seen a few potential hangups, most of which affect the setup equally if it is within the same datacenter or across datacenters. The difference there usually comes down to a greater chance of disconnects and “split-brain” scenarios when there are servers in multiple datacenters. In that case sharding (AKA cells, zones, etc.) is your friend to ensure that you can operate one site autonomously while disconnected from the others. Using DHCP servers in this way often reveals some bugs in the implementation depending on which server you are using. Fortunately I have seen several bugs get squashed in a couple of the open source implementations when members of my team reported them to the maintainers, so you should be confident in using one of the most common implementations (ISC, dnsmasq, a few others).You also need to make sure that your network routing infrastructure tends toward stability and stickiness, so the same client talks to the same server throughout a flow. Of course a failure in the middle of the flow will eventually lead to a failover, but anything in progress is unlikely to recover given the limited error correction and sanity checking in the mentioned protocols. Best to take this into account and plan for a number of retries on any failure. Also make sure to test that all your servers eventually reach consensus after you test failure scenarios, and come up with a plan to force synchronization if needed. Also, with IPv6 you want to make sure that if you are assigning multiple addresses to clients that all servers will offer the same set of IPv6 IPs. That can be a real headache to debug. You don’t necessarily need a DHCPv6 server to issue IPs at all depending on if your setup supports autoassignment (you’ll need the proper setup of route advertisers on your routers).Best of luck, I suspect it will work “like magic.” It does work but it flies in the face of past convention about how IP protocols are supposed to be used and requires control over areas that usually cross boundaries of responsibility (system admins vs. network admins vs. security admins).-Dan SneddonOn Feb 27, 2024, at 10:05 AM, Javier Gutierrez  wrote:






Thanks to you all for your answers, it has helped me a lot already.




My design is very simplistic, I have 2 sets of firewalls that I will have advertising a /32 unicast to the network at each location and it will have a TFTP server behind each firewall.




I have no intention to have this be part of the internet as it will be used to serve internal customers devices that require TFTP

For the setup where you are running Anycast on a datacenter, are you running it inside the datacenter only or across multiple datacenters? other than having to replicate IPs and file services between datacenters have you seen any other issues?




Kind regards,
 
Javier Gutierrez,
Network Architect – AS19016
https://www.peeringdb.com/net/4073
Westman Communications Group
1906
 Park Ave. • Brandon,
 MB • R7B
 0R9
204.720.1158
 gutierr...@westmancom.com

 
  
This e-mail and any attachments contain confidential and privileged information. If you are not the intended recipient, please
 notify the sender immediately by return e-mail, delete this e-mail and destroy any copies. Any dissemination or use of this information by a person other than intended recipient is unauthorized and may be illegal.
 



From: NANOG  on behalf of Bill Woodcock 
Sent: Saturday, February 24, 2024 1:09 AM
To: Ask Bjørn Hansen 
Cc: nanog@nanog.org 
Subject: Re: TFTP over anycast
 




CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and
 know the content is safe.

The system Ask is describing is the traditional method of using anycast to geographically load-balance long-lived flows.  The first time I did that was with FTP servers in Berkeley and Santa Cruz, in 1989. 


I did a bigger system, also load balancing FTP servers for Oracle, their public-facing documentation stores, with servers in San Jose and Washington DC, a couple of years later.   A couple of years further on and the World Wide Web was a thing, and everybody
 was doing it. 
    
                -Bill




On Feb 24, 2024, at 7:38 AM, Ask Bjørn Hansen  wrote:







On Feb 23, 2024, at 20:32, William Herrin  wrote:




The relay server `dhcplb` could, maybe, help in that scenario
(dhcplb runs on the anycast IP, the “real” DHCP servers on
unicast IPs behind dhcplb).


Although
 they used the word "anycast", they're just load balancing.




The idea is to run the relays on an anycasted IP (so the load balancer / relay IP is anycasted).



[….] Relying on ECMP for anycasted DHCP would be a disaster
during any sort of failure. Add or remove a single route from an ECMP
set and the hashed path selection changes for most of the connections.



Consistent hashing (which I thought was widely supported now in ECMP implementations) and a bit of automation in how an

Re: TFTP over anycast

[William Herrin]

On Tue, Feb 27, 2024 at 10:02 AM Javier Gutierrez
 wrote:
> My design is very simplistic, I have 2 sets of firewalls that I
> will have advertising a /32 unicast to the network at each
> location and it will have a TFTP server behind each firewall.

Hi Javier,

That sounds straightforward to me with no major failure modes. I would
make the firewall part of my OSPF network and then add the tftp
servers to OSPF using FRR. Then I'd write a script to monitor the
local tftp server and stop frr if it detects any problems with the
tftp server. The local tftp server will always be closer than the
remote one via OSPF link costs, unless it goes offline. I assume you
also have an encrypted channel between the firewalls to handle traffic
that stays "inside" your security boundary, as tftp generally should.

Where you could get into trouble is if you add a third or additional
sites. If there's ever an equal routing cost from any one site to two
others, there's a non-zero risk of the failover process failing... and
you won't know it until you need it.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: TFTP over anycast

[Javier Gutierrez]

Thanks to you all for your answers, it has helped me a lot already.

My design is very simplistic, I have 2 sets of firewalls that I will have 
advertising a /32 unicast to the network at each location and it will have a 
TFTP server behind each firewall.

I have no intention to have this be part of the internet as it will be used to 
serve internal customers devices that require TFTP
For the setup where you are running Anycast on a datacenter, are you running it 
inside the datacenter only or across multiple datacenters? other than having to 
replicate IPs and file services between datacenters have you seen any other 
issues?


Kind regards,



Javier Gutierrez,

Network Architect – AS19016
https://www.peeringdb.com/net/4073

Westman Communications Group

[cid:2db642a4-fcf9-40b4-a719-2afd8097f2e9]1906 Park Ave. • Brandon, MB • R7B 0R9

[cid:8862c057-cdef-45f6-a0e3-497508d0d64a]204.720.1158
[cid:6a35147d-b3b0-44cf-bc96-6822377f5231] 
gutierr...@westmancom.com<mailto:gutierr...@westmancom.com>

[A close up of a sign  Description automatically 
generated]<https://westmancom.com/personal>



[cid:486e0290-5d40-48dd-80eb-3be9a705b1e6]<https://www.facebook.com/WestmanCom>[cid:425d7b57-d7e3-491d-9d22-910d4072b88a]<https://twitter.com/WestmanCom>
  [cid:ee77dd48-8761-498b-b45b-82b00e5bf553] 
<https://www.youtube.com/user/WestmanCom>   
[cid:547ce68d-d61c-40e3-b150-39bff72b8d6b] 
<https://www.instagram.com/westmancom>   
[cid:ba4751b3-edc0-484e-bb40-731ca94e8c84] 
<https://www.linkedin.com/company/westmancom>

This e-mail and any attachments contain confidential and privileged 
information. If you are not the intended recipient, please notify the sender 
immediately by return e-mail, delete this e-mail and destroy any copies. Any 
dissemination or use of this information by a person other than intended 
recipient is unauthorized and may be illegal.




From: NANOG  on behalf of 
Bill Woodcock 
Sent: Saturday, February 24, 2024 1:09 AM
To: Ask Bjørn Hansen 
Cc: nanog@nanog.org 
Subject: Re: TFTP over anycast


CAUTION: This email is from an external source. Do not click links or open 
attachments unless you recognize the sender and know the content is safe.

The system Ask is describing is the traditional method of using anycast to 
geographically load-balance long-lived flows.  The first time I did that was 
with FTP servers in Berkeley and Santa Cruz, in 1989.

I did a bigger system, also load balancing FTP servers for Oracle, their 
public-facing documentation stores, with servers in San Jose and Washington DC, 
a couple of years later.   A couple of years further on and the World Wide Web 
was a thing, and everybody was doing it.

-Bill


On Feb 24, 2024, at 7:38 AM, Ask Bjørn Hansen  wrote:



On Feb 23, 2024, at 20:32, William Herrin  wrote:

The relay server `dhcplb` could, maybe, help in that scenario
(dhcplb runs on the anycast IP, the “real” DHCP servers on
unicast IPs behind dhcplb).

Although they used the word "anycast", they're just load balancing.

The idea is to run the relays on an anycasted IP (so the load balancer / relay 
IP is anycasted).

[….] Relying on ECMP for anycasted DHCP would be a disaster
during any sort of failure. Add or remove a single route from an ECMP
set and the hashed path selection changes for most of the connections.

Consistent hashing (which I thought was widely supported now in ECMP 
implementations) and a bit of automation in how announcements are added can 
greatly mitigate this.



Ask

Re: TFTP over anycast

[Dan Sneddon]

On Feb 22, 2024, at 10:47, Javier Gutierrez  wrote:Hi, I'm working on some DR design and we want to not only have this site as a DR but also performing some active/active for some of the services we hosts and I was wondering if someone had some experience with using anycast for TFTP or DHCP services?What are some of the pains/challenges you experienced and things we should lookout for?Any input is greatly appreciated.Kind regards, Javier GutierrezI have extensive experience using IP Anycast for TFTP and DHCP, in the area of cloud computing. My primary job role is the development of system and network provisioning in cloud infrastructure, and I’ve spent much of the last twelve years working in this area. This is one area where protocols like BGP and techniques like Anycast have a different set of assumtions and reputations for reliability when considered within a provider's network or on the Internet at large. Usually IP Anycast for DHCP and TFTP is done in a controlled environment (within a network operated by a single entity) and not done at global scale over the public Internet.I have designed or contributed to the design of several IP Anycast DHCP/TFTP implementations for cloud computing infrastructure (OpenStack, OpenShift/Kubernetes), using Quagga, Bird, or FRR for OSPF/BGP and Pound/HAProxy/NGinx/MetalLB for load balancing, along with custom ruby/python for DHCP or dnsmasq and typically standard Linux TFTP servers (either on bare metal or inside VMs or containers).It becomes complicated when you want to perform DHCP and/or TFTP across sites or WAN links, and downright tricky when you want to do it across the Internet. The DHCP servers may be configured with pre-allocated host IP reservations if the clients are known ahead of time, or all servers may use a shared database (often distributed using MariaDB, InfoBlox, or similar) to ensure that each DHCP server agrees about which IPs are assigned and can sync IP reservations and releases. It is usually necessary to ensure that all TFTP servers are offering identical images via TFTP.Some platforms that I have used IP Anycast DHCP and TFTP servers with:OpenStack Nova (KVM/QEMU virtual machines): https://docs.openstack.org/nova/latest/OpenStack Ironic (bare metal): https://docs.openstack.org/ironic/latest/Metal3 (an offspring of Ironic that works in Kubernetes clusters: https://metal3.io/As the initial DHCP request is usually done via broadcast request, the network hardware close to the client is often configured as a DHCP relay with either multiple unicast IPs as relay targets or one or more Anycast IPs (this may depend on what a particular vendor supports on a given make/model of network switch or router). In other cases a DHCP unicast IP is hard-coded into a custom image in firmware or microimage, or cached in the case of a running client making a renewal or release.Most projects use a micro-image booted over TFTP to prepare a second-stage loader that uses a more reliable protocol such as HTTPS. When using DHCPv6 there are a lot of unique challenges, especially when multiple IPv6 addresses are assigned or the client is a piece of embedded hardware (such as a bare metal IPMI controller or a NIC running PXE/iPXE firmware).Usually this is all done in either ”private” IP address space (local-scope or RFC1918). Now as far as running these same protocols over the Internet at large, I can’t speak about any personal experience. It is theoretically possible but I don’t know of any large scale examples or experiments.The underpinnings for IP Anycast that I have used were initially based on Quagga: https://github.com/QuaggaMore recently I’ve been working on projects that use a fork of Quagga called FRR (Free Range Router): https://docs.frrouting.Generally the TFTP and DHCP servers are not directly using IP Anycast, rather there is a load balancer in front of the servers at each site. Initially I used Pound for this, but then NGinX and then HAProxy became preferable. More recently MetalLB on Kubernetes has been the go-to load-balancer, and MetalLB integrates with BGP in a number of ways.I helped design and bootstrap a project to use BGP for IP Anycast in order to provide load-balanced DHCP and TFTP in OpenStack and OpenShift using OVN, which is related to OpenFlow. Here is the BGP plugin for OVN: https://docs.openstack.org/ovn-bgp-agent/latestI would be very curious to see any projects which are attempting to do this on the public Internet. Would you mind sharing a bit about your intended use case?Warm regards,-Dan Sneddon

Re: TFTP over anycast

[Bill Woodcock]

The system Ask is describing is the traditional method of using anycast to 
geographically load-balance long-lived flows.  The first time I did that was 
with FTP servers in Berkeley and Santa Cruz, in 1989. 

I did a bigger system, also load balancing FTP servers for Oracle, their 
public-facing documentation stores, with servers in San Jose and Washington DC, 
a couple of years later.   A couple of years further on and the World Wide Web 
was a thing, and everybody was doing it. 

-Bill


> On Feb 24, 2024, at 7:38 AM, Ask Bjørn Hansen  wrote:
> 
> 
> 
>>> On Feb 23, 2024, at 20:32, William Herrin  wrote:
>>> 
>>> The relay server `dhcplb` could, maybe, help in that scenario
>>> (dhcplb runs on the anycast IP, the “real” DHCP servers on
>>> unicast IPs behind dhcplb).
>> 
>> Although they used the word "anycast", they're just load balancing.
> 
> The idea is to run the relays on an anycasted IP (so the load balancer / 
> relay IP is anycasted).
> 
>> [….] Relying on ECMP for anycasted DHCP would be a disaster
>> during any sort of failure. Add or remove a single route from an ECMP
>> set and the hashed path selection changes for most of the connections.
> 
> 
> Consistent hashing (which I thought was widely supported now in ECMP 
> implementations) and a bit of automation in how announcements are added can 
> greatly mitigate this.
> 
> 
> 
> Ask

Re: TFTP over anycast

[Ask Bjørn Hansen]

> On Feb 23, 2024, at 20:32, William Herrin  wrote:
> 
>> The relay server `dhcplb` could, maybe, help in that scenario
>> (dhcplb runs on the anycast IP, the “real” DHCP servers on
>> unicast IPs behind dhcplb).
> 
> Although they used the word "anycast", they're just load balancing.

The idea is to run the relays on an anycasted IP (so the load balancer / relay 
IP is anycasted).

> [….] Relying on ECMP for anycasted DHCP would be a disaster
> during any sort of failure. Add or remove a single route from an ECMP
> set and the hashed path selection changes for most of the connections.


Consistent hashing (which I thought was widely supported now in ECMP 
implementations) and a bit of automation in how announcements are added can 
greatly mitigate this.



Ask

Re: TFTP over anycast

[William Herrin]

On Fri, Feb 23, 2024 at 6:34 PM Ask Bjørn Hansen  wrote:
> The relay server `dhcplb` could, maybe, help in that scenario
> (dhcplb runs on the anycast IP, the “real” DHCP servers on
> unicast IPs behind dhcplb).

Although they used the word "anycast", they're just load balancing.
Devices behind a load balancer are not "anycast," since the load
balancer explicitly decides which machine gets which transaction. Even
with clever load balancers like Linux Virtual Server in "routing" mode
where the back-end servers all share the virtual IP address, that's
load balancing, not anycast routing.

An IP is not "anycast" unless it moves via anycast routing. Anycast
routing means it's announced into the _routing protocol_ from multiple
sources on behalf of multiple distinct machines.

In their readme, they comment that their load balancer replaced
attempts to use anycast routing with equal cost multipath. That makes
good sense. Relying on ECMP for anycasted DHCP would be a disaster
during any sort of failure. Add or remove a single route from an ECMP
set and the hashed path selection changes for most of the connections.
All the DHCP renewals would very suddenly be going to the wrong DHCP
server. Where anycast works, it works because ECMP only rarely comes
into play.

Regards,
Bill Herrin


--
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: TFTP over anycast

[Ask Bjørn Hansen]

> On Feb 22, 2024, at 12:52, Thomas Mieslinger  wrote:
> 
> It becomes tricky for DHCP if a location has the same cost to more than
> one anycast Node. For this case we have setup a DHCP nodes in two
> datacenters using different local-preferences to simulate a failover
> active/passive setup.

The relay server `dhcplb` could, maybe, help in that scenario (dhcplb runs on 
the anycast IP, the “real” DHCP servers on unicast IPs behind dhcplb).

https://github.com/facebookincubator/dhcplb



Ask

RE: TFTP over anycast

[Adam Thompson]

Others have addressed some of the issues, but one easy win for DHCP (which is 
otherwise a PITA to make redundany in *any* way) is to (a) not block ICMP 
anywhere, including on the client devices, and (b) have the DHCP ping before 
assignment.  That’s not always on by default, and it’ll eliminate ~90% of the 
conflicts you would otherwise encounter if the anycast node isn’t extremely 
stable.  If you become aware of a distributed DHCP server that actually works 
well in this environment, that’s worth a post to the list all by itself.
-Adam

Adam Thompson
Consultant, Infrastructure Services
[cid:image001.png@01DA663E.AD5957D0]
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca<https://www.merlin.mb.ca/>
[cid:image002.png@01DA663E.AD5957D0]Chat with me on 
Teams<https://teams.microsoft.com/l/chat/0/0?users=athomp...@merlin.mb.ca>
[https://res-h3.public.cdn.office.net/assets/bookwithme/misc/CalendarPerson20px.png]<https://outlook.office.com/bookwithme/user/a3e87142ccc14503bf5ec7ea59afd...@merlin.mb.ca?anonymous&ep=bwmEmailSignature>
Book time to meet with 
me<https://outlook.office.com/bookwithme/user/a3e87142ccc14503bf5ec7ea59afd...@merlin.mb.ca?anonymous&ep=bwmEmailSignature>


From: NANOG  On Behalf Of 
Javier Gutierrez
Sent: Thursday, February 22, 2024 12:47 PM
To: nanog@nanog.org
Subject: TFTP over anycast

Hi,
I'm working on some DR design and we want to not only have this site as a DR 
but also performing some active/active for some of the services we hosts and I 
was wondering if someone had some experience with using anycast for TFTP or 
DHCP services?
What are some of the pains/challenges you experienced and things we should 
lookout for?

Any input is greatly appreciated.


Kind regards,



Javier Gutierrez

Re: TFTP over anycast

[Thomas Mieslinger]

I do NTP, DHCP, TFTP, DNS, HTTP anycast.

NTP, DNS and HTTP with ECMP, TFTP and DHCP as active/active on a per
Datacenter Basis.

These are small Datacenters with less than 50k Servers each.

In every datacenter an anycast node is active and the router just
chooses the shortest path.

It becomes tricky for DHCP if a location has the same cost to more than
one anycast Node. For this case we have setup a DHCP nodes in two
datacenters using different local-preferences to simulate a failover
active/passive setup.

Cheers
Thomas

Am 22.02.24 um 19:47 schrieb Javier Gutierrez:

Hi,
I'm working on some DR design and we want to not only have this site as
a DR but also performing some active/active for some of the services we
hosts and I was wondering if someone had some experience with using
anycast for TFTP or DHCP services?
What are some of the pains/challenges you experienced and things we
should lookout for?

Any input is greatly appreciated.

Kind regards,

*Javier Gutierrez*

Re: TFTP over anycast

[William Herrin]

On Thu, Feb 22, 2024 at 10:47 AM Javier Gutierrez
 wrote:
>  I was wondering if someone had some experience with using anycast for TFTP 
> or DHCP services?

Hi Javier,

Anycast for TFTP is more or less the same as anycast for TCP-based
protocols: it has corner cases which fail and fail hard, but otherwise
it works. Outside the corner cases, the failure mode for tftp clients
should be the same as the failure mode when the tftp server goes down
in the middle of a transfer and comes back up some time later.

The corner cases are variations on the theme that your routing causes
packets from a particular source to oscillate between the tftp
servers. In the corner cases, the tftp client can't communicate with
the -same- tftp server long enough to complete a transfer.

Experiments with anycast TCP suggest that the corner cases happen for
less than 1% of client sources, but when they do happen tend to be
persistent, affecting all communication between that client and the
anycast IP address for an extended duration, sometimes weeks or
months.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

TFTP over anycast

[Javier Gutierrez]

Hi,
I'm working on some DR design and we want to not only have this site as a DR 
but also performing some active/active for some of the services we hosts and I 
was wondering if someone had some experience with using anycast for TFTP or 
DHCP services?
What are some of the pains/challenges you experienced and things we should 
lookout for?

Any input is greatly appreciated.


Kind regards,



Javier Gutierrez

Re: Anycast but for egress

[Joel Jaeggli]

On 7/27/21 10:54, Vimal wrote:
> (Unsure if this is the right forum to ask this question, but here goes:)
>
> From what I understand, IP Anycast can be used to steer traffic into a
> server that's close to the client.
>
> I am curious if anyone here has/encountered a setup where they use
> anycast IP on their gateways... to have a predictable egress IP for
> their traffic, regardless of where they are located?

Stateless outbound load-balancing setups exist.

Example you have two  or more nat44 / nat64 / cgnat boxes behind a
common ecmp path with the same destination IP(s).  this is normally so
that you have more boxes that accumulate state rather than being bound
to a single one.

>
> For example, a search engine crawler could in principle have the same
> IP advertised all over the world, but it looks like they don't...  I
> wonder why?

So this is a somewhat different problem...

There's  no assurance that when you initiate a connection that the
return path will return to the same instance of your anycast
announcement  when the server on the other side  replies.

Effectively the initiating party needs a unicast address or you need
some out of band path to get an errant packet back to the correct host.

> -- 
> Vimal
>

Re: Anycast but for egress

[Christopher Morrow]

On Thu, Jul 29, 2021 at 4:58 PM Joe Maimon  wrote:

>
>
> Vimal wrote:
> > (Unsure if this is the right forum to ask this question, but here goes:)
> >
> > From what I understand, IP Anycast can be used to steer traffic into a
> > server that's close to the client.
> >
> > I am curious if anyone here has/encountered a setup where they use
> > anycast IP on their gateways... to have a predictable egress IP for
> > their traffic, regardless of where they are located?
> >
> > For example, a search engine crawler could in principle have the same
> > IP advertised all over the world, but it looks like they don't...  I
> > wonder why?
> >
> > --
> > Vimal
> >
> Its definitely possible, but would need a layer of software (kernel
> mode) on all the anycast holders synchronizing state to ensure
> asymmetric replies/connections get forwarded/shifted to the correct host.
>
>
is it actually that hard? isn't it more like:
  "use an outbound path local to that inbound path cone which NAT's (or
proxy's or...) to a small set of staticlly assigned addresses"

Provided you don't re-use the outbound addresses on different deployments
this should 'just work'[tm]

'anycast but outbound' is really: "get me local nat pools for my service by
locality"
I think this is, bascially, what every enterprise network in the world
does, effectively.


If the goals are worth that kind of effort is another question. And
> performance is likely to be "tricky".
>
>

Re: Anycast but for egress

[Joe Maimon]

Vimal wrote:

(Unsure if this is the right forum to ask this question, but here goes:)

From what I understand, IP Anycast can be used to steer traffic into a 
server that's close to the client.


I am curious if anyone here has/encountered a setup where they use 
anycast IP on their gateways... to have a predictable egress IP for 
their traffic, regardless of where they are located?


For example, a search engine crawler could in principle have the same 
IP advertised all over the world, but it looks like they don't...  I 
wonder why?


--
Vimal

Its definitely possible, but would need a layer of software (kernel 
mode) on all the anycast holders synchronizing state to ensure 
asymmetric replies/connections get forwarded/shifted to the correct host.


If the goals are worth that kind of effort is another question. And 
performance is likely to be "tricky".

Re: Anycast but for egress

[Vimal]

Great point.  We don't need geo-diversity for websites with the IP address
issue, so we could design for that case specially on a one-off basis.

For throughput it shouldn't be an issue where we're located, but we often
find websites serving different content based on the source IP of the
traffic.  So, having a presence closer to the user is useful.  But then
again, this is a different concern that's orthogonal to the original
question, because geo-ip doesn't make much sense with an anycast IP.  For
those websites that need a stable IP for NACLs *and* serve different
content based on source IP, we have to use the predictable 3-5 IPs per site
suggestion of yours.



On Wed, Jul 28, 2021 at 11:27 AM Glenn McGurrin via NANOG 
wrote:

> I'd had a similar thought/question, though keeping the geo diversity,
> you manage the crawlers, and are making contact individually with these
> sites from what you have stated (and so don't need a one size fit's all
> list for public posting), so why not have a restricted subset of the
> crawlers handle sites with these issues (which subset may be unique per
> site, which makes maintaining even load balancing not overly complex
> /limiting, especially as you are using nat anyway, so multiple servers
> can be behind each ip and that number can vary).  That let's you have
> geo diversity (or even multi cloud diversity) for every site, but each
> site that needs this IP whitelisting only needs 3-5 IP's at any site,
> but yet you can distribute load over a much larger overall set of
> machines and nat gateways.
>
> As I understand it even CDN's that anycast TCP (externally or internally
> [load balancing via routers and multi path]) do similar by spreading
> load over multiple IP's at the DNS layer first.
>
> As the transition to IPv6 happens you may have it easier as getting a
> large enough allocation to allow for splitting it out into multiple
> subnets advertised from different locations without providers dropping
> the route as too long a prefix is much easier on the v6 side, so you
> could give one /36 or /40 or even /44 out to whitelist but have /48's at
> each location.  For sites with ipv6 support that may help now, but it
> won't help all sites for quite some time, though the number that support
> v6 is slowly getting better.  For the foreseeable future you still need
> to handle the v4 side one way or another though.
>
> On 7/28/2021 10:21 AM, William Herrin wrote:
> > On Wed, Jul 28, 2021 at 6:04 AM Vimal  wrote:
> >> My intention is to run a web-crawling service on a public cloud. This
> service
> >> is geographically distributed, and therefore will run in multiple
> regions
> >> around the world inside AWS... this means there will be multiple AWS
> VPCs,
> >> each with their own NAT gateway, and traffic destined to websites
> >> that we crawl will appear to come from this NAT gateway's IP address.
> >
> > Hello,
> >
> > AWS does not provide the ability to attach anycasted IP addresses to a
> > NAT gateway, regardless of whether it would work, so that's the end of
> > your quest.
> >
> >> The reason I want a predictable IP is to communicate this IP to website
> >> owners so they can allow access from these IPs into their networks.
> >> I chose IP as an example; it can also be a subnet, but what I don't
> want to
> >> provide is a list of 100 different IP addresses without any
> predictability.
> >
> > If you bring your own IP addresses, you can attach a separate /24s of
> > them to your VPCs in each region, providing you with a single
> > predictable range of source addresses. You will find it difficult and
> > expensive to acquire that many IP addresses from the regional
> > registries for the purpose you describe.
> >
> >
> > Silly question but: for a web crawler, why do you care whether it has
> > the limited geographically distribution that a cloud service provides?
> > It's a parallel batch task. It doesn't exactly matter whether you have
> > minimum latency.
> >
> > Regards,
> > Bill Herrin
> >
> >
> >
>


-- 
Vimal

Re: Anycast but for egress

[Glenn McGurrin via NANOG]

I'd had a similar thought/question, though keeping the geo diversity, 
you manage the crawlers, and are making contact individually with these 
sites from what you have stated (and so don't need a one size fit's all 
list for public posting), so why not have a restricted subset of the 
crawlers handle sites with these issues (which subset may be unique per 
site, which makes maintaining even load balancing not overly complex 
/limiting, especially as you are using nat anyway, so multiple servers 
can be behind each ip and that number can vary).  That let's you have 
geo diversity (or even multi cloud diversity) for every site, but each 
site that needs this IP whitelisting only needs 3-5 IP's at any site, 
but yet you can distribute load over a much larger overall set of 
machines and nat gateways.


As I understand it even CDN's that anycast TCP (externally or internally 
[load balancing via routers and multi path]) do similar by spreading 
load over multiple IP's at the DNS layer first.


As the transition to IPv6 happens you may have it easier as getting a 
large enough allocation to allow for splitting it out into multiple 
subnets advertised from different locations without providers dropping 
the route as too long a prefix is much easier on the v6 side, so you 
could give one /36 or /40 or even /44 out to whitelist but have /48's at 
each location.  For sites with ipv6 support that may help now, but it 
won't help all sites for quite some time, though the number that support 
v6 is slowly getting better.  For the foreseeable future you still need 
to handle the v4 side one way or another though.


On 7/28/2021 10:21 AM, William Herrin wrote:

On Wed, Jul 28, 2021 at 6:04 AM Vimal  wrote:

My intention is to run a web-crawling service on a public cloud. This service
is geographically distributed, and therefore will run in multiple regions
around the world inside AWS... this means there will be multiple AWS VPCs,
each with their own NAT gateway, and traffic destined to websites
that we crawl will appear to come from this NAT gateway's IP address.


Hello,

AWS does not provide the ability to attach anycasted IP addresses to a
NAT gateway, regardless of whether it would work, so that's the end of
your quest.


The reason I want a predictable IP is to communicate this IP to website
owners so they can allow access from these IPs into their networks.
I chose IP as an example; it can also be a subnet, but what I don't want to
provide is a list of 100 different IP addresses without any predictability.


If you bring your own IP addresses, you can attach a separate /24s of
them to your VPCs in each region, providing you with a single
predictable range of source addresses. You will find it difficult and
expensive to acquire that many IP addresses from the regional
registries for the purpose you describe.


Silly question but: for a web crawler, why do you care whether it has
the limited geographically distribution that a cloud service provides?
It's a parallel batch task. It doesn't exactly matter whether you have
minimum latency.

Regards,
Bill Herrin

Re: Anycast but for egress

[Mark Tinka]

On 7/28/21 17:09, Bill Woodcock wrote:


I was about to say something about us having equal success over 105 or so 
countries, when I came to the realization that inviting quantitative 
comparisons of manhood with Mark is the very definition of folly.  :-)


Well, we are nowhere close to the 105 countries PCH boasts. That's a 
whole other level of scale :-).


Impressed!



Anyway, yeah, the folks who were scared of anycast in the 1990s were running 
from shadows, not basing it on experience or data.  In the real world, the 
number of stateful flows affected by route changes is dwarfed by those 
disrupted by other causes, and is immeasurably small.  And when they do crop up 
on the radar, it’s almost always someone’s equal-cost-multi-path gone wrong, 
rather than an actual shift.  So, not an issue at all in the real world, just 
in the imaginations of folks who thought TCP was a complex thing reserved for 
the specific use-cases that they’d already conceived of in the 1980s.  Took a 
while to get beyond their protestations, but here we are in the 21st century.  
Planck's principle holds.  Science progresses one funeral at a time.


100%.

Mark.

Re: Anycast but for egress

[Bill Woodcock]

> On Jul 28, 2021, at 3:21 AM, Mark Tinka  wrote:
> On 7/28/21 01:16, Daniel Corbe wrote:
> 
>>> This is interesting... I wonder whether Anycast will still have some 
>>> failure modes and break TCP connections if routing (configuration) were to 
>>> change?  I checked the PDF linked by Bill Woodcock... while the methodology 
>>> is the same from 20y ago, would the data still be the same (order of 
>>> magnitude)? :)
> 
> We are Anycast'ing DNS (authoritative and recursive), NTP and TACACS+. All 
> works well, across 11 or so countries.

I was about to say something about us having equal success over 105 or so 
countries, when I came to the realization that inviting quantitative 
comparisons of manhood with Mark is the very definition of folly.  :-)

Anyway, yeah, the folks who were scared of anycast in the 1990s were running 
from shadows, not basing it on experience or data.  In the real world, the 
number of stateful flows affected by route changes is dwarfed by those 
disrupted by other causes, and is immeasurably small.  And when they do crop up 
on the radar, it’s almost always someone’s equal-cost-multi-path gone wrong, 
rather than an actual shift.  So, not an issue at all in the real world, just 
in the imaginations of folks who thought TCP was a complex thing reserved for 
the specific use-cases that they’d already conceived of in the 1980s.  Took a 
while to get beyond their protestations, but here we are in the 21st century.  
Planck's principle holds.  Science progresses one funeral at a time.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: Anycast but for egress

[Bill Woodcock]

> On Jul 27, 2021, at 6:15 PM, Vimal  wrote:
> 
> AWS Global Accelerator gives anycast IPs that's good for ingress, but my 
> original question was about having predictable egress IPs.
> 
> It looks like having a few EIPs/a contiguous network block is the way to go.

Yes.  Predictable and unchanging (but each unique per location) static IP 
addresses is what you’re looking for.

It would be a huge convenience to others if you could specify a single 
contiguous CIDR block for others to “permit” in their access control lists, but 
alas that would be very difficult as well…  Since BGP announcements generally 
need to be aggregated up to at least a /24 or a /48 (though people are less 
strict on the v6 side), each group of hosts numbered from the same block of 
that size would need to have internally contiguous convex routing, meaning that 
it would have to be interconnected by its own network (albeit that could be 
tunnels) and accept inbound traffic at any point on the surface of that 
network, backhauling it to the appropriate location.  So if you wanted to be 
able to identify a single CIDR block with eight locations in it, you’d either 
need to specify a /24 that was 97% wasted, and was fully internally 
interconnected (i.e. no efficiencies in localizing traffic), or you’d need to 
advertise eight /24s, which would aggregate up to a single /21, which was 99.6% 
wasted.

So, you can see why the combination of scarce IPv4 addresses, scarce BGP 
routing slots, and content routing tricks often don’t play well together.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: Anycast but for egress

[William Herrin]

On Wed, Jul 28, 2021 at 6:04 AM Vimal  wrote:
> My intention is to run a web-crawling service on a public cloud. This service
> is geographically distributed, and therefore will run in multiple regions
> around the world inside AWS... this means there will be multiple AWS VPCs,
> each with their own NAT gateway, and traffic destined to websites
> that we crawl will appear to come from this NAT gateway's IP address.

Hello,

AWS does not provide the ability to attach anycasted IP addresses to a
NAT gateway, regardless of whether it would work, so that's the end of
your quest.

> The reason I want a predictable IP is to communicate this IP to website
> owners so they can allow access from these IPs into their networks.
> I chose IP as an example; it can also be a subnet, but what I don't want to
> provide is a list of 100 different IP addresses without any predictability.

If you bring your own IP addresses, you can attach a separate /24s of
them to your VPCs in each region, providing you with a single
predictable range of source addresses. You will find it difficult and
expensive to acquire that many IP addresses from the regional
registries for the purpose you describe.


Silly question but: for a web crawler, why do you care whether it has
the limited geographically distribution that a cloud service provides?
It's a parallel batch task. It doesn't exactly matter whether you have
minimum latency.

Regards,
Bill Herrin



-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Anycast but for egress

[Randy Bush]

we, verio, did anycast tcp streaming (hour long) of the tony awards in
about '96.  solid.

randy

---
ra...@psg.com
`gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
signatures are back, thanks to dmarc header butchery

Re: Anycast but for egress

[Vimal]

AWS Global Accelerator gives anycast IPs that's good for ingress, but my
original question was about having predictable egress IPs.

It looks like having a few EIPs/a contiguous network block is the way to go.

Thanks!

On Tue, Jul 27, 2021 at 4:30 PM Andras Toth  wrote:

> Since you mentioned AWS, have you tried AWS Global Accelerator? You get a
> pair of globally anycasted static IPs.
> https://aws.amazon.com/global-accelerator/
>
> Another alternative is to request a contiguous IP range of EIPs (/28 or
> /24 etc) that you can use for your EC2 instances or VPC resources.
>
> Andras
>
> On 28 Jul 2021, at 09:19, Daniel Corbe  wrote:
>
> 
>
> On Jul 27, 2021, at 17:20, Vimal  wrote:
>
>
> Hi all, great replies. :) Let me clarify my initial question, and then
> respond one by one:
>
>
> My intention is to run a web-crawling service on a public cloud. This
> service is geographically distributed, and therefore will run in multiple
> regions around the world inside AWS... this means there will be multiple
> AWS VPCs, each with their own NAT gateway, and traffic destined to websites
> that we crawl will appear to come from this NAT gateway's IP address.
>
>
> The reason I want a predictable IP is to communicate this IP to website
> owners so they can allow access from these IPs into their networks.  I
> chose IP as an example; it can also be a subnet, but what I don't want to
> provide is a list of 100 different IP addresses without any predictability.
>
>
> I understand that this is not perfect, and would frankly not be my
> preferred approach to solve the problem but we've had requests of this
> nature from websites to create an allowlist of a limited number of
> predictable IPs so it doesn't trip their IDSs/other systems they might
> have... so we're trying to see how well it would work in practice.  For the
> moment, let's set aside the issue as to whether AWS will even let me
> advertise the same IP on all my VPC NAT gateways, and just look at whether
> it's technically feasible.  My gut feeling is that this wouldn't work well
> in practice, but I wanted to ask the experts here...
>
>
> Also, pointers on what the best practices for solving this issue are most
> welcome, so I can reference those who ask for IP addresses to this
> discussion and follow recommendations here.
>
>
> Onto the responses:
>
>
> @o...@delong.com and @wo...@pch.net athomp...@merlin.mb.ca
>
> Because there’s no good/reliable way to get the replies back to the
> correct initiating host.
>
>
> When my clients make connections outbound to anycast addresses, the
> destination is more-or-less stable, and the replies come back to the
> client's unique IP, so anycast works in that direction.  The guarantees are
> not present in the reverse direction.
>
>
> Yes, this makes sense as the destination can be anywhere around the world,
> and that routing is asymmetric as others mentioned.  However, if the
> destination service is "close" (in the routing metric sense) to the
> initiating host, anycast return IP ought to work well, right?  I understand
> this is a very important caveat and impractical to implement correctly in
> the real world.
>
>
> We use our IGP (IS-IS) for our Anycast services. We find it to be very
>
> basic, and as such, very predictable.
>
>
> This is interesting... I wonder whether Anycast will still have some
> failure modes and break TCP connections if routing (configuration) were to
> change?  I checked the PDF linked by Bill Woodcock... while the methodology
> is the same from 20y ago, would the data still be the same (order of
> magnitude)? :)
>
>
> https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf (p38)
>
> "Limited operational data shows underlying instability to be on
>
> the order of one flow per ten thousand per hour of duration."
>
>
> @dan...@corbe.net, @m...@netfire.net,
>
> Unless you’re twisting knobs, egress traffic should already exit your
> network at the closest possible egress point to its origin.  Is your
> intention to carry the traffic for longer than that?
>
> No, but I hope my intention is more clear in this email.  It's to have a
> predictable egress IP to simplify firewall rules.
>
>
> thanks all!!
>
>
>
> On Tue, Jul 27, 2021 at 12:25 PM Adam Thompson 
> wrote:
>
> Without any sarcasm: to make it harder to block.
>
> If, say, Google, always crawled your site from 8.8.1.2 (random made-up
> example) then you would see a not-insignificant number of hosts and
> networks null-routing that IP.  I have no idea why someone would do so, but
> I've seen it done many time

Re: Anycast but for egress

[Vimal]

On AWS once we purchase EIPs, they are allocated to our account and so we
can assign them to VPC NAT gateways.  That's our current plan.

On Tue, Jul 27, 2021 at 4:16 PM Daniel Corbe  wrote:

>
> > On Jul 27, 2021, at 17:20, Vimal  wrote:
> >
> > Hi all, great replies. :) Let me clarify my initial question, and then
> respond one by one:
> >
> > My intention is to run a web-crawling service on a public cloud. This
> service is geographically distributed, and therefore will run in multiple
> regions around the world inside AWS... this means there will be multiple
> AWS VPCs, each with their own NAT gateway, and traffic destined to websites
> that we crawl will appear to come from this NAT gateway's IP address.
> >
> > The reason I want a predictable IP is to communicate this IP to website
> owners so they can allow access from these IPs into their networks.  I
> chose IP as an example; it can also be a subnet, but what I don't want to
> provide is a list of 100 different IP addresses without any predictability.
> >
> > I understand that this is not perfect, and would frankly not be my
> preferred approach to solve the problem but we've had requests of this
> nature from websites to create an allowlist of a limited number of
> predictable IPs so it doesn't trip their IDSs/other systems they might
> have... so we're trying to see how well it would work in practice.  For the
> moment, let's set aside the issue as to whether AWS will even let me
> advertise the same IP on all my VPC NAT gateways, and just look at whether
> it's technically feasible.  My gut feeling is that this wouldn't work well
> in practice, but I wanted to ask the experts here...
> >
> > Also, pointers on what the best practices for solving this issue are
> most welcome, so I can reference those who ask for IP addresses to this
> discussion and follow recommendations here.
> >
> > Onto the responses:
> >
> > @o...@delong.com and @wo...@pch.net athomp...@merlin.mb.ca
> > > Because there’s no good/reliable way to get the replies back to the
> correct initiating host.
> >
> > > When my clients make connections outbound to anycast addresses, the
> destination is more-or-less stable, and the replies come back to the
> client's unique IP, so anycast works in that direction.  The guarantees are
> not present in the reverse direction.
> >
> > Yes, this makes sense as the destination can be anywhere around the
> world, and that routing is asymmetric as others mentioned.  However, if the
> destination service is "close" (in the routing metric sense) to the
> initiating host, anycast return IP ought to work well, right?  I understand
> this is a very important caveat and impractical to implement correctly in
> the real world.
> >
> > > We use our IGP (IS-IS) for our Anycast services. We find it to be very
> > basic, and as such, very predictable.
> >
> > This is interesting... I wonder whether Anycast will still have some
> failure modes and break TCP connections if routing (configuration) were to
> change?  I checked the PDF linked by Bill Woodcock... while the methodology
> is the same from 20y ago, would the data still be the same (order of
> magnitude)? :)
> >
> > https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf (p38)
> > "Limited operational data shows underlying instability to be on
> > the order of one flow per ten thousand per hour of duration."
> >
> > @dan...@corbe.net, @m...@netfire.net,
> > > Unless you’re twisting knobs, egress traffic should already exit your
> network at the closest possible egress point to its origin.  Is your
> intention to carry the traffic for longer than that?
> > No, but I hope my intention is more clear in this email.  It's to have a
> predictable egress IP to simplify firewall rules.
> >
> > thanks all!!
> >
> >
> > On Tue, Jul 27, 2021 at 12:25 PM Adam Thompson 
> wrote:
> > Without any sarcasm: to make it harder to block.
> > If, say, Google, always crawled your site from 8.8.1.2 (random made-up
> example) then you would see a not-insignificant number of hosts and
> networks null-routing that IP.  I have no idea why someone would do so, but
> I've seen it done many times.  Mostly by people who don't understand how
> un-special they are on the internet.  Also it would trigger IDS/IPS systems
> all over the place, having gobs and gobs of connections coming from a
> single IP.
> >
> > That's setting aside the technical issues involved; routing is often
> asymmetric, i.e. the return packet takes a different pa

Re: Anycast but for egress

[Vimal]

Hi all, great replies. :) Let me clarify my initial question, and then
respond one by one:

My intention is to run a web-crawling service on a public cloud. This
service is geographically distributed, and therefore will run in multiple
regions around the world inside AWS... this means there will be multiple
AWS VPCs, each with their own NAT gateway, and traffic destined to websites
that we crawl will appear to come from this NAT gateway's IP address.

The reason I want a predictable IP is to communicate this IP to
website owners so they can allow access from these IPs into their
networks.  I chose IP as an example; it can also be a subnet, but what I
don't want to provide is a list of 100 different IP addresses without any
predictability.

I understand that this is not perfect, and would frankly not be my
preferred approach to solve the problem but we've had requests of this
nature from websites to create an allowlist of a limited number of
predictable IPs so it doesn't trip their IDSs/other systems they might
have... so we're trying to see how well it would work in practice.  For the
moment, let's set aside the issue as to whether AWS will even let me
advertise the same IP on all my VPC NAT gateways, and just look at whether
it's technically feasible.  My gut feeling is that this wouldn't work well
in practice, but I wanted to ask the experts here...

Also, pointers on what the best practices for solving this issue are most
welcome, so I can reference those who ask for IP addresses to this
discussion and follow recommendations here.

Onto the responses:

@o...@delong.com and @wo...@pch.net athomp...@merlin.mb.ca
> Because there’s no good/reliable way to get the replies back to the
correct initiating host.

> When my clients make connections outbound to anycast addresses, the
destination is more-or-less stable, and the replies come back to the
client's unique IP, so anycast works in that direction.  The guarantees are
not present in the reverse direction.

Yes, this makes sense as the destination can be anywhere around the world,
and that routing is asymmetric as others mentioned.  However, if the
destination service is "close" (in the routing metric sense) to the
initiating host, anycast return IP ought to work well, right?  I understand
this is a very important caveat and impractical to implement correctly in
the real world.

> We use our IGP (IS-IS) for our Anycast services. We find it to be very
basic, and as such, very predictable.

This is interesting... I wonder whether Anycast will still have some
failure modes and break TCP connections if routing (configuration) were to
change?  I checked the PDF linked by Bill Woodcock... while the methodology
is the same from 20y ago, would the data still be the same (order of
magnitude)? :)

https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf (p38)
"Limited operational data shows underlying instability to be on
the order of one flow per ten thousand per hour of duration."

@dan...@corbe.net, @m...@netfire.net,
> Unless you’re twisting knobs, egress traffic should already exit your
network at the closest possible egress point to its origin.  Is your
intention to carry the traffic for longer than that?
No, but I hope my intention is more clear in this email.  It's to have a
predictable egress IP to simplify firewall rules.

thanks all!!


On Tue, Jul 27, 2021 at 12:25 PM Adam Thompson 
wrote:

> Without any sarcasm: to make it harder to block.
> If, say, Google, always crawled your site from 8.8.1.2 (random made-up
> example) then you would see a not-insignificant number of hosts and
> networks null-routing that IP.  I have no idea why someone would do so, but
> I've seen it done many times.  Mostly by people who don't understand how
> un-special they are on the internet.  Also it would trigger IDS/IPS systems
> all over the place, having gobs and gobs of connections coming from a
> single IP.
>
> That's setting aside the technical issues involved; routing is often
> asymmetric, i.e. the return packet takes a different path than the inbound
> packet.  So it would, as Owen implied, be nearly impossible to ensure the
> reply packets got back to the correct TCP stack.  As an example, I'm
> multi-homed and use path-prepending, so if a packet claiming to be from
> 8.8.8.8 arrived on one of my commercial links, I would send the reply out
> the cheapest link, which in my case is a flat-rate R&E network (that has a
> path to Google), thus ensuring the reply does *not* get to the
> originating anycast node.
>
> When my clients make connections outbound to anycast addresses, the
> destination is more-or-less stable, and the replies come back to the
> client's unique IP, so anycast works in that direction.  The guarantees are
> not present in the reverse direction.
>
> The l

Re: Anycast but for egress

[Mark Tinka]

On 7/28/21 01:16, Daniel Corbe wrote:


This is interesting... I wonder whether Anycast will still have some failure 
modes and break TCP connections if routing (configuration) were to change?  I 
checked the PDF linked by Bill Woodcock... while the methodology is the same 
from 20y ago, would the data still be the same (order of magnitude)? :)


In our small experience, not at all.

We are Anycast'ing DNS (authoritative and recursive), NTP and TACACS+. 
All works well, across 11 or so countries.


Mark.

Re: Anycast but for egress

[Baldur Norddahl]

Here is what I think would happen if you were to try this setup. Let's
assume you deployed in eu-west-2 (London) and eu-central-1 (Frankfurt). You
would find that you could successfully connect to a number of networks but
also that some of them would work from the "wrong" site. Eg. you would have
clients in London that require you to use the Frankfurt instance to connect
to. You would also find a lot of networks that you could not connect to
from either site. And you would have some instability where some networks
at random switch between these states. For example you could have a client
that one day works from London, the next day it is Frankfurt and then
someday it won't be working at all.

As you add more sites, you also add more ways for the traffic to end up in
the wrong places. You could have something that works with just the two
sites above but then when you add eu-west-1 (Ireland) you could suddenly
not connect to them from any of the three sites.

There IS a possible solution which is to tunnel the traffic back to the
correct site. This still requires you to use unique IP addresses for each
site, but all addresses could be in the same subnet. You would have IP
a.b.c.1 to be London, a.b.c.2 Frankfurt, a.b.c.3 Ireland. Then if London
receives traffic to a.b.c.2 you would have a tunnel to send the traffic to
Frankfurt.

Regards,

Baldur


On Wed, Jul 28, 2021 at 11:07 AM Baldur Norddahl 
wrote:

>
>
>> > On Jul 27, 2021, at 17:20, Vimal  wrote:
>> > Yes, this makes sense as the destination can be anywhere around the
>> world, and that routing is asymmetric as others mentioned.  However, if the
>> destination service is "close" (in the routing metric sense) to the
>> initiating host, anycast return IP ought to work well, right?  I understand
>> this is a very important caveat and impractical to implement correctly in
>> the real world.
>>
>
> No, there is no such thing as "close". You could have a direct peering
> with some ISP and have them still deliver the responses on the other side
> of earth. You do not control the routing of other networks and can not be
> sure what they will do.
>
> For larger networks you may also have multiple peering points. Say you
> have a peering with them in city A and city B. How do you know which of
> their IP ranges are closer to A or B? You don't. And the same goes for
> them, they have no idea if you prefer A or B. Therefore you could select A
> and they may reply to B. They may even load balance between A and B if you
> are really unlucky.
>
> Routing is asymmetric. That means you have absolutely no idea where the
> replies end up going. Often it will not be what you think is "close".
>
> I do not run anycast, but I understand that the usual way of dealing with
> these issues is to do as little as possible with anycast before redirecting
> to a unicast address. For example you could have just your DNS on anycast
> and each site would reply with unique unicast addresses. Since DNS is just
> a single pair of UDP request/response, with the first packet originating
> from a unicast client, this works well.
>
> Regards,
>
> Baldur
>
>
>

Re: Anycast but for egress

[Baldur Norddahl]

>
> > On Jul 27, 2021, at 17:20, Vimal  wrote:
> > Yes, this makes sense as the destination can be anywhere around the
> world, and that routing is asymmetric as others mentioned.  However, if the
> destination service is "close" (in the routing metric sense) to the
> initiating host, anycast return IP ought to work well, right?  I understand
> this is a very important caveat and impractical to implement correctly in
> the real world.
>

No, there is no such thing as "close". You could have a direct peering with
some ISP and have them still deliver the responses on the other side of
earth. You do not control the routing of other networks and can not be sure
what they will do.

For larger networks you may also have multiple peering points. Say you have
a peering with them in city A and city B. How do you know which of their IP
ranges are closer to A or B? You don't. And the same goes for them, they
have no idea if you prefer A or B. Therefore you could select A and they
may reply to B. They may even load balance between A and B if you are
really unlucky.

Routing is asymmetric. That means you have absolutely no idea where the
replies end up going. Often it will not be what you think is "close".

I do not run anycast, but I understand that the usual way of dealing with
these issues is to do as little as possible with anycast before redirecting
to a unicast address. For example you could have just your DNS on anycast
and each site would reply with unique unicast addresses. Since DNS is just
a single pair of UDP request/response, with the first packet originating
from a unicast client, this works well.

Regards,

Baldur

Re: Anycast but for egress

[Andras Toth]

Since you mentioned AWS, have you tried AWS Global Accelerator? You get a pair 
of globally anycasted static IPs.
https://aws.amazon.com/global-accelerator/

Another alternative is to request a contiguous IP range of EIPs (/28 or /24 
etc) that you can use for your EC2 instances or VPC resources.

Andras

> On 28 Jul 2021, at 09:19, Daniel Corbe  wrote:
> 
> 
>> On Jul 27, 2021, at 17:20, Vimal  wrote:
>> 
>> Hi all, great replies. :) Let me clarify my initial question, and then 
>> respond one by one:
>> 
>> My intention is to run a web-crawling service on a public cloud. This 
>> service is geographically distributed, and therefore will run in multiple 
>> regions around the world inside AWS... this means there will be multiple AWS 
>> VPCs, each with their own NAT gateway, and traffic destined to websites that 
>> we crawl will appear to come from this NAT gateway's IP address.
>> 
>> The reason I want a predictable IP is to communicate this IP to website 
>> owners so they can allow access from these IPs into their networks.  I chose 
>> IP as an example; it can also be a subnet, but what I don't want to provide 
>> is a list of 100 different IP addresses without any predictability.
>> 
>> I understand that this is not perfect, and would frankly not be my preferred 
>> approach to solve the problem but we've had requests of this nature from 
>> websites to create an allowlist of a limited number of predictable IPs so it 
>> doesn't trip their IDSs/other systems they might have... so we're trying to 
>> see how well it would work in practice.  For the moment, let's set aside the 
>> issue as to whether AWS will even let me advertise the same IP on all my VPC 
>> NAT gateways, and just look at whether it's technically feasible.  My gut 
>> feeling is that this wouldn't work well in practice, but I wanted to ask the 
>> experts here...
>> 
>> Also, pointers on what the best practices for solving this issue are most 
>> welcome, so I can reference those who ask for IP addresses to this 
>> discussion and follow recommendations here.
>> 
>> Onto the responses:
>> 
>> @o...@delong.com and @wo...@pch.net athomp...@merlin.mb.ca
>>> Because there’s no good/reliable way to get the replies back to the correct 
>>> initiating host.
>> 
>>> When my clients make connections outbound to anycast addresses, the 
>>> destination is more-or-less stable, and the replies come back to the 
>>> client's unique IP, so anycast works in that direction.  The guarantees are 
>>> not present in the reverse direction.
>> 
>> Yes, this makes sense as the destination can be anywhere around the world, 
>> and that routing is asymmetric as others mentioned.  However, if the 
>> destination service is "close" (in the routing metric sense) to the 
>> initiating host, anycast return IP ought to work well, right?  I understand 
>> this is a very important caveat and impractical to implement correctly in 
>> the real world.
>> 
>>> We use our IGP (IS-IS) for our Anycast services. We find it to be very
>> basic, and as such, very predictable.
>> 
>> This is interesting... I wonder whether Anycast will still have some failure 
>> modes and break TCP connections if routing (configuration) were to change?  
>> I checked the PDF linked by Bill Woodcock... while the methodology is the 
>> same from 20y ago, would the data still be the same (order of magnitude)? :)
>> 
>> https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf (p38)
>> "Limited operational data shows underlying instability to be on
>> the order of one flow per ten thousand per hour of duration."
>> 
>> @dan...@corbe.net, @m...@netfire.net, 
>>> Unless you’re twisting knobs, egress traffic should already exit your 
>>> network at the closest possible egress point to its origin.  Is your 
>>> intention to carry the traffic for longer than that?
>> No, but I hope my intention is more clear in this email.  It's to have a 
>> predictable egress IP to simplify firewall rules.
>> 
>> thanks all!!
>> 
>> 
>> On Tue, Jul 27, 2021 at 12:25 PM Adam Thompson  
>> wrote:
>> Without any sarcasm: to make it harder to block.
>> If, say, Google, always crawled your site from 8.8.1.2 (random made-up 
>> example) then you would see a not-insignificant number of hosts and networks 
>> null-routing that IP.  I have no idea why someone would do so, but I've seen 
>> it done many times.  Mostly by people who don't und

Re: Anycast but for egress

[Daniel Corbe]

 
> On Jul 27, 2021, at 17:20, Vimal  wrote:
> 
> Hi all, great replies. :) Let me clarify my initial question, and then 
> respond one by one:
> 
> My intention is to run a web-crawling service on a public cloud. This service 
> is geographically distributed, and therefore will run in multiple regions 
> around the world inside AWS... this means there will be multiple AWS VPCs, 
> each with their own NAT gateway, and traffic destined to websites that we 
> crawl will appear to come from this NAT gateway's IP address.
> 
> The reason I want a predictable IP is to communicate this IP to website 
> owners so they can allow access from these IPs into their networks.  I chose 
> IP as an example; it can also be a subnet, but what I don't want to provide 
> is a list of 100 different IP addresses without any predictability.
> 
> I understand that this is not perfect, and would frankly not be my preferred 
> approach to solve the problem but we've had requests of this nature from 
> websites to create an allowlist of a limited number of predictable IPs so it 
> doesn't trip their IDSs/other systems they might have... so we're trying to 
> see how well it would work in practice.  For the moment, let's set aside the 
> issue as to whether AWS will even let me advertise the same IP on all my VPC 
> NAT gateways, and just look at whether it's technically feasible.  My gut 
> feeling is that this wouldn't work well in practice, but I wanted to ask the 
> experts here...
> 
> Also, pointers on what the best practices for solving this issue are most 
> welcome, so I can reference those who ask for IP addresses to this discussion 
> and follow recommendations here.
> 
> Onto the responses:
> 
> @o...@delong.com and @wo...@pch.net athomp...@merlin.mb.ca
> > Because there’s no good/reliable way to get the replies back to the correct 
> > initiating host. 
> 
> > When my clients make connections outbound to anycast addresses, the 
> > destination is more-or-less stable, and the replies come back to the 
> > client's unique IP, so anycast works in that direction.  The guarantees are 
> > not present in the reverse direction.
> 
> Yes, this makes sense as the destination can be anywhere around the world, 
> and that routing is asymmetric as others mentioned.  However, if the 
> destination service is "close" (in the routing metric sense) to the 
> initiating host, anycast return IP ought to work well, right?  I understand 
> this is a very important caveat and impractical to implement correctly in the 
> real world.
> 
> > We use our IGP (IS-IS) for our Anycast services. We find it to be very
> basic, and as such, very predictable.
> 
> This is interesting... I wonder whether Anycast will still have some failure 
> modes and break TCP connections if routing (configuration) were to change?  I 
> checked the PDF linked by Bill Woodcock... while the methodology is the same 
> from 20y ago, would the data still be the same (order of magnitude)? :)
> 
> https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf (p38)
> "Limited operational data shows underlying instability to be on
> the order of one flow per ten thousand per hour of duration."
> 
> @dan...@corbe.net, @m...@netfire.net, 
> > Unless you’re twisting knobs, egress traffic should already exit your 
> > network at the closest possible egress point to its origin.  Is your 
> > intention to carry the traffic for longer than that?
> No, but I hope my intention is more clear in this email.  It's to have a 
> predictable egress IP to simplify firewall rules.
> 
> thanks all!!
> 
> 
> On Tue, Jul 27, 2021 at 12:25 PM Adam Thompson  wrote:
> Without any sarcasm: to make it harder to block.
> If, say, Google, always crawled your site from 8.8.1.2 (random made-up 
> example) then you would see a not-insignificant number of hosts and networks 
> null-routing that IP.  I have no idea why someone would do so, but I've seen 
> it done many times.  Mostly by people who don't understand how un-special 
> they are on the internet.  Also it would trigger IDS/IPS systems all over the 
> place, having gobs and gobs of connections coming from a single IP.
> 
> That's setting aside the technical issues involved; routing is often 
> asymmetric, i.e. the return packet takes a different path than the inbound 
> packet.  So it would, as Owen implied, be nearly impossible to ensure the 
> reply packets got back to the correct TCP stack.  As an example, I'm 
> multi-homed and use path-prepending, so if a packet claiming to be from 
> 8.8.8.8 arrived on one of my commercial links, I would send the reply out the 

Re: Anycast but for egress

[Mark Tinka]

On 7/27/21 20:48, Bill Woodcock wrote:


In practice, that means that services are bound to a common shared address (an 
“anycast service address”) as those services are deployed on servers in 
different locations.  The service address is advertised into the BGP routing 
infrastructure.  Clients send packets to the service address, and the BGP 
routing infrastructure routes each packet on the shortest path to its 
destination, without knowing that there are multiple destinations.


We use our IGP (IS-IS) for our Anycast services. We find it to be very 
basic, and as such, very predictable.


Of course, if you don't run your own AS for the services you provide, or 
operate your own backbone, this is probably not a viable option.


Mark.

Re: Anycast but for egress

[Adam Thompson]

Without any sarcasm: to make it harder to block.
If, say, Google, always crawled your site from 8.8.1.2 (random made-up example) 
then you would see a not-insignificant number of hosts and networks 
null-routing that IP.  I have no idea why someone would do so, but I've seen it 
done many times.  Mostly by people who don't understand how un-special they are 
on the internet.  Also it would trigger IDS/IPS systems all over the place, 
having gobs and gobs of connections coming from a single IP.

That's setting aside the technical issues involved; routing is often 
asymmetric, i.e. the return packet takes a different path than the inbound 
packet.  So it would, as Owen implied, be nearly impossible to ensure the reply 
packets got back to the correct TCP stack.  As an example, I'm multi-homed and 
use path-prepending, so if a packet claiming to be from 8.8.8.8 arrived on one 
of my commercial links, I would send the reply out the cheapest link, which in 
my case is a flat-rate R&E network (that has a path to Google), thus ensuring 
the reply does not get to the originating anycast node.

When my clients make connections outbound to anycast addresses, the destination 
is more-or-less stable, and the replies come back to the client's unique IP, so 
anycast works in that direction.  The guarantees are not present in the reverse 
direction.

The logical extremity of this is that it would be nearly impossible for two 
anycast addresses to establish a TCP connection to each other.  (In general.  
There will be lots of local cases where it does happen to work, by coincidence.)

You'll find that even anycast nodes do not make connections outbound using 
their anycast address, pretty much for these reasons.

-Adam

Adam Thompson
Consultant, Infrastructure Services
[1593169877849]
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athomp...@merlin.mb.ca<mailto:athomp...@merlin.mb.ca>
www.merlin.mb.ca<http://www.merlin.mb.ca/>

From: NANOG  on behalf of Vimal 

Sent: July 27, 2021 12:54
To: nanog@nanog.org 
Subject: Anycast but for egress

(Unsure if this is the right forum to ask this question, but here goes:)

>From what I understand, IP Anycast can be used to steer traffic into a server 
>that's close to the client.

I am curious if anyone here has/encountered a setup where they use anycast IP 
on their gateways... to have a predictable egress IP for their traffic, 
regardless of where they are located?

For example, a search engine crawler could in principle have the same IP 
advertised all over the world, but it looks like they don't...  I wonder why?

--
Vimal

Re: Anycast but for egress

[Matt Harris]

Matt Harris|Infrastructure Lead
816-256-5446|Direct
Looking for help?
Helpdesk|Email Support
We build customized end-to-end technology solutions powered by NetFire Cloud.
On Tue, Jul 27, 2021 at 1:29 PM Vimal  wrote:

> (Unsure if this is the right forum to ask this question, but here goes:)
>
> From what I understand, IP Anycast can be used to steer traffic into a
> server that's close to the client.
>
> I am curious if anyone here has/encountered a setup where they use anycast
> IP on their gateways... to have a predictable egress IP for their traffic,
> regardless of where they are located?
>
> For example, a search engine crawler could in principle have the same IP
> advertised all over the world, but it looks like they don't...  I wonder
> why?
>

Hi Vimal,
I'm not sure I see what the benefit would be here. Maybe if you explain
what exactly you're trying to accomplish, someone can point you in the
right direction? What you're describing here isn't really analogous to how
anycast works. Anycast isn't really a protocol in and of itself, it's
simply the act of advertising IP space externally in a diverse way and then
routing that traffic to the nearest capable host once it's inside your
network, rather than to a single very specific host that exists in one
specific place.

- mdh

Re: Anycast but for egress

[Bill Woodcock]

> On Jul 27, 2021, at 10:54 AM, Vimal  wrote:
> 
> (Unsure if this is the right forum to ask this question

Sure, why not…  There isn’t anywhere more appropriate, really.

> From what I understand, IP Anycast can be used to steer traffic into a server 
> that's close to the client.

That’s the net effect, as it’s normally used.  But anycast is really very 
simple, and has no concept of client/server…  An IP address is assigned to 
multiple devices or processes, in locations which the routing topology views as 
diverse.

In practice, that means that services are bound to a common shared address (an 
“anycast service address”) as those services are deployed on servers in 
different locations.  The service address is advertised into the BGP routing 
infrastructure.  Clients send packets to the service address, and the BGP 
routing infrastructure routes each packet on the shortest path to its 
destination, without knowing that there are multiple destinations.

> I am curious if anyone here has/encountered a setup where they use anycast IP 
> on their gateways... to have a predictable egress IP for their traffic, 
> regardless of where they are located?
> 
> For example, a search engine crawler could in principle have the same IP 
> advertised all over the world, but it looks like they don't...  I wonder why?

I think you’re going to need to construct a clearer and more precise 
explanation of what you’re imagining, because my reading of these two lines is 
that they’re saying different things; I don’t see the connection between them 
that you see.  That said, a few reactions:

Anycast is often thought to _reduce_ predictability, since it offers multiple 
exclusive possible termination points for each packet, whereas unicast, 
multicast or broadcast would each have predictable outcomes by comparison: a 
specific node would receive the packet, a specific set of nodes would receive 
the packet, or all (in-scope broadcast domain) nodes would receive the packet.

If you’re asking whether it would make sense for border routers, which have 
access to full-table transit, to advertise that accessibility as an anycasted 
service, that’s what the special “default route” 0.0.0.0/0 is.  Many people 
configure full-transit BGP routers to redistribute a 0.0.0.0/0 default route 
into their IGP, their internal routing protocol (albeit that may well be iBGP, 
nowadays) in order to accommodate routers which haven’t the resources to hold 
or use full routes.

A search engine crawler depends upon a unicast return path in order to 
establish a TCP session with the web sites it’s crawling, and see the return 
traffic from them.  If a search engine crawler shared an anycast service 
address with other instances of itself in other locations, the outbound queries 
would head to web sites (which might be unicast or might be anycast, doesn’t 
matter), which would then try to reply.  If the source address of the query is 
an anycast service address, the reply will go to the nearest instance of that 
shared address, rather than to the specific instance which originated the query.

It’s for this reason that one normally assigns unique unicast addresses to 
network-facing interfaces which will originate packets, and anycast addresses 
to internal loopback interfaces, to which services are bound…  The server can 
receive packets addressed to the anycast shared address, but will originate 
packets using its unique address.

Here’s a tutorial from twenty years ago (when this was all less than fifteen 
years old!) that explains in some detail…  Things haven’t really changed since 
then:

https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: Anycast but for egress

[Daniel Corbe]

> On Jul 27, 2021, at 12:54, Vimal  wrote:
> 
> (Unsure if this is the right forum to ask this question, but here goes:)
> 
> From what I understand, IP Anycast can be used to steer traffic into a server 
> that's close to the client.
> 
> I am curious if anyone here has/encountered a setup where they use anycast IP 
> on their gateways... to have a predictable egress IP for their traffic, 
> regardless of where they are located?
> 
> For example, a search engine crawler could in principle have the same IP 
> advertised all over the world, but it looks like they don't...  I wonder why?
> 
> -- 
> Vimal
> 

Unless you’re twisting knobs, egress traffic should already exit your network 
at the closest possible egress point to its origin.  Is your intention to carry 
the traffic for longer than that?

Re: Anycast but for egress

[Owen DeLong via NANOG]

> On Jul 27, 2021, at 10:54 , Vimal  wrote:
> 
> (Unsure if this is the right forum to ask this question, but here goes:)
> 
> From what I understand, IP Anycast can be used to steer traffic into a server 
> that's close to the client.
> 
> I am curious if anyone here has/encountered a setup where they use anycast IP 
> on their gateways... to have a predictable egress IP for their traffic, 
> regardless of where they are located?

> For example, a search engine crawler could in principle have the same IP 
> advertised all over the world, but it looks like they don't...  I wonder why?

Because there’s no good/reliable way to get the replies back to the correct 
initiating host.

Owen

Anycast but for egress

[Vimal]

(Unsure if this is the right forum to ask this question, but here goes:)

>From what I understand, IP Anycast can be used to steer traffic into a
server that's close to the client.

I am curious if anyone here has/encountered a setup where they use anycast
IP on their gateways... to have a predictable egress IP for their traffic,
regardless of where they are located?

For example, a search engine crawler could in principle have the same IP
advertised all over the world, but it looks like they don't...  I wonder
why?

-- 
Vimal

Re: Layer 2 based anycast - Kind like GLBP - Research

[Dobbins, Roland]

> On 2 Jul 2021, at 01:04, Douglas Fischer  wrote:
> 
> Answering suggestions in advance:

As others have pointed out, what you’re describing isn’t anycast, nor anything 
directly to do with high availability.

There are multiple well-understood frameworks which can be used to do what 
you’re describing.

This is strictly a layer-7 issue, nothing to do with layer-2 nor anycast.  
There’s no need to get down into the networking weeds to accomplish what you 
appear to be trying to accomplish.

Just do it all at layer-7.


Roland Dobbins 

Re: Layer 2 based anycast - Kind like GLBP - Research

[Masataka Ohta]

Douglas Fischer wrote:


Yes... It probably solves my issues on a v6 only world.


No, not at all.

I'm afraid you don't understand what your issues are.

At least, neither L2 or L3 anycast has anything to do with
high reliability because reachability to a server and
availability of the server are different things.

Masataka Ohta

RE: Layer 2 based anycast - Kind like GLBP - Research

[Jean St-Laurent via NANOG]

Maybe a spine and leaf architecture could work for you. 

 

You could install 1 server per leaf or more. I believe this could achieve 
high-availability and load-balancing at layer 2.

 

There is a kind of layer 3 overlay, but for the hosts this is transparent and 
it feels like a real pure layer 2. 

 

I would go that route as it is also a very common setup these days. It scales 
well horizontally and no active/passive. It’s just 
active/active/active/active…./active.

 

Jean

Re: Layer 2 based anycast - Kind like GLBP - Research

[Douglas Fischer]

Hello William!

An ARP Controller to compose a L2 Cluster solution seems a good Idea to a
begging...
(I would include ND)

I will try to think a bit on that...

Any suggestions are welcome.

Em qui., 1 de jul. de 2021 às 16:06, William Herrin 
escreveu:

> On Thu, Jul 1, 2021 at 11:05 AM Douglas Fischer
>  wrote:
> > I'm looking for solutions do deploy some type of selective high
> availability and load balance based on the glue between Layer 2 and Layer 3
> (ARP or ND).
>
> Hi Douglas,
>
> Anycast is where you send to one network address and the "nearest"
> single server with that address receives the packet.
>
> By definition, every piece of equipment in an L2 broadcast domain is
> exactly one hop from every other -- no equipment is "nearer." So
> conceptually, there is no anycast.
>
> However, L2 domains aren't built with hubs any more; they're built
> with switches. There actually are variable distances between
> equipment, they're just not expressed in the protocols. So, in theory
> you could build an SDN controller for your switches which sets up
> different FIB entries in each switch to select which port receives the
> traffic for the designated "anycast" mac address. But you may face
> limitations where the hardware can't reasonably be programmed to give
> each port its own FIB allowing fine-grained control of which client
> reaches which server.
>
> Realistically... that approach would tend to be both expensive to
> build and very brittle. There's almost certainly a better way to
> accomplish your goal than trying to invent L2 anycast.
>
> If you're load balancing IP traffic, another approach might be a
> custom ARP controller which responds to ARP requests with different
> MAC addresses depending on the request source. There's no guaranteed
> timeout for ARP bindings but if you shared around a pool of MAC
> addresses guaranteeing that every MAC address in the pool gets
> assigned to a currently-working server it could work. You just have to
> keep in mind that gratuitous arp absolutely would not work in this
> sort of scenario so you have to have a plan for switching loads
> between servers without it.
>
> I don't think anybody has built that sort of arp controller (at least
> I haven't heard of one) so you'd have to invent it yourself.
>
> From what I understand of EVPN, it's about creating something
> equivalent to VLANs across a distributed virtual server
> infrastructure. Basically like what Amazon does under the hood for its
> virtual private cloud. Since you're trying to get the machines to
> appear on the same subnet, not separate them to different subnets, I
> don't think it's what you're looking for.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>


-- 
Douglas Fernando Fischer
Engº de Controle e Automação

Re: Layer 2 based anycast - Kind like GLBP - Research

[Douglas Fischer]

Hello Masataka!

Yes... It probably solves my issues on a v6 only world.
But unfortunately, in this scenario there is IPv4 also, and to make me cry
alone in the bathroom there are some IPv4 only...
So, I will need to provide some solution that covers IPv4 also.


Em qui., 1 de jul. de 2021 às 19:51, Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> escreveu:

> Douglas Fischer wrote:
>
> > I'm looking for solutions do deploy some type of selective high
> > availability and load balance based on the glue between Layer 2 and
> Layer 3
> > (ARP or ND).
>
> If you are looking for L2 anycast, it was purposelessly
> invented as a functionality of ND, though it does not
> satisfy your requirements. See rfcs 2461 and 4861.
>
> Glad to know it is totally ignored by most, if not all.
>
> Masataka Ohta
>


-- 
Douglas Fernando Fischer
Engº de Controle e Automação

Re: Layer 2 based anycast - Kind like GLBP - Research

[Masataka Ohta]

Douglas Fischer wrote:


I'm looking for solutions do deploy some type of selective high
availability and load balance based on the glue between Layer 2 and Layer 3
(ARP or ND).


If you are looking for L2 anycast, it was purposelessly
invented as a functionality of ND, though it does not
satisfy your requirements. See rfcs 2461 and 4861.

Glad to know it is totally ignored by most, if not all.

Masataka Ohta

Re: Layer 2 based anycast - Kind like GLBP - Research

[Baldur Norddahl]

tor. 1. jul. 2021 21.06 skrev William Herrin :

>
>
> From what I understand of EVPN, it's about creating something
> equivalent to VLANs across a distributed virtual server
> infrastructure. Basically like what Amazon does under the hood for its
> virtual private cloud. Since you're trying to get the machines to
> appear on the same subnet, not separate them to different subnets, I
> don't think it's what you're looking for.
>


EVPN creates a virtual layer 2 domain, aka a vlan, that can span the
internet or be used on a plain old layer 2 switch. It uses vxlan or mpls
tunnels and BGP to coordinate. EVPN has support for multiple active/active
exits, something almost like lacp. There can be load balancing using layer
3 headers as key, which might be exactly what OP is looking for.

EVPN elimates layer 2 flooding by keeping a database of MAC addresses in
BGP. Otherwise it behaves exactly like a vlan with extra features.

>

Re: Layer 2 based anycast - Kind like GLBP - Research

[William Herrin]

On Thu, Jul 1, 2021 at 11:05 AM Douglas Fischer
 wrote:
> I'm looking for solutions do deploy some type of selective high availability 
> and load balance based on the glue between Layer 2 and Layer 3 (ARP or ND).

Hi Douglas,

Anycast is where you send to one network address and the "nearest"
single server with that address receives the packet.

By definition, every piece of equipment in an L2 broadcast domain is
exactly one hop from every other -- no equipment is "nearer." So
conceptually, there is no anycast.

However, L2 domains aren't built with hubs any more; they're built
with switches. There actually are variable distances between
equipment, they're just not expressed in the protocols. So, in theory
you could build an SDN controller for your switches which sets up
different FIB entries in each switch to select which port receives the
traffic for the designated "anycast" mac address. But you may face
limitations where the hardware can't reasonably be programmed to give
each port its own FIB allowing fine-grained control of which client
reaches which server.

Realistically... that approach would tend to be both expensive to
build and very brittle. There's almost certainly a better way to
accomplish your goal than trying to invent L2 anycast.

If you're load balancing IP traffic, another approach might be a
custom ARP controller which responds to ARP requests with different
MAC addresses depending on the request source. There's no guaranteed
timeout for ARP bindings but if you shared around a pool of MAC
addresses guaranteeing that every MAC address in the pool gets
assigned to a currently-working server it could work. You just have to
keep in mind that gratuitous arp absolutely would not work in this
sort of scenario so you have to have a plan for switching loads
between servers without it.

I don't think anybody has built that sort of arp controller (at least
I haven't heard of one) so you'd have to invent it yourself.

>From what I understand of EVPN, it's about creating something
equivalent to VLANs across a distributed virtual server
infrastructure. Basically like what Amazon does under the hood for its
virtual private cloud. Since you're trying to get the machines to
appear on the same subnet, not separate them to different subnets, I
don't think it's what you're looking for.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Layer 2 based anycast - Kind like GLBP - Research

[Baldur Norddahl]

On Thu, Jul 1, 2021 at 8:04 PM Douglas Fischer 
wrote:

> I friend Suggested that EVPN could help-me, but I must confess that is a
> hard topic to me.
> Unless it can be used depending exclusively on software (no special
> hardware required), it won't fit.
>
>
Linux has EVPN support. Or you could use network switches with EVPN support.

EVPN will let each server or direct connected switch be a hidden layer 3
gateway. The application will not know about it and it will appear to be
layer 2 but routable like layer 3. Which means you can do anycast at layer
3. Physically the packets are moved in tunnels so you can use any layer 2
or layer 3 hardware at your choice.

Regards,

Baldur

Re: Layer 2 based anycast - Kind like GLBP - Research

[colin johnston]

Bigip with each host having two nics on public and private via inter switch 
shared vlan.
Should not cause issue so long as you know service comes via bigip to debug 
usage of kit via private ip side


Sent from my iPod

> On 1 Jul 2021, at 19:04, Douglas Fischer  wrote:
> 
> 
> I'm looking for solutions do deploy some type of selective high availability 
> and load balance based on the glue between Layer 2 and Layer 3 (ARP or ND).
> 
> And I'm coming here to ask help to avoid reinventing the wheel.
> 
> I know VRRP / Heartbeat, and their downside is the Active/Passive 
> characteristic.
>  -> But this project demands something that allows-me to have Active/Active 
> deployments.
> I know GLBP, and it almost fits on the needed requirements.
>  -> Except by his load-balancing methods that do not allow-me define priority 
> and affinity between server nodes and clients.
> 
> The basic ideia is something like Cisco GLBP with steroids:
>  - Multiple server nodes of same service running on a common bus and 
> answering the "L2 anycast requests" of the clients that are on the same bus 
> and same subnet.
>  - Some type of signaling between the multiple nodes making known the status 
> of the other nodes, their load. Maybe complementary information like "which 
> node is serving which client?"
>  - Resource Pools and Client Pools, and the crossing between then based on 
> priorities and affinities (Here is the Gotcha!).
> - I want to be able to say "Node X will priorly serve clients A, E, G, 
> and T. Node Y will serve priorly clients B, C D, F. And node Z will server 
> everyone else."
> 
> Answering suggestions in advance:
> (I discussed that with some friends and based on those talks I will try to 
> predict some suggestions that we already considered.)
> - No, unfortunately tradicional L3 anycast will not fit on the requirements. 
> Servers and clients to be at the same bus, on the same subnet. No L3 hops 
> between then.
> - No, the use of some type of connection broker in L2 does not satisfy one of 
> the requirements. Beyond the load balance, that this approach will address, 
> the high availability in case on L2 segregation is also needed.
> 
> 
> My v0 draft of idea was using GLBP, and L2 Firewall rules dynamically 
> adjusted, based on the Master-Status, to allow and block L2 communications 
> between each of those nodes and lists of client pools.
> (Actually, I'm coming back to this idea again... Since I still don't have any 
> other better idea until now.)
> 
> I friend Suggested that EVPN could help-me, but I must confess that is a hard 
> topic to me.
> Unless it can be used depending exclusively on software (no special hardware 
> required), it won't fit.
> 
> --
> Douglas Fernando Fischer
> Engº de Controle e Automação

Layer 2 based anycast - Kind like GLBP - Research

[Douglas Fischer]

I'm looking for solutions do deploy some type of selective high
availability and load balance based on the glue between Layer 2 and Layer 3
(ARP or ND).

And I'm coming here to ask help to avoid reinventing the wheel.

I know VRRP / Heartbeat, and their downside is the Active/Passive
characteristic.
 -> But this project demands something that allows-me to have Active/Active
deployments.
I know GLBP, and it almost fits on the needed requirements.
 -> Except by his load-balancing methods that do not allow-me define
priority and affinity between server nodes and clients.

The basic ideia is something like Cisco GLBP with steroids:
 - Multiple server nodes of same service running on a common bus and
answering the "L2 anycast requests" of the clients that are on the same bus
and same subnet.
 - Some type of signaling between the multiple nodes making known the
status of the other nodes, their load. Maybe complementary information like
"which node is serving which client?"
 - Resource Pools and Client Pools, and the crossing between then based on
priorities and affinities (Here is the Gotcha!).
- I want to be able to say "Node X will priorly serve clients A, E, G,
and T. Node Y will serve priorly clients B, C D, F. And node Z will server
everyone else."

Answering suggestions in advance:
(I discussed that with some friends and based on those talks I will try to
predict some suggestions that we already considered.)
- No, unfortunately tradicional L3 anycast will not fit on the
requirements. Servers and clients to be at the same bus, on the same
subnet. No L3 hops between then.
- No, the use of some type of connection broker in L2 does not satisfy one
of the requirements. Beyond the load balance, that this approach will
address, the high availability in case on L2 segregation is also needed.


My v0 draft of idea was using GLBP, and L2 Firewall rules dynamically
adjusted, based on the Master-Status, to allow and block L2
communications between each of those nodes and lists of client pools.
(Actually, I'm coming back to this idea again... Since I still don't have
any other better idea until now.)

I friend Suggested that EVPN could help-me, but I must confess that is a
hard topic to me.
Unless it can be used depending exclusively on software (no special
hardware required), it won't fit.

--
Douglas Fernando Fischer
Engº de Controle e Automação

bgp dampening and anycast networks (particularly cloudflare)

[David Hubbard]

Hi all, was curious if anyone has found it necessary to alter their route 
dampening rules related to anycast networks, and Cloudflare especially?  I’ve 
got a customer whose target web server has been going intermittently 
inaccessible from a very geographically distant Cloudflare location (AU), while 
no reports of issues from anywhere closer to the US.  I’m seeing a bunch of 
their /24’s dampened on my side in several locations, and they appear to be 
networks that favor or are specific to AU, so I’m thinking that’s the issue.  
I’m going to whitelist their ASN, but perhaps I need to work on the policy to 
be more tolerant of flaps compared to years past with the increase in anycast 
use?

Thanks,

David

Re: TCP and anycast (was Re: ECN)

[Scott Weeks]

--- ra...@psg.com wrote:
lots of good research lit on catchment topology of anycasted 
dns, which is very non-local.
---


For the others here that didn't know what that is and are 
curious.  I couldn't take it and just had to know... :)

https://tools.ietf.org/html/rfc4786

Catchment:  in physical geography, an area drained by a river, also
  known as a drainage basin.  By analogy, as used in this document,
  the topological region of a network within which packets directed
  at an Anycast Address are routed to one particular node.

scott

Re: TCP and anycast (was Re: ECN)

[William Herrin]

On Thu, Nov 14, 2019 at 1:10 AM Bill Woodcock  wrote:
> > On Nov 14, 2019, at 7:39 AM, Anoop Ghanwani 
wrote:
> > RFC 7094 (https://tools.ietf.org/html/rfc7094) describes the pitfalls &
risks of using TCP with an anycast address.  It recognizes that there are
valid use cases for it, though.
> > Specifically, section 3.1 says this:
> >Most stateful transport protocols (e.g., TCP), without modification,
do not understand the properties of anycast; hence, they will fail
> >probabilistically, but possibly catastrophically, when using anycast
addresses in the presence of "normal" routing dynamics.
> >This can lead  to a protocol working fine in, say, a test lab but
not in the global Internet.
> >
> > On Thu, Nov 14, 2019 at 12:25 AM Matt Corallo 
wrote:
> > > This sounds like a bug on Cloudflare’s end (cause trying to do
anycast TCP is... out of spec to say the least),
>
> No. We have been doing anycast TCP for more than _thirty years_, most of
that time on a global scale, without operational problems.

Hi Bill,

Not to put to fine a point on it but Baldur and Toke's scenario in which
anycast tcp failed, the one which started this thread, should probably be
classed as an operational problem.

It is possible to build an anycast TCP that works. YOU have not done so.
And Cloudflare certainly has not.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: TCP and anycast (was Re: ECN)

[Randy Bush]

>>> RFC 7094 (https://tools.ietf.org/html/rfc7094) describes the pitfalls
>>> & risks of using TCP with an anycast address.
>>
>> and two decades of operational experience are that prudent deployments
>> just work.
> 
> I agree with Bill/Randy here... this does just work if you understand
> your local topology and manage change properly.

agree, but would extend ...

sometimes s/local//

i.e. casting from your edge dumps directly to peers, keeping it off
your backbone.  but the topo set you have to keep in mind can be
large.

lots of good research lit on catchment topology of anycasted dns,
which is very non-local.

randy

Re: TCP and anycast (was Re: ECN)

[Christopher Morrow]

On Fri, Nov 15, 2019 at 1:54 AM Randy Bush  wrote:
>
> > RFC 7094 (https://tools.ietf.org/html/rfc7094) describes the pitfalls
> > & risks of using TCP with an anycast address.
>
> and two decades of operational experience are that prudent deployments
> just work.

I agree with Bill/Randy here... this does just work if you understand
your local topology and manage change properly.

Re: TCP and anycast (was Re: ECN)

[Randy Bush]

> RFC 7094 (https://tools.ietf.org/html/rfc7094) describes the pitfalls
> & risks of using TCP with an anycast address.

and two decades of operational experience are that prudent deployments
just work.

randy

Re: TCP and anycast (was Re: ECN)

[Bill Woodcock]

> On Nov 14, 2019, at 7:39 AM, Anoop Ghanwani  wrote:
> RFC 7094 (https://tools.ietf.org/html/rfc7094) describes the pitfalls & risks 
> of using TCP with an anycast address.  It recognizes that there are valid use 
> cases for it, though.
> Specifically, section 3.1 says this:
>Most stateful transport protocols (e.g., TCP), without modification, do 
> not understand the properties of anycast; hence, they will fail
>probabilistically, but possibly catastrophically, when using anycast 
> addresses in the presence of "normal" routing dynamics.
>This can lead  to a protocol working fine in, say, a test lab but not in 
> the global Internet.
> 
> On Thu, Nov 14, 2019 at 12:25 AM Matt Corallo  wrote:
> > This sounds like a bug on Cloudflare’s end (cause trying to do anycast TCP 
> > is... out of spec to say the least),

No. We have been doing anycast TCP for more than _thirty years_, most of that 
time on a global scale, without operational problems.

There were people who seemed gray-bearded at the time, who were scared of 
anycast because it used IP addresses _non uniquely_ and that wasn’t how they’d 
intended them to be used, and these kids these days, etc.  What you’re seeing 
is residuum of their pronouncements on the matter, carrying over from the 
mid-1990s.

It’s very true that anycast can be misused and abused in a myriad of ways, 
leading to unexpected or unpleasant results, but no more so than other routing 
techniques.  We and others have published on many or most of the potential 
issues and their solutions over the years.  That RFC has never actually been a 
comprehensive source of information on the topic, and it contains a lot of 
scare-mongering. 

-Bill

TCP and anycast (was Re: ECN)

[Anoop Ghanwani]

RFC 7094 (https://tools.ietf.org/html/rfc7094) describes the pitfalls &
risks of using TCP with an anycast address.  It recognizes that there are
valid use cases for it, though.

Specifically, section 3.1 says this:
>>>

   Most stateful transport protocols (e.g., TCP), without modification,
   do not understand the properties of anycast; hence, they will fail
   probabilistically, but possibly catastrophically, when using anycast
   addresses in the presence of "normal" routing dynamics.

...

   This can lead
   to a protocol working fine in, say, a test lab but not in the global
   Internet.

>>>

On Wed, Nov 13, 2019 at 3:33 PM Warren Kumari  wrote:

> On Thu, Nov 14, 2019 at 12:25 AM Matt Corallo  wrote:
> >
> > This sounds like a bug on Cloudflare’s end (cause trying to do anycast
> TCP is... out of spec to say the least), not a bug in ECN/ECMP.
>
> Err. I really don't think that there is any sort of spec that
> covers that :-P
>
> Using Anycast for TCP is incredibly common - the DNS root servers for
> one obvious example.
> More TCP centric well-known examples are Fastly and LinkedIn -
> LinkedIn in particular did a really good podcast on their experience
> with this.
>
> There is also a good NANOG talk from the ~2000s (?) on people using
> TCP anycast for long lived (serving ISO files, which were long-lived
> in those days) flows, and how reliable it is - perhaps that's the talk
> Todd mentioned?
>
> W
>
> >
> > > On Nov 13, 2019, at 11:07, Toke Høiland-Jørgensen via NANOG <
> nanog@nanog.org> wrote:
> > >
> > > 
> > >>
> > >> Hello
> > >>
> > >> I have a customer that believes my network has a ECN problem. We do
> > >> not, we just move packets. But how do I prove it?
> > >>
> > >> Is there a tool that checks for ECN trouble? Ideally something I could
> > >> run on the NLNOG Ring network.
> > >>
> > >> I believe it likely that it is the destination that has the problem.
> > >
> > > Hi Baldur
> > >
> > > I believe I may be that customer :)
> > >
> > > First of all, thank you for looking into the issue! We've been having
> > > great fun over on the ecn-sane mailing list trying to figure out what's
> > > going on. I'll summarise below, but see this thread for the discussion
> > > and debugging details:
> > >
> https://lists.bufferbloat.net/pipermail/ecn-sane/2019-November/000527.html
> > >
> > > The short version is that the problem appears to come from a
> combination
> > > of the ECMP routing in your network, and Cloudflare's heavy use of
> > > anycast. Specifically, a router in your network appears to be doing
> ECMP
> > > by hashing on the packet header, *including the ECN bits*. This breaks
> > > TCP connections with ECN because the TCP SYN (with no ECN bits set) end
> > > up taking a different path than the rest of the flow (which is marked
> as
> > > ECT(0)). When the destination is anycasted, this means that the data
> > > packets go to a different server than the SYN did. This second server
> > > doesn't recognise the connection, and so replies with a TCP RST. To fix
> > > this, simply exclude the ECN bits (or the whole TOS byte) from your
> > > router's ECMP hash.
> > >
> > > For a longer exposition, see below. You should be able to verify this
> > > from somewhere else in the network, but if there's anything else you
> > > want me to test, do let me know. Also, would you mind sharing the
> router
> > > make and model that does this? We're trying to collect real-world
> > > examples of network problems caused by ECN and this is definitely an
> > > interesting example.
> > >
> > > -Toke
> > >
> > >
> > >
> > > The long version:
> > >
> > > From my end I can see that I have two paths to Cloudflare; which is
> > > taken appears to be based on a hash of the packet header, as can be
> seen
> > > by varying the source port:
> > >
> > > $ traceroute -q 1 --sport=1 104.24.125.13
> > > traceroute to 104.24.125.13 (104.24.125.13), 30 hops max, 60 byte
> packets
> > > 1  _gateway (10.42.3.1)  0.357 ms
> > > 2  albertslund-edge1-lo.net.gigabit.dk (185.24.171.254)  4.707 ms
> > > 3  customer-185-24-168-46.ip4.gigabit.dk (185.24.168.46)  1.283 ms
> > > 4  te0-1-1-5.rcr21.cph01.atlas.cogentco.com (149.6.137.49)  1.667 ms
> > > 5  netnod-ix-cph-blue-900

Anycast operators

[Mehmet Akcin]

hey there,

I am looking to talk to anycast based service operators dns and non-dns
service operators to understand how virtualization is used in new
generation anycast deployments.

if you are running anycast based service (root-servers, tlds, etc.) please
contact me offlist.

mehmet

Re: well-known Anycast prefixes

[Bill Woodcock]

I imagine that the “description” of each entry in the list should include a 
machine-readable field indicating the use. 

There was a question about the use-case... I’m sure a lot of people in the ops 
community have their own reasons related to routing and filtering and so forth, 
but there’s also a huge demand for this kind of information, aggregated and 
sanity-checked, to support academic research at the graduate level. And the 
better we support those kids with real-world data, the more practical an 
education they receive, and the more ready they are to jump in to jobs we offer 
them in industry when they graduate. Supporting kids and networking graduate 
programs like that is a big part of our work, that tends not to be visible on 
the operations side. 

Academics downloaded routing-archive snapshots from us nearly 300 million 
times, last year, for example. 

-Bill


> On Mar 21, 2019, at 09:52, Ross Tajvar  wrote:
> 
> Not all any-casted prefixes are DNS resolvers and not all DNS resolvers are 
> anycasted. It sounds like you would be better served by a list of well-known 
> DNS resolvers.
> 
>> On Thu, Mar 21, 2019 at 12:35 PM Bryan Holloway  wrote:
>> 
>> On 3/21/19 10:59 AM, Frank Habicht wrote:
>> > Hi James,
>> > 
>> > On 20/03/2019 21:05, James Shank wrote:
>> >> I'm not clear on the use cases, though.  What are the imagined use cases?
>> >>
>> >> It might make sense to solve 'a method to request hot potato routing'
>> >> as a separate problem.  (Along the lines of Damian's point.)
>> > 
>> > my personal reason/motivation is this:
>> > Years ago I noticed that my traffic to the "I" DNS root server was
>> > traversing 4 continents. That's from Tanzania, East Africa.
>> > Not having a local instance (back then), we naturally sent the traffic
>> > to an upstream. That upstream happens to be in that club of those who
>> > don't have transit providers (which probably doesn't really matter, but
>> > means a "global" network).
>> 
>> /snip
>> 
>> > Greetings,
>> > Frank
>> > 
>> 
>> I can think of another ...
>> 
>> We rate-limit DNS from unknown quantities for reasons that should be 
>> obvious. We white-list traffic from known trusted (anycast) ones to 
>> prevent a DDoS attack from throttling legitimate queries. This would be 
>> a useful way to help auto-generate those ACLs.

Re: well-known Anycast prefixes

[Bryan Holloway]

On 3/21/19 11:52 AM, Ross Tajvar wrote:
Not all any-casted prefixes are DNS resolvers and not all DNS resolvers 
are anycasted. It sounds like you would be better served by a list of 
well-known DNS resolvers.


True on both counts, and that's why I said "help".


On Thu, Mar 21, 2019 at 12:35 PM Bryan Holloway <mailto:br...@shout.net>> wrote:



On 3/21/19 10:59 AM, Frank Habicht wrote:
 > Hi James,
 >
 > On 20/03/2019 21:05, James Shank wrote:
 >> I'm not clear on the use cases, though.  What are the imagined
use cases?
 >>
 >> It might make sense to solve 'a method to request hot potato
routing'
 >> as a separate problem.  (Along the lines of Damian's point.)
 >
 > my personal reason/motivation is this:
 > Years ago I noticed that my traffic to the "I" DNS root server was
 > traversing 4 continents. That's from Tanzania, East Africa.
 > Not having a local instance (back then), we naturally sent the
traffic
 > to an upstream. That upstream happens to be in that club of those who
 > don't have transit providers (which probably doesn't really
matter, but
 > means a "global" network).

/snip

 > Greetings,
 > Frank
 >

I can think of another ...

We rate-limit DNS from unknown quantities for reasons that should be
obvious. We white-list traffic from known trusted (anycast) ones to
prevent a DDoS attack from throttling legitimate queries. This would be
a useful way to help auto-generate those ACLs.

Re: well-known Anycast prefixes

[Ross Tajvar]

Not all any-casted prefixes are DNS resolvers and not all DNS resolvers are
anycasted. It sounds like you would be better served by a list of
well-known DNS resolvers.

On Thu, Mar 21, 2019 at 12:35 PM Bryan Holloway  wrote:

>
> On 3/21/19 10:59 AM, Frank Habicht wrote:
> > Hi James,
> >
> > On 20/03/2019 21:05, James Shank wrote:
> >> I'm not clear on the use cases, though.  What are the imagined use
> cases?
> >>
> >> It might make sense to solve 'a method to request hot potato routing'
> >> as a separate problem.  (Along the lines of Damian's point.)
> >
> > my personal reason/motivation is this:
> > Years ago I noticed that my traffic to the "I" DNS root server was
> > traversing 4 continents. That's from Tanzania, East Africa.
> > Not having a local instance (back then), we naturally sent the traffic
> > to an upstream. That upstream happens to be in that club of those who
> > don't have transit providers (which probably doesn't really matter, but
> > means a "global" network).
>
> /snip
>
> > Greetings,
> > Frank
> >
>
> I can think of another ...
>
> We rate-limit DNS from unknown quantities for reasons that should be
> obvious. We white-list traffic from known trusted (anycast) ones to
> prevent a DDoS attack from throttling legitimate queries. This would be
> a useful way to help auto-generate those ACLs.
>

Re: well-known Anycast prefixes

[Bryan Holloway]

On 3/21/19 10:59 AM, Frank Habicht wrote:

Hi James,

On 20/03/2019 21:05, James Shank wrote:

I'm not clear on the use cases, though.  What are the imagined use cases?

It might make sense to solve 'a method to request hot potato routing'
as a separate problem.  (Along the lines of Damian's point.)


my personal reason/motivation is this:
Years ago I noticed that my traffic to the "I" DNS root server was
traversing 4 continents. That's from Tanzania, East Africa.
Not having a local instance (back then), we naturally sent the traffic
to an upstream. That upstream happens to be in that club of those who
don't have transit providers (which probably doesn't really matter, but
means a "global" network).


/snip


Greetings,
Frank



I can think of another ...

We rate-limit DNS from unknown quantities for reasons that should be 
obvious. We white-list traffic from known trusted (anycast) ones to 
prevent a DDoS attack from throttling legitimate queries. This would be 
a useful way to help auto-generate those ACLs.

Re: well-known Anycast prefixes

[Job Snijders]

On Thu, Mar 21, 2019 at 06:59:18PM +0300, Frank Habicht wrote:
> On 20/03/2019 21:05, James Shank wrote:
> > I'm not clear on the use cases, though.  What are the imagined use cases?
> > 
> > It might make sense to solve 'a method to request hot potato routing'
> > as a separate problem.  (Along the lines of Damian's point.)
> 
> my personal reason/motivation is this:
> Years ago I noticed that my traffic to the "I" DNS root server was
> traversing 4 continents. That's from Tanzania, East Africa.
> Not having a local instance (back then), we naturally sent the traffic
> to an upstream. That upstream happens to be in that club of those who
> don't have transit providers (which probably doesn't really matter, but
> means a "global" network).

Luckily there are other root servers too! :)

> My Theory :
> So just because one I-root instance was hosted at a customer (or
> customer's customer), that got higher local-pref and now packets take
> the long way from Africa via Europe, NorthAmerica to Asia and that
> customer in Thailand. While closer I-root instances would obviously be
> along the way, just not from a paying customer, "only" from peering.
> 
> I don't know whether or not to blame that "carrier" for intentionally(?)
> carrying the traffic that far - presumably the $ they got for that from
> the I-root host in Thailand was worth it, and not enough customers
> complained enough about the latency?
> 
> But I think it would be worthwhile to give them an option and produce a
> mechanism of knowing what's anycasted.
> 
> Maybe (thinking of it) a solution for really well-known prefixes
> available at many instances/locations (like DNS root) would be to have
> their fixed set of direct transits at all the "global" nodes and
> everywhere else to tell peers to not advertise this to upstreams.

In all instances of what you mention you need cooperation from the
network which is routing in a (from your perspective) suboptimal way.

Either the customer of that upstream should use BGP communities to
localize the announcement, or the upstream themselves need to change
their routing policy to set 'same LOCAL_PREF everywhere' for some
prefixes. Of course any input channel into routing policy can be a
vector of abuse.

Even if you equalize the LOCAL_PREF attribute across your network edge,
you still have other tie breakers such as AS_PATH length. It is not
clear to me how a list of well-known anycast addresses, in practise,
would help swing the pendulum. In all cases you need cooperation from a
lot of networks, and the outcome is not clearly defined because we don't
have a true inter-domain 'shortest latency path' metric.

Kind regards,

Job

Re: well-known Anycast prefixes

[Frank Habicht]

Hi James,

On 20/03/2019 21:05, James Shank wrote:
> I'm not clear on the use cases, though.  What are the imagined use cases?
> 
> It might make sense to solve 'a method to request hot potato routing'
> as a separate problem.  (Along the lines of Damian's point.)

my personal reason/motivation is this:
Years ago I noticed that my traffic to the "I" DNS root server was
traversing 4 continents. That's from Tanzania, East Africa.
Not having a local instance (back then), we naturally sent the traffic
to an upstream. That upstream happens to be in that club of those who
don't have transit providers (which probably doesn't really matter, but
means a "global" network).

My Theory :
So just because one I-root instance was hosted at a customer (or
customer's customer), that got higher local-pref and now packets take
the long way from Africa via Europe, NorthAmerica to Asia and that
customer in Thailand. While closer I-root instances would obviously be
along the way, just not from a paying customer, "only" from peering.

I don't know whether or not to blame that "carrier" for intentionally(?)
carrying the traffic that far - presumably the $ they got for that from
the I-root host in Thailand was worth it, and not enough customers
complained enough about the latency?

But I think it would be worthwhile to give them an option and produce a
mechanism of knowing what's anycasted.

Maybe (thinking of it) a solution for really well-known prefixes
available at many instances/locations (like DNS root) would be to have
their fixed set of direct transits at all the "global" nodes and
everywhere else to tell peers to not advertise this to upstreams.

Greetings,
Frank

Re: well-known Anycast prefixes

[James Shank]

On 3/19/19 5:03 PM, Bill Woodcock wrote:
> 
> 
>> On Mar 19, 2019, at 1:55 PM, Frank Habicht  wrote:
>>
>> Hi,
>>
>> On 19/03/2019 23:13, Bill Woodcock wrote:
>>> Generally, static lists like that are difficult to maintain when
>>> they’re tracking multiple routes from multiple parties.
>>
>> agreed.
>> and on the other extreme, communities are very much prone to abuse.
>> I guess I could set any community on a number of prefixes (incl anycast)
>> right now
>>
>> So, I think a (moderated) BGP feed of prefixes a'la bogon from a trusted
>> {cymru[1], pch[2], ...}  could be good [3].
> 
> Ok, so, just trying to flesh out the idea to something that can be usefully 
> implemented…
> 
> 1) People send an eBGP multi-hop feed of well-known-community routes to a 
> collector, or send them over normal peering sessions to something that 
> aggregates…
> 
> 2) Because those are over BGP sessions, the counterparty is known, and can be 
> asked for details or clarification by the “moderator,” or the sender could 
> log in to an interface to add notes about the prefixes, as they would in the 
> IXPdir or PeeringDB.
> 
> 3) Known prefixes from known parties would be passed through in real-time, as 
> they were withdrawn and restored.
> 
> 4) New prefixes from known parties would be passed through in real-time if 
> they weren’t unusual (large/overlapping something else/previously announced 
> by other ASNs).
> 
> 5) New prefixes from known parties would be “moderated” if they were unusual.
> 
> 6) New prefixes from new parties would be “moderated” to establish that they 
> were legit and that there was some documentation explaining what they were.
> 
> 7) For anyone who really didn’t want to provide a community-tagged BGP feed, 
> a manual submission process would exist.
> 
> 8) Everything gets published as a real-time eBGP feed.
> 
> 9) Everything gets published as HTTPS-downloadable JSON.
> 
> 10) Everything gets published as a human-readable (and crawler-indexable) web 
> page.
> 
> Does that sound about right?
> 
> -Bill

Hi,

Interesting discussion and ideas.  I like how you've laid it out
above, Bill.

I'm not clear on the use cases, though.  What are the imagined use cases?

It might make sense to solve 'a method to request hot potato routing'
as a separate problem.  (Along the lines of Damian's point.)

Thanks!

James

-- 
James Shank
Senior Technical Advisor; Team Cymru, Inc.
jsh...@cymru.com; +1-847-378-3365; http://www.team-cymru.com/

Re: well-known Anycast prefixes

[Frank Habicht]

Hi,

On 20/03/2019 00:03, Bill Woodcock wrote:
> Ok, so, just trying to flesh out the idea to something that can be
> usefully implemented…
> 
> 1) People send an eBGP multi-hop feed of well-known-community routes
> to a collector, or send them over normal peering sessions to
> something that aggregates…

to clarify, eBGP multi-hop sounds good, that can be a dedicated session
for this purpose, BGP communities probably don't need to be added.
[for the case of 'normal peering sessions', it might seem "wasteful" to
use an additional IP on multiple peering LANs]

...


> Does that sound about right?

to me yes.

Frank

Re: well-known Anycast prefixes

[Bill Woodcock]

> On Mar 19, 2019, at 1:55 PM, Frank Habicht  wrote:
> 
> Hi,
> 
> On 19/03/2019 23:13, Bill Woodcock wrote:
>> Generally, static lists like that are difficult to maintain when
>> they’re tracking multiple routes from multiple parties.
> 
> agreed.
> and on the other extreme, communities are very much prone to abuse.
> I guess I could set any community on a number of prefixes (incl anycast)
> right now
> 
> So, I think a (moderated) BGP feed of prefixes a'la bogon from a trusted
> {cymru[1], pch[2], ...}  could be good [3].

Ok, so, just trying to flesh out the idea to something that can be usefully 
implemented…

1) People send an eBGP multi-hop feed of well-known-community routes to a 
collector, or send them over normal peering sessions to something that 
aggregates…

2) Because those are over BGP sessions, the counterparty is known, and can be 
asked for details or clarification by the “moderator,” or the sender could log 
in to an interface to add notes about the prefixes, as they would in the IXPdir 
or PeeringDB.

3) Known prefixes from known parties would be passed through in real-time, as 
they were withdrawn and restored.

4) New prefixes from known parties would be passed through in real-time if they 
weren’t unusual (large/overlapping something else/previously announced by other 
ASNs).

5) New prefixes from known parties would be “moderated” if they were unusual.

6) New prefixes from new parties would be “moderated” to establish that they 
were legit and that there was some documentation explaining what they were.

7) For anyone who really didn’t want to provide a community-tagged BGP feed, a 
manual submission process would exist.

8) Everything gets published as a real-time eBGP feed.

9) Everything gets published as HTTPS-downloadable JSON.

10) Everything gets published as a human-readable (and crawler-indexable) web 
page.

Does that sound about right?

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: well-known Anycast prefixes

[Frank Habicht]

Hi,

On 19/03/2019 23:13, Bill Woodcock wrote:
> Generally, static lists like that are difficult to maintain when
> they’re tracking multiple routes from multiple parties.

agreed.
and on the other extreme, communities are very much prone to abuse.
I guess I could set any community on a number of prefixes (incl anycast)
right now

So, I think a (moderated) BGP feed of prefixes a'la bogon from a trusted
{cymru[1], pch[2], ...}  could be good [3].

Frank Habicht
37084 / 33791
if that matters


{1] dealing with anycast?
[2] biased?
[3] speaking as someone not using (subscribing) any of the useful ones,
nor contributing to any...

Re: well-known Anycast prefixes

[Bill Woodcock]

> On Mar 19, 2019, at 1:11 PM, Grzegorz Janoszka  wrote:
> 
> On 2019-03-19 21:04, Hansen, Christoffer wrote:
>> https://github.com/netravnen/well-known-anycast-prefixes/blob/master/list.txt
>> PR's and/or suggestions appreciated! (Can be turned into $lirDB friendly
>> format->style RPSL)
> 
> Most DNS root servers are anycasted.

Right, yeah, I think he was just showing an example, since he had roughly a 
dozen, out of thousands.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: well-known Anycast prefixes

[Bill Woodcock]

> On Mar 19, 2019, at 1:04 PM, Hansen, Christoffer  
> wrote:
> 
> something like this?
> 
> https://github.com/netravnen/well-known-anycast-prefixes/blob/master/list.txt
> 
> PR's and/or suggestions appreciated! (Can be turned into $lirDB friendly
> format->style RPSL)

Generally, static lists like that are difficult to maintain when they’re 
tracking multiple routes from multiple parties.

Communities have been suggested, which works as long as they’re passed through 
to somewhere people can see.  Between PCH, RIS, and Route-Views, most should be 
visible somewhere, but not all.

I think a combination of the two is probably most useful…  people tag with a 
well-known community, then those get eBGP-multi-hopped to a common collector, 
and published as a clean machine-readable list.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: well-known Anycast prefixes

[Grzegorz Janoszka]

On 2019-03-19 21:04, Hansen, Christoffer wrote:

https://github.com/netravnen/well-known-anycast-prefixes/blob/master/list.txt
PR's and/or suggestions appreciated! (Can be turned into $lirDB friendly
format->style RPSL)


Most DNS root servers are anycasted.

--
Grzegorz Janoszka

Re: well-known Anycast prefixes

[Hansen, Christoffer]

something like this?

https://github.com/netravnen/well-known-anycast-prefixes/blob/master/list.txt

PR's and/or suggestions appreciated! (Can be turned into $lirDB friendly
format->style RPSL)

On 19/03/2019 18:12, Fredy Kuenzler wrote:
> I wonder whether anyone has ever compiled a list of well-known Anycast
> prefixes.
> 
> Such as
> 
> 1.1.1.0/24
> 8.8.8.0/24
> 9.9.9.0/24
> ...
> 
> Might be useful for a routing policy such as "always route hot-potato".
> 
> PS. this mail is not intended to start a flame war of hot vs. cold
> potato routing.
> 

-- 

Christoffer
0x18DD23C550293098DE07052A9DCF2CA008EBD2E8



signature.asc
Description: OpenPGP digital signature

Re: well-known Anycast prefixes

[Joe Provo]

On Tue, Mar 19, 2019 at 11:52:19AM -0700, Damian Menscher via NANOG wrote:
> Careful thought should be given into whether the BGP community means "this
> is an anycast prefix" vs "please hot-potato to this prefix".
> Latency-sensitive applications may prefer hot-potato to their network even
> if it's not technically an anycast range, as their private backbone may be
> faster (less congested) than the public internet.

To this point, it is pretty clear that any WK community covering this
will get [ab]used in a way that the prefix annoucer wishes. We'll then 
see operators only accepting the WKC if it matches their prefix lists
of known entities, getting us back to "hey maybe this should just be 
a registry I could reference".

Woody, maybe generate route-sets to publish in RPSL (RADB?), one per 
address-family, of observed anycasters?  It might be reasonable to do 
so in a format others can emulate if they wish to create/provide their 
own lists?

Cheers,

Joe

> Damian
> 
> On Tue, Mar 19, 2019 at 10:57 AM Siyuan Miao  wrote:
> 
> >
> > A Well-known BGP community will be better.
> >
> > You'll need to rewrite next hop or do something similar if AnyCast
> > prefixes are learnt from a multi hop BGP feed, and it made the
> > configuration more complicated and difficult to debug.
> >
> > On Wed, Mar 20, 2019, 01:48 Fredy Kuenzler  wrote:
> >
> >> Am 19.03.19 um 18:39 schrieb Bill Woodcock:
> >> >> On Mar 19, 2019, at 10:12 AM, Fredy Kuenzler 
> >> >> wrote: I wonder whether anyone has ever compiled a list of
> >> >> well-known Anycast prefixes.
> >> >
> >> > I don???t know of one.
> >> >
> >> > It seems like a good idea.
> >> >
> >> > BGP-multi-hop might be a reasonable way to collect them.
> >> >
> >> > If others agree that it???s a good idea, and it???s not stepping on
> >> > anyone???s toes, PCH would be happy to host/coordinate.
> >>
> >> Thanks for the effort, much appreciated.
> >>
> >> Am 19.03.19 um 18:40 schrieb Joe Provo:
> >> > I think one would want that internal and no rely upon someone else
> >> > maintaining it.  You might check if Oracle followed up on the
> >> > Renesys/Dyn work documented:
> >> > https://dyn.com/wp-content/uploads/2014/07/NANOG59_Anycast.pdf
> >> >
> >> > ...where there were ~600 anycast v4 prefixes at the time.
> >>
> >> That's a lot %-]
> >>
> >> Maybe a well-known community (similar to RFC7999) could be defined and
> >> every Anycast operator could tag his prefixes? That's likely a better
> >> idea than manually maintain some list somewhere.
> >>
> >> --
> >> Fredy Kuenzler
> >>
> >> Init7 (Switzerland) Ltd.
> >> AS13030
> >> Technoparkstrasse 5
> >> CH-8406 Winterthur
> >> Skype:   flyingpotato
> >> Phone:   +41 44 315 4400
> >> Fax: +41 44 315 4401
> >> Twitter: @init7 / @kuenzler
> >> http://www.init7.net/
> >>
> >>

-- 
Posted from my personal account - see X-Disclaimer header.
Joe Provo / Gweep / Earthling 

Re: well-known Anycast prefixes

[Damian Menscher via NANOG]

Careful thought should be given into whether the BGP community means "this
is an anycast prefix" vs "please hot-potato to this prefix".
Latency-sensitive applications may prefer hot-potato to their network even
if it's not technically an anycast range, as their private backbone may be
faster (less congested) than the public internet.

Damian

On Tue, Mar 19, 2019 at 10:57 AM Siyuan Miao  wrote:

>
> A Well-known BGP community will be better.
>
> You'll need to rewrite next hop or do something similar if AnyCast
> prefixes are learnt from a multi hop BGP feed, and it made the
> configuration more complicated and difficult to debug.
>
> On Wed, Mar 20, 2019, 01:48 Fredy Kuenzler  wrote:
>
>> Am 19.03.19 um 18:39 schrieb Bill Woodcock:
>> >> On Mar 19, 2019, at 10:12 AM, Fredy Kuenzler 
>> >> wrote: I wonder whether anyone has ever compiled a list of
>> >> well-known Anycast prefixes.
>> >
>> > I don’t know of one.
>> >
>> > It seems like a good idea.
>> >
>> > BGP-multi-hop might be a reasonable way to collect them.
>> >
>> > If others agree that it’s a good idea, and it’s not stepping on
>> > anyone’s toes, PCH would be happy to host/coordinate.
>>
>> Thanks for the effort, much appreciated.
>>
>> Am 19.03.19 um 18:40 schrieb Joe Provo:
>> > I think one would want that internal and no rely upon someone else
>> > maintaining it.  You might check if Oracle followed up on the
>> > Renesys/Dyn work documented:
>> > https://dyn.com/wp-content/uploads/2014/07/NANOG59_Anycast.pdf
>> >
>> > ...where there were ~600 anycast v4 prefixes at the time.
>>
>> That's a lot %-]
>>
>> Maybe a well-known community (similar to RFC7999) could be defined and
>> every Anycast operator could tag his prefixes? That's likely a better
>> idea than manually maintain some list somewhere.
>>
>> --
>> Fredy Kuenzler
>>
>> Init7 (Switzerland) Ltd.
>> AS13030
>> Technoparkstrasse 5
>> CH-8406 Winterthur
>> Skype:   flyingpotato
>> Phone:   +41 44 315 4400
>> Fax: +41 44 315 4401
>> Twitter: @init7 / @kuenzler
>> http://www.init7.net/
>>
>>

Re: well-known Anycast prefixes

[Siyuan Miao]

A Well-known BGP community will be better.

You'll need to rewrite next hop or do something similar if AnyCast prefixes
are learnt from a multi hop BGP feed, and it made the configuration more
complicated and difficult to debug.

On Wed, Mar 20, 2019, 01:48 Fredy Kuenzler  wrote:

> Am 19.03.19 um 18:39 schrieb Bill Woodcock:
> >> On Mar 19, 2019, at 10:12 AM, Fredy Kuenzler 
> >> wrote: I wonder whether anyone has ever compiled a list of
> >> well-known Anycast prefixes.
> >
> > I don’t know of one.
> >
> > It seems like a good idea.
> >
> > BGP-multi-hop might be a reasonable way to collect them.
> >
> > If others agree that it’s a good idea, and it’s not stepping on
> > anyone’s toes, PCH would be happy to host/coordinate.
>
> Thanks for the effort, much appreciated.
>
> Am 19.03.19 um 18:40 schrieb Joe Provo:
> > I think one would want that internal and no rely upon someone else
> > maintaining it.  You might check if Oracle followed up on the
> > Renesys/Dyn work documented:
> > https://dyn.com/wp-content/uploads/2014/07/NANOG59_Anycast.pdf
> >
> > ...where there were ~600 anycast v4 prefixes at the time.
>
> That's a lot %-]
>
> Maybe a well-known community (similar to RFC7999) could be defined and
> every Anycast operator could tag his prefixes? That's likely a better
> idea than manually maintain some list somewhere.
>
> --
> Fredy Kuenzler
>
> Init7 (Switzerland) Ltd.
> AS13030
> Technoparkstrasse 5
> CH-8406 Winterthur
> Skype:   flyingpotato
> Phone:   +41 44 315 4400
> Fax: +41 44 315 4401
> Twitter: @init7 / @kuenzler
> http://www.init7.net/
>
>

Re: well-known Anycast prefixes

[Fredy Kuenzler]

Am 19.03.19 um 18:39 schrieb Bill Woodcock:
>> On Mar 19, 2019, at 10:12 AM, Fredy Kuenzler  
>> wrote: I wonder whether anyone has ever compiled a list of 
>> well-known Anycast prefixes.
> 
> I don’t know of one.
> 
> It seems like a good idea.
> 
> BGP-multi-hop might be a reasonable way to collect them.
> 
> If others agree that it’s a good idea, and it’s not stepping on 
> anyone’s toes, PCH would be happy to host/coordinate.

Thanks for the effort, much appreciated.

Am 19.03.19 um 18:40 schrieb Joe Provo:
> I think one would want that internal and no rely upon someone else
> maintaining it.  You might check if Oracle followed up on the 
> Renesys/Dyn work documented: 
> https://dyn.com/wp-content/uploads/2014/07/NANOG59_Anycast.pdf
> 
> ...where there were ~600 anycast v4 prefixes at the time.

That's a lot %-]

Maybe a well-known community (similar to RFC7999) could be defined and
every Anycast operator could tag his prefixes? That's likely a better
idea than manually maintain some list somewhere.

-- 
Fredy Kuenzler

Init7 (Switzerland) Ltd.
AS13030
Technoparkstrasse 5
CH-8406 Winterthur
Skype:   flyingpotato
Phone:   +41 44 315 4400
Fax: +41 44 315 4401
Twitter: @init7 / @kuenzler
http://www.init7.net/



signature.asc
Description: OpenPGP digital signature

RE: well-known Anycast prefixes

[David Guo via NANOG]

Hi Fredy,

Our anycast prefixes for DNS resolver

185.222.222.0/24
2a09::/48

You can add them if someone will maintain a list.

Regards,

David

-Original Message-
From: NANOG  On Behalf Of Fredy Kuenzler
Sent: Wednesday, March 20, 2019 1:13 AM
To: nanog@nanog.org
Subject: well-known Anycast prefixes

I wonder whether anyone has ever compiled a list of well-known Anycast prefixes.

Such as

1.1.1.0/24
8.8.8.0/24
9.9.9.0/24
...

Might be useful for a routing policy such as "always route hot-potato".

PS. this mail is not intended to start a flame war of hot vs. cold potato 
routing.

--
Fredy Kuenzler

Init7 (Switzerland) Ltd.
AS13030
Technoparkstrasse 5
CH-8406 Winterthur
Twitter: @init7 / @kuenzler
http://www.init7.net/

Re: well-known Anycast prefixes

[Bill Woodcock]

> On Mar 19, 2019, at 10:12 AM, Fredy Kuenzler  wrote:
> 
> I wonder whether anyone has ever compiled a list of well-known Anycast
> prefixes.

I don’t know of one.

It seems like a good idea.

BGP-multi-hop might be a reasonable way to collect them.

If others agree that it’s a good idea, and it’s not stepping on anyone’s toes, 
PCH would be happy to host/coordinate.

-Bill



signature.asc
Description: Message signed with OpenPGP

well-known Anycast prefixes

[Fredy Kuenzler]

I wonder whether anyone has ever compiled a list of well-known Anycast
prefixes.

Such as

1.1.1.0/24
8.8.8.0/24
9.9.9.0/24
...

Might be useful for a routing policy such as "always route hot-potato".

PS. this mail is not intended to start a flame war of hot vs. cold
potato routing.

-- 
Fredy Kuenzler

Init7 (Switzerland) Ltd.
AS13030
Technoparkstrasse 5
CH-8406 Winterthur
Twitter: @init7 / @kuenzler
http://www.init7.net/



signature.asc
Description: OpenPGP digital signature

Re: Dedicated Server and IP anycast provider recommendation

[John Kristoff]

On Mon, 13 Aug 2018 12:31:44 +
Étienne via NANOG  wrote:

> Not sure you're still looking for something, but there's this 
> spreadsheet that has a few pointers: http://bgp.services/

Thanks again.  This is at least the third time someone has pointed this
web page out to me.  :-)

To summarize, NetActuate and Packet were two of the most commonly
suggested providers and probably the closest to what I was asking for.

Some well known providers can do this, but don't have a specific web
page / service offering describing it as a standard product offering.

Many others, such as the often mentioned Vultr, require you to bring
your own BGP speaker and prefix.

What I was asking for isn't a popular, mass-marketed product offering,
but is available.  It just requires some research and careful
evaluation.

I think I"m familiar many hosting providers, especially the so-called
low-end/cost providers for an entirely separate project. I'm always
interested in those too, but not specifically for their ability to
speak BGP.  I maintain my own list here of providers I use here:

  

Follow ups about that list are best sent to me directly or at that page
directly.

Thanks for all the on/off list replies and suggestions.

John

Re: Dedicated Server and IP anycast provider recommendation

[Damian Menscher via NANOG]

Not quite a dedicated server, but may meet your needs anyway:
https://cloud.google.com/load-balancing/

Damian

On Tue, Aug 7, 2018 at 6:50 AM John Kristoff  wrote:

> Friends,
>
> For those that may have used or know of a service like this.  I know
> some exist, but it doesn't seem to be that popular or widely advertised
> as a standard service.
>
> I'm interested in pointers to a hosting/network provider that leases
> dedicated servers and can provide an anycast IP address assignment to
> two or more US-diversely connected POPs, but with reasonably consistent
> routing (e.g. peering, transit).  A customer-shared prefix is OK. I'm
> interested in pointers to networks that would provide the prefix and
> handle all the routing.
>
> If you represent a network and sales is part of your job, I don't mind
> an off list pointer to a web page describing such a service, but please,
> this is not an invitation for "call me to discuss needs and options"
> replies nor an opportunity to get me on your customer prospect list.
> You likely ensure I never do business with you if you do either of
> those things.  :-)
>
> Thank you,
>
> John
>

Re: Dedicated Server and IP anycast provider recommendation

[Étienne via NANOG]

On 07/08/18 14:49, John Kristoff wrote:

Friends,

For those that may have used or know of a service like this.  I know
some exist, but it doesn't seem to be that popular or widely advertised
as a standard service.

I'm interested in pointers to a hosting/network provider that leases
dedicated servers and can provide an anycast IP address assignment to
two or more US-diversely connected POPs, but with reasonably consistent
routing (e.g. peering, transit).  A customer-shared prefix is OK. I'm
interested in pointers to networks that would provide the prefix and
handle all the routing.
Not sure you're still looking for something, but there's this 
spreadsheet that has a few pointers: http://bgp.services/


Cheers,

--
Étienne

Re: Dedicated Server and IP anycast provider recommendation

[Tommy Bowditch]

Hi all,

As it was mentioned just thought I'd chime in - I'm the operator of
https://bgp.services/ - if you have any suggestions/additions shoot me a
mail off-list and I'll be more than happy to add them.

Tom

On Tue, Aug 7, 2018 at 5:28 PM Yugandhar Veeramachaneni 
wrote:

> Hello,
>
> http://bgp.services is a community maintained spreadsheet with details of
> providers who offer BGP sessions across the world. I think it will be
> useful for you.
>
> Regards,
> Yugandhar
>
>
> On Tue, Aug 7, 2018 at 9:06 AM Anthony Leto  wrote:
>
> > Hi,
> >
> > I would checkout NetActuate. They are pretty awesome when it comes to
> > Anycast IPv4 /IPv6 and they do custom VM's.
> >
> > Anthony Leto
> >
> > On 8/7/2018 2:51:59 PM, John Kristoff  wrote:
> >
> > Friends,
> >
> > For those that may have used or know of a service like this. I know
> > some exist, but it doesn't seem to be that popular or widely advertised
> > as a standard service.
> >
> > I'm interested in pointers to a hosting/network provider that leases
> > dedicated servers and can provide an anycast IP address assignment to
> > two or more US-diversely connected POPs, but with reasonably consistent
> > routing (e.g. peering, transit). A customer-shared prefix is OK. I'm
> > interested in pointers to networks that would provide the prefix and
> > handle all the routing.
> >
> > If you represent a network and sales is part of your job, I don't mind
> > an off list pointer to a web page describing such a service, but please,
> > this is not an invitation for "call me to discuss needs and options"
> > replies nor an opportunity to get me on your customer prospect list.
> > You likely ensure I never do business with you if you do either of
> > those things. :-)
> >
> > Thank you,
> >
> > John
> >
>

Re: Dedicated Server and IP anycast provider recommendation

[A. Pishdadi]

Gigenet.com can be done in two or three locations , La, Chicago, ashburn,
and done with dedicated, colo or cloud.

On Tue, Aug 7, 2018 at 8:50 AM John Kristoff  wrote:

> Friends,
>
> For those that may have used or know of a service like this.  I know
> some exist, but it doesn't seem to be that popular or widely advertised
> as a standard service.
>
> I'm interested in pointers to a hosting/network provider that leases
> dedicated servers and can provide an anycast IP address assignment to
> two or more US-diversely connected POPs, but with reasonably consistent
> routing (e.g. peering, transit).  A customer-shared prefix is OK. I'm
> interested in pointers to networks that would provide the prefix and
> handle all the routing.
>
> If you represent a network and sales is part of your job, I don't mind
> an off list pointer to a web page describing such a service, but please,
> this is not an invitation for "call me to discuss needs and options"
> replies nor an opportunity to get me on your customer prospect list.
> You likely ensure I never do business with you if you do either of
> those things.  :-)
>
> Thank you,
>
> John
>

Re: Dedicated Server and IP anycast provider recommendation

[Philippe Bonvin via NANOG]

We use http://packet.net/ for our anycast setup, their pricing isn't cheap 
compared to Vultr but it worth try.

If you commit on long-term you can get custom pricing.


From: NANOG  on behalf of Siyuan Miao 

Sent: Tuesday, August 7, 2018 16:29
To: anth...@fms.io
Cc: nanog@nanog.org
Subject: Re: Dedicated Server and IP anycast provider recommendation

I would recommend Vultr, you can bring your own IP address and set up BGP
session using VM.

Their BGP service are fully automated and provide well-documented BGP
community for traffic engineering.

--
Siyuan Miao
Misaka Network, Inc | https://misaka.io

On Tue, Aug 7, 2018 at 10:06 PM Anthony Leto  wrote:

> Hi,
>
> I would checkout NetActuate. They are pretty awesome when it comes to
> Anycast IPv4 /IPv6 and they do custom VM's.
>
> Anthony Leto
>
> On 8/7/2018 2:51:59 PM, John Kristoff  wrote:
>
> Friends,
>
> For those that may have used or know of a service like this. I know
> some exist, but it doesn't seem to be that popular or widely advertised
> as a standard service.
>
> I'm interested in pointers to a hosting/network provider that leases
> dedicated servers and can provide an anycast IP address assignment to
> two or more US-diversely connected POPs, but with reasonably consistent
> routing (e.g. peering, transit). A customer-shared prefix is OK. I'm
> interested in pointers to networks that would provide the prefix and
> handle all the routing.
>
> If you represent a network and sales is part of your job, I don't mind
> an off list pointer to a web page describing such a service, but please,
> this is not an invitation for "call me to discuss needs and options"
> replies nor an opportunity to get me on your customer prospect list.
> You likely ensure I never do business with you if you do either of
> those things. :-)
>
> Thank you,
>
> John
>


[EDSI-Tech Sarl]<http://www.edsi-tech.com>
Philippe Bonvin, Directeur, Ing. MSc. in Computer Science, IPMA, eMBA
EDSI-Tech Sàrl<http://www.edsi-tech.com>
EPFL Innovation Park, Batiment C, 1015 Lausanne, Suisse | Téléphone: +41 (0) 21 
566 14 15, poste 99

Disclaimer:
This email is confidential and intended solely for the use of the individual to 
whom it is addressed. If you are not the intended recipient of this 
information, be advised that you have received this email in error and that any 
usage, disclosure, distribution, copying of the information or any part of it 
in any form whatsoever is strictly prohibited.
If you have received this email in error please notify the EDSI-Tech helpdesk 
by phone on +41 21 566 14 15 and then delete this e-mail.

Re: Dedicated Server and IP anycast provider recommendation

[Siyuan Miao]

I would recommend Vultr, you can bring your own IP address and set up BGP
session using VM.

Their BGP service are fully automated and provide well-documented BGP
community for traffic engineering.

--
Siyuan Miao
Misaka Network, Inc | https://misaka.io

On Tue, Aug 7, 2018 at 10:06 PM Anthony Leto  wrote:

> Hi,
>
> I would checkout NetActuate. They are pretty awesome when it comes to
> Anycast IPv4 /IPv6 and they do custom VM's.
>
> Anthony Leto
>
> On 8/7/2018 2:51:59 PM, John Kristoff  wrote:
>
> Friends,
>
> For those that may have used or know of a service like this. I know
> some exist, but it doesn't seem to be that popular or widely advertised
> as a standard service.
>
> I'm interested in pointers to a hosting/network provider that leases
> dedicated servers and can provide an anycast IP address assignment to
> two or more US-diversely connected POPs, but with reasonably consistent
> routing (e.g. peering, transit). A customer-shared prefix is OK. I'm
> interested in pointers to networks that would provide the prefix and
> handle all the routing.
>
> If you represent a network and sales is part of your job, I don't mind
> an off list pointer to a web page describing such a service, but please,
> this is not an invitation for "call me to discuss needs and options"
> replies nor an opportunity to get me on your customer prospect list.
> You likely ensure I never do business with you if you do either of
> those things. :-)
>
> Thank you,
>
> John
>

Re: Dedicated Server and IP anycast provider recommendation

[Yugandhar Veeramachaneni]

Hello,

http://bgp.services is a community maintained spreadsheet with details of
providers who offer BGP sessions across the world. I think it will be
useful for you.

Regards,
Yugandhar


On Tue, Aug 7, 2018 at 9:06 AM Anthony Leto  wrote:

> Hi,
>
> I would checkout NetActuate. They are pretty awesome when it comes to
> Anycast IPv4 /IPv6 and they do custom VM's.
>
> Anthony Leto
>
> On 8/7/2018 2:51:59 PM, John Kristoff  wrote:
>
> Friends,
>
> For those that may have used or know of a service like this. I know
> some exist, but it doesn't seem to be that popular or widely advertised
> as a standard service.
>
> I'm interested in pointers to a hosting/network provider that leases
> dedicated servers and can provide an anycast IP address assignment to
> two or more US-diversely connected POPs, but with reasonably consistent
> routing (e.g. peering, transit). A customer-shared prefix is OK. I'm
> interested in pointers to networks that would provide the prefix and
> handle all the routing.
>
> If you represent a network and sales is part of your job, I don't mind
> an off list pointer to a web page describing such a service, but please,
> this is not an invitation for "call me to discuss needs and options"
> replies nor an opportunity to get me on your customer prospect list.
> You likely ensure I never do business with you if you do either of
> those things. :-)
>
> Thank you,
>
> John
>

Re: Dedicated Server and IP anycast provider recommendation

[Mehmet Akcin]

+1 for Netactuate.

On Tue, Aug 7, 2018 at 8:33 AM, Owen DeLong  wrote:

> Netactuate has been doing this for years.
>
> I highly recommend them.
>
> http://netactuate.com
>
> Owen
>
>
> > On Aug 7, 2018, at 06:49, John Kristoff  wrote:
> >
> > Friends,
> >
> > For those that may have used or know of a service like this.  I know
> > some exist, but it doesn't seem to be that popular or widely advertised
> > as a standard service.
> >
> > I'm interested in pointers to a hosting/network provider that leases
> > dedicated servers and can provide an anycast IP address assignment to
> > two or more US-diversely connected POPs, but with reasonably consistent
> > routing (e.g. peering, transit).  A customer-shared prefix is OK. I'm
> > interested in pointers to networks that would provide the prefix and
> > handle all the routing.
> >
> > If you represent a network and sales is part of your job, I don't mind
> > an off list pointer to a web page describing such a service, but please,
> > this is not an invitation for "call me to discuss needs and options"
> > replies nor an opportunity to get me on your customer prospect list.
> > You likely ensure I never do business with you if you do either of
> > those things.  :-)
> >
> > Thank you,
> >
> > John
>

Re: Dedicated Server and IP anycast provider recommendation

[Owen DeLong]

Netactuate has been doing this for years. 

I highly recommend them. 

http://netactuate.com

Owen


> On Aug 7, 2018, at 06:49, John Kristoff  wrote:
> 
> Friends,
> 
> For those that may have used or know of a service like this.  I know
> some exist, but it doesn't seem to be that popular or widely advertised
> as a standard service.
> 
> I'm interested in pointers to a hosting/network provider that leases
> dedicated servers and can provide an anycast IP address assignment to
> two or more US-diversely connected POPs, but with reasonably consistent
> routing (e.g. peering, transit).  A customer-shared prefix is OK. I'm
> interested in pointers to networks that would provide the prefix and
> handle all the routing.
> 
> If you represent a network and sales is part of your job, I don't mind
> an off list pointer to a web page describing such a service, but please,
> this is not an invitation for "call me to discuss needs and options"
> replies nor an opportunity to get me on your customer prospect list.
> You likely ensure I never do business with you if you do either of
> those things.  :-)
> 
> Thank you,
> 
> John

Re: Dedicated Server and IP anycast provider recommendation

[Aaron Gould]

vultr ?  Is this the same vultr that appears to be hosting a lot of Sony 
PlayStation games ?

I've been tshooting PS4 CGNAT issues and seeing my test ps4 gaming console 
connecting to Vultr owned /27 address space all over the US Chicago, Miami, 
Seattle, etc

Aaron

> On Aug 7, 2018, at 9:43 AM, William Herrin  wrote:
> 
>> On Tue, Aug 7, 2018 at 9:58 AM, Jay Ford  wrote:
>> Depending on the details of what you're after, Packet Host and/or Vultr
>> might suffice.  They do BGP, IPv4 & IPv6 even. They have various flavors of
>> servers, some of which might meet your definition of "dedicated".  I do a
>> little BGP-based anycast DNS with both of them, with pretty decent results.
> 
> I can second Vultr. Their BGP+VPS product is inexpensive, it worked
> right the first time and it has continued working properly.
> 
> Regards,
> Bill Herrin
> 
> 
> 
> -- 
> William Herrin  her...@dirtside.com  b...@herrin.us
> Dirtside Systems . Web: <http://www.dirtside.com/>

Re: Dedicated Server and IP anycast provider recommendation

[William Herrin]

On Tue, Aug 7, 2018 at 9:58 AM, Jay Ford  wrote:
> Depending on the details of what you're after, Packet Host and/or Vultr
> might suffice.  They do BGP, IPv4 & IPv6 even. They have various flavors of
> servers, some of which might meet your definition of "dedicated".  I do a
> little BGP-based anycast DNS with both of them, with pretty decent results.

I can second Vultr. Their BGP+VPS product is inexpensive, it worked
right the first time and it has continued working properly.

Regards,
Bill Herrin



-- 
William Herrin  her...@dirtside.com  b...@herrin.us
Dirtside Systems . Web: <http://www.dirtside.com/>

Re: Dedicated Server and IP anycast provider recommendation

[Andrew Latham]

As mentioned https://www.packet.net/ is the MaaS provider that fits the
bill. You can optionally order servers like this from HE and other IXPs for
a price.

On Tue, Aug 7, 2018 at 8:51 AM John Kristoff  wrote:

> Friends,
>
> For those that may have used or know of a service like this.  I know
> some exist, but it doesn't seem to be that popular or widely advertised
> as a standard service.
>
> I'm interested in pointers to a hosting/network provider that leases
> dedicated servers and can provide an anycast IP address assignment to
> two or more US-diversely connected POPs, but with reasonably consistent
> routing (e.g. peering, transit).  A customer-shared prefix is OK. I'm
> interested in pointers to networks that would provide the prefix and
> handle all the routing.
>
> If you represent a network and sales is part of your job, I don't mind
> an off list pointer to a web page describing such a service, but please,
> this is not an invitation for "call me to discuss needs and options"
> replies nor an opportunity to get me on your customer prospect list.
> You likely ensure I never do business with you if you do either of
> those things.  :-)
>
> Thank you,
>
> John
>


-- 
- Andrew "lathama" Latham -

Re: Dedicated Server and IP anycast provider recommendation

[Anthony Leto]

Hi,

I would checkout NetActuate. They are pretty awesome when it comes to Anycast 
IPv4 /IPv6 and they do custom VM's.

Anthony Leto

On 8/7/2018 2:51:59 PM, John Kristoff  wrote:

Friends,

For those that may have used or know of a service like this. I know
some exist, but it doesn't seem to be that popular or widely advertised
as a standard service.

I'm interested in pointers to a hosting/network provider that leases
dedicated servers and can provide an anycast IP address assignment to
two or more US-diversely connected POPs, but with reasonably consistent
routing (e.g. peering, transit). A customer-shared prefix is OK. I'm
interested in pointers to networks that would provide the prefix and
handle all the routing.

If you represent a network and sales is part of your job, I don't mind
an off list pointer to a web page describing such a service, but please,
this is not an invitation for "call me to discuss needs and options"
replies nor an opportunity to get me on your customer prospect list.
You likely ensure I never do business with you if you do either of
those things. :-)

Thank you,

John