Re: russian prefixes

[Baldur Norddahl]

On Wed, Jul 28, 2021 at 11:29 PM Randy Bush  wrote:

>
> https://www.businessinsider.com/russia-cuts-self-off-from-global-internet-tests-defenses-rbc-2021-7
> says "Russia disconnected itself from the rest of the internet, a test
> of its new defense from cyber warfare, report says"
>

Would that even be effective? Surely a state sponsored cyber warfare attack
can use any domestic internet connection within Russia to continue the
attacks.

russian prefixes

[Randy Bush]

https://www.businessinsider.com/russia-cuts-self-off-from-global-internet-tests-defenses-rbc-2021-7
says "Russia disconnected itself from the rest of the internet, a test
of its new defense from cyber warfare, report says"

did this show up in bgp?  e.g. rv/ris?

randy

Industry Tools + As Heard on Twitter

[Nanog News]

*Tools + Resources for Industry Pros: IPv6 Website Validator*NANOG loves
sharing valuable tools and resources for our community of students and new
+ seasoned professionals.

*IPv6-test.com is a free service that checks your IPv6 and IPv4
connectivity and speed.* Diagnose connection problems, discover which
address(es) you are currently using to browse the Internet, and what is
your browser's protocol of choice when both v6 and v4 are available.

*IPv6 Website Validator *

*As Heard on Twitter *

John Kristoff
@jtkristoff

·
Jul 7 
Hello friends of
@NANOG 
Women in Tech, perhaps you can reach out and offer pointers and references?
Quote Tweet
Tracket Pacer
@TracketPacer
· Jul 4
I’m sad bc there are s many programs out there designed to get girls &
women interested in coding/programming… but really nothing for Network
Engineering. & NetEng is at LEAST as cool as coding. How do we fix
this? [image:
Thinking face]
Show this thread
*NANOG Stories of Women in Tech *

Re: Anycast but for egress

[Glenn McGurrin via NANOG]

I'd had a similar thought/question, though keeping the geo diversity, 
you manage the crawlers, and are making contact individually with these 
sites from what you have stated (and so don't need a one size fit's all 
list for public posting), so why not have a restricted subset of the 
crawlers handle sites with these issues (which subset may be unique per 
site, which makes maintaining even load balancing not overly complex 
/limiting, especially as you are using nat anyway, so multiple servers 
can be behind each ip and that number can vary).  That let's you have 
geo diversity (or even multi cloud diversity) for every site, but each 
site that needs this IP whitelisting only needs 3-5 IP's at any site, 
but yet you can distribute load over a much larger overall set of 
machines and nat gateways.


As I understand it even CDN's that anycast TCP (externally or internally 
[load balancing via routers and multi path]) do similar by spreading 
load over multiple IP's at the DNS layer first.


As the transition to IPv6 happens you may have it easier as getting a 
large enough allocation to allow for splitting it out into multiple 
subnets advertised from different locations without providers dropping 
the route as too long a prefix is much easier on the v6 side, so you 
could give one /36 or /40 or even /44 out to whitelist but have /48's at 
each location.  For sites with ipv6 support that may help now, but it 
won't help all sites for quite some time, though the number that support 
v6 is slowly getting better.  For the foreseeable future you still need 
to handle the v4 side one way or another though.


On 7/28/2021 10:21 AM, William Herrin wrote:

On Wed, Jul 28, 2021 at 6:04 AM Vimal  wrote:

My intention is to run a web-crawling service on a public cloud. This service
is geographically distributed, and therefore will run in multiple regions
around the world inside AWS... this means there will be multiple AWS VPCs,
each with their own NAT gateway, and traffic destined to websites
that we crawl will appear to come from this NAT gateway's IP address.


Hello,

AWS does not provide the ability to attach anycasted IP addresses to a
NAT gateway, regardless of whether it would work, so that's the end of
your quest.


The reason I want a predictable IP is to communicate this IP to website
owners so they can allow access from these IPs into their networks.
I chose IP as an example; it can also be a subnet, but what I don't want to
provide is a list of 100 different IP addresses without any predictability.


If you bring your own IP addresses, you can attach a separate /24s of
them to your VPCs in each region, providing you with a single
predictable range of source addresses. You will find it difficult and
expensive to acquire that many IP addresses from the regional
registries for the purpose you describe.


Silly question but: for a web crawler, why do you care whether it has
the limited geographically distribution that a cloud service provides?
It's a parallel batch task. It doesn't exactly matter whether you have
minimum latency.

Regards,
Bill Herrin

Re: Anycast but for egress

[Mark Tinka]

On 7/28/21 17:09, Bill Woodcock wrote:


I was about to say something about us having equal success over 105 or so 
countries, when I came to the realization that inviting quantitative 
comparisons of manhood with Mark is the very definition of folly.  :-)


Well, we are nowhere close to the 105 countries PCH boasts. That's a 
whole other level of scale :-).


Impressed!



Anyway, yeah, the folks who were scared of anycast in the 1990s were running 
from shadows, not basing it on experience or data.  In the real world, the 
number of stateful flows affected by route changes is dwarfed by those 
disrupted by other causes, and is immeasurably small.  And when they do crop up 
on the radar, it’s almost always someone’s equal-cost-multi-path gone wrong, 
rather than an actual shift.  So, not an issue at all in the real world, just 
in the imaginations of folks who thought TCP was a complex thing reserved for 
the specific use-cases that they’d already conceived of in the 1980s.  Took a 
while to get beyond their protestations, but here we are in the 21st century.  
Planck's principle holds.  Science progresses one funeral at a time.


100%.

Mark.

Call for Participation -- ICANN DNSSEC and Security Workshop for ICANN72 Virtual Annual General Meeting

[Jacques Latour]

Hello NANOG!
This is the CfP for our next DNSSEC & Security workshop @ ICANN72.
Jacques

Call for Participation -- ICANN DNSSEC and Security Workshop for ICANN72 
Virtual Annual General Meeting

In cooperation with the ICANN Security and Stability Advisory Committee (SSAC), 
we are planning a DNSSEC and Security Workshop for the ICANN72 Annual General 
Meeting being held virtually from 23-28 October 2021 in the Pacific Daylight 
Time Zone (UTC -7). This workshop date will be determined once ICANN creates a 
block schedule for us to follow; then we will be able to request a day and 
time. The DNSSEC and Security Workshop has been a part of ICANN meetings for 
several years and has provided a forum for both experienced and new people to 
meet, present and discuss current and future DNSSEC deployments.  For 
reference, the most recent session was held at the ICANN71 Virtual Meeting on 
14 June 2021. The presentations and transcripts are available at 
https://71.schedule.icann.org/meetings/3q22SHqif9XF5nFqG and 
https://71.schedule.icann.org/meetings/vv7XkuePvghwFaLgt

The DNSSEC Workshop Program Committee is developing a program.  Proposals will 
be considered for the following topic areas and included if space permits.  In 
addition, we welcome suggestions for additional topics either for inclusion in 
the ICANN72 workshop, or for consideration for future workshops.

1.  Global DNSSEC Activities Panel
For this panel, we are seeking participation from those who have been involved 
in DNSSEC deployment as well as from those who have not deployed DNSSEC but who 
have a keen interest in the challenges and benefits of deployment, including 
Root Key Signing Key (KSK) Rollover activities and plans.

2.  DNSSEC Best Practice
Now that DNSSEC has become an operational norm for many registries, registrars, 
and ISPs, what have we learned about how we manage DNSSEC?  Do you still 
submit/accept DS records with Digest Type 1? What is the best practice around 
key roll-overs?  What about Algorithm roll-overs? Do you use and support DNSKEY 
Algorithms 13-16? How often do you review your disaster recovery procedures? Is 
there operational familiarity within your customer support teams? What 
operational statistics have we gathered about DNSSEC? Are there experiences 
being documented in the form of best practices, or something similar, for 
transfer of signed zones?  Activities and issues related to DNSSEC in the DNS 
Root Zone are also desired.

3. DNSSEC Deployment Challenges
The program committee is seeking input from those that are interested in 
implementation of DNSSEC but have general or particular concerns with DNSSEC.  
In particular, we are seeking input from individuals that would be willing to 
participate in a panel that would discuss questions of the following nature:
- Are there any policies directly or indirectly impeding your DNSSEC 
deployment? (RRR model, CDS/CDNSKEY automation)
- What are your most significant concerns with DNSSEC, e.g., complexity, 
training, implementation, operation or something else?
- What do you expect DNSSEC to do for you and what doesn't it do?
- What do you see as the most important trade-offs with respect to doing or not 
doing DNSSEC?

4. Security Panel
The program committee is looking for presentations on DNS and Routing topics 
that could impact the security and/or stability of the internet. .
- DoH and DoT implementation issues, challenges and opportunities
- RPKI adoption and implementation  issues, challenges and opportunities
- BGP/routing/hijack issues, challenges and opportunities
- MANRS implementation challenges and opportunities
- Emerging threats that could impact (real or perceived)  the security and/or 
stability of the internet
- Domain hacking/hijacking prevention, best practice and techniques
- Browser related security implementations
- DMARC Challenges, opportunities and Best Practices
- BGP Flowspec challenges, opportunities and Best Practices

If you are interested in participating, please send a brief (1-2 sentence) 
description of your proposed presentation to dnssec-security-works...@icann.org 
by Friday, 10 September 2021

Thank you,
Kathy and Andrew
On behalf of the DNSSEC Workshop Program Committee:
Mark Elkins, DNS/ZACR
Jacques Latour, .CA
Russ Mundy, Parsons
Ondrej Filip, CZ.NIC
Yoshiro Yoneya, JPRS
Fred Baker, ISC
Dan York, Internet Society

Re: Anycast but for egress

[Bill Woodcock]

> On Jul 28, 2021, at 3:21 AM, Mark Tinka  wrote:
> On 7/28/21 01:16, Daniel Corbe wrote:
> 
>>> This is interesting... I wonder whether Anycast will still have some 
>>> failure modes and break TCP connections if routing (configuration) were to 
>>> change?  I checked the PDF linked by Bill Woodcock... while the methodology 
>>> is the same from 20y ago, would the data still be the same (order of 
>>> magnitude)? :)
> 
> We are Anycast'ing DNS (authoritative and recursive), NTP and TACACS+. All 
> works well, across 11 or so countries.

I was about to say something about us having equal success over 105 or so 
countries, when I came to the realization that inviting quantitative 
comparisons of manhood with Mark is the very definition of folly.  :-)

Anyway, yeah, the folks who were scared of anycast in the 1990s were running 
from shadows, not basing it on experience or data.  In the real world, the 
number of stateful flows affected by route changes is dwarfed by those 
disrupted by other causes, and is immeasurably small.  And when they do crop up 
on the radar, it’s almost always someone’s equal-cost-multi-path gone wrong, 
rather than an actual shift.  So, not an issue at all in the real world, just 
in the imaginations of folks who thought TCP was a complex thing reserved for 
the specific use-cases that they’d already conceived of in the 1980s.  Took a 
while to get beyond their protestations, but here we are in the 21st century.  
Planck's principle holds.  Science progresses one funeral at a time.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: Anycast but for egress

[Bill Woodcock]

> On Jul 27, 2021, at 6:15 PM, Vimal  wrote:
> 
> AWS Global Accelerator gives anycast IPs that's good for ingress, but my 
> original question was about having predictable egress IPs.
> 
> It looks like having a few EIPs/a contiguous network block is the way to go.

Yes.  Predictable and unchanging (but each unique per location) static IP 
addresses is what you’re looking for.

It would be a huge convenience to others if you could specify a single 
contiguous CIDR block for others to “permit” in their access control lists, but 
alas that would be very difficult as well…  Since BGP announcements generally 
need to be aggregated up to at least a /24 or a /48 (though people are less 
strict on the v6 side), each group of hosts numbered from the same block of 
that size would need to have internally contiguous convex routing, meaning that 
it would have to be interconnected by its own network (albeit that could be 
tunnels) and accept inbound traffic at any point on the surface of that 
network, backhauling it to the appropriate location.  So if you wanted to be 
able to identify a single CIDR block with eight locations in it, you’d either 
need to specify a /24 that was 97% wasted, and was fully internally 
interconnected (i.e. no efficiencies in localizing traffic), or you’d need to 
advertise eight /24s, which would aggregate up to a single /21, which was 99.6% 
wasted.

So, you can see why the combination of scarce IPv4 addresses, scarce BGP 
routing slots, and content routing tricks often don’t play well together.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: Anycast but for egress

[William Herrin]

On Wed, Jul 28, 2021 at 6:04 AM Vimal  wrote:
> My intention is to run a web-crawling service on a public cloud. This service
> is geographically distributed, and therefore will run in multiple regions
> around the world inside AWS... this means there will be multiple AWS VPCs,
> each with their own NAT gateway, and traffic destined to websites
> that we crawl will appear to come from this NAT gateway's IP address.

Hello,

AWS does not provide the ability to attach anycasted IP addresses to a
NAT gateway, regardless of whether it would work, so that's the end of
your quest.

> The reason I want a predictable IP is to communicate this IP to website
> owners so they can allow access from these IPs into their networks.
> I chose IP as an example; it can also be a subnet, but what I don't want to
> provide is a list of 100 different IP addresses without any predictability.

If you bring your own IP addresses, you can attach a separate /24s of
them to your VPCs in each region, providing you with a single
predictable range of source addresses. You will find it difficult and
expensive to acquire that many IP addresses from the regional
registries for the purpose you describe.


Silly question but: for a web crawler, why do you care whether it has
the limited geographically distribution that a cloud service provides?
It's a parallel batch task. It doesn't exactly matter whether you have
minimum latency.

Regards,
Bill Herrin



-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Anycast but for egress

[Randy Bush]

we, verio, did anycast tcp streaming (hour long) of the tony awards in
about '96.  solid.

randy

---
ra...@psg.com
`gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
signatures are back, thanks to dmarc header butchery

Re: Anycast but for egress

[Vimal]

AWS Global Accelerator gives anycast IPs that's good for ingress, but my
original question was about having predictable egress IPs.

It looks like having a few EIPs/a contiguous network block is the way to go.

Thanks!

On Tue, Jul 27, 2021 at 4:30 PM Andras Toth  wrote:

> Since you mentioned AWS, have you tried AWS Global Accelerator? You get a
> pair of globally anycasted static IPs.
> https://aws.amazon.com/global-accelerator/
>
> Another alternative is to request a contiguous IP range of EIPs (/28 or
> /24 etc) that you can use for your EC2 instances or VPC resources.
>
> Andras
>
> On 28 Jul 2021, at 09:19, Daniel Corbe  wrote:
>
> 
>
> On Jul 27, 2021, at 17:20, Vimal  wrote:
>
>
> Hi all, great replies. :) Let me clarify my initial question, and then
> respond one by one:
>
>
> My intention is to run a web-crawling service on a public cloud. This
> service is geographically distributed, and therefore will run in multiple
> regions around the world inside AWS... this means there will be multiple
> AWS VPCs, each with their own NAT gateway, and traffic destined to websites
> that we crawl will appear to come from this NAT gateway's IP address.
>
>
> The reason I want a predictable IP is to communicate this IP to website
> owners so they can allow access from these IPs into their networks.  I
> chose IP as an example; it can also be a subnet, but what I don't want to
> provide is a list of 100 different IP addresses without any predictability.
>
>
> I understand that this is not perfect, and would frankly not be my
> preferred approach to solve the problem but we've had requests of this
> nature from websites to create an allowlist of a limited number of
> predictable IPs so it doesn't trip their IDSs/other systems they might
> have... so we're trying to see how well it would work in practice.  For the
> moment, let's set aside the issue as to whether AWS will even let me
> advertise the same IP on all my VPC NAT gateways, and just look at whether
> it's technically feasible.  My gut feeling is that this wouldn't work well
> in practice, but I wanted to ask the experts here...
>
>
> Also, pointers on what the best practices for solving this issue are most
> welcome, so I can reference those who ask for IP addresses to this
> discussion and follow recommendations here.
>
>
> Onto the responses:
>
>
> @o...@delong.com and @wo...@pch.net athomp...@merlin.mb.ca
>
> Because there’s no good/reliable way to get the replies back to the
> correct initiating host.
>
>
> When my clients make connections outbound to anycast addresses, the
> destination is more-or-less stable, and the replies come back to the
> client's unique IP, so anycast works in that direction.  The guarantees are
> not present in the reverse direction.
>
>
> Yes, this makes sense as the destination can be anywhere around the world,
> and that routing is asymmetric as others mentioned.  However, if the
> destination service is "close" (in the routing metric sense) to the
> initiating host, anycast return IP ought to work well, right?  I understand
> this is a very important caveat and impractical to implement correctly in
> the real world.
>
>
> We use our IGP (IS-IS) for our Anycast services. We find it to be very
>
> basic, and as such, very predictable.
>
>
> This is interesting... I wonder whether Anycast will still have some
> failure modes and break TCP connections if routing (configuration) were to
> change?  I checked the PDF linked by Bill Woodcock... while the methodology
> is the same from 20y ago, would the data still be the same (order of
> magnitude)? :)
>
>
> https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf (p38)
>
> "Limited operational data shows underlying instability to be on
>
> the order of one flow per ten thousand per hour of duration."
>
>
> @dan...@corbe.net, @m...@netfire.net,
>
> Unless you’re twisting knobs, egress traffic should already exit your
> network at the closest possible egress point to its origin.  Is your
> intention to carry the traffic for longer than that?
>
> No, but I hope my intention is more clear in this email.  It's to have a
> predictable egress IP to simplify firewall rules.
>
>
> thanks all!!
>
>
>
> On Tue, Jul 27, 2021 at 12:25 PM Adam Thompson 
> wrote:
>
> Without any sarcasm: to make it harder to block.
>
> If, say, Google, always crawled your site from 8.8.1.2 (random made-up
> example) then you would see a not-insignificant number of hosts and
> networks null-routing that IP.  I have no idea why someone would do so, but
> I've seen it done many times.  Mostly by people who don't understand how
> un-special they are on the internet.  Also it would trigger IDS/IPS systems
> all over the place, having gobs and gobs of connections coming from a
> single IP.
>
>
> That's setting aside the technical issues involved; routing is often
> asymmetric, i.e. the return packet takes a different path than the inbound
> packet.  So it would, as Owen implied, be nearly impossible 

Re: Anycast but for egress

[Vimal]

On AWS once we purchase EIPs, they are allocated to our account and so we
can assign them to VPC NAT gateways.  That's our current plan.

On Tue, Jul 27, 2021 at 4:16 PM Daniel Corbe  wrote:

>
> > On Jul 27, 2021, at 17:20, Vimal  wrote:
> >
> > Hi all, great replies. :) Let me clarify my initial question, and then
> respond one by one:
> >
> > My intention is to run a web-crawling service on a public cloud. This
> service is geographically distributed, and therefore will run in multiple
> regions around the world inside AWS... this means there will be multiple
> AWS VPCs, each with their own NAT gateway, and traffic destined to websites
> that we crawl will appear to come from this NAT gateway's IP address.
> >
> > The reason I want a predictable IP is to communicate this IP to website
> owners so they can allow access from these IPs into their networks.  I
> chose IP as an example; it can also be a subnet, but what I don't want to
> provide is a list of 100 different IP addresses without any predictability.
> >
> > I understand that this is not perfect, and would frankly not be my
> preferred approach to solve the problem but we've had requests of this
> nature from websites to create an allowlist of a limited number of
> predictable IPs so it doesn't trip their IDSs/other systems they might
> have... so we're trying to see how well it would work in practice.  For the
> moment, let's set aside the issue as to whether AWS will even let me
> advertise the same IP on all my VPC NAT gateways, and just look at whether
> it's technically feasible.  My gut feeling is that this wouldn't work well
> in practice, but I wanted to ask the experts here...
> >
> > Also, pointers on what the best practices for solving this issue are
> most welcome, so I can reference those who ask for IP addresses to this
> discussion and follow recommendations here.
> >
> > Onto the responses:
> >
> > @o...@delong.com and @wo...@pch.net athomp...@merlin.mb.ca
> > > Because there’s no good/reliable way to get the replies back to the
> correct initiating host.
> >
> > > When my clients make connections outbound to anycast addresses, the
> destination is more-or-less stable, and the replies come back to the
> client's unique IP, so anycast works in that direction.  The guarantees are
> not present in the reverse direction.
> >
> > Yes, this makes sense as the destination can be anywhere around the
> world, and that routing is asymmetric as others mentioned.  However, if the
> destination service is "close" (in the routing metric sense) to the
> initiating host, anycast return IP ought to work well, right?  I understand
> this is a very important caveat and impractical to implement correctly in
> the real world.
> >
> > > We use our IGP (IS-IS) for our Anycast services. We find it to be very
> > basic, and as such, very predictable.
> >
> > This is interesting... I wonder whether Anycast will still have some
> failure modes and break TCP connections if routing (configuration) were to
> change?  I checked the PDF linked by Bill Woodcock... while the methodology
> is the same from 20y ago, would the data still be the same (order of
> magnitude)? :)
> >
> > https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf (p38)
> > "Limited operational data shows underlying instability to be on
> > the order of one flow per ten thousand per hour of duration."
> >
> > @dan...@corbe.net, @m...@netfire.net,
> > > Unless you’re twisting knobs, egress traffic should already exit your
> network at the closest possible egress point to its origin.  Is your
> intention to carry the traffic for longer than that?
> > No, but I hope my intention is more clear in this email.  It's to have a
> predictable egress IP to simplify firewall rules.
> >
> > thanks all!!
> >
> >
> > On Tue, Jul 27, 2021 at 12:25 PM Adam Thompson 
> wrote:
> > Without any sarcasm: to make it harder to block.
> > If, say, Google, always crawled your site from 8.8.1.2 (random made-up
> example) then you would see a not-insignificant number of hosts and
> networks null-routing that IP.  I have no idea why someone would do so, but
> I've seen it done many times.  Mostly by people who don't understand how
> un-special they are on the internet.  Also it would trigger IDS/IPS systems
> all over the place, having gobs and gobs of connections coming from a
> single IP.
> >
> > That's setting aside the technical issues involved; routing is often
> asymmetric, i.e. the return packet takes a different path than the inbound
> packet.  So it would, as Owen implied, be nearly impossible to ensure the
> reply packets got back to the correct TCP stack.  As an example, I'm
> multi-homed and use path-prepending, so if a packet claiming to be from
> 8.8.8.8 arrived on one of my commercial links, I would send the reply out
> the cheapest link, which in my case is a flat-rate R network (that has a
> path to Google), thus ensuring the reply does not get to the originating
> anycast node.
> >
> > 

Re: Anycast but for egress

[Vimal]

Hi all, great replies. :) Let me clarify my initial question, and then
respond one by one:

My intention is to run a web-crawling service on a public cloud. This
service is geographically distributed, and therefore will run in multiple
regions around the world inside AWS... this means there will be multiple
AWS VPCs, each with their own NAT gateway, and traffic destined to websites
that we crawl will appear to come from this NAT gateway's IP address.

The reason I want a predictable IP is to communicate this IP to
website owners so they can allow access from these IPs into their
networks.  I chose IP as an example; it can also be a subnet, but what I
don't want to provide is a list of 100 different IP addresses without any
predictability.

I understand that this is not perfect, and would frankly not be my
preferred approach to solve the problem but we've had requests of this
nature from websites to create an allowlist of a limited number of
predictable IPs so it doesn't trip their IDSs/other systems they might
have... so we're trying to see how well it would work in practice.  For the
moment, let's set aside the issue as to whether AWS will even let me
advertise the same IP on all my VPC NAT gateways, and just look at whether
it's technically feasible.  My gut feeling is that this wouldn't work well
in practice, but I wanted to ask the experts here...

Also, pointers on what the best practices for solving this issue are most
welcome, so I can reference those who ask for IP addresses to this
discussion and follow recommendations here.

Onto the responses:

@o...@delong.com and @wo...@pch.net athomp...@merlin.mb.ca
> Because there’s no good/reliable way to get the replies back to the
correct initiating host.

> When my clients make connections outbound to anycast addresses, the
destination is more-or-less stable, and the replies come back to the
client's unique IP, so anycast works in that direction.  The guarantees are
not present in the reverse direction.

Yes, this makes sense as the destination can be anywhere around the world,
and that routing is asymmetric as others mentioned.  However, if the
destination service is "close" (in the routing metric sense) to the
initiating host, anycast return IP ought to work well, right?  I understand
this is a very important caveat and impractical to implement correctly in
the real world.

> We use our IGP (IS-IS) for our Anycast services. We find it to be very
basic, and as such, very predictable.

This is interesting... I wonder whether Anycast will still have some
failure modes and break TCP connections if routing (configuration) were to
change?  I checked the PDF linked by Bill Woodcock... while the methodology
is the same from 20y ago, would the data still be the same (order of
magnitude)? :)

https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf (p38)
"Limited operational data shows underlying instability to be on
the order of one flow per ten thousand per hour of duration."

@dan...@corbe.net, @m...@netfire.net,
> Unless you’re twisting knobs, egress traffic should already exit your
network at the closest possible egress point to its origin.  Is your
intention to carry the traffic for longer than that?
No, but I hope my intention is more clear in this email.  It's to have a
predictable egress IP to simplify firewall rules.

thanks all!!


On Tue, Jul 27, 2021 at 12:25 PM Adam Thompson 
wrote:

> Without any sarcasm: to make it harder to block.
> If, say, Google, always crawled your site from 8.8.1.2 (random made-up
> example) then you would see a not-insignificant number of hosts and
> networks null-routing that IP.  I have no idea why someone would do so, but
> I've seen it done many times.  Mostly by people who don't understand how
> un-special they are on the internet.  Also it would trigger IDS/IPS systems
> all over the place, having gobs and gobs of connections coming from a
> single IP.
>
> That's setting aside the technical issues involved; routing is often
> asymmetric, i.e. the return packet takes a different path than the inbound
> packet.  So it would, as Owen implied, be nearly impossible to ensure the
> reply packets got back to the correct TCP stack.  As an example, I'm
> multi-homed and use path-prepending, so if a packet claiming to be from
> 8.8.8.8 arrived on one of my commercial links, I would send the reply out
> the cheapest link, which in my case is a flat-rate R network (that has a
> path to Google), thus ensuring the reply does *not* get to the
> originating anycast node.
>
> When my clients make connections outbound to anycast addresses, the
> destination is more-or-less stable, and the replies come back to the
> client's unique IP, so anycast works in that direction.  The guarantees are
> not present in the reverse direction.
>
> The logical extremity of this is that it would be nearly impossible for
> two anycast addresses to establish a TCP connection to each other.  (In
> general.  There will be lots of local cases 

Re: Anycast but for egress

[Mark Tinka]

On 7/28/21 01:16, Daniel Corbe wrote:


This is interesting... I wonder whether Anycast will still have some failure 
modes and break TCP connections if routing (configuration) were to change?  I 
checked the PDF linked by Bill Woodcock... while the methodology is the same 
from 20y ago, would the data still be the same (order of magnitude)? :)


In our small experience, not at all.

We are Anycast'ing DNS (authoritative and recursive), NTP and TACACS+. 
All works well, across 11 or so countries.


Mark.

Re: Anycast but for egress

[Baldur Norddahl]

Here is what I think would happen if you were to try this setup. Let's
assume you deployed in eu-west-2 (London) and eu-central-1 (Frankfurt). You
would find that you could successfully connect to a number of networks but
also that some of them would work from the "wrong" site. Eg. you would have
clients in London that require you to use the Frankfurt instance to connect
to. You would also find a lot of networks that you could not connect to
from either site. And you would have some instability where some networks
at random switch between these states. For example you could have a client
that one day works from London, the next day it is Frankfurt and then
someday it won't be working at all.

As you add more sites, you also add more ways for the traffic to end up in
the wrong places. You could have something that works with just the two
sites above but then when you add eu-west-1 (Ireland) you could suddenly
not connect to them from any of the three sites.

There IS a possible solution which is to tunnel the traffic back to the
correct site. This still requires you to use unique IP addresses for each
site, but all addresses could be in the same subnet. You would have IP
a.b.c.1 to be London, a.b.c.2 Frankfurt, a.b.c.3 Ireland. Then if London
receives traffic to a.b.c.2 you would have a tunnel to send the traffic to
Frankfurt.

Regards,

Baldur


On Wed, Jul 28, 2021 at 11:07 AM Baldur Norddahl 
wrote:

>
>
>> > On Jul 27, 2021, at 17:20, Vimal  wrote:
>> > Yes, this makes sense as the destination can be anywhere around the
>> world, and that routing is asymmetric as others mentioned.  However, if the
>> destination service is "close" (in the routing metric sense) to the
>> initiating host, anycast return IP ought to work well, right?  I understand
>> this is a very important caveat and impractical to implement correctly in
>> the real world.
>>
>
> No, there is no such thing as "close". You could have a direct peering
> with some ISP and have them still deliver the responses on the other side
> of earth. You do not control the routing of other networks and can not be
> sure what they will do.
>
> For larger networks you may also have multiple peering points. Say you
> have a peering with them in city A and city B. How do you know which of
> their IP ranges are closer to A or B? You don't. And the same goes for
> them, they have no idea if you prefer A or B. Therefore you could select A
> and they may reply to B. They may even load balance between A and B if you
> are really unlucky.
>
> Routing is asymmetric. That means you have absolutely no idea where the
> replies end up going. Often it will not be what you think is "close".
>
> I do not run anycast, but I understand that the usual way of dealing with
> these issues is to do as little as possible with anycast before redirecting
> to a unicast address. For example you could have just your DNS on anycast
> and each site would reply with unique unicast addresses. Since DNS is just
> a single pair of UDP request/response, with the first packet originating
> from a unicast client, this works well.
>
> Regards,
>
> Baldur
>
>
>

Re: Anycast but for egress

[Baldur Norddahl]

>
> > On Jul 27, 2021, at 17:20, Vimal  wrote:
> > Yes, this makes sense as the destination can be anywhere around the
> world, and that routing is asymmetric as others mentioned.  However, if the
> destination service is "close" (in the routing metric sense) to the
> initiating host, anycast return IP ought to work well, right?  I understand
> this is a very important caveat and impractical to implement correctly in
> the real world.
>

No, there is no such thing as "close". You could have a direct peering with
some ISP and have them still deliver the responses on the other side of
earth. You do not control the routing of other networks and can not be sure
what they will do.

For larger networks you may also have multiple peering points. Say you have
a peering with them in city A and city B. How do you know which of their IP
ranges are closer to A or B? You don't. And the same goes for them, they
have no idea if you prefer A or B. Therefore you could select A and they
may reply to B. They may even load balance between A and B if you are
really unlucky.

Routing is asymmetric. That means you have absolutely no idea where the
replies end up going. Often it will not be what you think is "close".

I do not run anycast, but I understand that the usual way of dealing with
these issues is to do as little as possible with anycast before redirecting
to a unicast address. For example you could have just your DNS on anycast
and each site would reply with unique unicast addresses. Since DNS is just
a single pair of UDP request/response, with the first packet originating
from a unicast client, this works well.

Regards,

Baldur