Re: [dns-operations] Route 53 Unexpected geo location behavior

[Dave Lawrence]

Dan McCombs via dns-operations writes:
> Ah, yes, so in this case the addresses given back when no edns
> subnet is provided are the addresses of servers in eu-west, whereas
> with the resolver's own IP (or /24 subnet, or the subnet of clients
> querying it) as the edns subnet gets more expected us-west responses
> since this resolver and clients are in San Francisco.

Yes, and this is the case for several other implementations.  I can't
say that it's true of *all* ECS implementations, but personally know
of two others besides AWS that make this distinction.

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Route 53 Unexpected geo location behavior

[Dan McCombs via dns-operations]

--- Begin Message ---
>
>  If there is a performance issue with one set of records versus another
> (you don't
> really say why the differing responses matter in your email), you might
> try contacting the nameserver operator directly to discuss the issue.
>

Ah, yes, so in this case the addresses given back when no edns subnet is
provided are the addresses of servers in eu-west, whereas with the
resolver's own IP (or /24 subnet, or the subnet of clients querying it) as
the edns subnet gets more expected us-west responses since this resolver
and clients are in San Francisco.

When contacting Atlassian, they seemed to shrug it off as Route 53 behavior
rather than something they control, so I'm curious if any Route 53
folks are here and could say whether this is expected behavior or not, or
if this could be something with Atlassian's DNS configuration in their
Route 53 service.

Thanks,

-Dan



Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com


On Sat, Jun 10, 2023 at 3:59 PM Robert Edmonds  wrote:

> Dan McCombs via dns-operations wrote:
> > Hi everyone,
> >
> > We've stumbled upon what seems like unexpected behavior with Route 53
> returning
> > answers based on IP geo location to our resolvers.
> >
> > According to their documentation:
> >
> > When a browser or other viewer uses a DNS resolver that does not
> support
> > edns-client-subnet, Route 53 uses the source IP address of the DNS
> resolver
> > to approximate the location of the user and responds to geolocation
> queries
> > with the DNS record for the resolver's location.
>
> Here is the page that text is from, and the description of the other
> case (when a resolver does send an EDNS Client Subnet payload):
>
>
> https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-edns0.html
>
> When a browser or other viewer uses a DNS resolver that does not
> support edns-client-subnet, Route 53 uses the source IP address of
> the DNS resolver to approximate the location of the user and
> responds to geolocation queries with the DNS record for the
> resolver's location.
>
> When a browser or other viewer uses a DNS resolver that does support
> edns-client-subnet, the DNS resolver sends Route 53 a truncated
> version of the user's IP address. Route 53 determines the location
> of the user based on the truncated IP address rather than the source
> IP address of the DNS resolver; this typically provides a more
> accurate estimate of the user's location. Route 53 then responds to
> geolocation queries with the DNS record for the user's location.
>
> That text seems to be entirely consistent with nameserver behavior that
> sends different records to a resolver when it supplies its own IP
> address (or its own subnet) via the ECS option versus when it does not,
> because the resolver is asking for two different things. In the former
> case, the resolver is asking for responses tailored to a precise IP (or
> a precise subnet). In the latter case, the resolver is asking for
> responses on behalf of the whole client population that utilizes the
> resolver. These are not necessarily the same.
>
> > If it were using the resolver's source IP address to determine
> geolocation when
> > no edns-client-subnet is sent, I would expect the same answers as when
> sending
> > that address as the edns-client-subnet. What's going on here?
>
> I don't see anything in these DNS responses that is inconsistent with
> their documentation, or with the ECS specification. If there is a
> performance issue with one set of records versus another (you don't
> really say why the differing responses matter in your email), you might
> try contacting the nameserver operator directly to discuss the issue.
>
> > Our resolvers are co-located with our user's instances in the same
> datacenters,
> > so we don't configure our resolvers to send edns-client-subnet since
> they're
> > not geographically different (and in fact in the same IP blocks). This
> is the
> > first time we've had a user contact us about this, so I'm not sure if
> something
> > changed with Route 53 recently, if this is being caused by configuration
> > specific to the atlassian.net zone, or if somehow we just haven't had
> users
> > notice that they were being affected by this for years.
>
> There are many ways for operators of ECS-enabled nameservers to decide
> how to tailor DNS responses when receiving an ECS-enabled query.
> Geolocation, network topology, and actual performance may all be
> relevant. Even if your resolver instances are receiving queries from
> customer instances located in the same physical data center, those
> customer instances may themselves be forwarding traffic from eyeballs
> located further away (e.g.: https://www.digitalocean.com/solutions/vpn).
> A data-driven approach on the part of the nameserver operator could
> plausibly choose to send different kinds of responses to resolvers that
> are serving 

Re: [dns-operations] Route 53 Unexpected geo location behavior

[Robert Edmonds]

Dan McCombs via dns-operations wrote:
> Hi everyone,
> 
> We've stumbled upon what seems like unexpected behavior with Route 53 
> returning
> answers based on IP geo location to our resolvers.
> 
> According to their documentation:
> 
> When a browser or other viewer uses a DNS resolver that does not support
> edns-client-subnet, Route 53 uses the source IP address of the DNS 
> resolver
> to approximate the location of the user and responds to geolocation 
> queries
> with the DNS record for the resolver's location.

Here is the page that text is from, and the description of the other
case (when a resolver does send an EDNS Client Subnet payload):


https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-edns0.html

When a browser or other viewer uses a DNS resolver that does not
support edns-client-subnet, Route 53 uses the source IP address of
the DNS resolver to approximate the location of the user and
responds to geolocation queries with the DNS record for the
resolver's location.

When a browser or other viewer uses a DNS resolver that does support
edns-client-subnet, the DNS resolver sends Route 53 a truncated
version of the user's IP address. Route 53 determines the location
of the user based on the truncated IP address rather than the source
IP address of the DNS resolver; this typically provides a more
accurate estimate of the user's location. Route 53 then responds to
geolocation queries with the DNS record for the user's location.

That text seems to be entirely consistent with nameserver behavior that
sends different records to a resolver when it supplies its own IP
address (or its own subnet) via the ECS option versus when it does not,
because the resolver is asking for two different things. In the former
case, the resolver is asking for responses tailored to a precise IP (or
a precise subnet). In the latter case, the resolver is asking for
responses on behalf of the whole client population that utilizes the
resolver. These are not necessarily the same.

> If it were using the resolver's source IP address to determine geolocation 
> when
> no edns-client-subnet is sent, I would expect the same answers as when sending
> that address as the edns-client-subnet. What's going on here?

I don't see anything in these DNS responses that is inconsistent with
their documentation, or with the ECS specification. If there is a
performance issue with one set of records versus another (you don't
really say why the differing responses matter in your email), you might
try contacting the nameserver operator directly to discuss the issue.

> Our resolvers are co-located with our user's instances in the same 
> datacenters,
> so we don't configure our resolvers to send edns-client-subnet since they're
> not geographically different (and in fact in the same IP blocks). This is the
> first time we've had a user contact us about this, so I'm not sure if 
> something
> changed with Route 53 recently, if this is being caused by configuration
> specific to the atlassian.net zone, or if somehow we just haven't had users
> notice that they were being affected by this for years.

There are many ways for operators of ECS-enabled nameservers to decide
how to tailor DNS responses when receiving an ECS-enabled query.
Geolocation, network topology, and actual performance may all be
relevant. Even if your resolver instances are receiving queries from
customer instances located in the same physical data center, those
customer instances may themselves be forwarding traffic from eyeballs
located further away (e.g.: https://www.digitalocean.com/solutions/vpn).
A data-driven approach on the part of the nameserver operator could
plausibly choose to send different kinds of responses to resolvers that
are serving eyeballs that are dispersed over a larger catchment area.

-- 
Robert Edmonds
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Route 53 Unexpected geo location behavior

[Viktor Dukhovni]

On Sat, Jun 10, 2023 at 05:52:27AM -0400, Dan McCombs via dns-operations wrote:

> > dig -b 64.227.108.32 +short +subnet=64.227.108.32/32 \
> >   @ns-1339.awsdns-39.org doitb-synthetic.atlassian.net

FWIW, though it seems to not matter with the server in question, note
that a scope prefix length of 32 is not recommended, a better choice
would be 24, mirroring the typical behaviour of ECS-enabled recursive
resolvers.

https://datatracker.ietf.org/doc/html/rfc7871#section-11.1

-- 
Viktor.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Route 53 Unexpected geo location behavior

[Dan McCombs via dns-operations]

--- Begin Message ---
Oops, thanks Crist. Yes, that was a typo in my example and this
inconsistency does happen even when it's correct. 

I retested for good measure:

> dig -b 64.227.108.32 @ns-1339.awsdns-39.org doitb-synthetic.atlassian.net
> +short +nosubnet

104.192.142.19
> 104.192.142.20
> 104.192.142.18


> dig -b 64.227.108.32 @ns-1339.awsdns-39.org doitb-synthetic.atlassian.net
> +short +subnet=64.227.108.32/32

104.192.138.12
> 104.192.138.13


Thanks,

-Dan


Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com


On Fri, Jun 9, 2023 at 11:57 PM Crist Clark  wrote:

> Well, in your example below, looks like a typo. You have the first octet
> in the subnet option set to 67, when it’s 64 for the server.
>
> Is that just a typo in the example below? Do you still see inconsistencies
> when it’s correct?
>
>
> On Fri, Jun 9, 2023 at 2:25 PM Dan McCombs via dns-operations <
> dns-operati...@dns-oarc.net> wrote:
>
>>
>>
>>
>> -- Forwarded message --
>> From: Dan McCombs 
>> To: dns-operations@lists.dns-oarc.net
>> Cc:
>> Bcc:
>> Date: Fri, 9 Jun 2023 16:58:51 -0400
>> Subject: Route 53 Unexpected geo location behavior
>> Hi everyone,
>>
>> We've stumbled upon what seems like unexpected behavior with Route 53
>> returning answers based on IP geo location to our resolvers.
>>
>> According to their documentation
>> <https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-edns0.html>
>> :
>>
>>> When a browser or other viewer uses a DNS resolver that does not support
>>> edns-client-subnet, Route 53 uses the source IP address of the DNS resolver
>>> to approximate the location of the user and responds to geolocation queries
>>> with the DNS record for the resolver's location.
>>>
>>
>> But that doesn't seem to be the case. On a resolver with the address
>> 64.227.108.32, if we query at an awsdns authoritative from 64.227.108.32
>> without edns client subnet, we get one set of answers:
>>
>>> > dig -b 64.227.108.32 @ns-1339.awsdns-39.org
>>> doitb-synthetic.atlassian.net +short +nosubnet
>>
>> 104.192.142.20
>>> 104.192.142.19
>>> 104.192.142.18
>>
>>
>> But if we send the resolver's own same IP in edns-client-subnet, we get a
>> different set of answers:
>>
>>> >  dig -b 64.227.108.32 @ns-1339.awsdns-39.org
>>> doitb-synthetic.atlassian.net +short +subnet=67.227.108.32/32
>>
>> 104.192.138.13
>>> 104.192.138.12
>>
>>
>> If it were using the resolver's source IP address to determine
>> geolocation when no edns-client-subnet is sent, I would expect the same
>> answers as when sending that address as the edns-client-subnet. What's
>> going on here?
>>
>> Our resolvers are co-located with our user's instances in the same
>> datacenters, so we don't configure our resolvers to send edns-client-subnet
>> since they're not geographically different (and in fact in the same IP
>> blocks). This is the first time we've had a user contact us about this, so
>> I'm not sure if something changed with Route 53 recently, if this is being
>> caused by configuration specific to the atlassian.net zone, or if
>> somehow we just haven't had users notice that they were being affected by
>> this for years.
>>
>> Any insights would be appreciated,
>>
>> -Dan
>>
>>
>> Dan McCombs
>> Senior Engineer I - DNS
>> dmcco...@digitalocean.com
>>
>>
>>
>> -- Forwarded message --
>> From: Dan McCombs via dns-operations 
>> To: dns-operations@lists.dns-oarc.net
>> Cc:
>> Bcc:
>> Date: Fri, 9 Jun 2023 16:58:51 -0400
>> Subject: [dns-operations] Route 53 Unexpected geo location behavior
>> ___
>> dns-operations mailing list
>> dns-operations@lists.dns-oarc.net
>> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>>
>
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

[dns-operations] Route 53 Unexpected geo location behavior

[Dan McCombs via dns-operations]

--- Begin Message ---
Hi everyone,

We've stumbled upon what seems like unexpected behavior with Route 53
returning answers based on IP geo location to our resolvers.

According to their documentation

:

> When a browser or other viewer uses a DNS resolver that does not support
> edns-client-subnet, Route 53 uses the source IP address of the DNS resolver
> to approximate the location of the user and responds to geolocation queries
> with the DNS record for the resolver's location.
>

But that doesn't seem to be the case. On a resolver with the address
64.227.108.32, if we query at an awsdns authoritative from 64.227.108.32
without edns client subnet, we get one set of answers:

> > dig -b 64.227.108.32 @ns-1339.awsdns-39.org
> doitb-synthetic.atlassian.net +short +nosubnet

104.192.142.20
> 104.192.142.19
> 104.192.142.18


But if we send the resolver's own same IP in edns-client-subnet, we get a
different set of answers:

> >  dig -b 64.227.108.32 @ns-1339.awsdns-39.org
> doitb-synthetic.atlassian.net +short +subnet=67.227.108.32/32

104.192.138.13
> 104.192.138.12


If it were using the resolver's source IP address to determine geolocation
when no edns-client-subnet is sent, I would expect the same answers as when
sending that address as the edns-client-subnet. What's going on here?

Our resolvers are co-located with our user's instances in the same
datacenters, so we don't configure our resolvers to send edns-client-subnet
since they're not geographically different (and in fact in the same IP
blocks). This is the first time we've had a user contact us about this, so
I'm not sure if something changed with Route 53 recently, if this is being
caused by configuration specific to the atlassian.net zone, or if somehow
we just haven't had users notice that they were being affected by this for
years.

Any insights would be appreciated,

-Dan


Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations