[dns-operations] Large amount of TXT records on cisco.com causing truncation

[Dan McCombs via dns-operations]

--- Begin Message ---
Hi all,

Is there anyone from Cisco on this mailing list?

I was looking through some queries earlier and noticed a large amount of
cache misses on our infra for TXT queries for cisco.com. There's a lot of
TXT records there that look like a lot of possibly stale site/domain
verification entries, and I'm wondering if those could be cleaned up to
save the extra queries hitting your nameservers.

The cisco.com nameservers are responding with the TC flag set and no
answer, so clients querying via UDP keep missing cache before retrying over
TCP. It receives a fair amount of traffic with a lot of unnecessary UDP
queries because of that. I added a rule to dnsdist to respond with TC
directly for now in that case, but I'm sure it would help both themselves
and others to clean up those records since it's a popular domain.

Take care,

-Dan


Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] hosting. nameservers partial unreachable

[Dan McCombs via dns-operations]

--- Begin Message ---
Just to follow up, these bad routes stopped being advertised around 20:00
UTC yesterday, and we're able to reach CentralNic addresses normally now
without any filters in place.

-Dan


Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com


On Mon, Jan 29, 2024 at 3:08 PM Dan McCombs 
wrote:

> We're seeing issues reaching CentralNic managed nameservers *.nic.(online,
> tech, xyl, space, etc.) in Europe as well. It looks like their prefixes are
> being hijacked by China Mobile and being advertised by FLAG telecom. We had
> to filter out those routes in order to reach them again.
>
> Anyone on this list from CentralNic that could look into things?
>
> -Dan
>
>
> Dan McCombs
> Senior Engineer I - DNS
> dmcco...@digitalocean.com
>
>
> On Mon, Jan 29, 2024 at 2:53 PM A. Schulze via dns-operations <
> dns-operati...@dns-oarc.net> wrote:
>
>>
>>
>>
>> -- Forwarded message --
>> From: "A. Schulze" 
>> To: "dns-operations@lists.dns-oarc.net" <
>> dns-operations@lists.dns-oarc.net>
>> Cc:
>> Bcc:
>> Date: Mon, 29 Jan 2024 20:50:00 +0100
>> Subject: hosting. nameservers partial unreachable
>> Hello,
>>
>> to day I noticed unreachable nameserver [a-d].nic.hosting. via IPv4
>> approved by at least two locations by such script:
>>
>> for transport in tcp notcp; do
>>for protocol in 4 6; do
>>  for host in a b c d; do
>>printf "${host}.nic.hosting/${protocol}/${transport}:"
>>if dig -${protocol} @${host}.nic.hosting. hosting. soa +timeout=1
>> +retry=1 +short +${transport} 2>/dev/null | grep --silent
>> hostmaster.centralnic.net; then
>>  printf "ok\n";
>>else
>>  printf "fail\n";
>>fi
>>  done
>>done
>> done
>>
>> from AS31334 and AS15451:
>>
>> a.nic.hosting/4/tcp:fail
>> b.nic.hosting/4/tcp:fail
>> c.nic.hosting/4/tcp:fail
>> d.nic.hosting/4/tcp:fail
>> a.nic.hosting/6/tcp:ok
>> b.nic.hosting/6/tcp:ok
>> c.nic.hosting/6/tcp:ok
>> d.nic.hosting/6/tcp:ok
>> a.nic.hosting/4/notcp:fail
>> b.nic.hosting/4/notcp:fail
>> c.nic.hosting/4/notcp:fail
>> d.nic.hosting/4/notcp:fail
>> a.nic.hosting/6/notcp:ok
>> b.nic.hosting/6/notcp:ok
>> c.nic.hosting/6/notcp:ok
>> d.nic.hosting/6/notcp:ok
>>
>> from other locations I got 16x ok
>> Sometimes I also saw some OK for IPv4, too bit resolver so not retry as
>> often.
>>
>> Any hints?
>> Andreas
>>
>>
>>
>> -- Forwarded message --
>> From: "A. Schulze via dns-operations" 
>> To: "dns-operations@lists.dns-oarc.net" <
>> dns-operations@lists.dns-oarc.net>
>> Cc:
>> Bcc:
>> Date: Mon, 29 Jan 2024 20:50:00 +0100
>> Subject: [dns-operations] hosting. nameservers partial unreachable
>> ___
>> dns-operations mailing list
>> dns-operations@lists.dns-oarc.net
>> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>>
>
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] hosting. nameservers partial unreachable

[Dan McCombs via dns-operations]

--- Begin Message ---
We're seeing issues reaching CentralNic managed nameservers *.nic.(online,
tech, xyl, space, etc.) in Europe as well. It looks like their prefixes are
being hijacked by China Mobile and being advertised by FLAG telecom. We had
to filter out those routes in order to reach them again.

Anyone on this list from CentralNic that could look into things?

-Dan


Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com


On Mon, Jan 29, 2024 at 2:53 PM A. Schulze via dns-operations <
dns-operati...@dns-oarc.net> wrote:

>
>
>
> -- Forwarded message --
> From: "A. Schulze" 
> To: "dns-operations@lists.dns-oarc.net"  >
> Cc:
> Bcc:
> Date: Mon, 29 Jan 2024 20:50:00 +0100
> Subject: hosting. nameservers partial unreachable
> Hello,
>
> to day I noticed unreachable nameserver [a-d].nic.hosting. via IPv4
> approved by at least two locations by such script:
>
> for transport in tcp notcp; do
>for protocol in 4 6; do
>  for host in a b c d; do
>printf "${host}.nic.hosting/${protocol}/${transport}:"
>if dig -${protocol} @${host}.nic.hosting. hosting. soa +timeout=1
> +retry=1 +short +${transport} 2>/dev/null | grep --silent
> hostmaster.centralnic.net; then
>  printf "ok\n";
>else
>  printf "fail\n";
>fi
>  done
>done
> done
>
> from AS31334 and AS15451:
>
> a.nic.hosting/4/tcp:fail
> b.nic.hosting/4/tcp:fail
> c.nic.hosting/4/tcp:fail
> d.nic.hosting/4/tcp:fail
> a.nic.hosting/6/tcp:ok
> b.nic.hosting/6/tcp:ok
> c.nic.hosting/6/tcp:ok
> d.nic.hosting/6/tcp:ok
> a.nic.hosting/4/notcp:fail
> b.nic.hosting/4/notcp:fail
> c.nic.hosting/4/notcp:fail
> d.nic.hosting/4/notcp:fail
> a.nic.hosting/6/notcp:ok
> b.nic.hosting/6/notcp:ok
> c.nic.hosting/6/notcp:ok
> d.nic.hosting/6/notcp:ok
>
> from other locations I got 16x ok
> Sometimes I also saw some OK for IPv4, too bit resolver so not retry as
> often.
>
> Any hints?
> Andreas
>
>
>
> -- Forwarded message --
> From: "A. Schulze via dns-operations" 
> To: "dns-operations@lists.dns-oarc.net"  >
> Cc:
> Bcc:
> Date: Mon, 29 Jan 2024 20:50:00 +0100
> Subject: [dns-operations] hosting. nameservers partial unreachable
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Route 53 Unexpected geo location behavior

[Dan McCombs via dns-operations]

--- Begin Message ---
>
>  If there is a performance issue with one set of records versus another
> (you don't
> really say why the differing responses matter in your email), you might
> try contacting the nameserver operator directly to discuss the issue.
>

Ah, yes, so in this case the addresses given back when no edns subnet is
provided are the addresses of servers in eu-west, whereas with the
resolver's own IP (or /24 subnet, or the subnet of clients querying it) as
the edns subnet gets more expected us-west responses since this resolver
and clients are in San Francisco.

When contacting Atlassian, they seemed to shrug it off as Route 53 behavior
rather than something they control, so I'm curious if any Route 53
folks are here and could say whether this is expected behavior or not, or
if this could be something with Atlassian's DNS configuration in their
Route 53 service.

Thanks,

-Dan



Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com


On Sat, Jun 10, 2023 at 3:59 PM Robert Edmonds  wrote:

> Dan McCombs via dns-operations wrote:
> > Hi everyone,
> >
> > We've stumbled upon what seems like unexpected behavior with Route 53
> returning
> > answers based on IP geo location to our resolvers.
> >
> > According to their documentation:
> >
> > When a browser or other viewer uses a DNS resolver that does not
> support
> > edns-client-subnet, Route 53 uses the source IP address of the DNS
> resolver
> > to approximate the location of the user and responds to geolocation
> queries
> > with the DNS record for the resolver's location.
>
> Here is the page that text is from, and the description of the other
> case (when a resolver does send an EDNS Client Subnet payload):
>
>
> https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-edns0.html
>
> When a browser or other viewer uses a DNS resolver that does not
> support edns-client-subnet, Route 53 uses the source IP address of
> the DNS resolver to approximate the location of the user and
> responds to geolocation queries with the DNS record for the
> resolver's location.
>
> When a browser or other viewer uses a DNS resolver that does support
> edns-client-subnet, the DNS resolver sends Route 53 a truncated
> version of the user's IP address. Route 53 determines the location
> of the user based on the truncated IP address rather than the source
> IP address of the DNS resolver; this typically provides a more
> accurate estimate of the user's location. Route 53 then responds to
> geolocation queries with the DNS record for the user's location.
>
> That text seems to be entirely consistent with nameserver behavior that
> sends different records to a resolver when it supplies its own IP
> address (or its own subnet) via the ECS option versus when it does not,
> because the resolver is asking for two different things. In the former
> case, the resolver is asking for responses tailored to a precise IP (or
> a precise subnet). In the latter case, the resolver is asking for
> responses on behalf of the whole client population that utilizes the
> resolver. These are not necessarily the same.
>
> > If it were using the resolver's source IP address to determine
> geolocation when
> > no edns-client-subnet is sent, I would expect the same answers as when
> sending
> > that address as the edns-client-subnet. What's going on here?
>
> I don't see anything in these DNS responses that is inconsistent with
> their documentation, or with the ECS specification. If there is a
> performance issue with one set of records versus another (you don't
> really say why the differing responses matter in your email), you might
> try contacting the nameserver operator directly to discuss the issue.
>
> > Our resolvers are co-located with our user's instances in the same
> datacenters,
> > so we don't configure our resolvers to send edns-client-subnet since
> they're
> > not geographically different (and in fact in the same IP blocks). This
> is the
> > first time we've had a user contact us about this, so I'm not sure if
> something
> > changed with Route 53 recently, if this is being caused by configuration
> > specific to the atlassian.net zone, or if somehow we just haven't had
> users
> > notice that they were being affected by this for years.
>
> There are many ways for operators of ECS-enabled nameservers to decide
> how to tailor DNS responses when receiving an ECS-enabled query.
> Geolocation, network topology, and actual performance may all be
> relevant. Even if your resolver instances are receiving queries from
> customer instances located in the same physical data center, t

Re: [dns-operations] Route 53 Unexpected geo location behavior

[Dan McCombs via dns-operations]

--- Begin Message ---
Oops, thanks Crist. Yes, that was a typo in my example and this
inconsistency does happen even when it's correct. 

I retested for good measure:

> dig -b 64.227.108.32 @ns-1339.awsdns-39.org doitb-synthetic.atlassian.net
> +short +nosubnet

104.192.142.19
> 104.192.142.20
> 104.192.142.18


> dig -b 64.227.108.32 @ns-1339.awsdns-39.org doitb-synthetic.atlassian.net
> +short +subnet=64.227.108.32/32

104.192.138.12
> 104.192.138.13


Thanks,

-Dan


Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com


On Fri, Jun 9, 2023 at 11:57 PM Crist Clark  wrote:

> Well, in your example below, looks like a typo. You have the first octet
> in the subnet option set to 67, when it’s 64 for the server.
>
> Is that just a typo in the example below? Do you still see inconsistencies
> when it’s correct?
>
>
> On Fri, Jun 9, 2023 at 2:25 PM Dan McCombs via dns-operations <
> dns-operati...@dns-oarc.net> wrote:
>
>>
>>
>>
>> -- Forwarded message --
>> From: Dan McCombs 
>> To: dns-operations@lists.dns-oarc.net
>> Cc:
>> Bcc:
>> Date: Fri, 9 Jun 2023 16:58:51 -0400
>> Subject: Route 53 Unexpected geo location behavior
>> Hi everyone,
>>
>> We've stumbled upon what seems like unexpected behavior with Route 53
>> returning answers based on IP geo location to our resolvers.
>>
>> According to their documentation
>> <https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-edns0.html>
>> :
>>
>>> When a browser or other viewer uses a DNS resolver that does not support
>>> edns-client-subnet, Route 53 uses the source IP address of the DNS resolver
>>> to approximate the location of the user and responds to geolocation queries
>>> with the DNS record for the resolver's location.
>>>
>>
>> But that doesn't seem to be the case. On a resolver with the address
>> 64.227.108.32, if we query at an awsdns authoritative from 64.227.108.32
>> without edns client subnet, we get one set of answers:
>>
>>> > dig -b 64.227.108.32 @ns-1339.awsdns-39.org
>>> doitb-synthetic.atlassian.net +short +nosubnet
>>
>> 104.192.142.20
>>> 104.192.142.19
>>> 104.192.142.18
>>
>>
>> But if we send the resolver's own same IP in edns-client-subnet, we get a
>> different set of answers:
>>
>>> >  dig -b 64.227.108.32 @ns-1339.awsdns-39.org
>>> doitb-synthetic.atlassian.net +short +subnet=67.227.108.32/32
>>
>> 104.192.138.13
>>> 104.192.138.12
>>
>>
>> If it were using the resolver's source IP address to determine
>> geolocation when no edns-client-subnet is sent, I would expect the same
>> answers as when sending that address as the edns-client-subnet. What's
>> going on here?
>>
>> Our resolvers are co-located with our user's instances in the same
>> datacenters, so we don't configure our resolvers to send edns-client-subnet
>> since they're not geographically different (and in fact in the same IP
>> blocks). This is the first time we've had a user contact us about this, so
>> I'm not sure if something changed with Route 53 recently, if this is being
>> caused by configuration specific to the atlassian.net zone, or if
>> somehow we just haven't had users notice that they were being affected by
>> this for years.
>>
>> Any insights would be appreciated,
>>
>> -Dan
>>
>>
>> Dan McCombs
>> Senior Engineer I - DNS
>> dmcco...@digitalocean.com
>>
>>
>>
>> -- Forwarded message --
>> From: Dan McCombs via dns-operations 
>> To: dns-operations@lists.dns-oarc.net
>> Cc:
>> Bcc:
>> Date: Fri, 9 Jun 2023 16:58:51 -0400
>> Subject: [dns-operations] Route 53 Unexpected geo location behavior
>> ___
>> dns-operations mailing list
>> dns-operations@lists.dns-oarc.net
>> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>>
>
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

[dns-operations] Route 53 Unexpected geo location behavior

[Dan McCombs via dns-operations]

--- Begin Message ---
Hi everyone,

We've stumbled upon what seems like unexpected behavior with Route 53
returning answers based on IP geo location to our resolvers.

According to their documentation

:

> When a browser or other viewer uses a DNS resolver that does not support
> edns-client-subnet, Route 53 uses the source IP address of the DNS resolver
> to approximate the location of the user and responds to geolocation queries
> with the DNS record for the resolver's location.
>

But that doesn't seem to be the case. On a resolver with the address
64.227.108.32, if we query at an awsdns authoritative from 64.227.108.32
without edns client subnet, we get one set of answers:

> > dig -b 64.227.108.32 @ns-1339.awsdns-39.org
> doitb-synthetic.atlassian.net +short +nosubnet

104.192.142.20
> 104.192.142.19
> 104.192.142.18


But if we send the resolver's own same IP in edns-client-subnet, we get a
different set of answers:

> >  dig -b 64.227.108.32 @ns-1339.awsdns-39.org
> doitb-synthetic.atlassian.net +short +subnet=67.227.108.32/32

104.192.138.13
> 104.192.138.12


If it were using the resolver's source IP address to determine geolocation
when no edns-client-subnet is sent, I would expect the same answers as when
sending that address as the edns-client-subnet. What's going on here?

Our resolvers are co-located with our user's instances in the same
datacenters, so we don't configure our resolvers to send edns-client-subnet
since they're not geographically different (and in fact in the same IP
blocks). This is the first time we've had a user contact us about this, so
I'm not sure if something changed with Route 53 recently, if this is being
caused by configuration specific to the atlassian.net zone, or if somehow
we just haven't had users notice that they were being affected by this for
years.

Any insights would be appreciated,

-Dan


Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations