Re: [dns-operations] Nameserver responses from different IP than destination of request

[P Vixie]

On Tue, Sep 01, 2020 at 01:28:17PM -0400, Dave Lawrence wrote:
> Stephane Bortzmeyer writes:
> >  P Vixie  wrote 
> > > you know that the plural of anecdote isn't data:
> > 
> > I recently discovered this english word and I love it:
> > https://en.wiktionary.org/wiki/anecdata
> 
> And one link more of relevance:
> 
> http://blog.danwin.com/don-t-forget-the-plural-of-anecdote-is-data/
> 
> It's just that anecdotes are not reasonably complete data, yet data
> they are.

to be clear, that's a religious war, and i won't get involved in it.

see also:

https://www.merriam-webster.com/dictionary/anecdote

-- 
Paul Vixie
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Dave Lawrence]

Stephane Bortzmeyer writes:
>  P Vixie  wrote 
> > you know that the plural of anecdote isn't data:
> 
> I recently discovered this english word and I love it:
> https://en.wiktionary.org/wiki/anecdata

And one link more of relevance:

http://blog.danwin.com/don-t-forget-the-plural-of-anecdote-is-data/

It's just that anecdotes are not reasonably complete data, yet data
they are.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Stephane Bortzmeyer]

On Tue, Sep 01, 2020 at 02:45:23AM +,
 P Vixie  wrote 
 a message of 22 lines which said:

> you know that the plural of anecdote isn't data:

I recently discovered this english word and I love it:

https://en.wiktionary.org/wiki/anecdata
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[P Vixie]

On Mon, Aug 31, 2020 at 08:00:00PM +0200, Florian Weimer wrote:
> * Puneet Sood via dns-operations:
> 
> > We would be interested in hearing other operator's experience here.
> > Are recursive servers seeing similar behavior from authoritative
> > servers? If yes, are you discarding these responses?
> > Are there authoritative server operators who still need the
> > flexibility afforded by RFC 1035?
> 
> If I recall correctly, while helping to run an academic network I
> encountered this issue on the authoritative server side.  That was
> close to twenty years ago, and even back then, it did not occur to us
> to push the resolvers to accept these incorrectly sourced responses,
> instead of getting the authoritative server operator to fix their
> setup. ...

right. same. this misbehaviour is why we added logic to BIND4.9 to
reject wrong-sourced responses, both in the stub and full resolvers.

> ...  Or maybe I'm not correctly remembering things, and it wasn't
> DNS but Sun RPC.  (Hard to believe that even early BIND 4 didn't get
> this right, and what else could they have been running?)

early BIND assumed singly-homed servers, and later BSD kernels used
the interface address as the source address for unbound UDP sockets.
however, earlier (4.3BSD and earlier) kernels always used the first
configured interface as the source address for unbound UDP sockets,
and this broke _everything_ on multi-homed hosts. including Sun RPC,
NTP, DNS of course, and NFS. Sun's fix for this in late SunOS was to
ignore the source address of UDP responses because it was unreliable.
anyone using SunOS on a multi-homed server ended up switching to open
source versions of the affected protocols. it was a real mess, easily
on-par with the .rhosts debacle that led to Sun's "fix" for the
gethostbyaddr() to have it call gethostbyname() on each returned name,
rather than doing this in ruserok() and similar. those were bad times.

> Anyway, in my current world, most recursive DNS servers operate behind
> some sort of stateful packet filter, so the server operators on their
> own cannot make these incorrectly source responses work because the
> systems under their direct control never receive them.

of the 27 million RDNS servers, 26.9 million of them are home gateways
with the oldest and worst DNS and kernel UDP implementations ever seen.
they do in fact need the relative safety of being behind stateful fire
walls. of the other ~100K, most are not behind stateful firewalls, and
about half are not behind anything other than a host-based firewall like
"pf". so i agree with your observation but i quibble with your method
of reaching your agreeable position.

-- 
Paul Vixie
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Viktor Dukhovni]

On Mon, Aug 31, 2020 at 02:19:08PM -0400, Warren Kumari wrote:

> The bit that I'm failing to understand is why these continue to exist
> -- if everyone (or, everyone other than Google) are ignoring /
> dropping these, how / why are they still on the Internet? Is it just
> the $whatever are sending these are always deployed next to something
> that ain't broke and the operator just hasn't noticed?
> Or are perhaps more things accepting these than we expect?

Quite likely the domains that are completely broken (none of the
nameservers respond from the right IP) are simply parked, and nobody
cares whether they they actually work or not.

The only reason you're seeing queries for them may be that folks doing
DNS measurements, query all the domains we can find including the parked
ones that nobody actually cares to have working.

Make these break, please!  Nobody has any just cause to complain, and
reducing the security of all other lookups to accommodate a tiny
minority of domains that have broken via most other resolvers for a
couple of decades is not a sound tradeoff.

-- 
Viktor.

P.S. I rather disagree with Paul that this is a operational practice
question.  The treatment of such response has an a priori *right*
answer, and not following the specifications is an operational error
that does not admit of an interpretation as an operational work-around.

If some such domains need to work, they know what they need to do,
or can be pointed in the right direction.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Florian Weimer]

* Warren Kumari:

> On Mon, Aug 31, 2020 at 2:11 PM Florian Weimer  wrote:
>>
>> * Puneet Sood via dns-operations:
>>
>> > We would be interested in hearing other operator's experience here.
>> > Are recursive servers seeing similar behavior from authoritative
>> > servers? If yes, are you discarding these responses?
>> > Are there authoritative server operators who still need the
>> > flexibility afforded by RFC 1035?
>>
>> If I recall correctly, while helping to run an academic network I
>> encountered this issue on the authoritative server side.  That was
>> close to twenty years ago, and even back then, it did not occur to us
>> to push the resolvers to accept these incorrectly sourced responses,
>> instead of getting the authoritative server operator to fix their
>> setup.
>
> The bit that I'm failing to understand is why these continue to exist
> -- if everyone (or, everyone other than Google) are ignoring /
> dropping these, how / why are they still on the Internet? Is it just
> the $whatever are sending these are always deployed next to something
> that ain't broke and the operator just hasn't noticed?
> Or are perhaps more things accepting these than we expect?

If such problems exist, they might not occur consistently for all
source addresses.  A subset of client addresses can route the response
in such a way that the expected source address is produced on the
public Internet.  Or the affected zones have other name servers that
hide the problem until you start looking for it.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Keith Mitchell]

On 8/31/20 12:40 PM, Puneet Sood via dns-operations wrote:

> Is there an online tool that does mark up on RFCs to show which other
> RFCs are referring to specific sections in it?

I suspect you may find:

https://powerdns.org/dns-camel/

helpful here.

Keith
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Florian Weimer]

* Puneet Sood via dns-operations:

> We would be interested in hearing other operator's experience here.
> Are recursive servers seeing similar behavior from authoritative
> servers? If yes, are you discarding these responses?
> Are there authoritative server operators who still need the
> flexibility afforded by RFC 1035?

If I recall correctly, while helping to run an academic network I
encountered this issue on the authoritative server side.  That was
close to twenty years ago, and even back then, it did not occur to us
to push the resolvers to accept these incorrectly sourced responses,
instead of getting the authoritative server operator to fix their
setup.  Or maybe I'm not correctly remembering things, and it wasn't
DNS but Sun RPC.  (Hard to believe that even early BIND 4 didn't get
this right, and what else could they have been running?)

Anyway, in my current world, most recursive DNS servers operate behind
some sort of stateful packet filter, so the server operators on their
own cannot make these incorrectly source responses work because the
systems under their direct control never receive them.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Puneet Sood via dns-operations]

--- Begin Message ---
On Sat, Aug 29, 2020 at 12:18 AM Robert Edmonds  wrote:
>
> Puneet Sood via dns-operations wrote:
> > RFC 1035 section 7.3 (https://tools.ietf.org/html/rfc1035)
> >  Some name servers send their responses from different
> >  addresses than the one used to receive the query.  That is, a
> >  resolver cannot rely that a response will come from the same
> >  address which it sent the corresponding query to.  This name
> >  server bug is typically encountered in UNIX systems.
> >
> > RFC 2181 (https://tools.ietf.org/html/rfc2181#section-4)
> >Most, if not all, DNS clients, expect the address from which a reply
> >is received to be the same address as that to which the query
> >eliciting the reply was sent.  This is true for servers acting as
> >clients for the purposes of recursive query resolution, as well as
> >simple resolver clients.  The address, along with the identifier (ID)
> >in the reply is used for disambiguating replies, and filtering
> >spurious responses.  This may, or may not, have been intended when
> >the DNS was designed, but is now a fact of life.
> >
> >Some multi-homed hosts running DNS servers generate a reply using a
> >source address that is not the same as the destination address from
> >the client's request packet.  Such replies will be discarded by the
> >client because the source address of the reply does not match that of
> >a host to which the client sent the original request.  That is, it
> >appears to be an unsolicited response.
>
> See also RFC 5452 section 9.1
> (https://tools.ietf.org/html/rfc5452#section-9.1) which puts the
> clarification in RFC 2181 into mandatory RFC 2119 language.
>
> 9.1.  Query Matching Rules
>
>A resolver implementation MUST match responses to all of the
>following attributes of the query:
>
>o  Source address against query destination address
>
>o  Destination address against query source address
>
>o  Destination port against query source port
>
>o  Query ID
>
>o  Query name
>
>o  Query class and type
>
>before applying DNS trustworthiness rules (see Section 5.4.1 of
>[RFC2181]).
>
>A mismatch and the response MUST be considered invalid.

Thanks Rob for that reference. It is quite clear. I didn't follow the
"Updated By" links from 1035 to this section :(

Is there an online tool that does mark up on RFCs to show which other
RFCs are referring to specific sections in it?

>
> --
> Robert Edmonds
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Thomas Mieslinger]

On 8/29/20 5:50 PM, Paul Hoffman wrote:

On Aug 28, 2020, at 3:24 PM, Puneet Sood via dns-operations 
 wrote:

We would be interested in hearing other operator's experience here.
Are recursive servers seeing similar behavior from authoritative
servers? If yes, are you discarding these responses?
Are there authoritative server operators who still need the
flexibility afforded by RFC 1035?


Please note that Puneet was asking for other operators' experiences, not the 
opinions of those of us who believe we should tell Google what to do. (And, 
yes, I certainly put myself in the latter category.) I, too, would like to hear 
if other resolver operators see this, and if possible to what extent they are 
seeing it, and if we're really lucky to hear at least a few names for which 
this is happening. The latter is not to name-and-shame, but instead to be able 
to talk to the authoritative operators about what their configuration is so 
that we can maybe guide others away from this path.


At my employer we discard this kind of responses. We could analyze how
often we see them but we wait until someone calls customer care for "DNS
not working".

To me this is similar to the endless discussion around "why can't I use
a cname in MX or NS".

RFC2181 is pretty clear about NS/MX or "Server Reply Source Address
Selection" and I don't see a reason why I should risk the stability of
my systems to make it work for a small fraction of the internet.

Just my 5¢

Thomas

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Robert Edmonds]

Puneet Sood via dns-operations wrote:
> RFC 1035 section 7.3 (https://tools.ietf.org/html/rfc1035)
>  Some name servers send their responses from different
>  addresses than the one used to receive the query.  That is, a
>  resolver cannot rely that a response will come from the same
>  address which it sent the corresponding query to.  This name
>  server bug is typically encountered in UNIX systems.
> 
> RFC 2181 (https://tools.ietf.org/html/rfc2181#section-4)
>Most, if not all, DNS clients, expect the address from which a reply
>is received to be the same address as that to which the query
>eliciting the reply was sent.  This is true for servers acting as
>clients for the purposes of recursive query resolution, as well as
>simple resolver clients.  The address, along with the identifier (ID)
>in the reply is used for disambiguating replies, and filtering
>spurious responses.  This may, or may not, have been intended when
>the DNS was designed, but is now a fact of life.
> 
>Some multi-homed hosts running DNS servers generate a reply using a
>source address that is not the same as the destination address from
>the client's request packet.  Such replies will be discarded by the
>client because the source address of the reply does not match that of
>a host to which the client sent the original request.  That is, it
>appears to be an unsolicited response.

See also RFC 5452 section 9.1
(https://tools.ietf.org/html/rfc5452#section-9.1) which puts the
clarification in RFC 2181 into mandatory RFC 2119 language.

9.1.  Query Matching Rules

   A resolver implementation MUST match responses to all of the
   following attributes of the query:

   o  Source address against query destination address

   o  Destination address against query source address

   o  Destination port against query source port

   o  Query ID

   o  Query name

   o  Query class and type

   before applying DNS trustworthiness rules (see Section 5.4.1 of
   [RFC2181]).

   A mismatch and the response MUST be considered invalid.

-- 
Robert Edmonds
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Paul Vixie]

Viktor Dukhovni wrote on 2020-08-28 18:46:

On Fri, Aug 28, 2020 at 06:24:40PM -0400, Puneet Sood via dns-operations wrote:


We (Google Public DNS) have noticed some instances of nameserver
responses for a query coming from a different IP. Our initial plan was
to consider these responses invalid and discard them. However after
reading the text in RFC 1035 and the update in RFC 2181, we wanted to
check what other recursive resolvers are seeing and how they are
handling such responses.

[...]



Not dropping them further weakens the already poor resistance of
non-DNSSEC replies to off-path cache poisoning attacks.  Please
drop these, the solution is up to the server operator.


+1. the robustness principle is 180deg out of phase in this case.


The operators of such domains need to clean up their network design.



that, too.

--
Sent from Postbox 

___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Viktor Dukhovni]

On Fri, Aug 28, 2020 at 06:24:40PM -0400, Puneet Sood via dns-operations wrote:

> We (Google Public DNS) have noticed some instances of nameserver
> responses for a query coming from a different IP. Our initial plan was
> to consider these responses invalid and discard them. However after
> reading the text in RFC 1035 and the update in RFC 2181, we wanted to
> check what other recursive resolvers are seeing and how they are
> handling such responses.
> 
> [...]

> We would be interested in hearing other operator's experience here.
> Are recursive servers seeing similar behavior from authoritative
> servers? If yes, are you discarding these responses?
> Are there authoritative server operators who still need the
> flexibility afforded by RFC 1035?

Not dropping them further weakens the already poor resistance of
non-DNSSEC replies to off-path cache poisoning attacks.  Please
drop these, the solution is up to the server operator.

They may have asymmetric inbound/outbound paths via firewalls doing NAT
between the Internet and their nameservers.  So even if the nameserver
is doing the right thing, the network middleboxes may introduce related,
but different IPs for the request and the response.

The operators of such domains need to clean up their network design.

-- 
Viktor.
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

Re: [dns-operations] Nameserver responses from different IP than destination of request

[Mark Andrews]

Google should definitely drop them as the rest of us then don’t have to deal 
with “but it works with Google’s servers”.

We (ISC) drop them.  They are basically the result of servers listening on *.53 
and failing to ensure the replies originate from the correct address.  In IPv4 
you need to listen explicitly on each address to force the reply to come from 
the correct address.  For IPv6 you need to use struct in6_pktinfo to get and 
set the destination / source addresses. They also don’t make it through 
stateful firewalls.

Mark

> On 29 Aug 2020, at 08:24, Puneet Sood via dns-operations 
>  wrote:
> 
> 
> From: Puneet Sood 
> Subject: Nameserver responses from different IP than destination of request
> Date: 29 August 2020 at 08:24:40 AEST
> To: dns-operations 
> Cc: Lu Zhao 
> 
> 
> Hello,
> 
> We (Google Public DNS) have noticed some instances of nameserver
> responses for a query coming from a different IP. Our initial plan was
> to consider these responses invalid and discard them. However after
> reading the text in RFC 1035 and the update in RFC 2181, we wanted to
> check what other recursive resolvers are seeing and how they are
> handling such responses.
> 
> RFC 1035 section 7.3 (https://tools.ietf.org/html/rfc1035)
> Some name servers send their responses from different
> addresses than the one used to receive the query.  That is, a
> resolver cannot rely that a response will come from the same
> address which it sent the corresponding query to.  This name
> server bug is typically encountered in UNIX systems.
> 
> RFC 2181 (https://tools.ietf.org/html/rfc2181#section-4)
>   Most, if not all, DNS clients, expect the address from which a reply
>   is received to be the same address as that to which the query
>   eliciting the reply was sent.  This is true for servers acting as
>   clients for the purposes of recursive query resolution, as well as
>   simple resolver clients.  The address, along with the identifier (ID)
>   in the reply is used for disambiguating replies, and filtering
>   spurious responses.  This may, or may not, have been intended when
>   the DNS was designed, but is now a fact of life.
> 
>   Some multi-homed hosts running DNS servers generate a reply using a
>   source address that is not the same as the destination address from
>   the client's request packet.  Such replies will be discarded by the
>   client because the source address of the reply does not match that of
>   a host to which the client sent the original request.  That is, it
>   appears to be an unsolicited response.
> 
> Couple of observations:
> 1. The difference between original and response nameserver IP happens
> for a small percentage of total queries (< 0.1%).
> 2. Where the original and response nameserver IPs are different,
> manual analysis of the top few response IPs showed that both IPs are
> on the same operator's network. So these are likely responses from the
> operator's DNS servers.
> 
> We would be interested in hearing other operator's experience here.
> Are recursive servers seeing similar behavior from authoritative
> servers? If yes, are you discarding these responses?
> Are there authoritative server operators who still need the
> flexibility afforded by RFC 1035?
> 
> Thanks,
> Puneet
> 
> 
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org


___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations

[dns-operations] Nameserver responses from different IP than destination of request

[Puneet Sood via dns-operations]

--- Begin Message ---
Hello,

We (Google Public DNS) have noticed some instances of nameserver
responses for a query coming from a different IP. Our initial plan was
to consider these responses invalid and discard them. However after
reading the text in RFC 1035 and the update in RFC 2181, we wanted to
check what other recursive resolvers are seeing and how they are
handling such responses.

RFC 1035 section 7.3 (https://tools.ietf.org/html/rfc1035)
 Some name servers send their responses from different
 addresses than the one used to receive the query.  That is, a
 resolver cannot rely that a response will come from the same
 address which it sent the corresponding query to.  This name
 server bug is typically encountered in UNIX systems.

RFC 2181 (https://tools.ietf.org/html/rfc2181#section-4)
   Most, if not all, DNS clients, expect the address from which a reply
   is received to be the same address as that to which the query
   eliciting the reply was sent.  This is true for servers acting as
   clients for the purposes of recursive query resolution, as well as
   simple resolver clients.  The address, along with the identifier (ID)
   in the reply is used for disambiguating replies, and filtering
   spurious responses.  This may, or may not, have been intended when
   the DNS was designed, but is now a fact of life.

   Some multi-homed hosts running DNS servers generate a reply using a
   source address that is not the same as the destination address from
   the client's request packet.  Such replies will be discarded by the
   client because the source address of the reply does not match that of
   a host to which the client sent the original request.  That is, it
   appears to be an unsolicited response.

Couple of observations:
1. The difference between original and response nameserver IP happens
for a small percentage of total queries (< 0.1%).
2. Where the original and response nameserver IPs are different,
manual analysis of the top few response IPs showed that both IPs are
on the same operator's network. So these are likely responses from the
operator's DNS servers.

We would be interested in hearing other operator's experience here.
Are recursive servers seeing similar behavior from authoritative
servers? If yes, are you discarding these responses?
Are there authoritative server operators who still need the
flexibility afforded by RFC 1035?

Thanks,
Puneet
--- End Message ---
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations