Re: [Pdns-users] pdns_recursor issue

[Otto Moerbeek via Pdns-users]

On Thu, Jan 26, 2023 at 10:57:21PM +0100, Arien Vijn wrote:

> 
> > On 26 Jan 2023, at 19:00, Otto Moerbeek  wrote:
> 
> [...]
> 
> > I expect the aggressive cache workaround to function.
> 
> It seems so indeed.
> 
> > What is happening is that a query of a non-existent type (e.g. )
> > for xdsl-c-serviceweb.gslb.kpn.com 
> > 
> > $ dig @ns1gslb.kpn.com.  xdsl-c-serviceweb.gslb.kpn.com 
> >   +dnssec
> > 
> > produces an NSEC3 record that denies all types except TXT and RRSIG:
> > 
> > cq026lgcduus730qu6cbhtrt7qpr2jnu.gslb.kpn.com 
> > . 86400 IN NSEC3 
> > 1 0 1 19623DE58C1E7E40 CQ026LGCDUUS730QU6CBHTRT7QPR2JNV TXT RRSIG
> > 
> > So when the A record expires and somebody has done an  query in
> > between, the aggressive cache concludes that the wanted A record  does
> > not exists and not even asks the auth for it.
> > 
> > The after a cache wipe it works because when the (aggressive) cache is
> > empty for that zone, there is also no NSEC3 record denying anything.
> > 
> > So in the end this is a misconfigured domain. Completely disabling the
> > aggressive cache is a bit of a big hammer, you can also add an NTA for
> > the specific problem domain, something like:
> > 
> > addNTA('gslb.kpn.com ', 'Invalid NSEC3 record served 
> > for xdsl-c-serviceweb.gslb.kpn.com 
> > ')
> > 
> > in your Lua config file. This effectively does disable DNSSEC for the
> > domain. And please also report this to KPN.
> 
> Thanks for the explanation! This is really useful because KPN pointed to our 
> DNS= servers.
> 
> We also saw this with other (KPN hosted) 'gslb-domains', which also show no 
> trouble anymore after disabling the
> aggressive cache. So, if we go the NTA-way then I am afraid that we'll have 
> to add a series of NTAs then :/
> 
> At any rate, I am really glad with this explanation. I hope that KPN, and the 
> parties they outsourced their DNS service to, wil appreciate this too :)
> 
> -- Arien

This gives background information and a link to a remedy to be
employed on the load balancer side.

https://en.blog.nic.cz/2019/07/10/error-in-dnssec-implementation-on-f5-big-ip-load-balancers/

-Otto



___
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users

Re: [Pdns-users] pdns_recursor issue

[Arien Vijn via Pdns-users]

> On 26 Jan 2023, at 19:00, Otto Moerbeek  wrote:

[...]

> I expect the aggressive cache workaround to function.

It seems so indeed.

> What is happening is that a query of a non-existent type (e.g. )
> for xdsl-c-serviceweb.gslb.kpn.com 
> 
> $ dig @ns1gslb.kpn.com.  xdsl-c-serviceweb.gslb.kpn.com 
>   +dnssec
> 
> produces an NSEC3 record that denies all types except TXT and RRSIG:
> 
> cq026lgcduus730qu6cbhtrt7qpr2jnu.gslb.kpn.com 
> . 86400 IN   NSEC3 
> 1 0 1 19623DE58C1E7E40 CQ026LGCDUUS730QU6CBHTRT7QPR2JNV TXT RRSIG
> 
> So when the A record expires and somebody has done an  query in
> between, the aggressive cache concludes that the wanted A record  does
> not exists and not even asks the auth for it.
> 
> The after a cache wipe it works because when the (aggressive) cache is
> empty for that zone, there is also no NSEC3 record denying anything.
> 
> So in the end this is a misconfigured domain. Completely disabling the
> aggressive cache is a bit of a big hammer, you can also add an NTA for
> the specific problem domain, something like:
> 
> addNTA('gslb.kpn.com ', 'Invalid NSEC3 record served 
> for xdsl-c-serviceweb.gslb.kpn.com ')
> 
> in your Lua config file. This effectively does disable DNSSEC for the
> domain. And please also report this to KPN.

Thanks for the explanation! This is really useful because KPN pointed to our 
DNS= servers.

We also saw this with other (KPN hosted) 'gslb-domains', which also show no 
trouble anymore after disabling the
aggressive cache. So, if we go the NTA-way then I am afraid that we'll have to 
add a series of NTAs then :/

At any rate, I am really glad with this explanation. I hope that KPN, and the 
parties they outsourced their DNS service to, wil appreciate this too :)

-- Arien





signature.asc
Description: Message signed with OpenPGP
___
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users

Re: [Pdns-users] pdns_recursor issue

[Otto Moerbeek via Pdns-users]

On Thu, Jan 26, 2023 at 05:37:12PM +0100, Arien Vijn via Pdns-users wrote:

> Hi Peter,
> 
> > On 26 Jan 2023, at 17:28, Peter van Dijk via Pdns-users 
> >  wrote:
> 
> [...]
> 
> > After some brief investigation we somewhat suspect this is aggressive
> > NSEC caching. Can you see if aggressive-nsec-cache-size=0 makes the
> > problem go away?
> 
> Thanks! I'll add this line to the configuration right away :)
> 
> -- Ari??n
> 

I expect the aggressive cache workaround to function.

What is happening is that a query of a non-existent type (e.g. )
for xdsl-c-serviceweb.gslb.kpn.com

$ dig @ns1gslb.kpn.com.  xdsl-c-serviceweb.gslb.kpn.com  +dnssec 

produces an NSEC3 record that denies all types except TXT and RRSIG:

cq026lgcduus730qu6cbhtrt7qpr2jnu.gslb.kpn.com. 86400 IN NSEC3 1 0 1 
19623DE58C1E7E40 CQ026LGCDUUS730QU6CBHTRT7QPR2JNV TXT RRSIG

So when the A record expires and somebody has done an  query in
between, the aggressive cache concludes that the wanted A record  does
not exists and not even asks the auth for it.

The after a cache wipe it works because when the (aggressive) cache is
empty for that zone, there is also no NSEC3 record denying anything.

So in the end this is a misconfigured domain. Completely disabling the
aggressive cache is a bit of a big hammer, you can also add an NTA for
the specific problem domain, something like:

addNTA('gslb.kpn.com', 'Invalid NSEC3 record served for 
xdsl-c-serviceweb.gslb.kpn.com')

in your Lua config file. This effectively does disable DNSSEC for the
domain. And please also report this to KPN.

-Otto



___
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users

Re: [Pdns-users] pdns_recursor issue

[Arien Vijn via Pdns-users]

Hi Peter,

> On 26 Jan 2023, at 17:28, Peter van Dijk via Pdns-users 
>  wrote:

[...]

> After some brief investigation we somewhat suspect this is aggressive
> NSEC caching. Can you see if aggressive-nsec-cache-size=0 makes the
> problem go away?

Thanks! I'll add this line to the configuration right away :)

-- Ariën



signature.asc
Description: Message signed with OpenPGP
___
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users

Re: [Pdns-users] pdns_recursor issue

[Arien Vijn via Pdns-users]

Hi Otto,

Thanks for checking. Here is the configuration:

local-address=0.0.0.0, ::
local-port=53
query-local-address=::,0.0.0.0
threads=8

allow-from=127.0.0.0/8, ::1/128, 10.0.0.0/8, 87.251.42.0/26, 2001:7b8:650::/48

dnssec=validate

lua-config-file=/etc/powerdns/recursor.lua

webserver=yes
webserver-port=8082
webserver-address=::
webserver-password=
webserver-allow-from=0.0.0.0/32,::/0
api-key=

pdns-distributes-queries=true
reuseport=yes
any-to-tcp=yes
root-nx-trust=no
version-string=powerdns
max-ns-per-resolve=5

-- Ariën


> On 26 Jan 2023, at 17:04, Otto Moerbeek  wrote:
> 
> Hi,
> 
> Please show your configuration.
> 
> I do not think your analysis is to the point.
> If I repeat a scenario, I see a correct retrieval of the A record.
> 
> So we have to find out what is different in your case.
> 
>   -Otto
> 
> 
> On Thu, Jan 26, 2023 at 01:30:54PM +0100, Arien Vijn via Pdns-users wrote:
> 
>> Greetings,
>> 
>> We recently upgraded pdns_recursor from version 4.4.5 to 4.8.0. It seems 
>> that we run in into the following issue ever since.
>> 
>> 1/ Client queries for an A-record for xdsl-serviceweb.kpn.com.
>> 2/ Recursor queries the domain tree and receives the CNAME-record that 
>> points to: xdsl-c-serviceweb.gslb.kpn.com. from the authoritative DNS server.
>> 3/ Recursor queries and receives the subsequent an A-record from the 
>> authoritative DNS server for that A-record.
>> 4/ Recursor answers the client mentioned in 1/.
>> 
>> So far so good, until the A-record of xdsl-c-serviceweb.gslb.kpn.com. 
>> expires out of the 'main record cache' but not from the 'main packet cache'. 
>> The CNAME remains in both caches. Please note this excerpt from: rec_control 
>> dump-cache below:
>> 
>>   ; main record cache dump follows
>>   ;
>>   xdsl-serviceweb.kpn.com. 300 -224 IN CNAME xdsl-c-serviceweb.gslb.kpn.com. 
>> ; (Secure) auth=1 zone=kpn.com from=194.151.228.10 nm= rtag= ss=0
>>   ; negcache dump follows
>> 
>>   [...]
>> 
>>   ; main packet cache dump from thread follows
>>   ;
>>   xdsl-c-serviceweb.gslb.kpn.com. -1803 A  ; tag 0 udp
>> 
>>   [...]
>> 
>>   ; main packet cache dump from thread follows
>>   ;
>>   xdsl-serviceweb.kpn.com. -470 A  ; tag 0 udp
>>   xdsl-serviceweb.kpn.com. 111 A  ; tag 0 udp
>>   xdsl-serviceweb.kpn.com. 111   ; tag 0 udp
>> 
>> 
>> From that point on, pdns_recursor replies on queries for the A-record with 
>> the SOA-record of the domain of the said A-record:
>> 
>>   ; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> 
>> xdsl-c-serviceweb.gslb.kpn.com. @localhost
>>   ;; global options: +cmd
>>   ;; Got answer:
>>   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36347
>>   ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
>> 
>>   ;; OPT PSEUDOSECTION:
>>   ; EDNS: version: 0, flags:; udp: 512
>>   ;; QUESTION SECTION:
>>   ;xdsl-c-serviceweb.gslb.kpn.com.IN  A
>> 
>>   ;; AUTHORITY SECTION:
>>   gslb.kpn.com.   79407   IN  SOA ns2gslb.kpn.com. 
>> netmaster.gslb.kpn.com. 2023011702 10800 3600 604800 86400
>> 
>>   ;; Query time: 0 msec
>>   ;; SERVER: ::1#53(::1)
>>   ;; WHEN: Thu Jan 26 12:10:13 CET 2023
>>   ;; MSG SIZE  rcvd: 113
>> 
>> 
>> This situation causes actual people to complain and is being resolved by 
>> removing the domain tree for the subdomain gslb.kpn.com. out of the caches. 
>> From then on the story starts again.
>> 
>> That the A-record xdsl-c-serviceweb.gslb.kpn.com. remains in the packet 
>> cache seems not good to me, but I don't know enough about DNS and 
>> pdns_recursor be sure. What could trigger this behaviour or is it perhaps a 
>> configuration issue because we made such a large jump in versions when we 
>> upgraded? Last but not least we see the same behaviour with at least one 
>> other hostname
>> 
>> -- Ari??n
>> 
> 
> 
> 
>> ___
>> Pdns-users mailing list
>> Pdns-users@mailman.powerdns.com 
>> https://mailman.powerdns.com/mailman/listinfo/pdns-users 
>> 


signature.asc
Description: Message signed with OpenPGP
___
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users

Re: [Pdns-users] pdns_recursor issue

[Peter van Dijk via Pdns-users]

Hi Arien,

On Thu, 2023-01-26 at 13:30 +0100, Arien Vijn via Pdns-users wrote:
> Greetings,
> 
> We recently upgraded pdns_recursor from version 4.4.5 to 4.8.0. It seems that 
> we run in into the following issue ever since.
> 
> 1/ Client queries for an A-record for xdsl-serviceweb.kpn.com.
> 2/ Recursor queries the domain tree and receives the CNAME-record that points 
> to: xdsl-c-serviceweb.gslb.kpn.com. from the authoritative DNS server.
> 3/ Recursor queries and receives the subsequent an A-record from the 
> authoritative DNS server for that A-record.
> 4/ Recursor answers the client mentioned in 1/.
> 
> So far so good, until the A-record of xdsl-c-serviceweb.gslb.kpn.com. expires 
> out of the 'main record cache' but not from the 'main packet cache'. The 
> CNAME remains in both caches. Please note this excerpt from: rec_control 
> dump-cache below:

After some brief investigation we somewhat suspect this is aggressive
NSEC caching. Can you see if aggressive-nsec-cache-size=0 makes the
problem go away?

Kind regards,
-- 
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/

___
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users

Re: [Pdns-users] pdns_recursor issue

[Otto Moerbeek via Pdns-users]

Hi,

Please show your configuration.

I do not think your analysis is to the point.
If I repeat a scenario, I see a correct retrieval of the A record.

So we have to find out what is different in your case.

-Otto


On Thu, Jan 26, 2023 at 01:30:54PM +0100, Arien Vijn via Pdns-users wrote:

> Greetings,
> 
> We recently upgraded pdns_recursor from version 4.4.5 to 4.8.0. It seems that 
> we run in into the following issue ever since.
> 
> 1/ Client queries for an A-record for xdsl-serviceweb.kpn.com.
> 2/ Recursor queries the domain tree and receives the CNAME-record that points 
> to: xdsl-c-serviceweb.gslb.kpn.com. from the authoritative DNS server.
> 3/ Recursor queries and receives the subsequent an A-record from the 
> authoritative DNS server for that A-record.
> 4/ Recursor answers the client mentioned in 1/.
> 
> So far so good, until the A-record of xdsl-c-serviceweb.gslb.kpn.com. expires 
> out of the 'main record cache' but not from the 'main packet cache'. The 
> CNAME remains in both caches. Please note this excerpt from: rec_control 
> dump-cache below:
> 
>; main record cache dump follows
>;
>xdsl-serviceweb.kpn.com. 300 -224 IN CNAME xdsl-c-serviceweb.gslb.kpn.com. 
> ; (Secure) auth=1 zone=kpn.com from=194.151.228.10 nm= rtag= ss=0
>; negcache dump follows
> 
>[...]
> 
>; main packet cache dump from thread follows
>;
>xdsl-c-serviceweb.gslb.kpn.com. -1803 A  ; tag 0 udp
> 
>[...]
> 
>; main packet cache dump from thread follows
>;
>xdsl-serviceweb.kpn.com. -470 A  ; tag 0 udp
>xdsl-serviceweb.kpn.com. 111 A  ; tag 0 udp
>xdsl-serviceweb.kpn.com. 111   ; tag 0 udp
> 
> 
> From that point on, pdns_recursor replies on queries for the A-record with 
> the SOA-record of the domain of the said A-record:
> 
>; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> 
> xdsl-c-serviceweb.gslb.kpn.com. @localhost
>;; global options: +cmd
>;; Got answer:
>;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36347
>;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
> 
>;; OPT PSEUDOSECTION:
>; EDNS: version: 0, flags:; udp: 512
>;; QUESTION SECTION:
>;xdsl-c-serviceweb.gslb.kpn.com.IN  A
> 
>;; AUTHORITY SECTION:
>gslb.kpn.com.   79407   IN  SOA ns2gslb.kpn.com. 
> netmaster.gslb.kpn.com. 2023011702 10800 3600 604800 86400
> 
>;; Query time: 0 msec
>;; SERVER: ::1#53(::1)
>;; WHEN: Thu Jan 26 12:10:13 CET 2023
>;; MSG SIZE  rcvd: 113
> 
> 
> This situation causes actual people to complain and is being resolved by 
> removing the domain tree for the subdomain gslb.kpn.com. out of the caches. 
> From then on the story starts again.
> 
> That the A-record xdsl-c-serviceweb.gslb.kpn.com. remains in the packet cache 
> seems not good to me, but I don't know enough about DNS and pdns_recursor be 
> sure. What could trigger this behaviour or is it perhaps a configuration 
> issue because we made such a large jump in versions when we upgraded? Last 
> but not least we see the same behaviour with at least one other hostname
> 
> -- Ari??n
> 



> ___
> Pdns-users mailing list
> Pdns-users@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users

___
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users