Re: apt temporary failure resolving deb.debian.org
On Mon, 10 Apr 2023, Tim Woodall wrote: On Mon, 10 Apr 2023, Lee wrote: Why are you using google as forwarders ? To eliminate as many variables as possible. delv talking to google works. delv talking to bind talking to google fails. When talking directly, delv is using udp to talk to google When talking via bind, bind is using tcp. And while google acks the DNSKEY request from bind, the data is not received. The seqnence number jumps from 1 on the ACK of the query to 1636 on the FIN where google closes the connection. Thats 1635 bytes of data gone missing. I managed to reproduce this talking to a remote bind server that I can control, running tcpdump on both ends. The DNS response was 1661 bytes split into two TCP packets with TCP segment len of 1208 and 455 (The other two bytes are the DNS response length itself) My router (at least I assume it's my router) is then dropping them. Change to use a non-standard port for the remote dns resolver and it works.
Re: apt temporary failure resolving deb.debian.org
On Mon, 10 Apr 2023, Lee wrote: Why are you using google as forwarders ? To eliminate as many variables as possible. delv talking to google works. delv talking to bind talking to google fails. When talking directly, delv is using udp to talk to google When talking via bind, bind is using tcp. And while google acks the DNSKEY request from bind, the data is not received. The seqnence number jumps from 1 on the ACK of the query to 1636 on the FIN where google closes the connection. Thats 1635 bytes of data gone missing. The mss on the original SYN packet is 1220, so that ought to be two (or more) packets gone missing. Interestingly if I use tcp to google servers it still works: (hmmm, capture suggest that it's only using TCP for the CNAME request, not the DNSKEY requests) delv -t cname deb.debian.org +rtrace +tcp @2001:4860:4860:: ;; fetch: deb.debian.org/CNAME ;; fetch: debian.org/DNSKEY ;; fetch: debian.org/DS ;; fetch: org/DNSKEY ;; fetch: org/DS ;; fetch: ./DNSKEY ; fully validated deb.debian.org. 3112IN CNAME debian.map.fastlydns.net. deb.debian.org. 3112IN RRSIG CNAME 8 3 3600 20230512040858 20230402034640 32728 debian.org. rFqk+TkAJPOXTbQl8irQJyMGjsL8yXMxFgxglzGC+7GaydpbQGEYaiOE FLHKy4dPshKq0pq5O8l+hw/gG3dgWg+fYkskltkGJyk8VNBnbgTM3Szm M2QjRR7x7hKitr61YrUkVCpZCroiKtZfat/0l42EWV24FewvatX9mBge VYzlUSrOchLHC7TjBOpxyA7Ta6ll4YIDDgMSZi4HxMMhjPdzGs2H/o8D CrKUmSE9VBhRoclczsBbMENUftKR0XOl while to my ISPs nameservers it doesn't! root@bind17:~# delv -t cname deb.debian.org +rtrace +tcp @2001:730:3ec2::10 ;; fetch: deb.debian.org/CNAME ;; fetch: debian.org/DNSKEY ;; resolution failed: timed out and I see exactly the same in the capture, 1635 bytes missing. bind works just fine for me with no forwarding: $ delv -t cname deb.debian.org +rtrace ;; fetch: deb.debian.org/CNAME ;; fetch: debian.org/DNSKEY ;; fetch: debian.org/DS ;; fetch: org/DNSKEY ;; fetch: org/DS ;; fetch: ./DNSKEY ; fully validated deb.debian.org. 3550IN CNAME debian.map.fastlydns.net. deb.debian.org. 3550IN RRSIG CNAME 8 3 3600 20230512040858 20230402034640 32728 debian.org. rFqk+TkAJPOXTbQl8irQJyMGjsL8yXMxFgxglzGC+7GaydpbQGEYaiOE FLHKy4dPshKq0pq5O8l+hw/gG3dgWg+fYksklt8VNBnbgTM3Szm M2QjRR7x7hKitr61YrUkVCpZCroiKtZfat/0l42EWV24FewvatX9mBge VYzlUSrOchLHC7TjBOpxyA7Ta6ll4YIDDgMSZi4HxMMhjPdzGs2H/o8D CrKUmSE9VBhRoclczsBbMENUftKR0XOl Regards, Lee
Re: apt temporary failure resolving deb.debian.org
On Sun, 9 Apr 2023, Andy Smith wrote:
Hi Badli,
On Sun, Apr 09, 2023 at 07:59:32AM +, Badli Al Rashid wrote:
I got a temporary failure resolving deb.debian.org and
www.debian.org since last week thursday. I can resolve other sites
like www.kernel.org and others.
Broke last monday for me.
When I switch to other DNS servers I can resolve www.debian.org.
Any clue in the logs of your bind9 resolver?
If you are able to install "delv", what does that say?
$ delv -t cname deb.debian.org
; fully validated
deb.debian.org. 3567IN CNAME debian.map.fastlydns.net.
deb.debian.org. 3567IN RRSIG CNAME 8 3 3600 20230512040858
20230402034640 32728 debian.org.
rFqk+TkAJPOXTbQl8irQJyMGjsL8yXMxFgxglzGC+7GaydpbQGEYaiOE
FLHKy4dPshKq0pq5O8l+hw/gG3dgWg+fYkskltkGJyk8VNBnbgTM3Szm
M2QjRR7x7hKitr61YrUkVCpZCroiKtZfat/0l42EWV24FewvatX9mBge
VYzlUSrOchLHC7TjBOpxyA7Ta6ll4YIDDgMSZi4HxMMhjPdzGs2H/o8D
CrKUmSE9VBhRoclczsBbMENUftKR0XOl
It does seem like your ;local resolver is at fault when it comes to
DNSSEC.
Cheers,
Andy
I suspect some weird pmtu issue or something like that
root@bind17:/etc/bind# delv -t cname www.microsoft.com +rtrace
;; fetch: www.microsoft.com/CNAME
;; fetch: com/DS
;; fetch: ./DNSKEY
;; fetch: microsoft.com/DS
;; fetch: com/DNSKEY
; unsigned answer
www.microsoft.com. 2858IN CNAME
www.microsoft.com-c-3.edgekey.net.
root@bind17:/etc/bind# delv -t cname deb.debian.org +rtrace
;; fetch: deb.debian.org/CNAME
;; fetch: debian.org/DNSKEY
;; resolution failed: timed out
root@bind17:/etc/bind#
And here's the really weird bit: that was with bind using google as
forwarders but...
root@bind17:/etc/bind# delv -6 -t cname deb.debian.org +rtrace
@2001:4860:4860::
;; fetch: deb.debian.org/CNAME
;; fetch: debian.org/DNSKEY
;; fetch: debian.org/DS
;; fetch: org/DNSKEY
;; fetch: org/DS
;; fetch: ./DNSKEY
; fully validated
deb.debian.org. 3284IN CNAME
debian.map.fastlydns.net.
deb.debian.org. 3284IN RRSIG CNAME 8 3 3600
20230512040858 20230402034640 32728 debian.org.
rFqk+TkAJPOXTbQl8irQJyMGjsL8yXMxFgxglzGC+7GaydpbQGEYaiOE
FLHKy4dPshKq0pq5O8l+hw/gG3dgWg+fYkskltkGJyk8VNBnbgTM3Szm
M2QjRR7x7hKitr61YrUkVCpZCroiKtZfat/0l42EWV24FewvatX9mBge
VYzlUSrOchLHC7TjBOpxyA7Ta6ll4YIDDgMSZi4HxMMhjPdzGs2H/o8D
CrKUmSE9VBhRoclczsBbMENUftKR0XOl
root@bind17:/etc/bind#
firewall17:~# tcpdump -n -i isp port 53
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on isp, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:24:02.676837 IP6 :::**00::1.50280 > 2001:4860:4860::.53:
27939+% [1au] DNSKEY? debian.org. (51)
03:24:02.686347 IP6 2001:4860:4860::.53 > :::**00::1.50280:
27939| 0/0/1 (39)
03:24:02.687485 IP6 :::**00::1.59395 > 2001:4860:4860::.53:
Flags [S], seq 2532653124, win 64660, options [mss 1220,sackOK,TS val 1661813206
ecr 0,nop,wscale 5], length 0
03:24:02.697849 IP6 2001:4860:4860::.53 > :::**00::1.59395:
Flags [S.], seq 2779959628, ack 2532653125, win 65535, options [mss 1440,sackOK,TS
val 1178061358 ecr 1661813206,nop,wscale 8], length 0
03:24:02.698472 IP6 :::**00::1.59395 > 2001:4860:4860::.53:
Flags [.], ack 1, win 2021, options [nop,nop,TS val 1661813217 ecr 1178061358],
length 0
03:24:02.698840 IP6 :::**00::1.59395 > 2001:4860:4860::.53:
Flags [P.], seq 1:54, ack 1, win 2021, options [nop,nop,TS val 1661813217 ecr
1178061358], length 53 16359+% [1au] DNSKEY? debian.org. (51)
03:24:02.708023 IP6 2001:4860:4860::.53 > :::**00::1.59395:
Flags [.], ack 54, win 256, options [nop,nop,TS val 1178061368 ecr 1661813217],
length 0
03:24:04.707378 IP6 2001:4860:4860::.53 > :::**00::1.59395:
Flags [F.], seq 1636, ack 54, win 256, options [nop,nop,TS val 1178063367 ecr
1661813217], length 0
03:24:04.708333 IP6 :::**00::1.59395 > 2001:4860:4860::.53:
Flags [.], ack 1, win 2021, options [nop,nop,TS val 1661815227 ecr
1178061368,nop,nop,sack 1 {1636:1637}], length 0
03:24:07.698316 IP6 :::**00::1.59395 > 2001:4860:4860::.53:
Flags [F.], seq 54, ack 1, win 2021, options [nop,nop,TS val 1661818217 ecr
1178061368,nop,nop,sack 1 {1636:1637}], length 0
03:24:07.708269 IP6 2001:4860:4860::.53 > :::**00::1.59395:
Flags [.], ack 55, win 256, options [nop,nop,TS val 1178066368 ecr 1661818217],
length 0
The result isn't getting back to me. Google shuts down the connection after 2
seconds.
And here's talking to google directly
firewall17:/etc/firewall# tcpdump -n -i isp port 53
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on isp, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:56:29.290782 IP6 :::**00::1.48805 > 2001:4860:4860::.53:
Flags [S], seq 3069515386, win 64660, options [mss 1220,sackOK,TS val 1663759806
ecr 0,nop,wscale 5], length 0
Re: apt temporary failure resolving deb.debian.org
On 4/9/23, Tim Woodall wrote: > On Sun, 9 Apr 2023, Badli Al Rashid wrote: > >> Hi All, >> >> Gooday everybody. Anyone having temporary failure when running apt update >> with own bind local resolver ? I got a temporary failure resolving >> deb.debian.org and www.debian.org since last week thursday. I can resolve >> other sites like www.kernel.org and others. >> >> When I switch to other DNS servers I can resolve www.debian.org. >> >> The command dig with +cd option I was able to resolve dwb.debian.org and >> www.debian.org. >> >> I am using bullseye bind packages and then upgraded to bind to sury to >> test. It is still the same. >> > > I've also been having severe problems resolving debian.org domains. > > I've now turned off dnssec validation on my bind server. > > > // > // If BIND logs error messages about the root key being expired, > // you will need to update your keys. See > // https://www.isc.org/bind-keys > > // > dnssec-validation no; If it was "yes" that might be the problem. dnssec-validation auto; # If dnssec-validation is set to auto, then a default trust anchor for the DNS root zone will be used. # If it is set to yes, however, then at least one trust anchor must be configured with a trusted-keys #or managed-keys statement in named.conf, or DNSSEC validation will not occur. # The default setting is yes. The only DNS issues I've noticed are NTP starting before BIND at boot time and all the N.debian.pool.ntp.org queries failing until bind is up and running. Regards Lee
Re: apt temporary failure resolving deb.debian.org
Hi Badli, On Sun, Apr 09, 2023 at 07:59:32AM +, Badli Al Rashid wrote: > I got a temporary failure resolving deb.debian.org and > www.debian.org since last week thursday. I can resolve other sites > like www.kernel.org and others. > > When I switch to other DNS servers I can resolve www.debian.org. Any clue in the logs of your bind9 resolver? If you are able to install "delv", what does that say? $ delv -t cname deb.debian.org ; fully validated deb.debian.org. 3567IN CNAME debian.map.fastlydns.net. deb.debian.org. 3567IN RRSIG CNAME 8 3 3600 20230512040858 20230402034640 32728 debian.org. rFqk+TkAJPOXTbQl8irQJyMGjsL8yXMxFgxglzGC+7GaydpbQGEYaiOE FLHKy4dPshKq0pq5O8l+hw/gG3dgWg+fYkskltkGJyk8VNBnbgTM3Szm M2QjRR7x7hKitr61YrUkVCpZCroiKtZfat/0l42EWV24FewvatX9mBge VYzlUSrOchLHC7TjBOpxyA7Ta6ll4YIDDgMSZi4HxMMhjPdzGs2H/o8D CrKUmSE9VBhRoclczsBbMENUftKR0XOl It does seem like your ;local resolver is at fault when it comes to DNSSEC. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: apt temporary failure resolving deb.debian.org
On Sun, 9 Apr 2023, Badli Al Rashid wrote: Hi All, Gooday everybody. Anyone having temporary failure when running apt update with own bind local resolver ? I got a temporary failure resolving deb.debian.org and www.debian.org since last week thursday. I can resolve other sites like www.kernel.org and others. When I switch to other DNS servers I can resolve www.debian.org. The command dig with +cd option I was able to resolve dwb.debian.org and www.debian.org. I am using bullseye bind packages and then upgraded to bind to sury to test. It is still the same. I've also been having severe problems resolving debian.org domains. I've now turned off dnssec validation on my bind server. // // If BIND logs error messages about the root key being expired, // you will need to update your keys. See // https://www.isc.org/bind-keys // dnssec-validation no; Tim.
