Re: [Dnsmasq-discuss] Dnsmasq cache does not fetch new value after cache expired for some record

[胡文峰]

HI all:

Thank you all.
The problem has been solved! It has nothing to do with dnsmasq or unbound.

It is the problem of the authoritative servers!


JasonHu/胡文峰
Mail: jaso...@boyaa.com<mailto:jaso...@boyaa.com>
深圳市东方博雅科技有限公司
Boyaa Interactive


From: Dnsmasq-discuss<mailto:dnsmasq-discuss-boun...@lists.thekelleys.org.uk>
Date: 2014-03-05 18:33
To: 
dnsmasq-discuss@lists.thekelleys.org.uk<mailto:dnsmasq-discuss@lists.thekelleys.org.uk>
Subject: Re: [Dnsmasq-discuss] Dnsmasq cache does not fetch new value after 
cache expired for some record

On 05/03/14 03:39, 胡文峰 wrote:
> It is repeatable! but I could not found the pattern!
>
> I didn't find any problem in the last few days, but now it happened again!
>
>
>
> The DNS api.m.duoku.com is a CNAME, and the upstream returned the CNAME
> to my local dnsmasq:
>
> Is it correct for the upstream to return CNAME to local dnsmasq?
>
> -
> # dig api.m.duoku.com
>
> ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5_8.6 <<>> api.m.duoku.com
> ;; global options:  printcmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 573
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;api.m.duoku.com. IN A
>
> ;; ANSWER SECTION:
> api.m.duoku.com. 0 IN CNAME gYVm2f8f533a.api.m.duoku.com.
>
> ;; AUTHORITY SECTION:
> duoku.com. 3600 IN SOA dns21.hichina.com. hostmaster.hichina.com.
> 2014022536 10800 2000 691200 5
>
> ;; Query time: 1091 msec
> ;; SERVER: 127.0.0.1#53(127.0.0.1)
> ;; WHEN: Wed Mar  5 10:55:29 2014
> ;; MSG SIZE  rcvd: 121
>
>
> 

Yes, that's a valid answer which says that api.m.duoku.com. is a CNAME
which points to gYVm2f8f533a.api.m.duoku.com. and
gYVm2f8f533a.api.m.duoku.com. doesn't exist (NXDOMAIN). Furthermore, the
numbers in the SOA record say that it's OK to cache this message for
2000 seconds.

>
> But when I do it again, it returned NXDOMAIN from dnsmasq,:
>
> 
> # dig api.m.duoku.com
>
> ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5_8.6 <<>> api.m.duoku.com
> ;; global options:  printcmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55686
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;api.m.duoku.com. IN A
>
> ;; Query time: 0 msec
> ;; SERVER: 127.0.0.1#53(127.0.0.1)
> ;; WHEN: Wed Mar  5 10:55:30 2014
> ;; MSG SIZE  rcvd: 33
>
>
> -

The protocol police might argue that dnsmasq should return exactly the
same answer as the one that came from upstream, but in this case (and
all realistic cases), the effect is exactly the same, the client sees an
answer that api.m.duoku.com doesn't exist.

When I do the same query, I see a different answer:

~$ dig @8.8.8.8  api.m.duoku.com

; <<>> DiG 9.7.3 <<>> @8.8.8.8 api.m.duoku.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21658
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;api.m.duoku.com.   IN  A

;; ANSWER SECTION:
api.m.duoku.com.3578IN  A   111.13.46.103

;; Query time: 27 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Mar  5 10:23:28 2014
;; MSG SIZE  rcvd: 4

So I think what's happening is that some problem with the upstream
nameserver or the authoritative nameserver for that domain is giving
transient bad answers. Dnsmasq is then caching the bad answer and
turning a transient problem into a long-lived one.

You can do two things to help with this

1) Complain to whoever runs the upstream nameservers you're using or
move to different one.s

2) Set the  --no-negcache flag in you dnsmasq configuration. That will
stop dnsmasq turning a transient failure into a long-lived one.


What version of dnsmasq are you using?


Cheers,

Simon.

>
> From the query log:
>
> # grep api.m.duoku.com /var/log/dnsmasq.log | grep '^Mar  5'
> Mar  5 10:55:28 dnsmasq[1513]: query[A] api.m.duoku.com from 127.0.0.1
> Mar  5 10:55:28 dnsmasq[1513]: forwarded api.m.duoku.com to 58.83.143.47
> Mar  5 10:55:29 dnsmasq[1513]: reply api.m.duoku.com is NXDOMAIN-IPv4
> Mar  5 10:55:30 dnsmasq[1513]: query[A] api.m.duoku.com from 127.0.0.1
> Mar  5 10:55:30 dnsmasq[1513]: cached api.m.duoku.com is NXDOMAIN-IPv4
> Mar  5 10:55:32 dnsmasq[1513]: query[A] api.m.duoku.com from 127.0.0.1
> Mar  5 10:55

Re: [Dnsmasq-discuss] Dnsmasq cache does not fetch new value after cache expired for some record

[Simon Kelley]

pi.m.duoku.com
> 4F   NX   Wed Mar  5 11:55:29 2014
>  
> -
>  
> Give dnsmasq a SIGUSR1, and found it really cached api.m.duoku.com as
> NXDOMAIN:
>  
>  
>  
> Mar  5 10:59:10 dnsmasq[1513]:
> api.m.duoku.com
> 4F   NX   Wed Mar  5 11:55:29 2014
>  
>  
>  
> 
>  
>  
>  
>  
> ------------------------------------
> JasonHu/胡文峰
> Mail: jaso...@boyaa.com <mailto:jaso...@boyaa.com>
> 深圳市东方博雅科技有限公司
> Boyaa Interactive
> 
>  
> *From:* Dnsmasq-discuss
> <mailto:dnsmasq-discuss-boun...@lists.thekelleys.org.uk>
> *Date:* 2014-02-28 18:33
> *To:* dnsmasq-discuss@lists.thekelleys.org.uk
> <mailto:dnsmasq-discuss@lists.thekelleys.org.uk>
> *Subject:* Re: [Dnsmasq-discuss] Dnsmasq cache does not fetch new value
> after cache expired for some record
> 
> On 27/02/14 06:43, hu jason wrote:
>> HI friends:
>>  I am using dnsmasq as the local dns cache. Everything is OK but we
>> found a very strange problem for one DNS records : api.m.duoku.com
>> <http://api.m.duoku.com>
>>
>>  It is ok to resolv this "api.m.duoku.com <http://api.m.duoku.com>"
>> as dnsmasq starts up. But after the cache expired(the TTL is 3600),
>> dnsmasq do not fetch new record result from upstream servers but return
>> a NXDOMAIN result:
>>
>>  For now, I have only found problem with his DNS record. It works
>> fine for other DNS records!
>>
>>
>> --
>> # dig api.m.duoku.com <http://api.m.duoku.com>
>>
>> ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5 <<>> api.m.duoku.com
>> <http://api.m.duoku.com>
>> ;; global options:  printcmd
>> ;; Got answer:
>> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 33100
>> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
>>
>> ;; QUESTION SECTION:
>> ;api.m.duoku.com <http://api.m.duoku.com>.INA
>>
>> ;; Query time: 0 msec
>> ;; SERVER: 127.0.0.1#53(127.0.0.1)
>> ;; WHEN: Thu Feb 27 12:39:21 2014
>> ;; MSG SIZE  rcvd: 33
>>
>> -
>>
>> After I restarted dnsmasq using /etc/init.d/dnsmasq, it returned to
> normal.
>> But after the cache expired, I need to restart again!
>>
>> The upstream servers are running Unbound, and they are OK to resolv
>> "api.m.duoku.com <http://api.m.duoku.com>", and I never have to restart
>> Unbound for the problelm!
>>
>> So I highly suspend it is the problem of dnsmasq!
>>
>>
>> --
>>
>> $ cat /etc/resolv.dnsmasq.conf
>> nameserver 120.132.147.28
>> nameserver 58.83.143.47
>> nameserver 216.12.201.200
>> nameserver 159.253.130.212
>> nameserver 208.43.166.72
>> nameserver 103.6.152.201
>>
>> --
>>
>> # cat /etc/dnsmasq.conf | grep -v '^#' | sed -e '/^$/d'
>> resolv-file=/etc/resolv.dnsmasq.conf
>> listen-address=127.0.0.1
>> no-dhcp-interface=127.0.0.1
>> bind-interfaces
>> no-hosts
>> 
>> cache-size=1000
>> log-queries
>> conf-dir=/etc/dnsmasq.d
>>
>> 
>>
>> # uname -a
>> Linux xxx.boyaa.com <http://xxx.boyaa.com> 2.6.18-348.1.1.el5 #1
>> SMP Tue Jan 22 16:19:19 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
>>
>>
>>
>> 
>>
>> # cat /etc/redhat-release
>> CentOS release 5.8 (Final) Tikanga
>>
>> # rpm -qa | grep dnsmasq
>> dnsmasq-2.45-1.1.el5_3
>>
>> -
>>
>> So anyone can help this problem?
>>
>> Thank you very much!
>>
> 
> Is this a one-off occurence, or a repeatable problem? The most likely
> explanation is that the TTL expired, and dnsmasq sent the query to an
> upstream server which replied with the NXDOMAIN due to some transient
> problem with the domain. The NXDOMAIN reply would then be cached by
> dnsmasq, so you'd see dnsmasq replying with NXDOMAIN until that record
> expired (The TTL for NXDOMAIN is not the same as for extant records, it
> comes from the SOA for the zone.)
> 
> If you can repeat this, the best thing to do is turn on dns-query
> logging (--log-queries) which should give you information about what's
> happening. Once you get to the situation where dnsmasq is returning
> NXDOMAIN, send the dnsmasq process SIGUSR1, which will make it dump the
> contents of the cache to the log too.
> 
> 
> Cheers,
> 
> Simon.
> 
> ___
> Dnsmasq-discuss mailing list
> Dnsmasq-discuss@lists.thekelleys.org.uk
> http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
> 
> 
> 
> ___
> Dnsmasq-discuss mailing list
> Dnsmasq-discuss@lists.thekelleys.org.uk
> http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss


___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss

Re: [Dnsmasq-discuss] Dnsmasq cache does not fetch new value after cache expired for some record

[胡文峰]

It is repeatable! but I could not found the pattern!

I didn't find any problem in the last few days, but now it happened again!



The DNS api.m.duoku.com is a CNAME, and the upstream returned the CNAME to my 
local dnsmasq:

Is it correct for the upstream to return CNAME to local dnsmasq?

-
# dig api.m.duoku.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5_8.6 <<>> api.m.duoku.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 573
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;api.m.duoku.com. IN A

;; ANSWER SECTION:
api.m.duoku.com. 0 IN CNAME gYVm2f8f533a.api.m.duoku.com.

;; AUTHORITY SECTION:
duoku.com. 3600 IN SOA dns21.hichina.com. hostmaster.hichina.com. 2014022536 
10800 2000 691200 5

;; Query time: 1091 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Mar  5 10:55:29 2014
;; MSG SIZE  rcvd: 121




But when I do it again, it returned NXDOMAIN from dnsmasq,:


# dig api.m.duoku.com

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5_8.6 <<>> api.m.duoku.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 55686
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;api.m.duoku.com. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Mar  5 10:55:30 2014
;; MSG SIZE  rcvd: 33


-

From the query log:

# grep api.m.duoku.com /var/log/dnsmasq.log | grep '^Mar  5'
Mar  5 10:55:28 dnsmasq[1513]: query[A] api.m.duoku.com from 127.0.0.1
Mar  5 10:55:28 dnsmasq[1513]: forwarded api.m.duoku.com to 58.83.143.47
Mar  5 10:55:29 dnsmasq[1513]: reply api.m.duoku.com is NXDOMAIN-IPv4
Mar  5 10:55:30 dnsmasq[1513]: query[A] api.m.duoku.com from 127.0.0.1
Mar  5 10:55:30 dnsmasq[1513]: cached api.m.duoku.com is NXDOMAIN-IPv4
Mar  5 10:55:32 dnsmasq[1513]: query[A] api.m.duoku.com from 127.0.0.1
Mar  5 10:55:32 dnsmasq[1513]: cached api.m.duoku.com is NXDOMAIN-IPv4
Mar  5 10:55:33 dnsmasq[1513]: query[A] api.m.duoku.com from 127.0.0.1
Mar  5 10:55:33 dnsmasq[1513]: cached api.m.duoku.com is NXDOMAIN-IPv4
Mar  5 10:55:34 dnsmasq[1513]: query[A] api.m.duoku.com from 127.0.0.1
Mar  5 10:55:34 dnsmasq[1513]: cached api.m.duoku.com is NXDOMAIN-IPv4
Mar  5 10:55:54 dnsmasq[1513]: query[A] api.m.duoku.com from 127.0.0.1
Mar  5 10:55:54 dnsmasq[1513]: cached api.m.duoku.com is NXDOMAIN-IPv4
Mar  5 10:59:10 dnsmasq[1513]: api.m.duoku.com  
   4F   NX   Wed Mar  5 11:55:29 2014

-

Give dnsmasq a SIGUSR1, and found it really cached api.m.duoku.com as NXDOMAIN:



Mar  5 10:59:10 dnsmasq[1513]: api.m.duoku.com  
   4F   NX   Wed Mar  5 11:55:29 2014









JasonHu/胡文峰
Mail: jaso...@boyaa.com<mailto:jaso...@boyaa.com>
深圳市东方博雅科技有限公司
Boyaa Interactive


From: Dnsmasq-discuss<mailto:dnsmasq-discuss-boun...@lists.thekelleys.org.uk>
Date: 2014-02-28 18:33
To: 
dnsmasq-discuss@lists.thekelleys.org.uk<mailto:dnsmasq-discuss@lists.thekelleys.org.uk>
Subject: Re: [Dnsmasq-discuss] Dnsmasq cache does not fetch new value after 
cache expired for some record

On 27/02/14 06:43, hu jason wrote:
> HI friends:
>  I am using dnsmasq as the local dns cache. Everything is OK but we
> found a very strange problem for one DNS records : api.m.duoku.com
> <http://api.m.duoku.com>
>
>  It is ok to resolv this "api.m.duoku.com <http://api.m.duoku.com>"
> as dnsmasq starts up. But after the cache expired(the TTL is 3600),
> dnsmasq do not fetch new record result from upstream servers but return
> a NXDOMAIN result:
>
>  For now, I have only found problem with his DNS record. It works
> fine for other DNS records!
>
>
> --
> # dig api.m.duoku.com <http://api.m.duoku.com>
>
> ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5 <<>> api.m.duoku.com
> <http://api.m.duoku.com>
> ;; global options:  printcmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 33100
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;api.m.duoku.com <http://api.m.duoku.com>.INA
>
> ;; Query time: 0 msec
> ;; SERVER: 127.0.0.1#53(127.0.0.1)
> ;; WHEN: Thu Feb 27 12:39:21 2014
> ;; MSG SIZE  rcvd

Re: [Dnsmasq-discuss] Dnsmasq cache does not fetch new value after cache expired for some record

[Simon Kelley]

On 27/02/14 06:43, hu jason wrote:
> HI friends:
>  I am using dnsmasq as the local dns cache. Everything is OK but we
> found a very strange problem for one DNS records : api.m.duoku.com
> 
> 
>  It is ok to resolv this "api.m.duoku.com "
> as dnsmasq starts up. But after the cache expired(the TTL is 3600),
> dnsmasq do not fetch new record result from upstream servers but return
> a NXDOMAIN result:
> 
>  For now, I have only found problem with his DNS record. It works
> fine for other DNS records!
> 
> 
> --
> # dig api.m.duoku.com 
> 
> ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-20.P1.el5 <<>> api.m.duoku.com
> 
> ;; global options:  printcmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 33100
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
> 
> ;; QUESTION SECTION:
> ;api.m.duoku.com .INA
> 
> ;; Query time: 0 msec
> ;; SERVER: 127.0.0.1#53(127.0.0.1)
> ;; WHEN: Thu Feb 27 12:39:21 2014
> ;; MSG SIZE  rcvd: 33
> 
> -
> 
> After I restarted dnsmasq using /etc/init.d/dnsmasq, it returned to normal.
> But after the cache expired, I need to restart again!
> 
> The upstream servers are running Unbound, and they are OK to resolv
> "api.m.duoku.com ", and I never have to restart
> Unbound for the problelm!
> 
> So I highly suspend it is the problem of dnsmasq!
> 
> 
> --
> 
> $ cat /etc/resolv.dnsmasq.conf
> nameserver 120.132.147.28
> nameserver 58.83.143.47
> nameserver 216.12.201.200
> nameserver 159.253.130.212
> nameserver 208.43.166.72
> nameserver 103.6.152.201
> 
> --
> 
> # cat /etc/dnsmasq.conf | grep -v '^#' | sed -e '/^$/d'
> resolv-file=/etc/resolv.dnsmasq.conf
> listen-address=127.0.0.1
> no-dhcp-interface=127.0.0.1
> bind-interfaces
> no-hosts
>  
> cache-size=1000
> log-queries
> conf-dir=/etc/dnsmasq.d
> 
> 
> 
> # uname -a
> Linux xxx.boyaa.com  2.6.18-348.1.1.el5 #1
> SMP Tue Jan 22 16:19:19 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> 
> 
> 
> # cat /etc/redhat-release
> CentOS release 5.8 (Final) Tikanga
> 
> # rpm -qa | grep dnsmasq
> dnsmasq-2.45-1.1.el5_3
> 
> -
> 
> So anyone can help this problem?
> 
> Thank you very much!
> 

Is this a one-off occurence, or a repeatable problem? The most likely
explanation is that the TTL expired, and dnsmasq sent the query to an
upstream server which replied with the NXDOMAIN due to some transient
problem with the domain. The NXDOMAIN reply would then be cached by
dnsmasq, so you'd see dnsmasq replying with NXDOMAIN until that record
expired (The TTL for NXDOMAIN is not the same as for extant records, it
comes from the SOA for the zone.)

If you can repeat this, the best thing to do is turn on dns-query
logging (--log-queries) which should give you information about what's
happening. Once you get to the situation where dnsmasq is returning
NXDOMAIN, send the dnsmasq process SIGUSR1, which will make it dump the
contents of the cache to the log too.


Cheers,

Simon.

___
Dnsmasq-discuss mailing list
Dnsmasq-discuss@lists.thekelleys.org.uk
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss