[Dnsmasq-discuss] server forwarding all traffic to parents after a successful PTR query of itself

2010-02-17 Thread Alberto Cuesta-Canada
Hi guys,
 
I saw a weird scenario in one of our dnsmasq servers yesterday. As the logs 
below show, the server was all happy doing its thing, until a set of PTR 
queries came from normal servers in our network. The last of it would ask for 
the hostname of the dns server giving the IP, and from that point dnsmasq would 
route all traffic to the parents. Restarting the dnsmasq service would restore 
the server to normal operations. This has happened 4 times in the last 10 days, 
always with the same pattern.
 
Feb 17 01:35:51 dnsmasq[28538]: query[A] grdvpm3.dselgrid.local from 
172.30.158.98

Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts grdvpm3.dselgrid.local is 
172.30.158.93

Feb 17 01:35:51 dnsmasq[28538]: query[PTR] 93.158.30.172.in-addr.arpa from 
172.30.158.98

Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts 172.30.158.93 is 
grdvpm3.dselgrid.local

Feb 17 01:35:51 dnsmasq[28538]: query[A] grdvpm3.dselgrid.local from 
172.30.158.98

Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts grdvpm3.dselgrid.local is 
172.30.158.93

Feb 17 01:37:16 dnsmasq[28538]: query[MX] smtpmail.daiwaeurope.local from 
127.0.0.1

Feb 17 01:37:16 dnsmasq[28538]: forwarded smtpmail.daiwaeurope.local to 
172.30.48.192

Feb 17 01:37:16 dnsmasq[28538]: query[MX] vsmtpmail.daiwaeurope.local from 
127.0.0.1

Feb 17 01:37:16 dnsmasq[28538]: forwarded vsmtpmail.daiwaeurope.local to 
172.30.48.192

Feb 17 01:37:16 dnsmasq[28538]: query[A] smtpmail.daiwaeurope.local from 
127.0.0.1

Feb 17 01:37:16 dnsmasq[28538]: forwarded smtpmail.daiwaeurope.local to 
172.30.48.192

Feb 17 01:37:16 dnsmasq[28538]: reply smtpmail.daiwaeurope.local is CNAME

Feb 17 01:37:16 dnsmasq[28538]: reply vsmtpmail.daiwaeurope.local is 
172.30.19.221

Feb 17 01:37:52 dnsmasq[28538]: query[PTR] 250.158.30.172.in-addr.arpa from 
172.30.158.94

Feb 17 01:37:52 dnsmasq[28538]: /etc/hosts 172.30.158.250 is 
grdxk-mgmt1.dselgrid.local

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Any idea what would be going on? Is that PTR query a signal that some other 
service could be asking the DNS server to stop reading the hosts file?
 
Many thanks,
 
Alberto Cuesta-Canada
GaaS Team Lead
Excelian Ltd.
+44 (0) 7942633361

The information contained in this email and any attached files are confidential 
and intended solely for the addressee(s). The email may be legally privileged 
or prohibited from disclosure and unauthorised use. If you are not the named 
addressee you may not use, copy, or disclose this information to any other 
person. If you received this message in error please notify the sender 
immediately and delete it from your system. 

Any opinion or views contained in this email message are those of the sender, 
and do not represent those of the Company in any way and reliance should not be 
placed upon its contents. Unless otherwise stated, this email message is not 
intended to be contractually binding. Where an Agreement exists between our 
respective companies and there is conflict between the contents of this email 
message and the Agreement then the terms of that Agreement shall prevail.

Excelian
50 Featherstone Street
London
EC1Y 8RT
Tel: +44 (0) 20 7336 9595
Fax: +44 (0) 20 7336 9596
www.Excelian.com
_
This e-mail has been scanned for viruses by MessageLabs. For further 
information visit http://www.messagelabs.com

Excelian subscribes to cleaner and greener methods of working. Help take 
responsibility for the environment. Please don't print this email unless you 
absolutely have to.

Re: [Dnsmasq-discuss] server forwarding all traffic to parents after a successful PTR query of itself

2010-02-17 Thread Simon Kelley

Alberto Cuesta-Canada wrote:

Hi guys,
 
I saw a weird scenario in one of our dnsmasq servers yesterday. As the 
logs below show, the server was all happy doing its thing, until a set 
of PTR queries came from normal servers in our network. The last of it 
would ask for the hostname of the dns server giving the IP, and from 
that point dnsmasq would route all traffic to the parents. Restarting 
the dnsmasq service would restore the server to normal operations. This 
has happened 4 times in the last 10 days, always with the same pattern.
 

Feb 17 01:35:51 dnsmasq[28538]: query[A] grdvpm3.dselgrid.local from 
172.30.158.98


Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts grdvpm3.dselgrid.local is 
172.30.158.93


Feb 17 01:35:51 dnsmasq[28538]: query[PTR] 93.158.30.172.in-addr.arpa 
from 172.30.158.98


Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts 172.30.158.93 is 
grdvpm3.dselgrid.local


Feb 17 01:35:51 dnsmasq[28538]: query[A] grdvpm3.dselgrid.local from 
172.30.158.98


Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts grdvpm3.dselgrid.local is 
172.30.158.93


Feb 17 01:37:16 dnsmasq[28538]: query[MX] smtpmail.daiwaeurope.local 
from 127.0.0.1


Feb 17 01:37:16 dnsmasq[28538]: forwarded smtpmail.daiwaeurope.local to 
172.30.48.192


Feb 17 01:37:16 dnsmasq[28538]: query[MX] vsmtpmail.daiwaeurope.local 
from 127.0.0.1


Feb 17 01:37:16 dnsmasq[28538]: forwarded vsmtpmail.daiwaeurope.local to 
172.30.48.192


Feb 17 01:37:16 dnsmasq[28538]: query[A] smtpmail.daiwaeurope.local from 
127.0.0.1


Feb 17 01:37:16 dnsmasq[28538]: forwarded smtpmail.daiwaeurope.local to 
172.30.48.192


Feb 17 01:37:16 dnsmasq[28538]: reply smtpmail.daiwaeurope.local is CNAME

Feb 17 01:37:16 dnsmasq[28538]: reply vsmtpmail.daiwaeurope.local is 
172.30.19.221


Feb 17 01:37:52 dnsmasq[28538]: query[PTR] 250.158.30.172.in-addr.arpa 
from 172.30.158.94


Feb 17 01:37:52 dnsmasq[28538]: /etc/hosts 172.30.158.250 is 
grdxk-mgmt1.dselgrid.local


Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Any idea what would be going on? Is that PTR query a signal that some 
other service could be asking the DNS server to stop reading the hosts file?
 


Which version of dnsmasq are you using?

Simon.



Re: [Dnsmasq-discuss] server forwarding all traffic to parents after a successful PTR query of itself

2010-02-17 Thread Alberto Cuesta-Canada
Hi Simon,
 
2.47
 
Cheers,
 
Alberto Cuesta-Canada
GaaS Team Lead
Excelian Ltd.
+44 (0) 7942633361



From: Simon Kelley [mailto:si...@thekelleys.org.uk]
Sent: Wed 17/02/2010 09:46
To: Alberto Cuesta-Canada
Cc: dnsmasq-discuss@lists.thekelleys.org.uk; Grid Support
Subject: Re: [Dnsmasq-discuss] server forwarding all traffic to parents after a 
successful PTR query of itself



Alberto Cuesta-Canada wrote:
 Hi guys,
 
 I saw a weird scenario in one of our dnsmasq servers yesterday. As the
 logs below show, the server was all happy doing its thing, until a set
 of PTR queries came from normal servers in our network. The last of it
 would ask for the hostname of the dns server giving the IP, and from
 that point dnsmasq would route all traffic to the parents. Restarting
 the dnsmasq service would restore the server to normal operations. This
 has happened 4 times in the last 10 days, always with the same pattern.
 

 Feb 17 01:35:51 dnsmasq[28538]: query[A] grdvpm3.dselgrid.local from
 172.30.158.98

 Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts grdvpm3.dselgrid.local is
 172.30.158.93

 Feb 17 01:35:51 dnsmasq[28538]: query[PTR] 93.158.30.172.in-addr.arpa
 from 172.30.158.98

 Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts 172.30.158.93 is
 grdvpm3.dselgrid.local

 Feb 17 01:35:51 dnsmasq[28538]: query[A] grdvpm3.dselgrid.local from
 172.30.158.98

 Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts grdvpm3.dselgrid.local is
 172.30.158.93

 Feb 17 01:37:16 dnsmasq[28538]: query[MX] smtpmail.daiwaeurope.local
 from 127.0.0.1

 Feb 17 01:37:16 dnsmasq[28538]: forwarded smtpmail.daiwaeurope.local to
 172.30.48.192

 Feb 17 01:37:16 dnsmasq[28538]: query[MX] vsmtpmail.daiwaeurope.local
 from 127.0.0.1

 Feb 17 01:37:16 dnsmasq[28538]: forwarded vsmtpmail.daiwaeurope.local to
 172.30.48.192

 Feb 17 01:37:16 dnsmasq[28538]: query[A] smtpmail.daiwaeurope.local from
 127.0.0.1

 Feb 17 01:37:16 dnsmasq[28538]: forwarded smtpmail.daiwaeurope.local to
 172.30.48.192

 Feb 17 01:37:16 dnsmasq[28538]: reply smtpmail.daiwaeurope.local is CNAME

 Feb 17 01:37:16 dnsmasq[28538]: reply vsmtpmail.daiwaeurope.local is
 172.30.19.221

 Feb 17 01:37:52 dnsmasq[28538]: query[PTR] 250.158.30.172.in-addr.arpa
 from 172.30.158.94

 Feb 17 01:37:52 dnsmasq[28538]: /etc/hosts 172.30.158.250 is
 grdxk-mgmt1.dselgrid.local

 Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

 Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

 Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

 Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

 Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

 Any idea what would be going on? Is that PTR query a signal that some
 other service could be asking the DNS server to stop reading the hosts file?
 

Which version of dnsmasq are you using?

Simon.



The information contained in this email and any attached files are confidential 
and intended solely for the addressee(s). The email may be legally privileged 
or prohibited from disclosure and unauthorised use. If you are not the named 
addressee you may not use, copy, or disclose this information to any other 
person. If you received this message in error please notify the sender 
immediately and delete it from your system. 

Any opinion or views contained in this email message are those of the sender, 
and do not represent those of the Company in any way and reliance should not be 
placed upon its contents. Unless otherwise stated, this email message is not 
intended to be contractually binding. Where an Agreement exists between our 
respective companies and there is conflict between the contents of this email 
message and the Agreement then the terms of that Agreement shall prevail.

Excelian
50 Featherstone Street
London
EC1Y 8RT
Tel: +44 (0) 20 7336 9595
Fax: +44 (0) 20 7336 9596
www.Excelian.com
_
This e-mail has been scanned for viruses by MessageLabs. For further 
information visit http://www.messagelabs.com

Excelian subscribes to cleaner and greener methods of working. Help take 
responsibility for the environment. Please don't print this email unless you 
absolutely have to.

Re: [Dnsmasq-discuss] server forwarding all traffic to parents after a successful PTR query of itself

2010-02-17 Thread Simon Kelley

Alberto Cuesta-Canada wrote:

Hi guys,
 
I saw a weird scenario in one of our dnsmasq servers yesterday. As the 
logs below show, the server was all happy doing its thing, until a set 
of PTR queries came from normal servers in our network. The last of it 
would ask for the hostname of the dns server giving the IP, and from 
that point dnsmasq would route all traffic to the parents. Restarting 
the dnsmasq service would restore the server to normal operations. This 
has happened 4 times in the last 10 days, always with the same pattern.
 

Feb 17 01:35:51 dnsmasq[28538]: query[A] grdvpm3.dselgrid.local from 
172.30.158.98


Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts grdvpm3.dselgrid.local is 
172.30.158.93


Feb 17 01:35:51 dnsmasq[28538]: query[PTR] 93.158.30.172.in-addr.arpa 
from 172.30.158.98


Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts 172.30.158.93 is 
grdvpm3.dselgrid.local


Feb 17 01:35:51 dnsmasq[28538]: query[A] grdvpm3.dselgrid.local from 
172.30.158.98


Feb 17 01:35:51 dnsmasq[28538]: /etc/hosts grdvpm3.dselgrid.local is 
172.30.158.93


Feb 17 01:37:16 dnsmasq[28538]: query[MX] smtpmail.daiwaeurope.local 
from 127.0.0.1


Feb 17 01:37:16 dnsmasq[28538]: forwarded smtpmail.daiwaeurope.local to 
172.30.48.192


Feb 17 01:37:16 dnsmasq[28538]: query[MX] vsmtpmail.daiwaeurope.local 
from 127.0.0.1


Feb 17 01:37:16 dnsmasq[28538]: forwarded vsmtpmail.daiwaeurope.local to 
172.30.48.192


Feb 17 01:37:16 dnsmasq[28538]: query[A] smtpmail.daiwaeurope.local from 
127.0.0.1


Feb 17 01:37:16 dnsmasq[28538]: forwarded smtpmail.daiwaeurope.local to 
172.30.48.192


Feb 17 01:37:16 dnsmasq[28538]: reply smtpmail.daiwaeurope.local is CNAME

Feb 17 01:37:16 dnsmasq[28538]: reply vsmtpmail.daiwaeurope.local is 
172.30.19.221


Feb 17 01:37:52 dnsmasq[28538]: query[PTR] 250.158.30.172.in-addr.arpa 
from 172.30.158.94


Feb 17 01:37:52 dnsmasq[28538]: /etc/hosts 172.30.158.250 is 
grdxk-mgmt1.dselgrid.local


Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Feb 17 01:37:52 dnsmasq[28538]: forwarded query to 172.30.48.192

Any idea what would be going on? Is that PTR query a signal that some 
other service could be asking the DNS server to stop reading the hosts file?
 



It's not clear to me what is going on here. How does the pattern 
continue? Do you just see  forwarded query to 172.30.48.192 from now 
on until the server is restarted, or do you still see query[A] and 
query[PTR} lines?


Do queries which get pushed upstream continue to work? How about queries 
which should be answered locally?


What is 172.30.158.94? Is it running anything that may generate odd 
DNS queries? The holy grail would be to able prod that machine to 
reproduce this at will.


What sort of machine are you running dnsmasq on? Does it have a 
reasonable amount of spare storage so that you could tcpdump all traffic 
to/from port 53,UDP for offline analysis?



Simon.



Re: [Dnsmasq-discuss] server forwarding all traffic to parents after a successful PTR query of itself

2010-02-17 Thread Alberto Cuesta-Canada
Hi Simon,
 
the parents of 250 (my dnsmasq server) have forwarding rules for the 
dselgrid.local domain, that I run. So I assumed that the queries pushed 
upstream would be routed down again, and timeout in a loop.
 
That said, in the logs I could still see successful PTR and A queries, 
outnumbered 10 to 1 by forwards. I'm not sure about the behaviour of local 
queries, I don't remember from yesterday, but I think they worked.
 
94 is a Platform Grid Master, that is a W2K3 machine which runs only one 
application. It keeps a cache of machines but it doesn't give DNS services, or 
anything similar. The interesting thing is that the PTR request doesn't always 
produce this effect. I have enterprise support for that software, so I will ask 
them.
 
dnsmasq is running in a quite complicated setup. We have a XenServer host 
running a Ubuntu 9.04 VM. I have just 1GB free on that machine and out of disk 
space scenarios are fatal, so I can't tcpdump. There is a rebuild of it coming 
in the next two weeks that will give me another 50GB.
 
Any idea on what to look for, or any hypothesis of what could be happening 
should be enough, I can keep investigating and contain it with workarounds for 
a while.
 
Many thanks,
 
Alberto Cuesta-Canada
GaaS Team Lead
Excelian Ltd.
+44 (0) 7942633361



From: Simon Kelley [mailto:si...@thekelleys.org.uk]
Sent: Wed 17/02/2010 10:04
To: Alberto Cuesta-Canada
Cc: dnsmasq-discuss@lists.thekelleys.org.uk; Grid Support
Subject: Re: [Dnsmasq-discuss] server forwarding all traffic to parents after a 
successful PTR query of itself



It's not clear to me what is going on here. How does the pattern
continue? Do you just see  forwarded query to 172.30.48.192 from now
on until the server is restarted, or do you still see query[A] and
query[PTR} lines?

Do queries which get pushed upstream continue to work? How about queries
which should be answered locally?

What is 172.30.158.94? Is it running anything that may generate odd
DNS queries? The holy grail would be to able prod that machine to
reproduce this at will.

What sort of machine are you running dnsmasq on? Does it have a
reasonable amount of spare storage so that you could tcpdump all traffic
to/from port 53,UDP for offline analysis?


Simon.



The information contained in this email and any attached files are confidential 
and intended solely for the addressee(s). The email may be legally privileged 
or prohibited from disclosure and unauthorised use. If you are not the named 
addressee you may not use, copy, or disclose this information to any other 
person. If you received this message in error please notify the sender 
immediately and delete it from your system. 

Any opinion or views contained in this email message are those of the sender, 
and do not represent those of the Company in any way and reliance should not be 
placed upon its contents. Unless otherwise stated, this email message is not 
intended to be contractually binding. Where an Agreement exists between our 
respective companies and there is conflict between the contents of this email 
message and the Agreement then the terms of that Agreement shall prevail.

Excelian
50 Featherstone Street
London
EC1Y 8RT
Tel: +44 (0) 20 7336 9595
Fax: +44 (0) 20 7336 9596
www.Excelian.com
_
This e-mail has been scanned for viruses by MessageLabs. For further 
information visit http://www.messagelabs.com

Excelian subscribes to cleaner and greener methods of working. Help take 
responsibility for the environment. Please don't print this email unless you 
absolutely have to.

Re: [Dnsmasq-discuss] server forwarding all traffic to parents after a successful PTR query of itself

2010-02-17 Thread Simon Kelley

Alberto Cuesta-Canada wrote:

Hi Simon,
 
the parents of 250 (my dnsmasq server) have forwarding rules for the 
dselgrid.local domain, that I run. So I assumed that the queries pushed 
upstream would be routed down again, and timeout in a loop.


Ahh, that could easily be the problem. If you generate a loop between 
two DNS servers, each forwarding to the other, then the queries can 
easily bounce back-and-forth forever. Dnsmasq will manage this situation 
reasonably well, and manage to server other traffic, but the circulating 
queries will eat bandwidth and CPU.


The logs would seem to show a strange query of some sort (dnsmasq 
can't parse a domain-name from the query, hence forwarded query rather 
than forwarded domain-name) If such queries can circulate forever 
then you have a problem.



 
That said, in the logs I could still see successful PTR and A queries, 
outnumbered 10 to 1 by forwards. I'm not sure about the behaviour of 
local queries, I don't remember from yesterday, but I think they worked.
 
94 is a Platform Grid Master, that is a W2K3 machine which runs only one 
application. It keeps a cache of machines but it doesn't give DNS 
services, or anything similar. The interesting thing is that the PTR 
request doesn't always produce this effect. I have enterprise support 
for that software, so I will ask them.
 
dnsmasq is running in a quite complicated setup. We have a XenServer 
host running a Ubuntu 9.04 VM. I have just 1GB free on that machine and 
out of disk space scenarios are fatal, so I can't tcpdump. There is a 
rebuild of it coming in the next two weeks that will give me another 50GB.
 
Any idea on what to look for, or any hypothesis of what could be 
happening should be enough, I can keep investigating and contain it with 
workarounds for a while.


See above. a loop, possibly only of odd queries.

Cheers,

Simon.

 
Many thanks,
 
*Alberto Cuesta-Canada*

GaaS Team Lead
Excelian Ltd.
+44 (0) 7942633361


*From:* Simon Kelley [mailto:si...@thekelleys.org.uk]
*Sent:* Wed 17/02/2010 10:04
*To:* Alberto Cuesta-Canada
*Cc:* dnsmasq-discuss@lists.thekelleys.org.uk; Grid Support
*Subject:* Re: [Dnsmasq-discuss] server forwarding all traffic to 
parents after a successful PTR query of itself


It's not clear to me what is going on here. How does the pattern
continue? Do you just see  forwarded query to 172.30.48.192 from now
on until the server is restarted, or do you still see query[A] and
query[PTR} lines?

Do queries which get pushed upstream continue to work? How about queries
which should be answered locally?

What is 172.30.158.94? Is it running anything that may generate odd
DNS queries? The holy grail would be to able prod that machine to
reproduce this at will.

What sort of machine are you running dnsmasq on? Does it have a
reasonable amount of spare storage so that you could tcpdump all traffic
to/from port 53,UDP for offline analysis?


Simon.


The information contained in this email and any attached files are 
confidential and intended solely for the addressee(s). The email may be 
legally privileged or prohibited from disclosure and unauthorised use. 
If you are not the named addressee you may not use, copy, or disclose 
this information to any other person. If you received this message in 
error please notify the sender immediately and delete it from your system.


Any opinion or views contained in this email message are those of the 
sender, and do not represent those of the Company in any way and 
reliance should not be placed upon its contents. Unless otherwise 
stated, this email message is not intended to be contractually binding. 
Where an Agreement exists between our respective companies and there is 
conflict between the contents of this email message and the Agreement 
then the terms of that Agreement shall prevail.


Excelian
50 Featherstone Street
London
EC1Y 8RT
Tel: +44 (0) 20 7336 9595
Fax: +44 (0) 20 7336 9596
www.Excelian.com
_
This e-mail has been scanned for viruses by MessageLabs. For further 
information visit http://www.messagelabs.com


Excelian subscribes to cleaner and greener methods of working. Help take 
responsibility for the environment. Please don't print this email unless 
you absolutely have to.





Re: [Dnsmasq-discuss] server forwarding all traffic to parents after a successful PTR query of itself

2010-02-17 Thread Alberto Cuesta-Canada
Cool, that makes a lot of sense. I'm actually reengineering the DNS 
infrastructure here, so it will be easy to account for and trace that at this 
stage.
 
I'll let you know when I find the rogue queries, many thanks,
 
Alberto Cuesta-Canada
GaaS Team Lead
Excelian Ltd.
+44 (0) 7942633361



From: Simon Kelley [mailto:si...@thekelleys.org.uk]
Sent: Wed 17/02/2010 10:30
To: Alberto Cuesta-Canada
Cc: dnsmasq-discuss@lists.thekelleys.org.uk; Grid Support
Subject: Re: [Dnsmasq-discuss] server forwarding all traffic to parents after a 
successful PTR query of itself



Alberto Cuesta-Canada wrote:
 Hi Simon,
 
 the parents of 250 (my dnsmasq server) have forwarding rules for the
 dselgrid.local domain, that I run. So I assumed that the queries pushed
 upstream would be routed down again, and timeout in a loop.

Ahh, that could easily be the problem. If you generate a loop between
two DNS servers, each forwarding to the other, then the queries can
easily bounce back-and-forth forever. Dnsmasq will manage this situation
reasonably well, and manage to server other traffic, but the circulating
queries will eat bandwidth and CPU.

The logs would seem to show a strange query of some sort (dnsmasq
can't parse a domain-name from the query, hence forwarded query rather
than forwarded domain-name) If such queries can circulate forever
then you have a problem.


 
 That said, in the logs I could still see successful PTR and A queries,
 outnumbered 10 to 1 by forwards. I'm not sure about the behaviour of
 local queries, I don't remember from yesterday, but I think they worked.
 
 94 is a Platform Grid Master, that is a W2K3 machine which runs only one
 application. It keeps a cache of machines but it doesn't give DNS
 services, or anything similar. The interesting thing is that the PTR
 request doesn't always produce this effect. I have enterprise support
 for that software, so I will ask them.
 
 dnsmasq is running in a quite complicated setup. We have a XenServer
 host running a Ubuntu 9.04 VM. I have just 1GB free on that machine and
 out of disk space scenarios are fatal, so I can't tcpdump. There is a
 rebuild of it coming in the next two weeks that will give me another 50GB.
 
 Any idea on what to look for, or any hypothesis of what could be
 happening should be enough, I can keep investigating and contain it with
 workarounds for a while.

See above. a loop, possibly only of odd queries.

Cheers,

Simon.

 
 Many thanks,
 
 *Alberto Cuesta-Canada*
 GaaS Team Lead
 Excelian Ltd.
 +44 (0) 7942633361

 
 *From:* Simon Kelley [mailto:si...@thekelleys.org.uk]
 *Sent:* Wed 17/02/2010 10:04
 *To:* Alberto Cuesta-Canada
 *Cc:* dnsmasq-discuss@lists.thekelleys.org.uk; Grid Support
 *Subject:* Re: [Dnsmasq-discuss] server forwarding all traffic to
 parents after a successful PTR query of itself

 It's not clear to me what is going on here. How does the pattern
 continue? Do you just see  forwarded query to 172.30.48.192 from now
 on until the server is restarted, or do you still see query[A] and
 query[PTR} lines?

 Do queries which get pushed upstream continue to work? How about queries
 which should be answered locally?

 What is 172.30.158.94? Is it running anything that may generate odd
 DNS queries? The holy grail would be to able prod that machine to
 reproduce this at will.

 What sort of machine are you running dnsmasq on? Does it have a
 reasonable amount of spare storage so that you could tcpdump all traffic
 to/from port 53,UDP for offline analysis?


 Simon.


 The information contained in this email and any attached files are
 confidential and intended solely for the addressee(s). The email may be
 legally privileged or prohibited from disclosure and unauthorised use.
 If you are not the named addressee you may not use, copy, or disclose
 this information to any other person. If you received this message in
 error please notify the sender immediately and delete it from your system.

 Any opinion or views contained in this email message are those of the
 sender, and do not represent those of the Company in any way and
 reliance should not be placed upon its contents. Unless otherwise
 stated, this email message is not intended to be contractually binding.
 Where an Agreement exists between our respective companies and there is
 conflict between the contents of this email message and the Agreement
 then the terms of that Agreement shall prevail.

 Excelian
 50 Featherstone Street
 London
 EC1Y 8RT
 Tel: +44 (0) 20 7336 9595
 Fax: +44 (0) 20 7336 9596
 www.Excelian.com
 _
 This e-mail has been scanned for viruses by MessageLabs. For further
 information visit http://www.messagelabs.com http://www.messagelabs.com/ 

 Excelian subscribes to cleaner and greener methods of working. Help take
 responsibility for the environment. Please don't