Re: [Pdns-users] Possible tcp listener issue
Hi Bert, bert hubert wrote: On Mon, Jul 26, 2010 at 02:24:03PM +0100, Simon Bedford wrote: I then just upgraded the version to our patched version of 3.2 but without specifying our usual 4 threads and the response times went up on the graphing straight away, upping the number of threads makes no difference and the response times stay high... How high? High should have said higher :) As explained earlier in the thread, the response times on Etch and 3.1.7-2 and then rise immediately after going to the patched 3.2 but still around 10ms, I will send you the graphs off list to peruse but they double essentially. Using my ISPs resolvers, which as yet do not run PowerDNS, bbc.co.uk has been slow to resolve for me as well. I will let the graph update some more and save it should you need it for more information. If you could show, this would be appreciated. I'll send you a few graphs from 2 different devices, thanks. Thanks. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Simon Bedford wrote: Hi Bert, Is there any update on this bug, interested to know if you have found the root cause of it... Also I have just upgraded one of our internal servers from Debian Etch and pdns-recursor-3.1.7-2 to Debian Lenny and the same patched version of pdns-recursor-3.2 that we use on the live customer DNS and the response time graphing for bbc.co.uk has gone up by at least double. Is this something you have seen before at all, I am investigating from this end as well but the upgrade definitely caused this, I can send you the graphing seperately if required. Bit more info on this for you, just done 3 more servers and going from Etch to Lenny and keeping pdns-recursor 3.1.7-2 was fine and response times were the same as before the OS upgrade. I then just upgraded the version to our patched version of 3.2 but without specifying our usual 4 threads and the response times went up on the graphing straight away, upping the number of threads makes no difference and the response times stay high... I will let the graph update some more and save it should you need it for more information. Its worth pointing out that even though they have double its still a good response time just not as awesome as the initial versions... Thanks Simon bert hubert wrote: The issue has now been reproduced on one of the PowerDNS servers, thanks to Brad and Christian. However, it is quite some work to make it happen and we've not yet been able to make sense of *why* it is happening. We'll keep you posted. Once this is solved, 3.3 will be released. Bert On Tue, Jul 06, 2010 at 04:11:35PM +0100, Simon Bedford wrote: Hi Christian, Have we had any joy in replicating this issue or highlighting the root cause of the issue? Thanks Simon Christian Hofstädtler wrote: Brad, Josh, I'm trying to reproduce the behaviour you see here, but fail to do so at this time. Maybe you can give me more details about your environment, at least OS version, architecture, and any compile options you've used to build the binary (say, STATIC or LUA). Also of interest would be the last SVN revision you've tried out. If there is anything else you can think of which is special in your environment, please let me know of that, too. Thanks, Christian ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Excellent news Bert, thanks for the update, good luck with the root cause squishing. Simon bert hubert wrote: The issue has now been reproduced on one of the PowerDNS servers, thanks to Brad and Christian. However, it is quite some work to make it happen and we've not yet been able to make sense of *why* it is happening. We'll keep you posted. Once this is solved, 3.3 will be released. Bert On Tue, Jul 06, 2010 at 04:11:35PM +0100, Simon Bedford wrote: Hi Christian, Have we had any joy in replicating this issue or highlighting the root cause of the issue? Thanks Simon Christian Hofstädtler wrote: Brad, Josh, I'm trying to reproduce the behaviour you see here, but fail to do so at this time. Maybe you can give me more details about your environment, at least OS version, architecture, and any compile options you've used to build the binary (say, STATIC or LUA). Also of interest would be the last SVN revision you've tried out. If there is anything else you can think of which is special in your environment, please let me know of that, too. Thanks, Christian ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Brad, Josh, I'm trying to reproduce the behaviour you see here, but fail to do so at this time. Maybe you can give me more details about your environment, at least OS version, architecture, and any compile options you've used to build the binary (say, STATIC or LUA). Also of interest would be the last SVN revision you've tried out. If there is anything else you can think of which is special in your environment, please let me know of that, too. Thanks, Christian ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Hi Christian, We are running a mixture of Debian Etch and Debian Lenny, we get the source tarball from your website and have then been patching from the appropriate changeset and then creating a static .deb package. the only thing we change in the process is the SEMI= flag in the rules file to static and also get rid of the distclean section for just a clean in the MAKE section, again in the rules file. We have followed this process for quite a while now and the package seems to create OK and function well apart from the closing of the TCP connections. Thanks Simon From: pdns-users-boun...@mailman.powerdns.com [pdns-users-boun...@mailman.powerdns.com] On Behalf Of Christian Hofstädtler [...@zeha.at] Sent: 30 June 2010 08:05 To: pdns-users@mailman.powerdns.com Cc: brad.dame...@clearwire.com Subject: Re: [Pdns-users] Possible tcp listener issue Brad, Josh, I'm trying to reproduce the behaviour you see here, but fail to do so at this time. Maybe you can give me more details about your environment, at least OS version, architecture, and any compile options you've used to build the binary (say, STATIC or LUA). Also of interest would be the last SVN revision you've tried out. If there is anything else you can think of which is special in your environment, please let me know of that, too. Thanks, Christian ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Hi Bert et al, bert hubert schreef: Simon, Josh, We're running out of things we can do without seeing packets. Could you try http://wiki.powerdns.com/trac/changeset/1652 ? This is the last thing we can come up with that might explain the issue you are seeing. This fix is necessary, but unlikely to be the cause of the problems you are seeing. I sent a message about a similar issue seen using the auth-server. The problem occured again about two weeks ago and I noticed in our case that tcp-sockets weren't closing. It being an auth-server I didn't check the remote end of the connection but I will next time around. Is it possible that the remote dns-client is the problem? Maybe it just isn't closing the connection. Regard, Ton ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
bert hubert schreef: On Mon, Jun 28, 2010 at 01:13:16PM +0200, Ton van Rosmalen wrote: This is the last thing we can come up with that might explain the issue you are seeing. This fix is necessary, but unlikely to be the cause of the problems you are seeing. I sent a message about a similar issue seen using the auth-server. The problem occured again about two weeks ago and I noticed in our case that tcp-sockets weren't closing. It being an auth-server I didn't check the remote end of the connection but I will next time around. Ton, The authoritative server TCP/IP mechanics are completely different from the mechanics of the Recursor. Which version of the Authoritative Server do you run? I'm running version 2.9.22. In our case the remote connections where all coming from argus.sidn.nl. This is a machine (as you probably know) of the .nl-registry which AFAIK excutes zonechecks for registered .nl-domains. It hasn't happened since but I will do some more testing next around, assuming a next time will come. I'll start a new thread if/when the problem occurs again. If I need to collect specfic information just let me know and I'll try. Ton ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
-Original Message- So far, Josh Simon are the only ones reporting this issue though, so something must be going on. What Laurent Papier reported has been fixed already (although he may not know it yet ;-)) Bert Bert, Us too!! We're able to reproduce what may be this same TCP listener issue that is being discussed on the email list with little effort, using the latest 3.3 pre-release code. We use the queryfile-example-3million file from nominum : ftp://ftp.nominum.com/pub/nominum/dnsperf/data/queryfile-example-3millio n.gz and the script below to fire off a flood of TCP requests, 1000 at a time. Within a few minutes, pdns_recursor locks up and refuses to answer any more TCP queries. UDP continues fine. Our last run, it queried only 16K addresses before hanging. Environment is a HP G6 blade running CentOS 5.1 Linux: # uname -a Linux MOC-LAB001-HPB-DNS-NTP-1 2.6.18-53.1.14.el5 #1 SMP Wed Mar 5 11:37:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux Invoked with the following arguments: /usr/sbin/pdns_recursor --local-address=172.26.68.42,127.0.0.1 --allow-from= --max-cache-entries=300 --log-common-errors=no --threads=4 --socket-dir=/var/run/recursor1 --daemon --dont-query= #!/bin/ksh MAX=1000 integer n=0 zcat queryfile-example-3million.gz | while read x ; do let n=n+1 if [ $n -le $MAX ] ; then print dig @127.0.0.1 +tcp +short $x out 2out.err else print wait let n=0; fi done # USAGE: # genrun.sh run # sh ./run # # Use the following script to monitor progress: # #!/bin/ksh # while : ; do # nt=$(rec_control --socket-dir=/var/run/recursor1 get tcp-questions) # nl=$(netstat -an|grep :53|grep -v LISTEN|wc -l) # print $nt queries $nl open # sleep 5 # done These errors were noted in the logs: pdns_recursor[28030]: segfault at 7ffefff8 rip 00455740 rsp 2aaecbb0 error 4 pdns_recursor[28082] general protection rip:455740 rsp:2aaecbb0 error:0 Let us know if you have any new patches to test! Thanks, Brad This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Thanks Bert, Appreciate you looking into this, would be nice to nail it once and for all. Simon From: bert.hub...@netherlabs.nl [bert.hub...@netherlabs.nl] Sent: 22 June 2010 15:50 To: Simon Bedford Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Possible tcp listener issue So much for my beliefs then :) Will take another good look at how else this could happen. The patch we made at least solves A bug just not your bug. Sent from my phone. - Reply message - From: Simon Bedford sbedf...@plus.net Date: Tue, Jun 22, 2010 09:44 Subject: [Pdns-users] Possible tcp listener issue To: bert hubert bert.hub...@netherlabs.nl Cc: pdns-users@mailman.powerdns.com pdns-users@mailman.powerdns.com bert hubert wrote: Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you do not see the log messages, I'm pretty sure this is it. Bert Morning, I created and installed the package yesterday (with the following option STATIC=full) and the tcp clients still appears to be growing, less than 24 hours later and the count is at 54 already although if I monitor it more frequently I do see the number fluctuate a little put the pattern is still growth as though not cleaning up all client connections The following stats are entered in every 5 mins :- 51 51 51 51 51 51 51 51 51 51 51 51 51 52 52 52 52 52 52 53 53 53 53 53 53 54 54 54 54 54 Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Simon, Can you grep your logfiles for 'epoll' or 'unlisted' or 'exception'? Bert On Wed, Jun 23, 2010 at 11:00:12AM +0100, Simon Bedford wrote: Thanks Bert, Appreciate you looking into this, would be nice to nail it once and for all. Simon From: bert.hub...@netherlabs.nl [bert.hub...@netherlabs.nl] Sent: 22 June 2010 15:50 To: Simon Bedford Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Possible tcp listener issue So much for my beliefs then :) Will take another good look at how else this could happen. The patch we made at least solves A bug just not your bug. Sent from my phone. - Reply message - From: Simon Bedford sbedf...@plus.net Date: Tue, Jun 22, 2010 09:44 Subject: [Pdns-users] Possible tcp listener issue To: bert hubert bert.hub...@netherlabs.nl Cc: pdns-users@mailman.powerdns.com pdns-users@mailman.powerdns.com bert hubert wrote: Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you do not see the log messages, I'm pretty sure this is it. Bert Morning, I created and installed the package yesterday (with the following option STATIC=full) and the tcp clients still appears to be growing, less than 24 hours later and the count is at 54 already although if I monitor it more frequently I do see the number fluctuate a little put the pattern is still growth as though not cleaning up all client connections The following stats are entered in every 5 mins :- 51 51 51 51 51 51 51 51 51 51 51 51 51 52 52 52 52 52 52 53 53 53 53 53 53 54 54 54 54 54 Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Simon is away this afternoon, but to keep things moving... On the server that he is monitoring on there is only one thing in the logs for those, and its when pDNS is starting up (this is the only restart in the logs): /var/log/syslog.1.gz:Jun 21 11:22:18 pcl-cachedns01 pdns_recursor[31939]: Enabled 'epoll' multiplexer Ta, Josh From: pdns-users-boun...@mailman.powerdns.com [pdns-users-boun...@mailman.powerdns.com] On Behalf Of bert hubert [bert.hub...@netherlabs.nl] Sent: 23 June 2010 12:47 To: Simon Bedford Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Possible tcp listener issue Simon, Can you grep your logfiles for 'epoll' or 'unlisted' or 'exception'? Bert On Wed, Jun 23, 2010 at 11:00:12AM +0100, Simon Bedford wrote: Thanks Bert, Appreciate you looking into this, would be nice to nail it once and for all. Simon From: bert.hub...@netherlabs.nl [bert.hub...@netherlabs.nl] Sent: 22 June 2010 15:50 To: Simon Bedford Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Possible tcp listener issue So much for my beliefs then :) Will take another good look at how else this could happen. The patch we made at least solves A bug just not your bug. Sent from my phone. - Reply message - From: Simon Bedford sbedf...@plus.net Date: Tue, Jun 22, 2010 09:44 Subject: [Pdns-users] Possible tcp listener issue To: bert hubert bert.hub...@netherlabs.nl Cc: pdns-users@mailman.powerdns.com pdns-users@mailman.powerdns.com bert hubert wrote: Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you do not see the log messages, I'm pretty sure this is it. Bert Morning, I created and installed the package yesterday (with the following option STATIC=full) and the tcp clients still appears to be growing, less than 24 hours later and the count is at 54 already although if I monitor it more frequently I do see the number fluctuate a little put the pattern is still growth as though not cleaning up all client connections The following stats are entered in every 5 mins :- 51 51 51 51 51 51 51 51 51 51 51 51 51 52 52 52 52 52 52 53 53 53 53 53 53 54 54 54 54 54 Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Wed, Jun 23, 2010 at 01:21:03PM +0100, Josh Berry wrote: Simon is away this afternoon, but to keep things moving... On the server that he is monitoring on there is only one thing in the logs for those, and its when pDNS is starting up (this is the only restart in the logs): /var/log/syslog.1.gz:Jun 21 11:22:18 pcl-cachedns01 pdns_recursor[31939]: Enabled 'epoll' multiplexer Ok - anything along the lines of EOF writing TCP answer to Error writing TCP answer to Oops, partial answer sent to ? Bert Ta, Josh From: pdns-users-boun...@mailman.powerdns.com [pdns-users-boun...@mailman.powerdns.com] On Behalf Of bert hubert [bert.hub...@netherlabs.nl] Sent: 23 June 2010 12:47 To: Simon Bedford Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Possible tcp listener issue Simon, Can you grep your logfiles for 'epoll' or 'unlisted' or 'exception'? Bert On Wed, Jun 23, 2010 at 11:00:12AM +0100, Simon Bedford wrote: Thanks Bert, Appreciate you looking into this, would be nice to nail it once and for all. Simon From: bert.hub...@netherlabs.nl [bert.hub...@netherlabs.nl] Sent: 22 June 2010 15:50 To: Simon Bedford Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Possible tcp listener issue So much for my beliefs then :) Will take another good look at how else this could happen. The patch we made at least solves A bug just not your bug. Sent from my phone. - Reply message - From: Simon Bedford sbedf...@plus.net Date: Tue, Jun 22, 2010 09:44 Subject: [Pdns-users] Possible tcp listener issue To: bert hubert bert.hub...@netherlabs.nl Cc: pdns-users@mailman.powerdns.com pdns-users@mailman.powerdns.com bert hubert wrote: Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you do not see the log messages, I'm pretty sure this is it. Bert Morning, I created and installed the package yesterday (with the following option STATIC=full) and the tcp clients still appears to be growing, less than 24 hours later and the count is at 54 already although if I monitor it more frequently I do see the number fluctuate a little put the pattern is still growth as though not cleaning up all client connections The following stats are entered in every 5 mins :- 51 51 51 51 51 51 51 51 51 51 51 51 51 52 52 52 52 52 52 53 53 53 53 53 53 54 54 54 54 54 Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
bert hubert wrote: Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you do not see the log messages, I'm pretty sure this is it. Bert Morning, I created and installed the package yesterday (with the following option STATIC=full) and the tcp clients still appears to be growing, less than 24 hours later and the count is at 54 already although if I monitor it more frequently I do see the number fluctuate a little put the pattern is still growth as though not cleaning up all client connections The following stats are entered in every 5 mins :- 51 51 51 51 51 51 51 51 51 51 51 51 51 52 52 52 52 52 52 53 53 53 53 53 53 54 54 54 54 54 Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
So much for my beliefs then :) Will take another good look at how else this could happen. The patch we made at least solves A bug just not your bug. Sent from my phone. - Reply message - From: Simon Bedford sbedf...@plus.net Date: Tue, Jun 22, 2010 09:44 Subject: [Pdns-users] Possible tcp listener issue To: bert hubert bert.hub...@netherlabs.nl Cc: pdns-users@mailman.powerdns.com pdns-users@mailman.powerdns.com bert hubert wrote: Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you do not see the log messages, I'm pretty sure this is it. Bert Morning, I created and installed the package yesterday (with the following option STATIC=full) and the tcp clients still appears to be growing, less than 24 hours later and the count is at 54 already although if I monitor it more frequently I do see the number fluctuate a little put the pattern is still growth as though not cleaning up all client connections The following stats are entered in every 5 mins :- 51 51 51 51 51 51 51 51 51 51 51 51 51 52 52 52 52 52 52 53 53 53 53 53 53 54 54 54 54 54 Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Morning Bert, We had a crash on one of the upgraded servers over the weekend and as suspected it happened around the default max tcp-clients, please see the entry from my log below that was running to query the amount of clients :- 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 129 0 restarted here 6 0 0 0 1 0 0 0 0 Also the grep you asked for returned no lines for any of the stated messages. This looks to be an issue with closing down certain kinds of tcp clients from the debug. Thanks Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you do not see the log messages, I'm pretty sure this is it. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
To everybody seeing this issue (all two of you :)), can you grep the logfile on: startDoResolve problem: DNS parser error: STL error: Any other exception in a resolver context Without the quotes. If you see any of these lines, it might explain the problem. Thanks! On Thu, Jun 17, 2010 at 04:16:50PM +0200, bert hubert wrote: On Thu, Jun 17, 2010 at 03:14:39PM +0100, Simon Bedford wrote: Bert, its now climbing and not seeming to close any clients on both servers now, although one is much worse than the other, one is at 11 and the other is at 108 tcp-clients at the moment. Could you run: # tcpdump -s 0 -w tcpdns port 53 and tcp For a while (few hours) And then send me the output of: # tcpdump -n -r tcpdns You can obfuscate IP addresses, as long as I can see patterns that would be great. Thanks! Simon Simon Bedford wrote: Please see answers below :- bert hubert wrote: On Wed, Jun 16, 2010 at 11:10:29AM +0100, Simon Bedford wrote: I have been running the 'get tcp-clients' every 5 mins in cron on both servers, one is very low 10 and fluctuates, the other is just growing and growing and is currently at 55, hope this info is of value for debug. Is there any difference between these two servers in terms of: * Operating system No, both Debian Etch * Number of processors No, slightly faster processors in the one with the most clients * Network configuration (load balancer?) Different data centre but same network architecture, both loadbalanced. * Use (mail servers versus residential internet connections?) No, config is identical and use is only as a cachingDNS and NTP server * Anything else that comes to mind Nothing appears to be different, they are from a build that should be identical across the platform, although we will be looking to upgrade to Lenny in the near future. Thanks! Thanks Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Bert, its now climbing and not seeming to close any clients on both servers now, although one is much worse than the other, one is at 11 and the other is at 108 tcp-clients at the moment. Simon Simon Bedford wrote: Please see answers below :- bert hubert wrote: On Wed, Jun 16, 2010 at 11:10:29AM +0100, Simon Bedford wrote: I have been running the 'get tcp-clients' every 5 mins in cron on both servers, one is very low 10 and fluctuates, the other is just growing and growing and is currently at 55, hope this info is of value for debug. Is there any difference between these two servers in terms of: * Operating system No, both Debian Etch * Number of processors No, slightly faster processors in the one with the most clients * Network configuration (load balancer?) Different data centre but same network architecture, both loadbalanced. * Use (mail servers versus residential internet connections?) No, config is identical and use is only as a cachingDNS and NTP server * Anything else that comes to mind Nothing appears to be different, they are from a build that should be identical across the platform, although we will be looking to upgrade to Lenny in the near future. Thanks! Thanks Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
bert hubert wrote: On Tue, Jun 15, 2010 at 02:23:04PM +0100, Simon Bedford wrote: It contains some other exciting stuff too, and it appears to be stable for production use. I have now built a static package and installed to 2 of the 8 servers after testing in isolation, they appear to be running fine and I can use the rec_control get tcp-clients now to see the number reported. We will monitor the new package for a week before rolling out to the rest of the servers and let you know how that goes. Ok - please check the 'get tcp-clients' every once in a while. I personally expect that this version will still see tcp support die after a while under your conditions. Bert I have been running the 'get tcp-clients' every 5 mins in cron on both servers, one is very low 10 and fluctuates, the other is just growing and growing and is currently at 55, hope this info is of value for debug. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Wed, Jun 16, 2010 at 11:10:29AM +0100, Simon Bedford wrote: I have been running the 'get tcp-clients' every 5 mins in cron on both servers, one is very low 10 and fluctuates, the other is just growing and growing and is currently at 55, hope this info is of value for debug. Is there any difference between these two servers in terms of: * Operating system * Number of processors * Network configuration (load balancer?) * Use (mail servers versus residential internet connections?) * Anything else that comes to mind Thanks! ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Please see answers below :- bert hubert wrote: On Wed, Jun 16, 2010 at 11:10:29AM +0100, Simon Bedford wrote: I have been running the 'get tcp-clients' every 5 mins in cron on both servers, one is very low 10 and fluctuates, the other is just growing and growing and is currently at 55, hope this info is of value for debug. Is there any difference between these two servers in terms of: * Operating system No, both Debian Etch * Number of processors No, slightly faster processors in the one with the most clients * Network configuration (load balancer?) Different data centre but same network architecture, both loadbalanced. * Use (mail servers versus residential internet connections?) No, config is identical and use is only as a cachingDNS and NTP server * Anything else that comes to mind Nothing appears to be different, they are from a build that should be identical across the platform, although we will be looking to upgrade to Lenny in the near future. Thanks! Thanks Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Simon, There is now a 3.3 prerelease that contains all the stuff you'd need for further trouble shooting. Please find it on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 It contains some other exciting stuff too, and it appears to be stable for production use. Bert Hi, I have now built a static package and installed to 2 of the 8 servers after testing in isolation, they appear to be running fine and I can use the rec_control get tcp-clients now to see the number reported. We will monitor the new package for a week before rolling out to the rest of the servers and let you know how that goes. Regards Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Tue, Jun 15, 2010 at 02:23:04PM +0100, Simon Bedford wrote: It contains some other exciting stuff too, and it appears to be stable for production use. I have now built a static package and installed to 2 of the 8 servers after testing in isolation, they appear to be running fine and I can use the rec_control get tcp-clients now to see the number reported. We will monitor the new package for a week before rolling out to the rest of the servers and let you know how that goes. Ok - please check the 'get tcp-clients' every once in a while. I personally expect that this version will still see tcp support die after a while under your conditions. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Mon, Jun 07, 2010 at 04:08:17PM +0100, Simon Bedford wrote: Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically run 'rec_control get tcp-clients' ? Hi Bert, I have just returned from annual leave and began to look into this earlier but when I download the latest source from your website and put the files from the changeset into place (following the same procedure as worked previously) I get the following error when trying to create the debian package:- Simon, There is now a 3.3 prerelease that contains all the stuff you'd need for further trouble shooting. Please find it on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 It contains some other exciting stuff too, and it appears to be stable for production use. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On 30/05/2010 19:31, bert hubert wrote: On Thu, May 20, 2010 at 11:12:29AM +0100, Simon Bedford wrote: This has happened a further twice in the last week, output sent off list, please let me know if you need any further information. Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically run 'rec_control get tcp-clients' ? (Sending again as not sure if this got to list, problems with email yesterday) Hi Bert, I have just returned from annual leave and began to look into this earlier but when I download the latest source from your website and put the files from the changeset into place (following the same procedure as worked previously) I get the following error when trying to create the debian package:- g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o logger.o logger.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o arguments.o arguments.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o lwres.o lwres.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o pdns_recursor.o pdns_recursor.cc pdns_recursor.cc: In function ‘void makeTCPServerSockets()’ pdns_recursor.cc:895: error: ‘makeIPv6sockaddr’ was not declared in this scope pdns_recursor.cc: In function ‘void makeUDPServerSockets()’: pdns_recursor.cc:959: error: ‘makeIPv6sockaddr’ was not declared in this scope make[1]: *** [pdns_recursor.o] Error 1 make[1]: Leaving directory `/home/***/applications/pdns-recursor' make: *** [build-stamp] Error 2 I will continue to try and troubleshoot this issue but thought I would post back to list to show that I am looking into and so that you may give me a pointer if this is a known problem or something you have come across before. A point worth noting it that it also errors if I just try to compile the software rather than creating a Debian package. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On 30/05/2010 19:31, bert hubert wrote: On Thu, May 20, 2010 at 11:12:29AM +0100, Simon Bedford wrote: This has happened a further twice in the last week, output sent off list, please let me know if you need any further information. Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically run 'rec_control get tcp-clients' ? Hi Bert, I have just returned from annual leave and began to look into this earlier but when I download the latest source from your website and put the files from the changeset into place (following the same procedure as worked previously) I get the following error when trying to create the debian package:- g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o logger.o logger.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o arguments.o arguments.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o lwres.o lwres.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o pdns_recursor.o pdns_recursor.cc pdns_recursor.cc: In function ‘void makeTCPServerSockets()’ pdns_recursor.cc:895: error: ‘makeIPv6sockaddr’ was not declared in this scope pdns_recursor.cc: In function ‘void makeUDPServerSockets()’: pdns_recursor.cc:959: error: ‘makeIPv6sockaddr’ was not declared in this scope make[1]: *** [pdns_recursor.o] Error 1 make[1]: Leaving directory `/home/***/applications/pdns-recursor' make: *** [build-stamp] Error 2 I will continue to try and troubleshoot this issue but thought I would post back to list to show that I am looking into and so that you may give me a pointer if this is a known problem or something you have come across before. A point worth noting it that it also errors if I just try to compile the software rather than creating a Debian package. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically run 'rec_control get tcp-clients' ? Hi Bert, I have just returned from annual leave and began to look into this earlier but when I download the latest source from your website and put the files from the changeset into place (following the same procedure as worked previously) I get the following error when trying to create the debian package:- g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o logger.o logger.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o arguments.o arguments.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o lwres.o lwres.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o pdns_recursor.o pdns_recursor.cc pdns_recursor.cc: In function ‘void makeTCPServerSockets()’: pdns_recursor.cc:895: error: ‘makeIPv6sockaddr’ was not declared in this scope pdns_recursor.cc: In function ‘void makeUDPServerSockets()’: pdns_recursor.cc:959: error: ‘makeIPv6sockaddr’ was not declared in this scope make[1]: *** [pdns_recursor.o] Error 1 make[1]: Leaving directory `/home/***/applications/pdns-recursor' make: *** [build-stamp] Error 2 I will continue to try and troubleshoot this issue but thought I would post back to list to show that I am looking into and so that you may give me a pointer if this is a known problem or something you have come across before. A point worth noting it that it also errors if I just try to compile the software rather than creating a Debian package. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically run 'rec_control get tcp-clients' ? Hi Bert, I have just returned from annual leave and began to look into this earlier but when I download the latest source from your website and put the files from the changeset into place (following the same procedure as worked previously) I get the following error when trying to create the debian package:- g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o logger.o logger.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o arguments.o arguments.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o lwres.o lwres.cc g++ -g -O2 -Wall -O3 -pthread -D_GNU_SOURCE -c -o pdns_recursor.o pdns_recursor.cc pdns_recursor.cc: In function ‘void makeTCPServerSockets()’: pdns_recursor.cc:895: error: ‘makeIPv6sockaddr’ was not declared in this scope pdns_recursor.cc: In function ‘void makeUDPServerSockets()’: pdns_recursor.cc:959: error: ‘makeIPv6sockaddr’ was not declared in this scope make[1]: *** [pdns_recursor.o] Error 1 make[1]: Leaving directory `/home/***/applications/pdns-recursor' make: *** [build-stamp] Error 2 I will continue to try and troubleshoot this issue but thought I would post back to list to show that I am looking into and so that you may give me a pointer if this is a known problem or something you have come across before. A point worth noting it that it also errors if I just try to compile the software rather than creating a Debian package. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Thu, May 20, 2010 at 11:12:29AM +0100, Simon Bedford wrote: This has happened a further twice in the last week, output sent off list, please let me know if you need any further information. Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically run 'rec_control get tcp-clients' ? If you see that number increase very slowly, this is a powerful hint where we need to look for the problem. If you do not see it increase very slowly, it means we have to look somewhere else.. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Simon Bedford wrote: bert hubert wrote: On Mon, May 10, 2010 at 05:20:18PM +0100, Simon Bedford wrote: It has happened again over the weekend and a simple restart fixed the issue. Simon, We have a new theory why this might be happening, to verify, if the issue ever occcurs again, can you run: $ netstat -an | grep ^tcp | grep :53 And send me the output? Please do this *before* a restart. Thanks! I have mailled you the output Bert, thanks. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users This has happened a further twice in the last week, output sent off list, please let me know if you need any further information. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello, I didn't read the whole thread. But we had the same problem in past. Is it maybee a timeout mysql connection. We had this only system with out load. Ciao Marco Am 06.05.10 07:45, schrieb bert hubert: On Wed, May 05, 2010 at 11:27:13AM +0200, Ton van Rosmalen wrote: Our primary auth pdns didn't respond to tcp queries anymore where udp queries still worked. A restart solved the problem. I didn't think anything of it until I read these new messages on this thread. Could it be a shared problem between the recursor and auth server? Highly unlikely. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users - -- Marco Schrieck Bereichsleiter Entwicklung - -- InterNetX GmbH Maximilianstr. 6 93047 Regensburg Germany Tel. +49 941 59559-0 Fax +49 941 59579-050 www.internetx.com www.facebook.com/InterNetX www.twitter.com/InterNetX Geschäftsführer/CEO: Thomas Mörz Amtsgericht Regensburg, HRB 7142 ICQ: 232016987 -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJL8RUTAAoJEN9yMHEBd2HnGRAIAMBtTvOoXVRUgT9sl59uWCSR ZDfJJTotO0ZoIyI3PrksCK95nQKmb9n+KnYrz+ZYPI9F7cCv5tnqGHUvQ28IFoXW xzxnVm6T50n+68XRW2/kj6l139U08ebuJ6tzKC0gYLQW0IdbfayBKJbdRCnBOPDs AKprkh9fWqyFPdUb/AEOfOzCRX5rnBwKP/5jzm4J1xT00LKApfo2LaK4/GwQpWIH ujsj/fuP81Kz680IrU7STHroaiREFJRpzlk4ehDlqDr7lnLuu6VuZqnJcquTpxwV GNMK/3Dwhaa+JQupU+hS+Zm1AhZunOkv0iO/mJ15ETz1trrzM/xLAgKTzbH64/s= =6nTU -END PGP SIGNATURE- ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
bert hubert wrote: On Mon, May 10, 2010 at 05:20:18PM +0100, Simon Bedford wrote: It has happened again over the weekend and a simple restart fixed the issue. Simon, We have a new theory why this might be happening, to verify, if the issue ever occcurs again, can you run: $ netstat -an | grep ^tcp | grep :53 And send me the output? Please do this *before* a restart. Thanks! I have mailled you the output Bert, thanks. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
bert hubert wrote: On Tue, May 04, 2010 at 05:37:10PM +0100, Simon Bedford wrote: How many TCP/IP queries do you see per second, roughly? A quick run of rec_control should dig this up. date;rec_control get tcp-questions Tue May 4 17:34:22 BST 2010 79371 date;rec_control get tcp-questions Tue May 4 17:35:53 BST 2010 79924 Gives us rough stats from a short period of 6 tcp requests per second per server, theres 8 servers. Ok, this is a good clue. Can you share 'grep tcp /etc/powerdns/recursor.conf', or whereever your configuration is? Thanks. Bert, I believe we have the defaults for this option which is 0 :- # client-tcp-timeoutTimeout in seconds when talking to TCP clients # client-tcp-timeout=2 # max-tcp-clients Maximum number of simultaneous TCP clients # max-tcp-clients=128 # max-tcp-per-client If set, maximum number of TCP sessions per client (IP address) max-tcp-per-client=0 Thanks Simon -- | Simon BedfordBroadband Solutions for | Senior Operations Engineer Home Business @ | PlusNet plc www.plus.net | Desk: 0870 705 8017 | Mobile: 07803 040392 | Registered Office: Internet House, 2 Tenter Street, Sheffield, S1 4BY | Registered in England no: 3279013 +-- Plusnet - ISPA Best Consumer ISP 2008 + ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Hi, bert hubert schreef: On Tue, May 04, 2010 at 05:04:51PM +0100, Simon Bedford wrote: [snip] Not trying to hijack the thread but I'm using the authorative server and saw a similar issue last Monday. Our primary auth pdns didn't respond to tcp queries anymore where udp queries still worked. A restart solved the problem. I didn't think anything of it until I read these new messages on this thread. Could it be a shared problem between the recursor and auth server? Regards, Ton ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Wed, May 05, 2010 at 11:27:13AM +0200, Ton van Rosmalen wrote: Our primary auth pdns didn't respond to tcp queries anymore where udp queries still worked. A restart solved the problem. I didn't think anything of it until I read these new messages on this thread. Could it be a shared problem between the recursor and auth server? Highly unlikely. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Tue, May 04, 2010 at 05:37:10PM +0100, Simon Bedford wrote: How many TCP/IP queries do you see per second, roughly? A quick run of rec_control should dig this up. date;rec_control get tcp-questions Tue May 4 17:34:22 BST 2010 79371 date;rec_control get tcp-questions Tue May 4 17:35:53 BST 2010 79924 Gives us rough stats from a short period of 6 tcp requests per second per server, theres 8 servers. Ok, this is a good clue. Can you share 'grep tcp /etc/powerdns/recursor.conf', or whereever your configuration is? Thanks. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Simon Bedford wrote: Laurent Papier wrote: Le Wed, 31 Mar 2010 20:42:33 +0200 bert hubert bert.hub...@netherlabs.nl écrit: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I can help test this patch Bert. It can be found on: http://wiki.powerdns.com/trac/changeset/1546 or http://wiki.powerdns.com/trac/changeset/1546?format=diffnew=1546 The 'applying patches' section on this page explains how to apply: http://wiki.powerdns.com/trac/wiki/HACKING I hope this solves the issue - it sure sounds like it! 3 days without problem, your patch solves the issue for me. We created a patched version in the end and rolled to one server back on the 8th April, no fault seen since and now rolled to the entire platform as of today, looking good. Thanks Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users We have now had 2 more failures of the tcp listener on 2 seperate servers in the last week, a simple reload of the software fixes this. the thing I have noticed when this occurs is that when telnetting to the port you can then type in anything and try and escape, etc but the connection never closes, whereas on a working server the session will close after a short while of typing and pressing Enter. The logs do not show anything or give any pointers as to the cause... Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Tue, May 04, 2010 at 05:04:51PM +0100, Simon Bedford wrote: We created a patched version in the end and rolled to one server back on the 8th April, no fault seen since and now rolled to the entire platform as of today, looking good. We have now had 2 more failures of the tcp listener on 2 seperate servers in the last week, a simple reload of the software fixes So this is with the patched servers? this. the thing I have noticed when this occurs is that when telnetting to the port you can then type in anything and try and escape, etc but the connection never closes, whereas on a working server the session will close after a short while of typing and pressing Enter. The logs do not show anything or give any pointers as to the cause... Please remind us of your operating system, RHEL4 I think? Or 5? Is there anything special about your workload? Residential internet connections I think? How many TCP/IP queries do you see per second, roughly? A quick run of rec_control should dig this up. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Please see answers below, thanks. bert hubert wrote: On Tue, May 04, 2010 at 05:04:51PM +0100, Simon Bedford wrote: We created a patched version in the end and rolled to one server back on the 8th April, no fault seen since and now rolled to the entire platform as of today, looking good. We have now had 2 more failures of the tcp listener on 2 seperate servers in the last week, a simple reload of the software fixes So this is with the patched servers? Yes this. the thing I have noticed when this occurs is that when telnetting to the port you can then type in anything and try and escape, etc but the connection never closes, whereas on a working server the session will close after a short while of typing and pressing Enter. The logs do not show anything or give any pointers as to the cause... Please remind us of your operating system, RHEL4 I think? Or 5? Is there anything special about your workload? Residential internet connections I think? Debian Etchnhalf and Lenny. Just run of the mill DSL connection DNS requests. How many TCP/IP queries do you see per second, roughly? A quick run of rec_control should dig this up. date;rec_control get tcp-questions Tue May 4 17:34:22 BST 2010 79371 date;rec_control get tcp-questions Tue May 4 17:35:53 BST 2010 79924 Gives us rough stats from a short period of 6 tcp requests per second per server, theres 8 servers. Bert Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Laurent Papier wrote: Le Wed, 31 Mar 2010 20:42:33 +0200 bert hubert bert.hub...@netherlabs.nl écrit: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I can help test this patch Bert. It can be found on: http://wiki.powerdns.com/trac/changeset/1546 or http://wiki.powerdns.com/trac/changeset/1546?format=diffnew=1546 The 'applying patches' section on this page explains how to apply: http://wiki.powerdns.com/trac/wiki/HACKING I hope this solves the issue - it sure sounds like it! 3 days without problem, your patch solves the issue for me. We created a patched version in the end and rolled to one server back on the 8th April, no fault seen since and now rolled to the entire platform as of today, looking good. Thanks Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Le Wed, 31 Mar 2010 20:42:33 +0200 bert hubert bert.hub...@netherlabs.nl écrit: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I can help test this patch Bert. It can be found on: http://wiki.powerdns.com/trac/changeset/1546 or http://wiki.powerdns.com/trac/changeset/1546?format=diffnew=1546 The 'applying patches' section on this page explains how to apply: http://wiki.powerdns.com/trac/wiki/HACKING I hope this solves the issue - it sure sounds like it! 3 days without problem, your patch solves the issue for me. -- Laurent Papier - 03 88 75 80 50 Admin. système - SdV Plurimedia - http://www.sdv.fr/ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
bert hubert wrote: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I can help test this patch Bert. It can be found on: http://wiki.powerdns.com/trac/changeset/1546 or http://wiki.powerdns.com/trac/changeset/1546?format=diffnew=1546 The 'applying patches' section on this page explains how to apply: http://wiki.powerdns.com/trac/wiki/HACKING I hope this solves the issue - it sure sounds like it! Bert I have patched and created a deb package but I have the same dependency issue that I had with the semi-static package for the initial 3.2 upgrade before Bert provided a static deb package. dpkg -i /var/tmp/pdns-recursor_3.2-2_i386.deb (Reading database ... 24171 files and directories currently installed.) Preparing to replace pdns-recursor 3.2-1 (using .../pdns-recursor_3.2-2_i386.deb) ... Unpacking replacement pdns-recursor ... dpkg: dependency problems prevent configuration of pdns-recursor: pdns-recursor depends on libc6 (= 2.7-1); however: Version of libc6 on system is 2.3.6.ds1-13. pdns-recursor depends on libstdc++6 (= 4.2.1); however: Version of libstdc++6 on system is 4.1.1-21. Do we have a static version available as yet? Thanks Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Le Thu, 8 Apr 2010 12:01:28 +0100 Simon Bedford sbedf...@plus.net écrit: bert hubert wrote: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I can help test this patch Bert. It can be found on: http://wiki.powerdns.com/trac/changeset/1546 or http://wiki.powerdns.com/trac/changeset/1546?format=diffnew=1546 The 'applying patches' section on this page explains how to apply: http://wiki.powerdns.com/trac/wiki/HACKING I hope this solves the issue - it sure sounds like it! Bert I have patched and created a deb package but I have the same dependency issue that I had with the semi-static package for the initial 3.2 upgrade before Bert provided a static deb package. dpkg -i /var/tmp/pdns-recursor_3.2-2_i386.deb (Reading database ... 24171 files and directories currently installed.) Preparing to replace pdns-recursor 3.2-1 (using .../pdns-recursor_3.2-2_i386.deb) ... Unpacking replacement pdns-recursor ... dpkg: dependency problems prevent configuration of pdns-recursor: pdns-recursor depends on libc6 (= 2.7-1); however: Version of libc6 on system is 2.3.6.ds1-13. pdns-recursor depends on libstdc++6 (= 4.2.1); however: Version of libstdc++6 on system is 4.1.1-21. I have patched and created a custom rpm package for our systems. I have pushed it in production this morning. Just for info, running unpatched 3.2 version with threads=1 really fixes the issue. No problem here for more than a week. -- Laurent Papier - 03 88 75 80 50 Admin. système - SdV Plurimedia - http://www.sdv.fr/ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I can help test this patch Bert. It can be found on: http://wiki.powerdns.com/trac/changeset/1546 or http://wiki.powerdns.com/trac/changeset/1546?format=diffnew=1546 The 'applying patches' section on this page explains how to apply: http://wiki.powerdns.com/trac/wiki/HACKING I hope this solves the issue - it sure sounds like it! Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Hi Bert, We have one of the instances on our non live server happening at the moment so I have not restarted it as yet, I have performed the following on it and recorded the responses :- UDP REQUEST ptn-cdns03:/usr/local/nagios/libexec# dig google.com @212.159.6.117 any ; DiG 9.5.1-P3 google.com @212.159.6.117 any ;; global options: printcmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 27597 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; QUESTION SECTION: ;google.com.IN ANY ;; ANSWER SECTION: google.com. 26792 IN NS ns2.google.com. google.com. 26792 IN NS ns4.google.com. google.com. 26792 IN NS ns1.google.com. google.com. 26792 IN NS ns3.google.com. ;; ADDITIONAL SECTION: ns4.google.com. 81336 IN A 216.239.38.10 ns2.google.com. 81336 IN A 216.239.34.10 ns3.google.com. 81336 IN A 216.239.36.10 ns1.google.com. 81336 IN A 216.239.32.10 ;; Query time: 3 msec ;; SERVER: 212.159.6.117#53(212.159.6.117) ;; WHEN: Tue Mar 30 14:49:06 2010 ;; MSG SIZE rcvd: 164 TCP REQUEST ptn-cdns03:/usr/local/nagios/libexec# dig +tcp google.com @212.159.6.117 any ; DiG 9.5.1-P3 +tcp google.com @212.159.6.117 any ;; global options: printcmd ;; connection timed out; no servers could be reached Netstat shows it to be still listening:- ptn-cdns03:/usr/local/nagios/libexec# netstat -anp | grep LISTEN | grep 212.159.6.117:53 tcp0 0 212.159.6.117:530.0.0.0:* LISTEN 14214/pdns_recursor Telnet shows the socket to be there :- ptn-cdns03:/usr/local/nagios/libexec# telnet 212.159.6.117 53 Trying 212.159.6.117... Connected to 212.159.6.117. Escape character is '^]'. Pressing enter has no effect on this also. This happens on port 53, 5300, 5301, etc as we use different ports for some of our services whereas udp works fine on all ports. Hope this info is useful Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? Bert On Tue, Mar 30, 2010 at 02:58:16PM +0100, Simon Bedford wrote: We have one of the instances on our non live server happening at the moment so I have not restarted it as yet, I have performed the following on it and recorded the responses :- UDP REQUEST ptn-cdns03:/usr/local/nagios/libexec# dig google.com @212.159.6.117 any ; DiG 9.5.1-P3 google.com @212.159.6.117 any ;; global options: printcmd ;; Got answer: ;; -HEADER- opcode: QUERY, status: NOERROR, id: 27597 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 ;; QUESTION SECTION: ;google.com. IN ANY ;; ANSWER SECTION: google.com. 26792 IN NS ns2.google.com. google.com. 26792 IN NS ns4.google.com. google.com. 26792 IN NS ns1.google.com. google.com. 26792 IN NS ns3.google.com. ;; ADDITIONAL SECTION: ns4.google.com. 81336 IN A 216.239.38.10 ns2.google.com. 81336 IN A 216.239.34.10 ns3.google.com. 81336 IN A 216.239.36.10 ns1.google.com. 81336 IN A 216.239.32.10 ;; Query time: 3 msec ;; SERVER: 212.159.6.117#53(212.159.6.117) ;; WHEN: Tue Mar 30 14:49:06 2010 ;; MSG SIZE rcvd: 164 TCP REQUEST ptn-cdns03:/usr/local/nagios/libexec# dig +tcp google.com @212.159.6.117 any ; DiG 9.5.1-P3 +tcp google.com @212.159.6.117 any ;; global options: printcmd ;; connection timed out; no servers could be reached Netstat shows it to be still listening:- ptn-cdns03:/usr/local/nagios/libexec# netstat -anp | grep LISTEN | grep 212.159.6.117:53 tcp0 0 212.159.6.117:530.0.0.0:* LISTEN 14214/pdns_recursor Telnet shows the socket to be there :- ptn-cdns03:/usr/local/nagios/libexec# telnet 212.159.6.117 53 Trying 212.159.6.117... Connected to 212.159.6.117. Escape character is '^]'. Pressing enter has no effect on this also. This happens on port 53, 5300, 5301, etc as we use different ports for some of our services whereas udp works fine on all ports. Hope this info is useful Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
bert hubert wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? Bert I can help test this patch Bert. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Hi, Simon is off at the moment, the details of our installation are: Debian Etch (2.6.24-etchnhalf.1-686-bigmem) and Lenny (2.6.26-2-686-bigmem). We have used the supplied binary version of 3.2 and are seing the TCP listener fail every few days (but not at regular intervals). If you need any further information please let me know. Michael Now I don't know anything about this issue specifically, but it's customary to provide some extra information when reporting bugs, what OS and OS version are you using for example ? Did you download a Linux-distribution binary ? What kernel version are you using ? Or did you build from an updated BSD-ports. Did you do your own build ? If so, what compiler did you use ? And so on. Some information would be better then no information. :-) Just so you know. Have a nice day, Leen. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users . -- | Michael Presson Broadband Solutions for | | Network and Systems LeadHome Business@ | | Plusnet PLC www.plus.net | | Registered in England no:3279013 | | Registered Office:Internet House, 2 Tenter Street, Sheffield, S1 4BY | +--- Plusnet - ISPA Best Consumer ISP 2008 + ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Le Thu, 25 Mar 2010 19:34:11 +0100 bert hubert bert.hub...@netherlabs.nl écrit: Ok, that makes two reports of the same thing. When this happens, do you experience timeouts on TCP queries? Or connection refused? Or just servfail answers? Can you check if this happens again before restarting? Hi, I have more info about this problem. Since Friday, I have switched to threads=1. The problem did not happen again. So the problem may be related to tcp+thread!=1. -- Laurent Papier - 03 88 75 80 50 Admin. système - SdV Plurimedia - http://www.sdv.fr/ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
[Pdns-users] Possible tcp listener issue
Guys, We have upgraded our customer caching name servers to pdns recursor 3.2 (which is working very well), this has now been running for 4 days but in the last 24 hours we have seen the tcp listener stop answering queries on 2 seperate servers. Our monitoring servers flag this up for us and restarting the recursor fixes it. Now I know tcp isn't used that much but I was wondering if anyone else has experienced this, due to the size of the logs we would generate we have the quiet option set to yes in the config so only get basic logging (which showed nothing out of the ordinary). In order to get the logging more detailed I tried the following :- mkdir /local/log edit /etc/syslog.conf and add the following line local0.* /local/log/recursor.log /etc/init.d/sysklogd restart Then alter the recursor.conf file to log to local0 and turn off quiet :- uncomment logging-facility and change it to 0 logging-facility=0 also change quiet=yes to no quiet=no /etc/init.d/pdns-recursor restart But for some reason this slowed the caching name server down noticeably and caused 0ms queries to become 200-500ms sometimes more and when looking at the pdns processes in top the cache memory usage was very small and not moving up as expected. This has only happened to one of 7 live servers at the moment and when I try and replicate on a test server the above does not happen so could be load through syslog level as the test server is very quiet. Thoughts appreciated.. Simon ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Le Thu, 25 Mar 2010 15:51:29 + Simon Bedford sbedf...@plus.net écrit: Guys, We have upgraded our customer caching name servers to pdns recursor 3.2 (which is working very well), this has now been running for 4 days but in the last 24 hours we have seen the tcp listener stop answering queries on 2 seperate servers. Our monitoring servers flag this up for us and restarting the recursor fixes it. Now I know tcp isn't used that much but I was wondering if anyone else has experienced this, due to the size of the logs we would generate we have the quiet option set to yes in the config so only get basic logging (which showed nothing out of the ordinary). Hi, I have also upgraded to pdns recursor 3.2 yesterday. And today, I have a strange problem on some of my systems. It seems be related to tcp DNS as the only thing that stopped working is using tcp dns queries. The rest of the system worked fine. I have restarted pdns recursor and it fixes the problem. I will do further testing if the problem happened again. -- Laurent Papier - 03 88 75 80 50 Admin. système - SdV Plurimedia - http://www.sdv.fr/ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On 03/25/2010 05:54 PM, Laurent Papier wrote: Le Thu, 25 Mar 2010 15:51:29 + Simon Bedfordsbedf...@plus.net écrit: Guys, We have upgraded our customer caching name servers to pdns recursor 3.2 (which is working very well), this has now been running for 4 days but in the last 24 hours we have seen the tcp listener stop answering queries on 2 seperate servers. Our monitoring servers flag this up for us and restarting the recursor fixes it. Now I know tcp isn't used that much but I was wondering if anyone else has experienced this, due to the size of the logs we would generate we have the quiet option set to yes in the config so only get basic logging (which showed nothing out of the ordinary). Hi, I have also upgraded to pdns recursor 3.2 yesterday. And today, I have a strange problem on some of my systems. It seems be related to tcp DNS as the only thing that stopped working is using tcp dns queries. The rest of the system worked fine. I have restarted pdns recursor and it fixes the problem. I will do further testing if the problem happened again. Hello Simon and Laurant, Now I don't know anything about this issue specifically, but it's customary to provide some extra information when reporting bugs, what OS and OS version are you using for example ? Did you download a Linux-distribution binary ? What kernel version are you using ? Or did you build from an updated BSD-ports. Did you do your own build ? If so, what compiler did you use ? And so on. Some information would be better then no information. :-) Just so you know. Have a nice day, Leen. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
On Thu, Mar 25, 2010 at 05:54:23PM +0100, Laurent Papier wrote: I have also upgraded to pdns recursor 3.2 yesterday. And today, I have a strange problem on some of my systems. It seems be related to tcp DNS as the only thing that stopped working is using tcp dns queries. The rest of the system worked fine. Ok, that makes two reports of the same thing. When this happens, do you experience timeouts on TCP queries? Or connection refused? Or just servfail answers? Can you check if this happens again before restarting? In addition, OS details would be appreciated (as Leen has pointed out too). Bert I have restarted pdns recursor and it fixes the problem. I will do further testing if the problem happened again. -- Laurent Papier - 03 88 75 80 50 Admin. système - SdV Plurimedia - http://www.sdv.fr/ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Possible tcp listener issue
Le Thu, 25 Mar 2010 19:34:11 +0100 bert hubert bert.hub...@netherlabs.nl écrit: On Thu, Mar 25, 2010 at 05:54:23PM +0100, Laurent Papier wrote: I have also upgraded to pdns recursor 3.2 yesterday. And today, I have a strange problem on some of my systems. It seems be related to tcp DNS as the only thing that stopped working is using tcp dns queries. The rest of the system worked fine. Ok, that makes two reports of the same thing. When this happens, do you experience timeouts on TCP queries? Or connection refused? Or just servfail answers? When this happened, I have quickly checked if dns recursor still answer dns queries with dig. And it still replied even to the big MX entry I suspect. Maybe I was too fast or did not read the dig output well enough. I need to double check this next time it happens. But Simon report makes me think that something could be wrong with tcp in version 3.2. I'm sorry that I don't have more precise report for the moment. Can you check if this happens again before restarting? Sure I will. In addition, OS details would be appreciated (as Leen has pointed out too). Some technical info : - home made and compiled RPM for Redhat system (gcc 3.3, boost 1.34.1, kernel 2.6.24.5). - threads=2 setting, listen to 127.0.0.1 only. I have switched to threads=1, in the hope that the problem maybe related to thread (version 3.1.7.2 works very well on the same system). I'm also thinking of a solution to change the MX entry fit in a UDP dns packet as it seems that the packet cache is only used for UDP. -- Laurent Papier - 03 88 75 80 50 Admin. système - SdV Plurimedia - http://www.sdv.fr/ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
