Re: [Pdns-users] Possible tcp listener issue

2010-07-27 Thread Simon Bedford
Hi Bert, bert hubert wrote: On Mon, Jul 26, 2010 at 02:24:03PM +0100, Simon Bedford wrote: I then just upgraded the version to our patched version of 3.2 but without specifying our usual 4 threads and the response times went up on the graphing straight away, upping the number of threads makes

Re: [Pdns-users] Possible tcp listener issue

2010-07-26 Thread Simon Bedford
Simon Bedford wrote: Hi Bert, Is there any update on this bug, interested to know if you have found the root cause of it... Also I have just upgraded one of our internal servers from Debian Etch and pdns-recursor-3.1.7-2 to Debian Lenny and the same patched version of pdns-recursor-3.2

Re: [Pdns-users] Possible tcp listener issue

2010-07-06 Thread Simon Bedford
Excellent news Bert, thanks for the update, good luck with the root cause squishing. Simon bert hubert wrote: The issue has now been reproduced on one of the PowerDNS servers, thanks to Brad and Christian. However, it is quite some work to make it happen and we've not yet been able to make

Re: [Pdns-users] Possible tcp listener issue

2010-06-30 Thread Christian Hofstädtler
Brad, Josh, I'm trying to reproduce the behaviour you see here, but fail to do so at this time. Maybe you can give me more details about your environment, at least OS version, architecture, and any compile options you've used to build the binary (say, STATIC or LUA). Also of interest would be

Re: [Pdns-users] Possible tcp listener issue

2010-06-30 Thread Simon Bedford
From: pdns-users-boun...@mailman.powerdns.com [pdns-users-boun...@mailman.powerdns.com] On Behalf Of Christian Hofstädtler [...@zeha.at] Sent: 30 June 2010 08:05 To: pdns-users@mailman.powerdns.com Cc: brad.dame...@clearwire.com Subject: Re: [Pdns-users] Possible tcp

Re: [Pdns-users] Possible tcp listener issue

2010-06-28 Thread Ton van Rosmalen
Hi Bert et al, bert hubert schreef: Simon, Josh, We're running out of things we can do without seeing packets. Could you try http://wiki.powerdns.com/trac/changeset/1652 ? This is the last thing we can come up with that might explain the issue you are seeing. This fix is necessary, but

Re: [Pdns-users] Possible tcp listener issue

2010-06-28 Thread Ton van Rosmalen
bert hubert schreef: On Mon, Jun 28, 2010 at 01:13:16PM +0200, Ton van Rosmalen wrote: This is the last thing we can come up with that might explain the issue you are seeing. This fix is necessary, but unlikely to be the cause of the problems you are seeing. I sent a message about

Re: [Pdns-users] Possible tcp listener issue

2010-06-28 Thread Brad Dameron
-Original Message- So far, Josh Simon are the only ones reporting this issue though, so something must be going on. What Laurent Papier reported has been fixed already (although he may not know it yet ;-)) Bert Bert, Us too!! We're able to reproduce what may be this same

Re: [Pdns-users] Possible tcp listener issue

2010-06-23 Thread Simon Bedford
Subject: [Pdns-users] Possible tcp listener issue To: bert hubert bert.hub...@netherlabs.nl Cc: pdns-users@mailman.powerdns.com pdns-users@mailman.powerdns.com bert hubert wrote: Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you

Re: [Pdns-users] Possible tcp listener issue

2010-06-23 Thread bert hubert
From: bert.hub...@netherlabs.nl [bert.hub...@netherlabs.nl] Sent: 22 June 2010 15:50 To: Simon Bedford Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Possible tcp listener issue So much for my beliefs then :) Will take another good look at how else this could happen. The patch

Re: [Pdns-users] Possible tcp listener issue

2010-06-23 Thread Josh Berry
Subject: Re: [Pdns-users] Possible tcp listener issue Simon, Can you grep your logfiles for 'epoll' or 'unlisted' or 'exception'? Bert On Wed, Jun 23, 2010 at 11:00:12AM +0100, Simon Bedford wrote: Thanks Bert, Appreciate you looking into this, would be nice to nail it once and for all

Re: [Pdns-users] Possible tcp listener issue

2010-06-23 Thread bert hubert
From: pdns-users-boun...@mailman.powerdns.com [pdns-users-boun...@mailman.powerdns.com] On Behalf Of bert hubert [bert.hub...@netherlabs.nl] Sent: 23 June 2010 12:47 To: Simon Bedford Cc: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Possible tcp listener issue Simon

Re: [Pdns-users] Possible tcp listener issue

2010-06-22 Thread Simon Bedford
bert hubert wrote: Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you do not see the log messages, I'm pretty sure this is it. Bert Morning, I created and installed the package yesterday (with the following option

Re: [Pdns-users] Possible tcp listener issue

2010-06-22 Thread bert.hub...@netherlabs.nl
So much for my beliefs then :) Will take another good look at how else this could happen. The patch we made at least solves A bug just not your bug. Sent from my phone. - Reply message - From: Simon Bedford sbedf...@plus.net Date: Tue, Jun 22, 2010 09:44 Subject: [Pdns-users] Possible

Re: [Pdns-users] Possible tcp listener issue

2010-06-21 Thread Simon Bedford
Morning Bert, We had a crash on one of the upgraded servers over the weekend and as suspected it happened around the default max tcp-clients, please see the entry from my log below that was running to query the amount of clients :- 128 128 128 128 128 128 128 128 128 128 128 128 128 128 128

Re: [Pdns-users] Possible tcp listener issue

2010-06-21 Thread bert hubert
Simon, the solution to your issue is almost certainly in http://wiki.powerdns.com/trac/changeset/1640 Even though you do not see the log messages, I'm pretty sure this is it. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com

Re: [Pdns-users] Possible tcp listener issue

2010-06-18 Thread bert hubert
To everybody seeing this issue (all two of you :)), can you grep the logfile on: startDoResolve problem: DNS parser error: STL error: Any other exception in a resolver context Without the quotes. If you see any of these lines, it might explain the problem. Thanks! On Thu, Jun 17, 2010 at

Re: [Pdns-users] Possible tcp listener issue

2010-06-17 Thread Simon Bedford
Bert, its now climbing and not seeming to close any clients on both servers now, although one is much worse than the other, one is at 11 and the other is at 108 tcp-clients at the moment. Simon Simon Bedford wrote: Please see answers below :- bert hubert wrote: On Wed, Jun 16, 2010 at

Re: [Pdns-users] Possible tcp listener issue

2010-06-16 Thread Simon Bedford
bert hubert wrote: On Tue, Jun 15, 2010 at 02:23:04PM +0100, Simon Bedford wrote: It contains some other exciting stuff too, and it appears to be stable for production use. I have now built a static package and installed to 2 of the 8 servers after testing in isolation, they appear to be

Re: [Pdns-users] Possible tcp listener issue

2010-06-16 Thread bert hubert
On Wed, Jun 16, 2010 at 11:10:29AM +0100, Simon Bedford wrote: I have been running the 'get tcp-clients' every 5 mins in cron on both servers, one is very low 10 and fluctuates, the other is just growing and growing and is currently at 55, hope this info is of value for debug. Is there any

Re: [Pdns-users] Possible tcp listener issue

2010-06-16 Thread Simon Bedford
Please see answers below :- bert hubert wrote: On Wed, Jun 16, 2010 at 11:10:29AM +0100, Simon Bedford wrote: I have been running the 'get tcp-clients' every 5 mins in cron on both servers, one is very low 10 and fluctuates, the other is just growing and growing and is currently at 55, hope

Re: [Pdns-users] Possible tcp listener issue

2010-06-15 Thread Simon Bedford
Simon, There is now a 3.3 prerelease that contains all the stuff you'd need for further trouble shooting. Please find it on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 It contains some other exciting stuff too, and it appears to be stable for production use. Bert

Re: [Pdns-users] Possible tcp listener issue

2010-06-15 Thread bert hubert
On Tue, Jun 15, 2010 at 02:23:04PM +0100, Simon Bedford wrote: It contains some other exciting stuff too, and it appears to be stable for production use. I have now built a static package and installed to 2 of the 8 servers after testing in isolation, they appear to be running fine and I can

Re: [Pdns-users] Possible tcp listener issue

2010-06-13 Thread bert hubert
On Mon, Jun 07, 2010 at 04:08:17PM +0100, Simon Bedford wrote: Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically run 'rec_control get tcp-clients' ? Hi Bert, I have just returned from annual leave and began to look into this

Re: [Pdns-users] Possible tcp listener issue

2010-06-09 Thread Simon Bedford
On 30/05/2010 19:31, bert hubert wrote: On Thu, May 20, 2010 at 11:12:29AM +0100, Simon Bedford wrote: This has happened a further twice in the last week, output sent off list, please let me know if you need any further information. Simon, Could you apply this patch:

Re: [Pdns-users] Possible tcp listener issue

2010-06-08 Thread Simon Bedford
On 30/05/2010 19:31, bert hubert wrote: On Thu, May 20, 2010 at 11:12:29AM +0100, Simon Bedford wrote: This has happened a further twice in the last week, output sent off list, please let me know if you need any further information. Simon, Could you apply this patch:

Re: [Pdns-users] Possible tcp listener issue

2010-06-07 Thread Simon Bedford
Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically run 'rec_control get tcp-clients' ? Hi Bert, I have just returned from annual leave and began to look into this earlier but when I download the latest source from your website and put the

Re: [Pdns-users] Possible tcp listener issue

2010-06-07 Thread Simon Bedford
Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically run 'rec_control get tcp-clients' ? Hi Bert, I have just returned from annual leave and began to look into this earlier but when I download the latest source from your website and put the

Re: [Pdns-users] Possible tcp listener issue

2010-05-30 Thread bert hubert
On Thu, May 20, 2010 at 11:12:29AM +0100, Simon Bedford wrote: This has happened a further twice in the last week, output sent off list, please let me know if you need any further information. Simon, Could you apply this patch: http://wiki.powerdns.com/trac/changeset/1623 And periodically

Re: [Pdns-users] Possible tcp listener issue

2010-05-20 Thread Simon Bedford
Simon Bedford wrote: bert hubert wrote: On Mon, May 10, 2010 at 05:20:18PM +0100, Simon Bedford wrote: It has happened again over the weekend and a simple restart fixed the issue. Simon, We have a new theory why this might be happening, to verify, if the issue ever occcurs again, can you

Re: [Pdns-users] Possible tcp listener issue

2010-05-17 Thread InterNetX - Marco Schrieck
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello, I didn't read the whole thread. But we had the same problem in past. Is it maybee a timeout mysql connection. We had this only system with out load. Ciao Marco Am 06.05.10 07:45, schrieb bert hubert: On Wed, May 05, 2010 at 11:27:13AM

Re: [Pdns-users] Possible tcp listener issue

2010-05-14 Thread Simon Bedford
bert hubert wrote: On Mon, May 10, 2010 at 05:20:18PM +0100, Simon Bedford wrote: It has happened again over the weekend and a simple restart fixed the issue. Simon, We have a new theory why this might be happening, to verify, if the issue ever occcurs again, can you run: $ netstat -an |

Re: [Pdns-users] Possible tcp listener issue

2010-05-06 Thread Simon Bedford
bert hubert wrote: On Tue, May 04, 2010 at 05:37:10PM +0100, Simon Bedford wrote: How many TCP/IP queries do you see per second, roughly? A quick run of rec_control should dig this up. date;rec_control get tcp-questions Tue May 4 17:34:22 BST 2010 79371 date;rec_control get tcp-questions Tue

Re: [Pdns-users] Possible tcp listener issue

2010-05-05 Thread Ton van Rosmalen
Hi, bert hubert schreef: On Tue, May 04, 2010 at 05:04:51PM +0100, Simon Bedford wrote: [snip] Not trying to hijack the thread but I'm using the authorative server and saw a similar issue last Monday. Our primary auth pdns didn't respond to tcp queries anymore where udp queries still

Re: [Pdns-users] Possible tcp listener issue

2010-05-05 Thread bert hubert
On Wed, May 05, 2010 at 11:27:13AM +0200, Ton van Rosmalen wrote: Our primary auth pdns didn't respond to tcp queries anymore where udp queries still worked. A restart solved the problem. I didn't think anything of it until I read these new messages on this thread. Could it be a shared

Re: [Pdns-users] Possible tcp listener issue

2010-05-05 Thread bert hubert
On Tue, May 04, 2010 at 05:37:10PM +0100, Simon Bedford wrote: How many TCP/IP queries do you see per second, roughly? A quick run of rec_control should dig this up. date;rec_control get tcp-questions Tue May 4 17:34:22 BST 2010 79371 date;rec_control get tcp-questions Tue May 4

Re: [Pdns-users] Possible tcp listener issue

2010-05-04 Thread Simon Bedford
Simon Bedford wrote: Laurent Papier wrote: Le Wed, 31 Mar 2010 20:42:33 +0200 bert hubert bert.hub...@netherlabs.nl écrit: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in

Re: [Pdns-users] Possible tcp listener issue

2010-05-04 Thread bert hubert
On Tue, May 04, 2010 at 05:04:51PM +0100, Simon Bedford wrote: We created a patched version in the end and rolled to one server back on the 8th April, no fault seen since and now rolled to the entire platform as of today, looking good. We have now had 2 more failures of the tcp listener on

Re: [Pdns-users] Possible tcp listener issue

2010-05-04 Thread Simon Bedford
Please see answers below, thanks. bert hubert wrote: On Tue, May 04, 2010 at 05:04:51PM +0100, Simon Bedford wrote: We created a patched version in the end and rolled to one server back on the 8th April, no fault seen since and now rolled to the entire platform as of today, looking good. We

Re: [Pdns-users] Possible tcp listener issue

2010-04-21 Thread Simon Bedford
Laurent Papier wrote: Le Wed, 31 Mar 2010 20:42:33 +0200 bert hubert bert.hub...@netherlabs.nl écrit: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch?

Re: [Pdns-users] Possible tcp listener issue

2010-04-12 Thread Laurent Papier
Le Wed, 31 Mar 2010 20:42:33 +0200 bert hubert bert.hub...@netherlabs.nl écrit: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I can help test

Re: [Pdns-users] Possible tcp listener issue

2010-04-08 Thread Simon Bedford
bert hubert wrote: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I can help test this patch Bert. It can be found on:

Re: [Pdns-users] Possible tcp listener issue

2010-04-08 Thread Laurent Papier
Le Thu, 8 Apr 2010 12:01:28 +0100 Simon Bedford sbedf...@plus.net écrit: bert hubert wrote: On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I

Re: [Pdns-users] Possible tcp listener issue

2010-03-31 Thread bert hubert
On Tue, Mar 30, 2010 at 03:09:08PM +0100, Simon Bedford wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? I can help test this patch Bert. It can be found on: http://wiki.powerdns.com/trac/changeset/1546 or

Re: [Pdns-users] Possible tcp listener issue

2010-03-30 Thread Simon Bedford
Hi Bert, We have one of the instances on our non live server happening at the moment so I have not restarted it as yet, I have performed the following on it and recorded the responses :- UDP REQUEST ptn-cdns03:/usr/local/nagios/libexec# dig google.com @212.159.6.117 any ; DiG 9.5.1-P3

Re: [Pdns-users] Possible tcp listener issue

2010-03-30 Thread bert hubert
A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? Bert On Tue, Mar 30, 2010 at 02:58:16PM +0100, Simon Bedford wrote: We have one of the instances on our non live server happening at the moment so I have

Re: [Pdns-users] Possible tcp listener issue

2010-03-30 Thread Simon Bedford
bert hubert wrote: A quick update - I've traced down an issue that may have been causing this. Would you be interested in testing this small patch? Bert I can help test this patch Bert. Simon ___ Pdns-users mailing list

Re: [Pdns-users] Possible tcp listener issue

2010-03-29 Thread Michael Presson
Hi, Simon is off at the moment, the details of our installation are: Debian Etch (2.6.24-etchnhalf.1-686-bigmem) and Lenny (2.6.26-2-686-bigmem). We have used the supplied binary version of 3.2 and are seing the TCP listener fail every few days (but not at regular intervals). If you need any

Re: [Pdns-users] Possible tcp listener issue

2010-03-29 Thread Laurent Papier
Le Thu, 25 Mar 2010 19:34:11 +0100 bert hubert bert.hub...@netherlabs.nl écrit: Ok, that makes two reports of the same thing. When this happens, do you experience timeouts on TCP queries? Or connection refused? Or just servfail answers? Can you check if this happens again before

[Pdns-users] Possible tcp listener issue

2010-03-25 Thread Simon Bedford
Guys, We have upgraded our customer caching name servers to pdns recursor 3.2 (which is working very well), this has now been running for 4 days but in the last 24 hours we have seen the tcp listener stop answering queries on 2 seperate servers. Our monitoring servers flag this up for us

Re: [Pdns-users] Possible tcp listener issue

2010-03-25 Thread Laurent Papier
Le Thu, 25 Mar 2010 15:51:29 + Simon Bedford sbedf...@plus.net écrit: Guys, We have upgraded our customer caching name servers to pdns recursor 3.2 (which is working very well), this has now been running for 4 days but in the last 24 hours we have seen the tcp listener stop answering

Re: [Pdns-users] Possible tcp listener issue

2010-03-25 Thread Leen Besselink
On 03/25/2010 05:54 PM, Laurent Papier wrote: Le Thu, 25 Mar 2010 15:51:29 + Simon Bedfordsbedf...@plus.net écrit: Guys, We have upgraded our customer caching name servers to pdns recursor 3.2 (which is working very well), this has now been running for 4 days but in the last 24 hours

Re: [Pdns-users] Possible tcp listener issue

2010-03-25 Thread bert hubert
On Thu, Mar 25, 2010 at 05:54:23PM +0100, Laurent Papier wrote: I have also upgraded to pdns recursor 3.2 yesterday. And today, I have a strange problem on some of my systems. It seems be related to tcp DNS as the only thing that stopped working is using tcp dns queries. The rest of the

Re: [Pdns-users] Possible tcp listener issue

2010-03-25 Thread Laurent Papier
Le Thu, 25 Mar 2010 19:34:11 +0100 bert hubert bert.hub...@netherlabs.nl écrit: On Thu, Mar 25, 2010 at 05:54:23PM +0100, Laurent Papier wrote: I have also upgraded to pdns recursor 3.2 yesterday. And today, I have a strange problem on some of my systems. It seems be related to tcp DNS