Re: [Pdns-users] tcp listener issue - hopefully fixed
Wonderful news! This truly closes this truly longstanding bug ;-) Bert On Tue, Nov 30, 2010 at 09:43:24AM +, Simon Bedford wrote: Bert, I have now rolled out the new 3.3 version to our entire estate of servers and have seen no further bugs, the process has been stable for a number of weeks now and all looks good, many thanks for all of your help. Simon From: pdns-users-boun...@mailman.powerdns.com [pdns-users-boun...@mailman.powerdns.com] On Behalf Of Brad Dameron [brad.dame...@clearwire.com] Sent: 09 September 2010 20:04 To: bert hubert Cc: Mike; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed Bert, I re-downloaded the src and rebuilt my RPM. It appears to be up and running now. Unknown why the first compiles had this issue. I have been testing the last 2 hours and so far so good. I'll let you know tomorrow the final outcome. Thanks, Brad -Original Message- From: bert hubert [mailto:bert.hub...@netherlabs.nl] Sent: Thursday, September 09, 2010 5:06 AM To: Brad Dameron Cc: Mike; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed On Wed, Sep 08, 2010 at 11:16:41PM +, Brad Dameron wrote: Bert, I can't seem to get this latest version to run right. I build my RPM. I launch it with the following: This is not good! Does it crash immediately, even without accepting any traffic? So if you bind it to port 54 (for example), does it crash too? Bert /usr/sbin/pdns_recursor --local-address=172.26.68.42,127.0.0.1 / --allow-from= --max-cache-entries=300 --log-common-errors=no / --threads=4 --socket-dir=/var/run/recursor1 --daemon --dont-query= pdns_recursor[32283]: Operating in 64 bits mode pdns_recursor[32283]: Reading random entropy from '/dev/urandom' pdns_recursor[32283]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32283]: Inserting rfc 1918 private space zones pdns_recursor[32283]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32283]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32283]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32283]: Calling daemonize, going to background pdns_recursor[32284]: Launching 4 threads pdns_recursor[32284]: Done priming cache with root hints kernel: pdns_recursor[32288] general protection rip:4ea75c rsp:42802840 error:0 pdns_recursor[32284]: Done priming cache with root hints pdns_recursor[32298]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32298]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32298]: Operating in 64 bits mode pdns_recursor[32298]: Reading random entropy from '/dev/urandom' pdns_recursor[32298]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32298]: Inserting rfc 1918 private space zones pdns_recursor[32298]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32298]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32298]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32298]: Calling daemonize, going to background pdns_recursor[32299]: Launching 4 threads pdns_recursor[32299]: Done priming cache with root hints kernel: pdns_recursor[32300] general protection rip:4ea75c rsp:409ff840 error:0 pdns_recursor[32307]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32307]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32307]: Operating in 64 bits mode pdns_recursor[32307]: Reading random entropy from '/dev/urandom' pdns_recursor[32307]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32307]: Inserting rfc 1918 private space zones pdns_recursor[32307]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32307]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32307]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32307]: Calling daemonize, going to background pdns_recursor[32308]: Launching 4 threads pdns_recursor[32308
Re: [Pdns-users] tcp listener issue - hopefully fixed
On Wed, Sep 08, 2010 at 11:16:41PM +, Brad Dameron wrote: Bert, I can’t seem to get this latest version to run right. I build my RPM. I launch it with the following: This is not good! Does it crash immediately, even without accepting any traffic? So if you bind it to port 54 (for example), does it crash too? Bert /usr/sbin/pdns_recursor --local-address=172.26.68.42,127.0.0.1 / --allow-from= --max-cache-entries=300 --log-common-errors=no / --threads=4 --socket-dir=/var/run/recursor1 --daemon --dont-query= pdns_recursor[32283]: Operating in 64 bits mode pdns_recursor[32283]: Reading random entropy from '/dev/urandom' pdns_recursor[32283]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32283]: Inserting rfc 1918 private space zones pdns_recursor[32283]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32283]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32283]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32283]: Calling daemonize, going to background pdns_recursor[32284]: Launching 4 threads pdns_recursor[32284]: Done priming cache with root hints kernel: pdns_recursor[32288] general protection rip:4ea75c rsp:42802840 error:0 pdns_recursor[32284]: Done priming cache with root hints pdns_recursor[32298]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32298]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32298]: Operating in 64 bits mode pdns_recursor[32298]: Reading random entropy from '/dev/urandom' pdns_recursor[32298]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32298]: Inserting rfc 1918 private space zones pdns_recursor[32298]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32298]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32298]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32298]: Calling daemonize, going to background pdns_recursor[32299]: Launching 4 threads pdns_recursor[32299]: Done priming cache with root hints kernel: pdns_recursor[32300] general protection rip:4ea75c rsp:409ff840 error:0 pdns_recursor[32307]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32307]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32307]: Operating in 64 bits mode pdns_recursor[32307]: Reading random entropy from '/dev/urandom' pdns_recursor[32307]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32307]: Inserting rfc 1918 private space zones pdns_recursor[32307]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32307]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32307]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32307]: Calling daemonize, going to background pdns_recursor[32308]: Launching 4 threads pdns_recursor[32308]: Done priming cache with root hints pdns_recursor[32308]: Enabled 'epoll' multiplexer kernel: pdns_recursor[32309]: segfault at 00723a41 rip 00723a41 rsp 409ffb18 error 15 Thanks, Brad From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users-boun...@mailman.powerdns.com] On Behalf Of bert.hub...@netherlabs.nl Sent: Tuesday, September 07, 2010 5:45 AM To: Mike Cc: Brad Dameron; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed Simon, Brief reply, am on the road. 3.3 will be released the moment you, brad or laurent confirm the issue is truly gone. What I can do is make packages that will be binary identical to the real 3.3 once you 'bless' them as having solved your issue. This would save you an upgrade. Would this work for you? Bert. Sent from my phone. - Reply message - From: Simon Bedford sbedf...@plus.net Date: Mon, Sep 6, 2010 14:16 Subject: tcp listener issue - hopefully fixed To: bert hubert bert.hub...@netherlabs.nl Cc: Brad Dameron brad.dame...@clearwire.com, pdns-users@mailman.powerdns.com pdns-users@mailman.powerdns.com Hi Bert, Apologies for the delay in replying I have been assigned to some different work for the moment, I
Re: [Pdns-users] tcp listener issue - hopefully fixed
Bert, I re-downloaded the src and rebuilt my RPM. It appears to be up and running now. Unknown why the first compiles had this issue. I have been testing the last 2 hours and so far so good. I'll let you know tomorrow the final outcome. Thanks, Brad -Original Message- From: bert hubert [mailto:bert.hub...@netherlabs.nl] Sent: Thursday, September 09, 2010 5:06 AM To: Brad Dameron Cc: Mike; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed On Wed, Sep 08, 2010 at 11:16:41PM +, Brad Dameron wrote: Bert, I can't seem to get this latest version to run right. I build my RPM. I launch it with the following: This is not good! Does it crash immediately, even without accepting any traffic? So if you bind it to port 54 (for example), does it crash too? Bert /usr/sbin/pdns_recursor --local-address=172.26.68.42,127.0.0.1 / --allow-from= --max-cache-entries=300 --log-common-errors=no / --threads=4 --socket-dir=/var/run/recursor1 --daemon --dont-query= pdns_recursor[32283]: Operating in 64 bits mode pdns_recursor[32283]: Reading random entropy from '/dev/urandom' pdns_recursor[32283]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32283]: Inserting rfc 1918 private space zones pdns_recursor[32283]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32283]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32283]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32283]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32283]: Calling daemonize, going to background pdns_recursor[32284]: Launching 4 threads pdns_recursor[32284]: Done priming cache with root hints kernel: pdns_recursor[32288] general protection rip:4ea75c rsp:42802840 error:0 pdns_recursor[32284]: Done priming cache with root hints pdns_recursor[32298]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32298]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32298]: Operating in 64 bits mode pdns_recursor[32298]: Reading random entropy from '/dev/urandom' pdns_recursor[32298]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32298]: Inserting rfc 1918 private space zones pdns_recursor[32298]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32298]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32298]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32298]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32298]: Calling daemonize, going to background pdns_recursor[32299]: Launching 4 threads pdns_recursor[32299]: Done priming cache with root hints kernel: pdns_recursor[32300] general protection rip:4ea75c rsp:409ff840 error:0 pdns_recursor[32307]: PowerDNS recursor 3.3-pre (C) 2001-2010 PowerDNS.COM BV (Sep 8 2010, 22:53:00, gcc 4.1.2 20080704 (Red Hat pdns_recursor[32307]: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it ac the GPL version 2. pdns_recursor[32307]: Operating in 64 bits mode pdns_recursor[32307]: Reading random entropy from '/dev/urandom' pdns_recursor[32307]: WARNING: Allowing queries from all IP addresses - this can be a security risk! pdns_recursor[32307]: Inserting rfc 1918 private space zones pdns_recursor[32307]: Listening for UDP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for UDP queries on 127.0.0.1:53 pdns_recursor[32307]: Enabled TCP data-ready filter for (slight) DoS protection pdns_recursor[32307]: Listening for TCP queries on 172.26.68.42:53 pdns_recursor[32307]: Listening for TCP queries on 127.0.0.1:53 pdns_recursor[32307]: Calling daemonize, going to background pdns_recursor[32308]: Launching 4 threads pdns_recursor[32308]: Done priming cache with root hints pdns_recursor[32308]: Enabled 'epoll' multiplexer kernel: pdns_recursor[32309]: segfault at 00723a41 rip 00723a41 rsp 409ffb18 error 15 Thanks, Brad From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users-boun...@mailman.powerdns.com] On Behalf Of bert.hub...@netherlabs.nl Sent: Tuesday, September 07, 2010 5:45 AM To: Mike Cc: Brad Dameron; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed Simon, Brief reply, am on the road. 3.3 will be released the moment you, brad or laurent confirm the issue is truly gone. What I can do is make packages that will be binary identical to the real 3.3 once you 'bless' them as having solved your issue
Re: [Pdns-users] tcp listener issue - hopefully fixed
On Sun, Aug 29, 2010 at 09:17:01PM +, Brad Dameron wrote: The release process for 3.3 can now start - only 1 feature request left to finish. Good to hear Bert. I'll run it through the ringer on Monday and see if we can reproduce the problem. Cross fingers that it is fixed. Brad, Any news? If I get confirmation from you, Simon or Laurent that the problem is gone, we can go for release. Thanks ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] tcp listener issue - hopefully fixed
On Mon, Aug 23, 2010 at 11:37:42AM +0100, Simon Bedford wrote: Just returned from annual leave to this welcome news, has anyone had a chance to try it as yet and investigate whether the bug still manifests? And does it no longer happen when running Brad's test script? Simon, Sadly we found that the previous code drop, while stable, still exhibited the same problem. We've fixed yet another bug that might be causing the issue, and we are currently testing that. This drop can be found on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 If you compile the program below, and run it on your pdns server, you'll probably see it print '0' and then a very large number. If you could share that output with us, it could tell us if we are on the right track. http://pastie.org/1124088 http://pastie.org/1124088.txt To compile, save as .cpp file and compile with 'g++ -O2 testing.cpp -o testing', and run with './testing'. Good luck! Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] tcp listener issue - hopefully fixed
On Sun, Aug 29, 2010 at 04:56:14PM +0200, bert hubert wrote: We've fixed yet another bug that might be causing the issue, and we are currently testing that. This drop can be found on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 Our testing shows that the problem disappeared! Many thanks to Christian Hofstaedtler. If this was the final problem, this also explains why only some people are seeing it. There are hardware platforms that are highly immune to this bug (Michel Stol measured this), but we've seen an Openvirtuozo hosted Linux being very vulnerable to it. On my own testing hardware, the problem was relatively rare. For the people who care, the root cause appeared to be doing non-atomic updates to the TCP client count. The release process for 3.3 can now start - only 1 feature request left to finish. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] tcp listener issue - hopefully fixed
-Original Message- From: pdns-users-boun...@mailman.powerdns.com [mailto:pdns-users- boun...@mailman.powerdns.com] On Behalf Of bert hubert Sent: Sunday, August 29, 2010 11:31 AM To: Mike Cc: Brad Dameron; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] tcp listener issue - hopefully fixed On Sun, Aug 29, 2010 at 04:56:14PM +0200, bert hubert wrote: We've fixed yet another bug that might be causing the issue, and we are currently testing that. This drop can be found on http://svn.powerdns.com/snapshots/pdns-recursor-3.3-pre.tar.bz2 Our testing shows that the problem disappeared! Many thanks to Christian Hofstaedtler. If this was the final problem, this also explains why only some people are seeing it. There are hardware platforms that are highly immune to this bug (Michel Stol measured this), but we've seen an Openvirtuozo hosted Linux being very vulnerable to it. On my own testing hardware, the problem was relatively rare. For the people who care, the root cause appeared to be doing non-atomic updates to the TCP client count. The release process for 3.3 can now start - only 1 feature request left to finish. Bert Good to hear Bert. I'll run it through the ringer on Monday and see if we can reproduce the problem. Cross fingers that it is fixed. Thanks, Brad This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
