backtrace attached. [root@c605 pdns-recursor]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 3873 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 8192 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 3873 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
On 05/21/2014 12:00 AM, Aki Tuomi wrote: > Can you install the debuginfo package and run it with gdb to get stack trace? > Also, can > you give us ulimit -a? > > Aki > > On Tue, May 20, 2014 at 11:41:21PM +0300, Imre Gergely wrote: >> [root@c605 ~]# cat /etc/pdns-recursor/recursor.conf |grep -v "^#" | grep >> -v "^$" >> setuid=pdns-recursor >> setgid=pdns-recursor >> daemon=no >> local-address=127.0.0.1 >> threads=1 >> >> [root@c605 ~]# strace /usr/sbin/pdns_recursor > >> /tmp/strace-pdns-recursor.txt 2>&1 >> Segmentation fault >> [root@c605 ~]# >> >> [root@c605 ~]# ip a |grep inet |wc >> 4576 27451 233679 >> >> Attached. If this is not what you had in mind, please let me know. >> >> On 05/20/2014 11:31 PM, bert hubert wrote: >>> Imre, >>> >>> Can you strace the startup with threads=1? >>> >>> Bert >>> >>> On May 20, 2014 10:25 PM, Imre Gergely <[email protected]> wrote: >>>> Hi >>>> >>>> I did manage to reproduce this in a VM. Installed a CentOS 6.5, and >>>> recursor 3.5.3 from EPEL. Then I did this: >>>> >>>> for i in `seq 1 16`; do for j in `seq 1 254`; do ip a a 10.0.$i.$j/16 dev >>>> eth0; done; done >>>> >>>> Then I started the recursor, everything went just fine, did a bunch of >>>> digs, no problems. >>>> >>>> Then I added some more IPs: >>>> >>>> for i in `seq 17 32`; do for j in `seq 1 254`; do ip a a 10.0.$i.$j/16 dev >>>> eth0; done; done >>>> >>>> And then init.d/pdns-recursor restart: >>>> >>>> May 20 23:18:24 c605 pdns_recursor[21341]: PowerDNS recursor 3.5.3 (C) >>>> 2001-2013 PowerDNS.COM BV (Feb 10 2014, 17:26:52, gcc 4.4.7 20120313 (Red >>>> Hat 4.4.7-4)) >>>> starting up >>>> May 20 23:18:24 c605 pdns_recursor[21341]: PowerDNS comes with ABSOLUTELY >>>> NO WARRANTY. This is free software, and you are welcome to redistribute it >>>> according to the terms of the GPL version 2. >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Operating in 32 bits mode >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Reading random entropy from >>>> '/dev/urandom' >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Only allowing queries from: >>>> 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, >>>> 172.16.0.0/12, ::1/128, fe80::/10 >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Will not send queries to: >>>> 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, >>>> 172.16.0.0/12, ::1/128, fe80::/10, 0.0.0.0, :: >>>> May 20 23:18:24 c605 pdns_recursor[21341]: NOT using IPv6 for outgoing >>>> queries - set 'query-local-address6=::' to enable >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Inserting rfc 1918 private >>>> space zones >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Listening for UDP queries on >>>> 127.0.0.1:53 >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Enabled TCP data-ready filter >>>> for (slight) DoS protection >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Listening for TCP queries on >>>> 127.0.0.1:53 >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Calling daemonize, going to >>>> background >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Set effective group id to 499 >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Set effective user id to 498 >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Launching 2 threads >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Done priming cache with root >>>> hints >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Enabled 'epoll' multiplexer >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Done priming cache with root >>>> hints >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Refreshed . records >>>> May 20 23:18:25 c605 kernel: pdns_recursor[21345]: segfault at ffff01d4 ip >>>> 080b1626 sp b6397890 error 4 in pdns_recursor[8048000+112000] >>>> >>>> [root@c605 ~]# ip a |grep inet |wc >>>> 4322 25927 220579 >>>> [root@c605 ~]# /etc/init.d/pdns-recursor start >>>> Starting pdns-recursor: [ OK ] <-- >>>> starts OK >>>> [root@c605 ~]# /etc/init.d/pdns-recursor stop >>>> Stopping pdns-recursor: [ OK ] >>>> >>>> Adding one more /24: >>>> >>>> [root@c605 ~]# for j in `seq 1 254`; do ip a a 10.0.18.$j/16 dev eth0; done >>>> [root@c605 ~]# /etc/init.d/pdns-recursor start >>>> Starting pdns-recursor: [ OK ] >>>> [root@c605 ~]# ip a |grep inet |wc >>>> 4576 27451 233679 >>>> >>>> It says it starts, but it doesn't, just segfaults. >>>> >>>> [root@c605 ~]# file /bin/bash >>>> /bin/bash: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), >>>> dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped >>>> >>>> >>>> On 05/20/2014 10:58 PM, James Baer wrote: >>>>> Hi All - I'm experiencing an issue that I am unsure if it is a bug or >>>>> just something I need to adjust on my systems to account for. >>>>> >>>>> I have 2 servers, both running pdns_recursor (3.5.3) on Centos 6.5, >>>>> installed from epel repository. The recursor is only listening on >>>>> localhost on each system. >>>>> >>>>> I am experiencing somewhat random crashes of the recursor with the >>>>> following error: >>>>> >>>>> kernel: pdns_recursor[21993]: segfault at 200001fc8 ip 0000000000472780 >>>>> sp 00007f3f9c03f690 error 4 in pdns_recursor[400000+111000] >>>>> >>>>> Both servers have a large number of ip addresses bound to them, in the >>>>> range of 3-4k. I was able to replicate the segfaults on one of the >>>>> servers by adding additional ip addresses. When I got to around 4k ip >>>>> addresses the recursor simply would not even start, just segafulted right >>>>> away. I was able to get it to start again, by removing some ip addresses, >>>>> so i know it has something to do with how many addresses I have bound the >>>>> server. >>>>> >>>>> Any body have an ideas what I can do to correct this problem? I really >>>>> don't see a reason why the recursor would care how many ip addresses I >>>>> have on a system. >>>>> >>>>> thank you >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Pdns-users mailing list >>>>> [email protected] >>>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users >>>>> >>>> -- >>>> >>>> Imre Gergely >>>> >>>> http://havaz.net >>>> >>>> gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 >>>> >> -- >> Imre Gergely >> http://havaz.net >> gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 >> > >> _______________________________________________ >> Pdns-users mailing list >> [email protected] >> http://mailman.powerdns.com/mailman/listinfo/pdns-users -- Imre Gergely http://havaz.net gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/pdns_recursor...Reading symbols from /usr/lib/debug/usr/sbin/pdns_recursor.debug...done. done. [?1034h(gdb) run Starting program: /usr/sbin/pdns_recursor [Thread debugging using libthread_db enabled] Program received signal SIGSEGV, Segmentation fault. 0x003b629e in make_request () from /lib/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.i686 libgcc-4.4.7-4.el6.i686 libstdc++-4.4.7-4.el6.i686 lua-5.1.4-4.1.el6.i686 (gdb) backtrace #0 0x003b629e in make_request () from /lib/libc.so.6 #1 0x003b64e9 in __check_pf () from /lib/libc.so.6 #2 0x003754af in getaddrinfo () from /lib/libc.so.6 #3 0x0807af9b in makeIPv6sockaddr (addr="2001:503:ba3e::2:30", ret=0xb7ff5d20) at misc.cc:717 #4 0x080ba3d5 in ComboAddress (rr=...) at iputils.hh:122 #5 DNSRR2String (rr=...) at recursor_cache.cc:75 #6 0x080bca00 in MemRecursorCache::replace (this=0x81655c8, now=1400620340, qname="a.root-servers.net.", qt=..., content=std::set with 1 elements = {...}, auth=true) at recursor_cache.cc:249 #7 0x0805a2b2 in SyncRes::doResolveAt (this=0xb7ff6b60, nameservers=std::set with 13 elements = {...}, auth=".", flawedNSSet=false, qname=".", qtype=..., ret=std::vector of length 0, capacity 0, depth=0, beenthere=std::set with 1 elements = {...}) at syncres.cc:1043 #8 0x08056472 in SyncRes::doResolve (this=0xb7ff6b60, qname=".", qtype=..., ret=std::vector of length 0, capacity 0, depth=0, beenthere=std::set with 1 elements = {...}) at syncres.cc:440 #9 0x08062847 in SyncRes::beginResolve (this=0xb7ff6b60, qname=".", qtype=..., qclass=1, ret=std::vector of length 0, capacity 0) at syncres.cc:126 #10 0x08095cdc in houseKeeping () at pdns_recursor.cc:1179 #11 0x080ac2be in MTasker<PacketID, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >::threadWrapper (self1=0, self2=135687208, tf=0x8095bc0 <houseKeeping(void*)>, tid=0, val1=0, val2=0) at mtasker.cc:380 #12 0x002e9b9b in makecontext () from /lib/libc.so.6 #13 0x00000000 in ?? () (gdb) #0 0x003b629e in make_request () from /lib/libc.so.6 #1 0x003b64e9 in __check_pf () from /lib/libc.so.6 #2 0x003754af in getaddrinfo () from /lib/libc.so.6 #3 0x0807af9b in makeIPv6sockaddr (addr="2001:503:ba3e::2:30", ret=0xb7ff5d20) at misc.cc:717 #4 0x080ba3d5 in ComboAddress (rr=...) at iputils.hh:122 #5 DNSRR2String (rr=...) at recursor_cache.cc:75 #6 0x080bca00 in MemRecursorCache::replace (this=0x81655c8, now=1400620340, qname="a.root-servers.net.", qt=..., content=std::set with 1 elements = {...}, auth=true) at recursor_cache.cc:249 #7 0x0805a2b2 in SyncRes::doResolveAt (this=0xb7ff6b60, nameservers=std::set with 13 elements = {...}, auth=".", flawedNSSet=false, qname=".", qtype=..., ret=std::vector of length 0, capacity 0, depth=0, beenthere=std::set with 1 elements = {...}) at syncres.cc:1043 #8 0x08056472 in SyncRes::doResolve (this=0xb7ff6b60, qname=".", qtype=..., ret=std::vector of length 0, capacity 0, depth=0, beenthere=std::set with 1 elements = {...}) at syncres.cc:440 #9 0x08062847 in SyncRes::beginResolve (this=0xb7ff6b60, qname=".", qtype=..., qclass=1, ret=std::vector of length 0, capacity 0) at syncres.cc:126 #10 0x08095cdc in houseKeeping () at pdns_recursor.cc:1179 #11 0x080ac2be in MTasker<PacketID, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >::threadWrapper (self1=0, self2=135687208, tf=0x8095bc0 <houseKeeping(void*)>, tid=0, val1=0, val2=0) at mtasker.cc:380 #12 0x002e9b9b in makecontext () from /lib/libc.so.6 #13 0x00000000 in ?? () (gdb) quit A debugging session is active. Inferior 1 [process 5896] will be killed. Quit anyway? (y or n)
_______________________________________________ Pdns-users mailing list [email protected] http://mailman.powerdns.com/mailman/listinfo/pdns-users
