Thank you, this is pretty much what happens. Just need to figure out why it crashes. No obvious reason stands out, other than why it's using make_request for numeric host is not something I understand, and why it happens if you increase the amount of local addresses.
is the recursor used by localhost? does anything change if you set query-local-address to some IP address? Aki On Wed, May 21, 2014 at 12:16:22AM +0300, Imre Gergely wrote: > > backtrace attached. > > [root@c605 pdns-recursor]# ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 3873 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 8192 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 3873 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > On 05/21/2014 12:00 AM, Aki Tuomi wrote: > > Can you install the debuginfo package and run it with gdb to get stack > > trace? Also, can > > you give us ulimit -a? > > > > Aki > > > > On Tue, May 20, 2014 at 11:41:21PM +0300, Imre Gergely wrote: > >> [root@c605 ~]# cat /etc/pdns-recursor/recursor.conf |grep -v "^#" | grep > >> -v "^$" > >> setuid=pdns-recursor > >> setgid=pdns-recursor > >> daemon=no > >> local-address=127.0.0.1 > >> threads=1 > >> > >> [root@c605 ~]# strace /usr/sbin/pdns_recursor > > >> /tmp/strace-pdns-recursor.txt 2>&1 > >> Segmentation fault > >> [root@c605 ~]# > >> > >> [root@c605 ~]# ip a |grep inet |wc > >> 4576 27451 233679 > >> > >> Attached. If this is not what you had in mind, please let me know. > >> > >> On 05/20/2014 11:31 PM, bert hubert wrote: > >>> Imre, > >>> > >>> Can you strace the startup with threads=1? > >>> > >>> Bert > >>> > >>> On May 20, 2014 10:25 PM, Imre Gergely <[email protected]> wrote: > >>>> Hi > >>>> > >>>> I did manage to reproduce this in a VM. Installed a CentOS 6.5, and > >>>> recursor 3.5.3 from EPEL. Then I did this: > >>>> > >>>> for i in `seq 1 16`; do for j in `seq 1 254`; do ip a a 10.0.$i.$j/16 > >>>> dev eth0; done; done > >>>> > >>>> Then I started the recursor, everything went just fine, did a bunch of > >>>> digs, no problems. > >>>> > >>>> Then I added some more IPs: > >>>> > >>>> for i in `seq 17 32`; do for j in `seq 1 254`; do ip a a 10.0.$i.$j/16 > >>>> dev eth0; done; done > >>>> > >>>> And then init.d/pdns-recursor restart: > >>>> > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: PowerDNS recursor 3.5.3 (C) > >>>> 2001-2013 PowerDNS.COM BV (Feb 10 2014, 17:26:52, gcc 4.4.7 20120313 > >>>> (Red Hat 4.4.7-4)) > >>>> starting up > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: PowerDNS comes with > >>>> ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to > >>>> redistribute it according to the terms of the GPL version 2. > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Operating in 32 bits mode > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Reading random entropy from > >>>> '/dev/urandom' > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Only allowing queries from: > >>>> 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, > >>>> 172.16.0.0/12, ::1/128, fe80::/10 > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Will not send queries to: > >>>> 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, > >>>> 172.16.0.0/12, ::1/128, fe80::/10, 0.0.0.0, :: > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: NOT using IPv6 for outgoing > >>>> queries - set 'query-local-address6=::' to enable > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Inserting rfc 1918 private > >>>> space zones > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Listening for UDP queries on > >>>> 127.0.0.1:53 > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Enabled TCP data-ready filter > >>>> for (slight) DoS protection > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Listening for TCP queries on > >>>> 127.0.0.1:53 > >>>> May 20 23:18:24 c605 pdns_recursor[21341]: Calling daemonize, going to > >>>> background > >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Set effective group id to 499 > >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Set effective user id to 498 > >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Launching 2 threads > >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Done priming cache with root > >>>> hints > >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Enabled 'epoll' multiplexer > >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Done priming cache with root > >>>> hints > >>>> May 20 23:18:24 c605 pdns_recursor[21342]: Refreshed . records > >>>> May 20 23:18:25 c605 kernel: pdns_recursor[21345]: segfault at ffff01d4 > >>>> ip 080b1626 sp b6397890 error 4 in pdns_recursor[8048000+112000] > >>>> > >>>> [root@c605 ~]# ip a |grep inet |wc > >>>> 4322 25927 220579 > >>>> [root@c605 ~]# /etc/init.d/pdns-recursor start > >>>> Starting pdns-recursor: [ OK ] <-- > >>>> starts OK > >>>> [root@c605 ~]# /etc/init.d/pdns-recursor stop > >>>> Stopping pdns-recursor: [ OK ] > >>>> > >>>> Adding one more /24: > >>>> > >>>> [root@c605 ~]# for j in `seq 1 254`; do ip a a 10.0.18.$j/16 dev eth0; > >>>> done > >>>> [root@c605 ~]# /etc/init.d/pdns-recursor start > >>>> Starting pdns-recursor: [ OK ] > >>>> [root@c605 ~]# ip a |grep inet |wc > >>>> 4576 27451 233679 > >>>> > >>>> It says it starts, but it doesn't, just segfaults. > >>>> > >>>> [root@c605 ~]# file /bin/bash > >>>> /bin/bash: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), > >>>> dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped > >>>> > >>>> > >>>> On 05/20/2014 10:58 PM, James Baer wrote: > >>>>> Hi All - I'm experiencing an issue that I am unsure if it is a bug or > >>>>> just something I need to adjust on my systems to account for. > >>>>> > >>>>> I have 2 servers, both running pdns_recursor (3.5.3) on Centos 6.5, > >>>>> installed from epel repository. The recursor is only listening on > >>>>> localhost on each system. > >>>>> > >>>>> I am experiencing somewhat random crashes of the recursor with the > >>>>> following error: > >>>>> > >>>>> kernel: pdns_recursor[21993]: segfault at 200001fc8 ip 0000000000472780 > >>>>> sp 00007f3f9c03f690 error 4 in pdns_recursor[400000+111000] > >>>>> > >>>>> Both servers have a large number of ip addresses bound to them, in the > >>>>> range of 3-4k. I was able to replicate the segfaults on one of the > >>>>> servers by adding additional ip addresses. When I got to around 4k ip > >>>>> addresses the recursor simply would not even start, just segafulted > >>>>> right away. I was able to get it to start again, by removing some ip > >>>>> addresses, so i know it has something to do with how many addresses I > >>>>> have bound the server. > >>>>> > >>>>> Any body have an ideas what I can do to correct this problem? I really > >>>>> don't see a reason why the recursor would care how many ip addresses I > >>>>> have on a system. > >>>>> > >>>>> thank you > >>>>> > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> Pdns-users mailing list > >>>>> [email protected] > >>>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users > >>>>> > >>>> -- > >>>> > >>>> Imre Gergely > >>>> > >>>> http://havaz.net > >>>> > >>>> gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 > >>>> > >> -- > >> Imre Gergely > >> http://havaz.net > >> gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 > >> > > > >> _______________________________________________ > >> Pdns-users mailing list > >> [email protected] > >> http://mailman.powerdns.com/mailman/listinfo/pdns-users > > -- > Imre Gergely > http://havaz.net > gpg --keyserver subkeys.pgp.net --recv-keys 0x34525305 > > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) > Copyright (C) 2010 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "i686-redhat-linux-gnu". > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>... > Reading symbols from /usr/sbin/pdns_recursor...Reading symbols from > /usr/lib/debug/usr/sbin/pdns_recursor.debug...done. > done. > [?1034h(gdb) run > Starting program: /usr/sbin/pdns_recursor > [Thread debugging using libthread_db enabled] > > Program received signal SIGSEGV, Segmentation fault. > 0x003b629e in make_request () from /lib/libc.so.6 > Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6.i686 > libgcc-4.4.7-4.el6.i686 libstdc++-4.4.7-4.el6.i686 lua-5.1.4-4.1.el6.i686 > (gdb) backtrace > #0 0x003b629e in make_request () from /lib/libc.so.6 > #1 0x003b64e9 in __check_pf () from /lib/libc.so.6 > #2 0x003754af in getaddrinfo () from /lib/libc.so.6 > #3 0x0807af9b in makeIPv6sockaddr (addr="2001:503:ba3e::2:30", > ret=0xb7ff5d20) at misc.cc:717 > #4 0x080ba3d5 in ComboAddress (rr=...) at iputils.hh:122 > #5 DNSRR2String (rr=...) at recursor_cache.cc:75 > #6 0x080bca00 in MemRecursorCache::replace (this=0x81655c8, now=1400620340, > qname="a.root-servers.net.", qt=..., content=std::set with 1 elements = > {...}, auth=true) at recursor_cache.cc:249 > #7 0x0805a2b2 in SyncRes::doResolveAt (this=0xb7ff6b60, nameservers=std::set > with 13 elements = {...}, auth=".", flawedNSSet=false, qname=".", qtype=..., > ret=std::vector of length 0, capacity 0, depth=0, beenthere=std::set with 1 > elements = {...}) at syncres.cc:1043 > #8 0x08056472 in SyncRes::doResolve (this=0xb7ff6b60, qname=".", qtype=..., > ret=std::vector of length 0, capacity 0, depth=0, beenthere=std::set with 1 > elements = {...}) at syncres.cc:440 > #9 0x08062847 in SyncRes::beginResolve (this=0xb7ff6b60, qname=".", > qtype=..., qclass=1, ret=std::vector of length 0, capacity 0) at > syncres.cc:126 > #10 0x08095cdc in houseKeeping () at pdns_recursor.cc:1179 > #11 0x080ac2be in MTasker<PacketID, std::basic_string<char, > std::char_traits<char>, std::allocator<char> > >::threadWrapper (self1=0, > self2=135687208, tf=0x8095bc0 <houseKeeping(void*)>, tid=0, val1=0, val2=0) > at mtasker.cc:380 > #12 0x002e9b9b in makecontext () from /lib/libc.so.6 > #13 0x00000000 in ?? () > (gdb) > #0 0x003b629e in make_request () from /lib/libc.so.6 > #1 0x003b64e9 in __check_pf () from /lib/libc.so.6 > #2 0x003754af in getaddrinfo () from /lib/libc.so.6 > #3 0x0807af9b in makeIPv6sockaddr (addr="2001:503:ba3e::2:30", > ret=0xb7ff5d20) at misc.cc:717 > #4 0x080ba3d5 in ComboAddress (rr=...) at iputils.hh:122 > #5 DNSRR2String (rr=...) at recursor_cache.cc:75 > #6 0x080bca00 in MemRecursorCache::replace (this=0x81655c8, now=1400620340, > qname="a.root-servers.net.", qt=..., content=std::set with 1 elements = > {...}, auth=true) at recursor_cache.cc:249 > #7 0x0805a2b2 in SyncRes::doResolveAt (this=0xb7ff6b60, nameservers=std::set > with 13 elements = {...}, auth=".", flawedNSSet=false, qname=".", qtype=..., > ret=std::vector of length 0, capacity 0, depth=0, beenthere=std::set with 1 > elements = {...}) at syncres.cc:1043 > #8 0x08056472 in SyncRes::doResolve (this=0xb7ff6b60, qname=".", qtype=..., > ret=std::vector of length 0, capacity 0, depth=0, beenthere=std::set with 1 > elements = {...}) at syncres.cc:440 > #9 0x08062847 in SyncRes::beginResolve (this=0xb7ff6b60, qname=".", > qtype=..., qclass=1, ret=std::vector of length 0, capacity 0) at > syncres.cc:126 > #10 0x08095cdc in houseKeeping () at pdns_recursor.cc:1179 > #11 0x080ac2be in MTasker<PacketID, std::basic_string<char, > std::char_traits<char>, std::allocator<char> > >::threadWrapper (self1=0, > self2=135687208, tf=0x8095bc0 <houseKeeping(void*)>, tid=0, val1=0, val2=0) > at mtasker.cc:380 > #12 0x002e9b9b in makecontext () from /lib/libc.so.6 > #13 0x00000000 in ?? () > (gdb) quit > A debugging session is active. > > Inferior 1 [process 5896] will be killed. > > Quit anyway? (y or n)
signature.asc
Description: Digital signature
_______________________________________________ Pdns-users mailing list [email protected] http://mailman.powerdns.com/mailman/listinfo/pdns-users
