Package: pdns-recursor Version: 4.0.3-4 Severity: important Dear maintainers,
Since the upgrade from 4.0.3-3 to 4.0.3-4 I have been experiencing intermittent crashes in pdns-recursor. I have so far not managed to get a core dump from the daemon, so have resorted to running the process within gdb to catch a backtrace. I have included this below: (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:58 #1 0x00007f405816440a in __GI_abort () at abort.c:89 #2 0x00007f40581a0bd0 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f4058295c70 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007f40581a6fa6 in malloc_printerr (action=3, str=0x7f4058295d80 "free(): invalid next size (fast)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5046 #4 0x00007f40581a779e in _int_free (av=0x7f404c000020, p=0x7f404c9985f0, have_lock=0) at malloc.c:3902 #5 0x000055b3dcea35e3 in __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<DNSName const, double> > >::deallocate (this=<optimized out>, __p=<optimized out>) at /usr/include/c++/6/ext/new_allocator.h:110 #6 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<DNSName const, double> > > >::deallocate (__a=..., __n=1, __p=<optimized out>) at /usr/include/c++/6/bits/alloc_traits.h:442 #7 std::_Rb_tree<DNSName, std::pair<DNSName const, double>, std::_Select1st<std::pair<DNSName const, double> >, std::less<DNSName>, std::allocator<std::pair<DNSName const, double> > >::_M_put_node (this=0x7f404cb6bb60, __p=<optimized out>) at /usr/include/c++/6/bits/stl_tree.h:509 #8 std::_Rb_tree<DNSName, std::pair<DNSName const, double>, std::_Select1st<std::pair<DNSName const, double> >, std::less<DNSName>, std::allocator<std::pair<DNSName const, double> > >::_M_drop_node (__p=<optimized out>, this=0x7f404cb6bb60) at /usr/include/c++/6/bits/stl_tree.h:576 #9 std::_Rb_tree<DNSName, std::pair<DNSName const, double>, std::_Select1st<std::pair<DNSName const, double> >, std::less<DNSName>, std::allocator<std::pair<DNSName const, double> > >::_M_erase (this=this@entry=0x7f404cb6bb60, __x=<optimized out>) at /usr/include/c++/6/bits/stl_tree.h:1640 #10 0x000055b3dcea35f4 in std::_Rb_tree<DNSName, std::pair<DNSName const, double>, std::_Select1st<std::pair<DNSName const, double> >, std::less<DNSName>, std::allocator<std::pair<DNSName const, double> > >::_M_erase (this=this@entry=0x7f404cb6bb60, __x=0x7f404c0e4d70) at /usr/include/c++/6/bits/stl_tree.h:1638 #11 0x000055b3dcea35f4 in std::_Rb_tree<DNSName, std::pair<DNSName const, double>, std::_Select1st<std::pair<DNSName const, double> >, std::less<DNSName>, std::allocator<std::pair<DNSName const, double> > >::_M_erase (this=this@entry=0x7f404cb6bb60, __x=0x7f404c7b3220) at /usr/include/c++/6/bits/stl_tree.h:1638 #12 0x000055b3dcea35f4 in std::_Rb_tree<DNSName, std::pair<DNSName const, double>, std::_Select1st<std::pair<DNSName const, double> >, std::less<DNSName>, std::allocator<std::pair<DNSName const, double> > >::_M_erase (this=this@entry=0x7f404cb6bb60, __x=0x7f404c9fdc30) at /usr/include/c++/6/bits/stl_tree.h:1638 #13 0x000055b3dcea35f4 in std::_Rb_tree<DNSName, std::pair<DNSName const, double>, std::_Select1st<std::pair<DNSName const, double> >, std::less<DNSName>, std::allocator<std::pair<DNSName const, double> > >::_M_erase (this=this@entry=0x7f404cb6bb60, __x=0x7f404c9fc200) at /usr/include/c++/6/bits/stl_tree.h:1638 #14 0x000055b3dcea9a43 in std::_Rb_tree<DNSName, std::pair<DNSName const, double>, std::_Select1st<std::pair<DNSName const, double> >, std::less<DNSName>, std::allocator<std::pair<DNSName const, double> > >::~_Rb_tree (this=0x7f404cb6bb60, __in_chrg=<optimized out>) at /usr/include/c++/6/bits/stl_tree.h:873 #15 std::map<DNSName, double, std::less<DNSName>, std::allocator<std::pair<DNSName const, double> > >::~map (this=0x7f404cb6bb60, __in_chrg=<optimized out>) at /usr/include/c++/6/bits/stl_map.h:96 #16 SyncRes::shuffleInSpeedOrder (this=this@entry=0x7f404cb76c60, tnameservers=..., prefix=...) at syncres.cc:885 #17 0x000055b3dce96276 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=..., qtype=..., ret=..., depth=<optimized out>, beenthere=...) at syncres.cc:989 #18 0x000055b3dce9fd5d in SyncRes::doResolve (this=this@entry=0x7f404cb76c60, qname=..., qtype=..., ret=..., depth=depth@entry=15, beenthere=...) at syncres.cc:470 #19 0x000055b3dcea19d8 in SyncRes::getAddrs (this=this@entry=0x7f404cb76c60, qname=..., depth=depth@entry=14, beenthere=...) at syncres.cc:517 #20 0x000055b3dce98ed2 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=..., qtype=..., ret=..., depth=<optimized out>, beenthere=...) at syncres.cc:1041 #21 0x000055b3dce9fd5d in SyncRes::doResolve (this=this@entry=0x7f404cb76c60, qname=..., qtype=..., ret=..., depth=depth@entry=12, beenthere=...) at syncres.cc:470 #22 0x000055b3dcea19d8 in SyncRes::getAddrs (this=this@entry=0x7f404cb76c60, qname=..., depth=depth@entry=11, beenthere=...) at syncres.cc:517 #23 0x000055b3dce98ed2 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=..., qtype=..., ret=..., depth=<optimized out>, beenthere=...) at syncres.cc:1041 #24 0x000055b3dce9fd5d in SyncRes::doResolve (this=this@entry=0x7f404cb76c60, qname=..., qtype=..., ret=..., depth=depth@entry=9, beenthere=...) at syncres.cc:470 #25 0x000055b3dcea19d8 in SyncRes::getAddrs (this=this@entry=0x7f404cb76c60, qname=..., depth=depth@entry=8, beenthere=...) at syncres.cc:517 #26 0x000055b3dce98ed2 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=..., qtype=..., ret=..., depth=<optimized out>, beenthere=...) at syncres.cc:1041 #27 0x000055b3dce9fd5d in SyncRes::doResolve (this=this@entry=0x7f404cb76c60, qname=..., qtype=..., ret=..., depth=depth@entry=6, beenthere=...) at syncres.cc:470 #28 0x000055b3dcea19d8 in SyncRes::getAddrs (this=this@entry=0x7f404cb76c60, qname=..., depth=depth@entry=5, beenthere=...) at syncres.cc:517 #29 0x000055b3dce98ed2 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=..., qtype=..., ret=..., depth=<optimized out>, beenthere=...) at syncres.cc:1041 #30 0x000055b3dce9fd5d in SyncRes::doResolve (this=this@entry=0x7f404cb76c60, qname=..., qtype=..., ret=..., depth=depth@entry=3, beenthere=...) at syncres.cc:470 #31 0x000055b3dcea19d8 in SyncRes::getAddrs (this=this@entry=0x7f404cb76c60, qname=..., depth=depth@entry=2, beenthere=...) at syncres.cc:517 #32 0x000055b3dce98ed2 in SyncRes::doResolveAt (this=this@entry=0x7f404cb76c60, nameservers=..., auth=..., flawedNSSet=flawedNSSet@entry=false, qname=..., qtype=..., ret=..., depth=<optimized out>, beenthere=...) at syncres.cc:1041 #33 0x000055b3dce9fd5d in SyncRes::doResolve (this=this@entry=0x7f404cb76c60, qname=..., qtype=..., ret=..., depth=depth@entry=0, beenthere=...) at syncres.cc:470 #34 0x000055b3dcea0df3 in SyncRes::beginResolve (this=this@entry=0x7f404cb76c60, qname=..., qtype=..., qclass=qclass@entry=1, ret=...) at syncres.cc:180 #35 0x000055b3dceb96cd in SRRecordOracle::get (this=0x7f404cb780a0, qname=..., qtype=<optimized out>) at validate-recursor.cc:24 #36 0x000055b3dceaef15 in getZoneCuts (begin=..., end=..., dro=...) at validate.cc:130 #37 0x000055b3dceb1f49 in getKeysFor (dro=..., zone=..., keyset=...) at validate.cc:273 #38 0x000055b3dceb8918 in validateRecords (recs=...) at validate-recursor.cc:104 #39 0x000055b3dcdfda2b in startDoResolve (p=0x7f404ca8e5a0) at pdns_recursor.cc:951 #40 0x000055b3dce1385b in MTasker<PacketID, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::makeThread(void (*)(void*), void*)::{lambda()#1}::operator()() const (__closure=<optimized out>) at mtasker.cc:270 #41 boost::detail::function::void_function_obj_invoker0<MTasker<PacketID, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::makeThread(void (*)(void*), void*)::{lambda()#1}, void>::invoke(boost::detail::function::function_buffer&) (function_obj_ptr=...) at /usr/include/boost/function/function_template.hpp:159 #42 0x000055b3dcddf1a9 in boost::function0<void>::operator() (this=0x7f404cb791e0) at /usr/include/boost/function/function_template.hpp:771 #43 threadWrapper (ctx0=32576, ctx1=1285765232, fun0=<optimized out>, fun1=<optimized out>) at mtasker_ucontext.cc:77 #44 0x00007f4058173fe0 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6 #45 0x00007f404c001d10 in ?? () #46 0x722e30736e203a65 in ?? () #47 0x6362622e766f7362 in ?? () #48 0x00000000000727e1 in ?? () #49 0x00007f404c000838 in ?? () #50 0x00007f404c000838 in ?? () #51 0x00007f404cb79260 in ?? () #52 0x00007f404cb79260 in ?? () #53 0x2020202020202020 in ?? () #54 0x62722e30736e2020 in ?? () #55 0x2e6362622e766f73 in ?? () #56 0x62203a6b752e6f63 in ?? () #57 0x65726568746e6565 in ?? () #58 0x62722e30736e203a in ?? () #59 0x2e6362622e766f73 in ?? () #60 0x20417c6b752e6f63 in ?? () #61 0x202020200a293828 in ?? () #62 0x2020202020202020 in ?? () #63 0x2020202020202020 in ?? () #64 0x2020202020202020 in ?? () #65 0x2020202020202020 in ?? () #66 0x6f7362722e30736e in ?? () #67 0x6f632e6362622e76 in ?? () #68 0x656562203a6b752e in ?? () #69 0x203a65726568746e in ?? () #70 0x6f7362722e30736e in ?? () #71 0x6f632e6362622e76 in ?? () #72 0x342820417c6b752e in ?? () #73 0x2020202020200a29 in ?? () #74 0x2020202020202020 in ?? () #75 0x2020202020202020 in ?? () #76 0x2020202020202020 in ?? () #77 0x736e202020202020 in ?? () #78 0x2e766f7362722e30 in ?? () #79 0x752e6f632e636262 in ?? () #80 0x746e656562203a6b in ?? () #81 0x736e203a65726568 in ?? () #82 0x2e766f7362722e30 in ?? () #83 0x752e6f632e636262 in ?? () #84 0x2820414141417c6b in ?? () #85 0x20202020200a2938 in ?? () #86 0x2020202020202020 in ?? () #87 0x2020202020202020 in ?? () #88 0x2020202020202020 in ?? () #89 0x6e20202020202020 in ?? () #90 0x766f7362722e3073 in ?? () #91 0x2e6f632e6362622e in ?? () #92 0x6e656562203a6b75 in ?? () #93 0x203a2a6572656874 in ?? () #94 0x6f7362722e30736e in ?? () #95 0x6f632e6362622e76 in ?? () #96 0x414141417c6b752e in ?? () #97 0x2020200a29342820 in ?? () #98 0x2020202020202020 in ?? () #99 0x2020202020202020 in ?? () #100 0x2020202020202020 in ?? () #101 0x2020202020202020 in ?? () #102 0x7362722e30736e20 in ?? () #103 0x632e6362622e766f in ?? () #104 0x6562203a6b752e6f in ?? () #105 0x3a65726568746e65 in ?? () #106 0x6362622e33736e20 in ?? () At this point the stack seems to recurse very deep, perhaps infinitely. I stopped at 3768 frames of repetition. I have two identical machines running pdns_recursor behind dnsdist, and both suffer from this issue. I hope this helps. Regards, Chris -- System Information: Debian Release: stretch/sid APT prefers testing APT policy: (990, 'testing'), (500, 'unstable') Architecture: amd64 (x86_64) Kernel: Linux 4.8.0-2-amd64 (SMP w/2 CPU cores) Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages pdns-recursor depends on: ii adduser 3.115 ii dns-root-data 2015052300+h+1 ii init-system-helpers 1.46 ii libatomic1 6.2.1-5 ii libc6 2.24-8 ii libgcc1 1:6.2.1-5 ii liblua5.2-0 5.2.4-1.1+b1 ii libprotobuf10 3.0.0-9 ii libssl1.1 1.1.0c-2 ii libstdc++6 6.2.1-5 ii libsystemd0 232-8 pdns-recursor recommends no packages. pdns-recursor suggests no packages. -- Configuration Files: /etc/powerdns/recursor.conf changed: allow-from=<<<< redacted>>>> carbon-server=2a02:2770:8::2635:0:1 config-dir=/etc/powerdns dnssec=validate dnssec-log-bogus=yes hint-file=/usr/share/dns/root.hints include-dir=/etc/powerdns/recursor.d local-address=::,0.0.0.0 local-port=8053 log-common-errors=yes query-local-address6=:: quiet=yes security-poll-suffix= server-down-max-fails=0 server-down-throttle-time=0 setgid=pdns setuid=pdns trace=fail -- no debconf information