Thanks! can you open a bug report on github and mention @janmejay who will most probably be able to check on this.
To get a useful valgrind run, you need to add the "-n" command line option to rsyslog (else it backgrounds). Sorry for missing to mention this. Rainer 2016-06-22 15:17 GMT+02:00 Christopher Racky <[email protected]>: > Hi, > > Issues seems to be related to the Destruction of lookup-Table: > > (gdb) where > #0 0x00007f4d6558b213 in pthread_join () from /lib64/libpthread.so.0 > #1 0x00007f4d65c1b857 in lookupDestroyCnf () > #2 0x00007f4d65bf5142 in rsconfDestruct () > #3 0x00007f4d65c2ba4b in main () > > Here the full Output of core-file Analysis: > --------------------------------------------------------------------------------------------- > > [root@server1 ~]# gdb /sbin/rsyslogd /var/tmp/core.14983 > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-90.el6) > Copyright (C) 2010 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>... > Reading symbols from /sbin/rsyslogd...(no debugging symbols found)...done. > [New Thread 14983] > [New Thread 14990] > [New Thread 14993] > [New Thread 14992] > [New Thread 14989] > Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. > Loaded symbols for /lib64/libz.so.1 > Reading symbols from /lib64/libpthread.so.0...(no debugging symbols > found)...done. > [Thread debugging using libthread_db enabled] > Loaded symbols for /lib64/libpthread.so.0 > Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. > Loaded symbols for /lib64/libdl.so.2 > Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. > Loaded symbols for /lib64/librt.so.1 > Reading symbols from /usr/lib64/libestr.so.0...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libestr.so.0 > Reading symbols from /usr/lib64/libfastjson.so.3...done. > Loaded symbols for /usr/lib64/libfastjson.so.3 > Reading symbols from /lib64/libuuid.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libuuid.so.1 > Reading symbols from /usr/lib64/liblogging-stdlog.so.0...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/liblogging-stdlog.so.0 > Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libgcc_s.so.1 > Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. > Loaded symbols for /lib64/libc.so.6 > Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging > symbols found)...done. > Loaded symbols for /lib64/ld-linux-x86-64.so.2 > Reading symbols from /lib64/rsyslog/lmnet.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/rsyslog/lmnet.so > Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libnss_files.so.2 > Reading symbols from /lib64/rsyslog/imuxsock.so...(no debugging > symbols found)...done. > Loaded symbols for /lib64/rsyslog/imuxsock.so > Reading symbols from /lib64/rsyslog/imklog.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/rsyslog/imklog.so > Reading symbols from /lib64/rsyslog/impstats.so...(no debugging > symbols found)...done. > Loaded symbols for /lib64/rsyslog/impstats.so > Reading symbols from /lib64/rsyslog/imtcp.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/rsyslog/imtcp.so > Reading symbols from /lib64/rsyslog/lmnetstrms.so...(no debugging > symbols found)...done. > Loaded symbols for /lib64/rsyslog/lmnetstrms.so > Reading symbols from /lib64/rsyslog/lmtcpsrv.so...(no debugging > symbols found)...done. > Loaded symbols for /lib64/rsyslog/lmtcpsrv.so > Reading symbols from /lib64/rsyslog/imudp.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/rsyslog/imudp.so > Reading symbols from /lib64/rsyslog/ommysql.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/rsyslog/ommysql.so > Reading symbols from /usr/lib64/mysql/libmysqlclient.so.16...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/mysql/libmysqlclient.so.16 > Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libcrypt.so.1 > Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done. > Loaded symbols for /lib64/libnsl.so.1 > Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. > Loaded symbols for /lib64/libm.so.6 > Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libssl.so.10 > Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/libcrypto.so.10 > Reading symbols from /lib64/libfreebl3.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libfreebl3.so > Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging > symbols found)...done. > Loaded symbols for /lib64/libgssapi_krb5.so.2 > Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libkrb5.so.3 > Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libcom_err.so.2 > Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libk5crypto.so.3 > Reading symbols from /lib64/libkrb5support.so.0...(no debugging > symbols found)...done. > Loaded symbols for /lib64/libkrb5support.so.0 > Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libkeyutils.so.1 > Reading symbols from /lib64/libresolv.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libresolv.so.2 > Reading symbols from /lib64/libselinux.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libselinux.so.1 > Reading symbols from /lib64/rsyslog/omudpspoof.so...(no debugging > symbols found)...done. > Loaded symbols for /lib64/rsyslog/omudpspoof.so > Reading symbols from /lib64/libnet.so.1...(no debugging symbols found)...done. > Loaded symbols for /lib64/libnet.so.1 > Reading symbols from /lib64/rsyslog/pmlastmsg.so...(no debugging > symbols found)...done. > Loaded symbols for /lib64/rsyslog/pmlastmsg.so > Reading symbols from /lib64/rsyslog/pmaixforwardedfrom.so...(no > debugging symbols found)...done. > Loaded symbols for /lib64/rsyslog/pmaixforwardedfrom.so > Reading symbols from /lib64/rsyslog/lmtcpclt.so...(no debugging > symbols found)...done. > Loaded symbols for /lib64/rsyslog/lmtcpclt.so > Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libnss_dns.so.2 > Core was generated by `/sbin/rsyslogd -i /var/run/syslogd.pid'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007f4d6558b213 in pthread_join () from /lib64/libpthread.so.0 > Missing separate debuginfos, use: debuginfo-install > rsyslog-8.19.0-1.el6.x86_64 > (gdb) where > #0 0x00007f4d6558b213 in pthread_join () from /lib64/libpthread.so.0 > #1 0x00007f4d65c1b857 in lookupDestroyCnf () > #2 0x00007f4d65bf5142 in rsconfDestruct () > #3 0x00007f4d65c2ba4b in main () > --------------------------------------------------------------------------------------------- > > Attached the log-file Output from the command: > > valgrind --log-file=/var/log/valgrindlog /sbin/rsyslogd -f /etc/rsyslog.conf > (I'm not sure how to Analysis this correctly as this is a daemon and > it is just reacting in kill situtation) > > regards, > Chris > > OUTPUT: > --------------------------------------------------------------------------- > > ==9342== Memcheck, a memory error detector > ==9342== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. > ==9342== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info > ==9342== Command: /sbin/rsyslogd -f /etc/rsyslog.conf > ==9342== Parent PID: 30783 > ==9342== > ==9342== Conditional jump or move depends on uninitialised value(s) > ==9342== at 0x16BB54: cnfexprDestruct (in /sbin/rsyslogd) > ==9342== by 0x1711AD: cnfstmtOptimize (in /sbin/rsyslogd) > ==9342== by 0x147687: ??? (in /sbin/rsyslogd) > ==9342== by 0x12F69E: llExecFunc (in /sbin/rsyslogd) > ==9342== by 0x1474D6: rulesetOptimizeAll (in /sbin/rsyslogd) > ==9342== by 0x124F8A: load (in /sbin/rsyslogd) > ==9342== by 0x15A0E8: initAll (in /sbin/rsyslogd) > ==9342== by 0x15A8A2: main (in /sbin/rsyslogd) > ==9342== > ==9344== Warning: invalid file descriptor 131062 in syscall close() > ==9342== > ==9342== HEAP SUMMARY: > ==9342== in use at exit: 225,937 bytes in 3,225 blocks > ==9342== total heap usage: 9,312 allocs, 6,087 frees, 1,019,793 > bytes allocated > ==9342== > ==9342== LEAK SUMMARY: > ==9342== definitely lost: 26 bytes in 1 blocks > ==9342== indirectly lost: 0 bytes in 0 blocks > ==9342== possibly lost: 608 bytes in 1 blocks > ==9342== still reachable: 225,303 bytes in 3,223 blocks > ==9342== suppressed: 0 bytes in 0 blocks > ==9342== Rerun with --leak-check=full to see details of leaked memory > ==9342== > ==9342== For counts of detected and suppressed errors, rerun with: -v > ==9342== Use --track-origins=yes to see where uninitialised values come from > ==9342== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 98 from 9) > --------------------------------------------------------------------------- > > regards > Chris > > 2016-06-21 13:23 GMT+02:00 Rainer Gerhards <[email protected]>: >> 2016-06-21 13:21 GMT+02:00 Christopher Racky <[email protected]>: >>> Hi, >>> >>> On my RHEL 6.8 Servers I get a segfault when I try to restart or stop >>> rsyslog after some runetime. >>> >>> See here: >>> >>> [root@server1 log]# rsyslogd -N2 >>> rsyslogd: version 8.19.0, config validation run (level 2), master >>> config /etc/rsyslog.conf >>> rsyslogd: End of config validation run. Bye. >>> [root@server1 log]# /etc/init.d/rsyslog restart >>> Shutting down system logger: [ OK ] >>> Starting system logger: [ OK ] >>> >>> [root@server1 log]# dmesg >>> rsyslogd[16603]: segfault at 7fb717ec79d0 ip 00007fb71d353213 sp >>> 00007fffc35c2a40 error 4 in libpthread-2.12.so[7fb71d34b000+17000] >>> >>> [root@server1 log]# ps -ef | grep rsyslog >>> root 17729 1 0 13:13 ? 00:00:00 /sbin/rsyslogd -i >>> /var/run/syslogd.pid >>> root 17780 13962 0 13:14 pts/1 00:00:00 grep rsyslog >>> >>> [root@server1 log]# cat /etc/redhat-release >>> Red Hat Enterprise Linux Server release 6.8 (Santiago) >>> >>> Unfortunately this segfault does not occure each stop... >>> >>> This Problems seems to persinst since rsyslog 8.16 >>> >>> >>> My configuration is quite complex with rulesets, lookuptable and the >>> following modules: >>> >>> Module(load="imuxsock") # provides support for local system logging >>> (e.g. via logger command) >>> Module(load="imklog") # provides kernel logging support (previously >>> done by rklogd) >>> Module (load="impstats" interval="120" severity="7" log.syslog="on" >>> ruleset="pstats") >>> Module (load="imtcp" KeepAlive="on" KeepAlive.Probes="1" >>> KeepAlive.Interval="2" KeepAlive.Time="20" MaxSessions="5000") >>> Module (load="imudp") >>> Module (load="ommysql") >>> Module (load="omudpspoof") >>> Module (load="pmlastmsg") >>> Module (load="pmaixforwardedfrom") >>> >>> >>> Do you have any hint or is this a known bug? >> >> Can you run it under valgrind control, so that we can see why it >> aborts? Or do you have a least a coredump, which could tell us at what >> location it aborted? >> Rainer >> >>> >>> >>> regards >>> Chris >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
