Hi, Issues seems to be related to the Destruction of lookup-Table:
(gdb) where #0 0x00007f4d6558b213 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f4d65c1b857 in lookupDestroyCnf () #2 0x00007f4d65bf5142 in rsconfDestruct () #3 0x00007f4d65c2ba4b in main () Here the full Output of core-file Analysis: --------------------------------------------------------------------------------------------- [root@server1 ~]# gdb /sbin/rsyslogd /var/tmp/core.14983 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-90.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /sbin/rsyslogd...(no debugging symbols found)...done. [New Thread 14983] [New Thread 14990] [New Thread 14993] [New Thread 14992] [New Thread 14989] Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /usr/lib64/libestr.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libestr.so.0 Reading symbols from /usr/lib64/libfastjson.so.3...done. Loaded symbols for /usr/lib64/libfastjson.so.3 Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /usr/lib64/liblogging-stdlog.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/liblogging-stdlog.so.0 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/rsyslog/lmnet.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/lmnet.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/rsyslog/imuxsock.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/imuxsock.so Reading symbols from /lib64/rsyslog/imklog.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/imklog.so Reading symbols from /lib64/rsyslog/impstats.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/impstats.so Reading symbols from /lib64/rsyslog/imtcp.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/imtcp.so Reading symbols from /lib64/rsyslog/lmnetstrms.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/lmnetstrms.so Reading symbols from /lib64/rsyslog/lmtcpsrv.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/lmtcpsrv.so Reading symbols from /lib64/rsyslog/imudp.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/imudp.so Reading symbols from /lib64/rsyslog/ommysql.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/ommysql.so Reading symbols from /usr/lib64/mysql/libmysqlclient.so.16...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/mysql/libmysqlclient.so.16 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libnsl.so.1 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libfreebl3.so Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/rsyslog/omudpspoof.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/omudpspoof.so Reading symbols from /lib64/libnet.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libnet.so.1 Reading symbols from /lib64/rsyslog/pmlastmsg.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/pmlastmsg.so Reading symbols from /lib64/rsyslog/pmaixforwardedfrom.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/pmaixforwardedfrom.so Reading symbols from /lib64/rsyslog/lmtcpclt.so...(no debugging symbols found)...done. Loaded symbols for /lib64/rsyslog/lmtcpclt.so Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_dns.so.2 Core was generated by `/sbin/rsyslogd -i /var/run/syslogd.pid'. Program terminated with signal 11, Segmentation fault. #0 0x00007f4d6558b213 in pthread_join () from /lib64/libpthread.so.0 Missing separate debuginfos, use: debuginfo-install rsyslog-8.19.0-1.el6.x86_64 (gdb) where #0 0x00007f4d6558b213 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f4d65c1b857 in lookupDestroyCnf () #2 0x00007f4d65bf5142 in rsconfDestruct () #3 0x00007f4d65c2ba4b in main () --------------------------------------------------------------------------------------------- Attached the log-file Output from the command: valgrind --log-file=/var/log/valgrindlog /sbin/rsyslogd -f /etc/rsyslog.conf (I'm not sure how to Analysis this correctly as this is a daemon and it is just reacting in kill situtation) regards, Chris OUTPUT: --------------------------------------------------------------------------- ==9342== Memcheck, a memory error detector ==9342== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. ==9342== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info ==9342== Command: /sbin/rsyslogd -f /etc/rsyslog.conf ==9342== Parent PID: 30783 ==9342== ==9342== Conditional jump or move depends on uninitialised value(s) ==9342== at 0x16BB54: cnfexprDestruct (in /sbin/rsyslogd) ==9342== by 0x1711AD: cnfstmtOptimize (in /sbin/rsyslogd) ==9342== by 0x147687: ??? (in /sbin/rsyslogd) ==9342== by 0x12F69E: llExecFunc (in /sbin/rsyslogd) ==9342== by 0x1474D6: rulesetOptimizeAll (in /sbin/rsyslogd) ==9342== by 0x124F8A: load (in /sbin/rsyslogd) ==9342== by 0x15A0E8: initAll (in /sbin/rsyslogd) ==9342== by 0x15A8A2: main (in /sbin/rsyslogd) ==9342== ==9344== Warning: invalid file descriptor 131062 in syscall close() ==9342== ==9342== HEAP SUMMARY: ==9342== in use at exit: 225,937 bytes in 3,225 blocks ==9342== total heap usage: 9,312 allocs, 6,087 frees, 1,019,793 bytes allocated ==9342== ==9342== LEAK SUMMARY: ==9342== definitely lost: 26 bytes in 1 blocks ==9342== indirectly lost: 0 bytes in 0 blocks ==9342== possibly lost: 608 bytes in 1 blocks ==9342== still reachable: 225,303 bytes in 3,223 blocks ==9342== suppressed: 0 bytes in 0 blocks ==9342== Rerun with --leak-check=full to see details of leaked memory ==9342== ==9342== For counts of detected and suppressed errors, rerun with: -v ==9342== Use --track-origins=yes to see where uninitialised values come from ==9342== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 98 from 9) --------------------------------------------------------------------------- regards Chris 2016-06-21 13:23 GMT+02:00 Rainer Gerhards <[email protected]>: > 2016-06-21 13:21 GMT+02:00 Christopher Racky <[email protected]>: >> Hi, >> >> On my RHEL 6.8 Servers I get a segfault when I try to restart or stop >> rsyslog after some runetime. >> >> See here: >> >> [root@server1 log]# rsyslogd -N2 >> rsyslogd: version 8.19.0, config validation run (level 2), master >> config /etc/rsyslog.conf >> rsyslogd: End of config validation run. Bye. >> [root@server1 log]# /etc/init.d/rsyslog restart >> Shutting down system logger: [ OK ] >> Starting system logger: [ OK ] >> >> [root@server1 log]# dmesg >> rsyslogd[16603]: segfault at 7fb717ec79d0 ip 00007fb71d353213 sp >> 00007fffc35c2a40 error 4 in libpthread-2.12.so[7fb71d34b000+17000] >> >> [root@server1 log]# ps -ef | grep rsyslog >> root 17729 1 0 13:13 ? 00:00:00 /sbin/rsyslogd -i >> /var/run/syslogd.pid >> root 17780 13962 0 13:14 pts/1 00:00:00 grep rsyslog >> >> [root@server1 log]# cat /etc/redhat-release >> Red Hat Enterprise Linux Server release 6.8 (Santiago) >> >> Unfortunately this segfault does not occure each stop... >> >> This Problems seems to persinst since rsyslog 8.16 >> >> >> My configuration is quite complex with rulesets, lookuptable and the >> following modules: >> >> Module(load="imuxsock") # provides support for local system logging >> (e.g. via logger command) >> Module(load="imklog") # provides kernel logging support (previously >> done by rklogd) >> Module (load="impstats" interval="120" severity="7" log.syslog="on" >> ruleset="pstats") >> Module (load="imtcp" KeepAlive="on" KeepAlive.Probes="1" >> KeepAlive.Interval="2" KeepAlive.Time="20" MaxSessions="5000") >> Module (load="imudp") >> Module (load="ommysql") >> Module (load="omudpspoof") >> Module (load="pmlastmsg") >> Module (load="pmaixforwardedfrom") >> >> >> Do you have any hint or is this a known bug? > > Can you run it under valgrind control, so that we can see why it > aborts? Or do you have a least a coredump, which could tell us at what > location it aborted? > Rainer > >> >> >> regards >> Chris >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
