Thank you very much for all your great support :) I created https://github.com/rsyslog/rsyslog/issues/1071
regards Chris 2016-06-22 15:20 GMT+02:00 Rainer Gerhards <[email protected]>: > Thanks! can you open a bug report on github and mention @janmejay who > will most probably be able to check on this. > > To get a useful valgrind run, you need to add the "-n" command line > option to rsyslog (else it backgrounds). Sorry for missing to mention > this. > > Rainer > > 2016-06-22 15:17 GMT+02:00 Christopher Racky <[email protected]>: >> Hi, >> >> Issues seems to be related to the Destruction of lookup-Table: >> >> (gdb) where >> #0 0x00007f4d6558b213 in pthread_join () from /lib64/libpthread.so.0 >> #1 0x00007f4d65c1b857 in lookupDestroyCnf () >> #2 0x00007f4d65bf5142 in rsconfDestruct () >> #3 0x00007f4d65c2ba4b in main () >> >> Here the full Output of core-file Analysis: >> --------------------------------------------------------------------------------------------- >> >> [root@server1 ~]# gdb /sbin/rsyslogd /var/tmp/core.14983 >> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-90.el6) >> Copyright (C) 2010 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-redhat-linux-gnu". >> For bug reporting instructions, please see: >> <http://www.gnu.org/software/gdb/bugs/>... >> Reading symbols from /sbin/rsyslogd...(no debugging symbols found)...done. >> [New Thread 14983] >> [New Thread 14990] >> [New Thread 14993] >> [New Thread 14992] >> [New Thread 14989] >> Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. >> Loaded symbols for /lib64/libz.so.1 >> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols >> found)...done. >> [Thread debugging using libthread_db enabled] >> Loaded symbols for /lib64/libpthread.so.0 >> Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. >> Loaded symbols for /lib64/libdl.so.2 >> Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. >> Loaded symbols for /lib64/librt.so.1 >> Reading symbols from /usr/lib64/libestr.so.0...(no debugging symbols >> found)...done. >> Loaded symbols for /usr/lib64/libestr.so.0 >> Reading symbols from /usr/lib64/libfastjson.so.3...done. >> Loaded symbols for /usr/lib64/libfastjson.so.3 >> Reading symbols from /lib64/libuuid.so.1...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libuuid.so.1 >> Reading symbols from /usr/lib64/liblogging-stdlog.so.0...(no debugging >> symbols found)...done. >> Loaded symbols for /usr/lib64/liblogging-stdlog.so.0 >> Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libgcc_s.so.1 >> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. >> Loaded symbols for /lib64/libc.so.6 >> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/ld-linux-x86-64.so.2 >> Reading symbols from /lib64/rsyslog/lmnet.so...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/rsyslog/lmnet.so >> Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libnss_files.so.2 >> Reading symbols from /lib64/rsyslog/imuxsock.so...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/rsyslog/imuxsock.so >> Reading symbols from /lib64/rsyslog/imklog.so...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/rsyslog/imklog.so >> Reading symbols from /lib64/rsyslog/impstats.so...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/rsyslog/impstats.so >> Reading symbols from /lib64/rsyslog/imtcp.so...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/rsyslog/imtcp.so >> Reading symbols from /lib64/rsyslog/lmnetstrms.so...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/rsyslog/lmnetstrms.so >> Reading symbols from /lib64/rsyslog/lmtcpsrv.so...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/rsyslog/lmtcpsrv.so >> Reading symbols from /lib64/rsyslog/imudp.so...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/rsyslog/imudp.so >> Reading symbols from /lib64/rsyslog/ommysql.so...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/rsyslog/ommysql.so >> Reading symbols from /usr/lib64/mysql/libmysqlclient.so.16...(no >> debugging symbols found)...done. >> Loaded symbols for /usr/lib64/mysql/libmysqlclient.so.16 >> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libcrypt.so.1 >> Reading symbols from /lib64/libnsl.so.1...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libnsl.so.1 >> Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. >> Loaded symbols for /lib64/libm.so.6 >> Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols >> found)...done. >> Loaded symbols for /usr/lib64/libssl.so.10 >> Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging >> symbols found)...done. >> Loaded symbols for /usr/lib64/libcrypto.so.10 >> Reading symbols from /lib64/libfreebl3.so...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libfreebl3.so >> Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/libgssapi_krb5.so.2 >> Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libkrb5.so.3 >> Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libcom_err.so.2 >> Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libk5crypto.so.3 >> Reading symbols from /lib64/libkrb5support.so.0...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/libkrb5support.so.0 >> Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libkeyutils.so.1 >> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libresolv.so.2 >> Reading symbols from /lib64/libselinux.so.1...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libselinux.so.1 >> Reading symbols from /lib64/rsyslog/omudpspoof.so...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/rsyslog/omudpspoof.so >> Reading symbols from /lib64/libnet.so.1...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libnet.so.1 >> Reading symbols from /lib64/rsyslog/pmlastmsg.so...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/rsyslog/pmlastmsg.so >> Reading symbols from /lib64/rsyslog/pmaixforwardedfrom.so...(no >> debugging symbols found)...done. >> Loaded symbols for /lib64/rsyslog/pmaixforwardedfrom.so >> Reading symbols from /lib64/rsyslog/lmtcpclt.so...(no debugging >> symbols found)...done. >> Loaded symbols for /lib64/rsyslog/lmtcpclt.so >> Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols >> found)...done. >> Loaded symbols for /lib64/libnss_dns.so.2 >> Core was generated by `/sbin/rsyslogd -i /var/run/syslogd.pid'. >> Program terminated with signal 11, Segmentation fault. >> #0 0x00007f4d6558b213 in pthread_join () from /lib64/libpthread.so.0 >> Missing separate debuginfos, use: debuginfo-install >> rsyslog-8.19.0-1.el6.x86_64 >> (gdb) where >> #0 0x00007f4d6558b213 in pthread_join () from /lib64/libpthread.so.0 >> #1 0x00007f4d65c1b857 in lookupDestroyCnf () >> #2 0x00007f4d65bf5142 in rsconfDestruct () >> #3 0x00007f4d65c2ba4b in main () >> --------------------------------------------------------------------------------------------- >> >> Attached the log-file Output from the command: >> >> valgrind --log-file=/var/log/valgrindlog /sbin/rsyslogd -f /etc/rsyslog.conf >> (I'm not sure how to Analysis this correctly as this is a daemon and >> it is just reacting in kill situtation) >> >> regards, >> Chris >> >> OUTPUT: >> --------------------------------------------------------------------------- >> >> ==9342== Memcheck, a memory error detector >> ==9342== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. >> ==9342== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info >> ==9342== Command: /sbin/rsyslogd -f /etc/rsyslog.conf >> ==9342== Parent PID: 30783 >> ==9342== >> ==9342== Conditional jump or move depends on uninitialised value(s) >> ==9342== at 0x16BB54: cnfexprDestruct (in /sbin/rsyslogd) >> ==9342== by 0x1711AD: cnfstmtOptimize (in /sbin/rsyslogd) >> ==9342== by 0x147687: ??? (in /sbin/rsyslogd) >> ==9342== by 0x12F69E: llExecFunc (in /sbin/rsyslogd) >> ==9342== by 0x1474D6: rulesetOptimizeAll (in /sbin/rsyslogd) >> ==9342== by 0x124F8A: load (in /sbin/rsyslogd) >> ==9342== by 0x15A0E8: initAll (in /sbin/rsyslogd) >> ==9342== by 0x15A8A2: main (in /sbin/rsyslogd) >> ==9342== >> ==9344== Warning: invalid file descriptor 131062 in syscall close() >> ==9342== >> ==9342== HEAP SUMMARY: >> ==9342== in use at exit: 225,937 bytes in 3,225 blocks >> ==9342== total heap usage: 9,312 allocs, 6,087 frees, 1,019,793 >> bytes allocated >> ==9342== >> ==9342== LEAK SUMMARY: >> ==9342== definitely lost: 26 bytes in 1 blocks >> ==9342== indirectly lost: 0 bytes in 0 blocks >> ==9342== possibly lost: 608 bytes in 1 blocks >> ==9342== still reachable: 225,303 bytes in 3,223 blocks >> ==9342== suppressed: 0 bytes in 0 blocks >> ==9342== Rerun with --leak-check=full to see details of leaked memory >> ==9342== >> ==9342== For counts of detected and suppressed errors, rerun with: -v >> ==9342== Use --track-origins=yes to see where uninitialised values come from >> ==9342== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 98 from 9) >> --------------------------------------------------------------------------- >> >> regards >> Chris >> >> 2016-06-21 13:23 GMT+02:00 Rainer Gerhards <[email protected]>: >>> 2016-06-21 13:21 GMT+02:00 Christopher Racky <[email protected]>: >>>> Hi, >>>> >>>> On my RHEL 6.8 Servers I get a segfault when I try to restart or stop >>>> rsyslog after some runetime. >>>> >>>> See here: >>>> >>>> [root@server1 log]# rsyslogd -N2 >>>> rsyslogd: version 8.19.0, config validation run (level 2), master >>>> config /etc/rsyslog.conf >>>> rsyslogd: End of config validation run. Bye. >>>> [root@server1 log]# /etc/init.d/rsyslog restart >>>> Shutting down system logger: [ OK ] >>>> Starting system logger: [ OK ] >>>> >>>> [root@server1 log]# dmesg >>>> rsyslogd[16603]: segfault at 7fb717ec79d0 ip 00007fb71d353213 sp >>>> 00007fffc35c2a40 error 4 in libpthread-2.12.so[7fb71d34b000+17000] >>>> >>>> [root@server1 log]# ps -ef | grep rsyslog >>>> root 17729 1 0 13:13 ? 00:00:00 /sbin/rsyslogd -i >>>> /var/run/syslogd.pid >>>> root 17780 13962 0 13:14 pts/1 00:00:00 grep rsyslog >>>> >>>> [root@server1 log]# cat /etc/redhat-release >>>> Red Hat Enterprise Linux Server release 6.8 (Santiago) >>>> >>>> Unfortunately this segfault does not occure each stop... >>>> >>>> This Problems seems to persinst since rsyslog 8.16 >>>> >>>> >>>> My configuration is quite complex with rulesets, lookuptable and the >>>> following modules: >>>> >>>> Module(load="imuxsock") # provides support for local system logging >>>> (e.g. via logger command) >>>> Module(load="imklog") # provides kernel logging support (previously >>>> done by rklogd) >>>> Module (load="impstats" interval="120" severity="7" log.syslog="on" >>>> ruleset="pstats") >>>> Module (load="imtcp" KeepAlive="on" KeepAlive.Probes="1" >>>> KeepAlive.Interval="2" KeepAlive.Time="20" MaxSessions="5000") >>>> Module (load="imudp") >>>> Module (load="ommysql") >>>> Module (load="omudpspoof") >>>> Module (load="pmlastmsg") >>>> Module (load="pmaixforwardedfrom") >>>> >>>> >>>> Do you have any hint or is this a known bug? >>> >>> Can you run it under valgrind control, so that we can see why it >>> aborts? Or do you have a least a coredump, which could tell us at what >>> location it aborted? >>> Rainer >>> >>>> >>>> >>>> regards >>>> Chris >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
