Hello, I'm running rsyslog-8.4.2 (Adiscon package) on CentOS 6.5. The rsyslogd daemon crashes intermittently on a set of our systems.These hosts run apps which write their own log files, which are fed to rsyslog via the IMFILE feature. I've patched the OS (CentOS 6.5) on these hosts with all updates as of a few days ago. Often the daemon won't start, post-crash, unless (1) it's run under strace, or (2) I remove (some of) the /var/log/srvrfwd-*.queue.* files. Perhaps some bug parsing/flushing queue files. I can post the rsyslog configuration if necessary, but there are a crapload of small include .conf files (around 40, to IMFILE various application logs) but nothing really fancy is being done. Here is the package info:
$ rpm -qvi rsyslog Name : rsyslog Relocations: (not relocatable) Version : 8.4.2 Vendor: (none) Release : 1.el6 Build Date: Thu 02 Oct 2014 03:25:33 AM EDT Install Date: Mon 20 Oct 2014 04:59:59 PM EDT Build Host: vmrpm.adiscon.com Group : System Environment/Daemons Source RPM: rsyslog-8.4.2-1.el6.src.rpm Size : 2186241 License: (GPLv3+ and ASL 2.0) Signature : RSA/SHA1, Thu 02 Oct 2014 03:28:29 AM EDT, Key ID e0f233b3e00b8985 URL : http://www.rsyslog.com/ Summary : Enhanced system logging and kernel message trapping daemon Description : Rsyslog is an enhanced, multi-threaded syslog daemon. It supports MySQL, syslog/TCP, RFC 3195, permitted sender lists, filtering on any message part, and fine grain output format control. It is compatible with stock sysklogd and can be used as a drop-in replacement. Rsyslog is simple to set up, with advanced features suitable for enterprise-class, encryption-protected syslog relay chains. Attached is the suggested GDB dump from a core file that was produced by the crash. Any ideas on how to fix this, are appreciated. Thanks! Ray
gdb /sbin/rsyslogd GNU gdb (GDB) Red Hat Enterprise Linux (7.2-64.el6_5.2) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /sbin/rsyslogd...Reading symbols from /usr/lib/debug/sbin/rsyslogd.debug...done. done. (gdb) core /core.27411 [New Thread 11028] [New Thread 27412] [New Thread 27414] [New Thread 27411] [New Thread 27418] [New Thread 27423] [New Thread 27419] [New Thread 27415] [New Thread 27417] [New Thread 27413] [New Thread 27424] [New Thread 27420] [New Thread 27416] [New Thread 27422] [New Thread 27421] [New Thread 11029] Missing separate debuginfo for Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/54/74f0d8daf3d6177e2c4b06f3892745cb43b4d5 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /usr/lib64/libestr.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libestr.so.0 Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libuuid.so.1 Reading symbols from /usr/lib64/liblogging-stdlog.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/liblogging-stdlog.so.0 Reading symbols from /usr/lib64/libjson-c.so.2...done. Loaded symbols for /usr/lib64/libjson-c.so.2 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/rsyslog/lmnet.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/lmnet.so.debug...done. done. Loaded symbols for /lib64/rsyslog/lmnet.so Reading symbols from /lib64/rsyslog/imuxsock.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/imuxsock.so.debug...done. done. Loaded symbols for /lib64/rsyslog/imuxsock.so Reading symbols from /lib64/rsyslog/imklog.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/imklog.so.debug...done. done. Loaded symbols for /lib64/rsyslog/imklog.so Reading symbols from /lib64/rsyslog/immark.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/immark.so.debug...done. done. Loaded symbols for /lib64/rsyslog/immark.so Reading symbols from /lib64/rsyslog/imudp.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/imudp.so.debug...done. done. Loaded symbols for /lib64/rsyslog/imudp.so Reading symbols from /lib64/rsyslog/imtcp.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/imtcp.so.debug...done. done. Loaded symbols for /lib64/rsyslog/imtcp.so Reading symbols from /lib64/rsyslog/lmnetstrms.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/lmnetstrms.so.debug...done. done. Loaded symbols for /lib64/rsyslog/lmnetstrms.so Reading symbols from /lib64/rsyslog/lmtcpsrv.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/lmtcpsrv.so.debug...done. done. Loaded symbols for /lib64/rsyslog/lmtcpsrv.so Reading symbols from /lib64/rsyslog/imfile.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/imfile.so.debug...done. done. Loaded symbols for /lib64/rsyslog/imfile.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/rsyslog/lmtcpclt.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/lmtcpclt.so.debug...done. done. Loaded symbols for /lib64/rsyslog/lmtcpclt.so Reading symbols from /lib64/rsyslog/lmnsd_ptcp.so...Reading symbols from /usr/lib/debug/lib64/rsyslog/lmnsd_ptcp.so.debug...done. done. Loaded symbols for /lib64/rsyslog/lmnsd_ptcp.so Core was generated by `/sbin/rsyslogd -i /var/run/syslogd.pid'. Program terminated with signal 11, Segmentation fault. #0 strmSetWCntr (pThis=0x0, pWCnt=0x7fb908d18c18) at stream.c:1918 1918 pThis->pUsrWCntr = pWCnt; Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.132.el6_5.4.x86_64 json-c-0.11-3.el6.x86_64 libestr-0.1.9-1.el6.x86_64 libgcc-4.4.7-4.el6.x86_64 liblogging-1.0.4-1.el6.x86_64 libuuid-2.17.2-12.14.el6_5.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) info thread 16 Thread 0x7fb9035fe700 (LWP 11029) 0x00007fb914ead5c7 in unlink () from /lib64/libc.so.6 15 Thread 0x7fb90b7fd700 (LWP 27421) 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 14 Thread 0x7fb90b3fc700 (LWP 27422) 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 13 Thread 0x7fb91171a700 (LWP 27416) 0x00007fb914eb9e63 in epoll_wait () from /lib64/libc.so.6 12 Thread 0x7fb90bbfe700 (LWP 27420) 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 11 Thread 0x7fb90971a700 (LWP 27424) 0x00007fb915faa264 in __lll_lock_wait () from /lib64/libpthread.so.0 10 Thread 0x7fb912b1c700 (LWP 27413) 0x00007fb915faa75d in read () from /lib64/libpthread.so.0 9 Thread 0x7fb903fff700 (LWP 27417) 0x00007fb914eb22e3 in select () from /lib64/libc.so.6 8 Thread 0x7fb91211b700 (LWP 27415) 0x00007fb914eb9e63 in epoll_wait () from /lib64/libc.so.6 7 Thread 0x7fb90bfff700 (LWP 27419) 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 6 Thread 0x7fb90affb700 (LWP 27423) 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 5 Thread 0x7fb910d19700 (LWP 27418) 0x00007fb915faa264 in __lll_lock_wait () from /lib64/libpthread.so.0 4 Thread 0x7fb9165dd740 (LWP 27411) 0x00007fb914eb22e3 in select () from /lib64/libc.so.6 3 Thread 0x7fb90a11b700 (LWP 27414) 0x00007fb914eb22e3 in select () from /lib64/libc.so.6 2 Thread 0x7fb91351d700 (LWP 27412) 0x00007fb914eb22e3 in select () from /lib64/libc.so.6 * 1 Thread 0x7fb908d19700 (LWP 11028) strmSetWCntr (pThis=0x0, pWCnt=0x7fb908d18c18) at stream.c:1918 (gdb) thread apply all bt full Thread 16 (Thread 0x7fb9035fe700 (LWP 11029)): #0 0x00007fb914ead5c7 in unlink () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fb91663a843 in strmCloseFile (pThis=0x7fb917166300) at stream.c:444 currOffs = <value optimized out> iRet = RS_RET_OK #2 0x00007fb91663ad73 in strmDestruct (ppThis=0x7fb917163c60) at stream.c:919 pThis = 0x7fb917166300 i = <value optimized out> #3 0x00007fb91663d895 in qDestructDisk (pThis=0x7fb9171639e0) at queue.c:918 No locals. #4 0x00007fb9166425f6 in queueSwitchToEmergencyMode (pThis=0x7fb9171639e0, pWti=<value optimized out>) at queue.c:711 No locals. #5 ConsumerReg (pThis=0x7fb9171639e0, pWti=<value optimized out>) at queue.c:1880 iCancelStateSave = <value optimized out> bNeedReLock = 0 iRet = RS_RET_FILE_NOT_FOUND #6 0x00007fb91663cb52 in wtiWorker (pThis=0x7fb917167780) at wti.c:334 __clframe = {__cancel_routine = 0x7fb91663c920 <wtiWorkerCancelCleanup>, __cancel_arg = 0x7fb917167780, __do_it = 1, __cancel_type = 0} pWtp = 0x7fb917167630 pAction = <value optimized out> bInactivityTOOccured = 0 localRet = <value optimized out> terminateRet = RS_RET_TERMINATE_WHEN_IDLE wrkrInfo = <value optimized out> iCancelStateSave = <value optimized out> i = <value optimized out> j = <value optimized out> k = <value optimized out> #7 0x00007fb91663c6a2 in wtpWorker (arg=0x7fb917167780) at wtp.c:390 __clframe = {__cancel_routine = 0x7fb91663c750 <wtpWrkrExecCancelCleanup>, __cancel_arg = 0x7fb917167780, __do_it = 1, __cancel_type = 0} pThis = 0x7fb917167630 sigSet = {__val = {1048576, 0 <repeats 15 times>}} ---Type <return> to continue, or q <return> to quit--- pszDbgHdr = <value optimized out> thrdName = "rs:action 2 queue[DA]:R\000\000\000\000\000\000\000\000" #8 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #9 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 15 (Thread 0x7fb90b7fd700 (LWP 27421)): #0 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fb913d4193f in wrkr (myself=0x7fb913f44c10) at tcpsrv.c:632 me = 0x7fb913f44c10 #2 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 14 (Thread 0x7fb90b3fc700 (LWP 27422)): #0 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fb913d4193f in wrkr (myself=0x7fb913f44c78) at tcpsrv.c:632 me = 0x7fb913f44c78 #2 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 13 (Thread 0x7fb91171a700 (LWP 27416)): #0 0x00007fb914eb9e63 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fb913521096 in Wait (pNsdpoll=0x7fb8f0002610, timeout=-1, numEntries=0x7fb91171992c, workset=0x7fb911719110) at nsdpoll_ptcp.c:242 pThis = 0x7fb8f0002610 pOurEvt = <value optimized out> event = {{events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 50 times>, {events = 0, data = {ptr = 0x2800000000, fd = 0, u32 = 0, u64 = 171798691840}}, {events = 48, data = { ---Type <return> to continue, or q <return> to quit--- ptr = 0x7fb911719080, fd = 292655232, u32 = 292655232, u64 = 140432838332544}}, {events = 292655040, data = {ptr = 0x7fb9, fd = 32697, u32 = 32697, u64 = 32697}}, {events = 0, data = { ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 11 times>, {events = 3, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, { events = 80, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x300000000, fd = 0, u32 = 0, u64 = 12884901888}}, {events = 48, data = {ptr = 0x7c0000005b, fd = 91, u32 = 91, u64 = 532575944795}}, {events = 119, data = {ptr = 0x10000006e, fd = 110, u32 = 110, u64 = 4294967406}}, {events = 0, data = {ptr = 0x1, fd = 1, u32 = 1, u64 = 1}}, {events = 133552, data = {ptr = 0xf000265000000000, fd = 0, u32 = 0, u64 = 17293864694141943808}}, { events = 32696, data = {ptr = 0x28, fd = 40, u32 = 40, u64 = 40}}, {events = 4026531872, data = {ptr = 0x15160e8000007fb8, fd = 32696, u32 = 32696, u64 = 1519417867202756536}}, { events = 32697, data = {ptr = 0x7fb914e4b346, fd = 350532422, u32 = 350532422, u64 = 140432896209734}}, {events = 292655248, data = {ptr = 0x100007fb9, fd = 32697, u32 = 32697, u64 = 4294999993}}, {events = 0, data = {ptr = 0x7fb9170d1fb0, fd = 386736048, u32 = 386736048, u64 = 140432932413360}}, {events = 386946784, data = {ptr = 0xf000261000007fb9, fd = 32697, u32 = 32697, u64 = 17293864419264069561}}, {events = 32696, data = {ptr = 0x7fb916882000, fd = 378019840, u32 = 378019840, u64 = 140432923697152}}, {events = 1, data = { ptr = 0x1352133c00000000, fd = 0, u32 = 0, u64 = 1392196383229870080}}, {events = 32697, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x7fb8f0002630, fd = -268425680, u32 = 4026541616, u64 = 140432277251632}}, {events = 0, data = {ptr = 0x4, fd = 4, u32 = 4, u64 = 4}}, { events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, { events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0xffffffff00000000, fd = 0, u32 = 0, u64 = 18446744069414584320}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x1515e44000000000, fd = 0, u32 = 0, u64 = 1519371412836450304}}, { events = 32697, data = {ptr = 0x3, fd = 3, u32 = 3, u64 = 3}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x50, fd = 80, u32 = 80, u64 = 80}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x2, fd = 2, u32 = 2, u64 = 2}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, { events = 0, data = {ptr = 0x40, fd = 64, u32 = 64, u64 = 64}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, { events = 2, data = {ptr = 0x5b00000030, fd = 48, u32 = 48, u64 = 390842023984}}, {events = 124, data = {ptr = 0x6e00000077, fd = 119, u32 = 119, u64 = 472446402679}}, {events = 5, data = {ptr = 0x100007fb9, fd = 32697, u32 = 32697, u64 = 4294999993}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 112, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x500000000, fd = 0, u32 = 0, u64 = 21474836480}}, {events = 49, data = { ptr = 0x7c0000005b, fd = 91, u32 = 91, u64 = 532575944795}}, {events = 119, data = {ptr = 0x117190380000006e, fd = 110, u32 = 110, u64 = 1256944341205647470}}, {events = 32697, data = { ptr = 0x7fb8f0002550, fd = -268425904, u32 = 4026541392, u64 = 140432277251408}}, {events = 133632, data = {ptr = 0xf000260000000000, fd = 0, u32 = 0, u64 = 17293864350544560128}}, { events = 32696, data = {ptr = 0x48, fd = 72, u32 = 72, u64 = 72}}, {events = 4026531872, data = {ptr = 0x15160e8000007fb8, fd = 32696, u32 = 32696, u64 = 1519417867202756536}}, { events = 32697, data = {ptr = 0x7fb914e4b346, fd = 350532422, u32 = 350532422, u64 = 140432896209734}}, {events = 334787609, data = {ptr = 0xf000255000007fb9, fd = 32697, u32 = 32697, u64 = 17293863594630348729}}, {events = 32696, data = {ptr = 0x7fb8f0002610, fd = -268425712, u32 = 4026541584, u64 = 140432277251600}}, {events = 324146427, data = { ptr = 0x11719d3000007fb9, fd = 32697, u32 = 32697, u64 = 1256958600497102777}}, {events = 32697, data = {ptr = 0x7fb8f0002550, fd = -268425904, u32 = 4026541392, u64 = 140432277251408}}, {events = 4026541584, data = {ptr = 0x135215c800007fb8, fd = 32696, u32 = 32696, u64 = 1392199183548579768}}, {events = 32697, data = {ptr = 0x7fb911719d30, fd = 292658480, u32 = 292658480, u64 = 140432838335792}}, {events = 4026541376, data = {ptr = 0x1171909000007fb8, fd = 32696, u32 = 32696, u64 = 1256944719162802104}}, {events = 32697, data = {ptr = 0x7fb913f47232, fd = 334787122, u32 = 334787122, u64 = 140432880464434}}, {events = 353767040, data = {ptr = 0x14e4b46200007fb9, fd = 32697, u32 = 32697, u64 = 1505526508448415673}}, {events = 32697, data = {ptr = 0x6c6f7064736e6d6c, fd = 1936616812, u32 = 1936616812, u64 = 7813587455246757228}}, {events = 1953521516, data = { ptr = 0x170d13e000007063, fd = 28771, u32 = 28771, u64 = 1661005690363211875}}, {events = 32697, data = {ptr = 0xf14e1cce485d3524, fd = 1214068004, u32 = 1214068004, u64 = 17387866883626972452}}} nfds = <value optimized out> i = <value optimized out> ---Type <return> to continue, or q <return> to quit--- iRet = RS_RET_OK #2 0x00007fb913d415fd in Run (pThis=0x7fb9170d13e0) at tcpsrv.c:889 iRet = RS_RET_OK i = <value optimized out> workset = {{id = 0, pUsr = 0x0} <repeats 128 times>} numEntries = 128 pPoll = 0x7fb8f0002540 localRet = <value optimized out> #3 0x00007fb91664baa3 in thrdStarter (arg=0x7fb91716a950) at ../threads.c:212 iRet = RS_RET_OK thrdName = "in:imtcp", '\000' <repeats 23 times> sigSet = {__val = {1048576, 0 <repeats 15 times>}} #4 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 12 (Thread 0x7fb90bbfe700 (LWP 27420)): #0 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fb913d4193f in wrkr (myself=0x7fb913f44ba8) at tcpsrv.c:632 me = 0x7fb913f44ba8 #2 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 11 (Thread 0x7fb90971a700 (LWP 27424)): #0 0x00007fb915faa264 in __lll_lock_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fb915fa5508 in _L_lock_854 () from /lib64/libpthread.so.0 No symbol table info available. #2 0x00007fb915fa53d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 No symbol table info available. ---Type <return> to continue, or q <return> to quit--- #3 0x00007fb91664252e in ConsumerReg (pThis=0x7fb9170f96e0, pWti=<value optimized out>) at queue.c:1918 iCancelStateSave = 1 bNeedReLock = 1 iRet = RS_RET_OK #4 0x00007fb91663cb52 in wtiWorker (pThis=0x7fb917163100) at wti.c:334 __clframe = {__cancel_routine = 0x7fb91663c920 <wtiWorkerCancelCleanup>, __cancel_arg = 0x7fb917163100, __do_it = 1, __cancel_type = 0} pWtp = 0x7fb917162fb0 pAction = <value optimized out> bInactivityTOOccured = 0 localRet = <value optimized out> terminateRet = RS_RET_OK wrkrInfo = <value optimized out> iCancelStateSave = <value optimized out> i = <value optimized out> j = <value optimized out> k = <value optimized out> #5 0x00007fb91663c6a2 in wtpWorker (arg=0x7fb917163100) at wtp.c:390 __clframe = {__cancel_routine = 0x7fb91663c750 <wtpWrkrExecCancelCleanup>, __cancel_arg = 0x7fb917163100, __do_it = 1, __cancel_type = 0} pThis = 0x7fb917162fb0 sigSet = {__val = {1048576, 0 <repeats 15 times>}} pszDbgHdr = <value optimized out> thrdName = "rs:action 2 queue:Reg\000\000\000\000\000\000\000\000\000\000" #6 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #7 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 10 (Thread 0x7fb912b1c700 (LWP 27413)): #0 0x00007fb915faa75d in read () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fb91475c8aa in read (pModConf=0x7fb9170d4ff0) at /usr/include/bits/unistd.h:45 No locals. #2 readklog (pModConf=0x7fb9170d4ff0) at bsd.c:230 p = <value optimized out> ---Type <return> to continue, or q <return> to quit--- q = <value optimized out> len = 0 i = <value optimized out> iMaxLine = 8096 bufRcv = '\000' <repeats 131072 times> errmsg = '\000' <repeats 1888 times>, "f\000\000\000\000\000\000\000^\000\000\000\000\000\000\000\002", '\000' <repeats 23 times>, "@", '\000' <repeats 31 times>, "\002\000\000\000\060\000\000\000[\000\000\000|\000\000\000w\000\000\000n\000\000\000\300%\210\026\271\177\000\000\000\000\000\000\000\000\000\000wh\372\025\271\177\000\000 \000\000\004\271\177\000\000\300ɱ\022\271\177\000\000\000\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000\261\266\344\024\271\177\000" pRcv = 0x7fb912afb530 "" #3 0x00007fb91475ca29 in klogLogKMsg (pModConf=<value optimized out>) at bsd.c:288 No locals. #4 0x00007fb91475c043 in runInput (pThrd=0x7fb91716a350) at imklog.c:282 iRet = RS_RET_OK #5 0x00007fb91664baa3 in thrdStarter (arg=0x7fb91716a350) at ../threads.c:212 iRet = RS_RET_OK thrdName = "in:imklog", '\000' <repeats 22 times> sigSet = {__val = {1048576, 0 <repeats 15 times>}} #6 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #7 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 9 (Thread 0x7fb903fff700 (LWP 27417)): #0 0x00007fb914eb22e3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fb916631053 in srSleep (iSeconds=<value optimized out>, iuSeconds=<value optimized out>) at srutils.c:493 tvSelectTimeout = {tv_sec = 9, tv_usec = 951049} #2 0x00007fb913b3953e in doPolling (pThrd=<value optimized out>) at imfile.c:964 i = <value optimized out> bHadFileData = 0 #3 runInput (pThrd=<value optimized out>) at imfile.c:1364 iRet = RS_RET_OK #4 0x00007fb91664baa3 in thrdStarter (arg=0x7fb91716ab50) at ../threads.c:212 iRet = RS_RET_OK ---Type <return> to continue, or q <return> to quit--- thrdName = "in:imfile", '\000' <repeats 22 times> sigSet = {__val = {1048576, 0 <repeats 15 times>}} #5 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 8 (Thread 0x7fb91211b700 (LWP 27415)): #0 0x00007fb914eb9e63 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fb9143523a4 in rcvMainLoop (pWrkr=0x7fb9145556e0) at imudp.c:786 iRet = RS_RET_OK nfds = <value optimized out> efd = 8 i = <value optimized out> frominetPrev = {ss_family = 0, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>} bIsPermitted = 0 udpEPollEvt = 0x7fb8f8002230 currEvt = {{events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = {ptr = 0x7fb91211aa60, fd = 303147616, u32 = 303147616, u64 = 140432848824928}}, {events = 303147600, data = {ptr = 0x800007fb9, fd = 32697, u32 = 32697, u64 = 34359771065}}, {events = 0, data = {ptr = 0x7fb91211abd0, fd = 303147984, u32 = 303147984, u64 = 140432848825296}}, {events = 339030162, data = {ptr = 0x14e1517700007fb9, fd = 32697, u32 = 32697, u64 = 1504573322061447097}}, {events = 32697, data = {ptr = 0x7fb914353093, fd = 339030163, u32 = 339030163, u64 = 140432884707475}}} errStr = '\000' <repeats 311 times>, "0", '\000' <repeats 40 times>, "(\000\000\000\060\000\000\000\260\254\021\022\271\177\000\000\360\253\021\022\271\177", '\000' <repeats 130 times>"\266, \202=\026\271\177\000\000\000\000\000\000\000\000\000\000\220\252\021\022\271\177\000\000\000\000\000\000\000\000\000\000\220\252\021\022\271\177\000\000\f\000\000\000\000\000\000\000\300\v^\026\271\177\000\000\207\360\226|\000\000\000\000`\252\021\022\271\177\000\000\300\254\021\022\271\177\000\000P\253\021\022\271\177\000\000\001\000\000\000\000\000\000\000Ы\021\022\271\177\000\000\037\000\000\000\000\000\000\000\211\060\065\024\271\177\000\000\300\254\021\022\271\177\000\000\240\n\355\024\271\177\000\000\001\200\255\373\000\000\000\000\300\254\021\022\271\177\000\000\300\254\021\022\271\177\000\000\300\254\021\022\271\177\000\000\300\254\021\022\271\177\000\000ɬ\021\022\271\177\000\000߬\021\022\271\177\000\000\300\254\021\022\271\177\000\000߬\021\022\271\177", '\000' <repeats 42 times>... lstn = <value optimized out> nLstn = <value optimized out> #2 0x00007fb914352bb2 in wrkr (myself=0x7fb9145556e0) at imudp.c:1172 pWrkr = 0x7fb9145556e0 thrdName = "imudp(w0)\000\000\000\000\000\000\000\340VU\024\271\177\000\000\000\000@\000\000\000\000" ---Type <return> to continue, or q <return> to quit--- #3 0x00007fb914352cea in runInput (pThrd=0x7fb91716a750) at imudp.c:1197 i = <value optimized out> wrkrThrdAttr = {__size = '\000' <repeats 17 times>, "\020", '\000' <repeats 16 times>, "@", '\000' <repeats 20 times>, __align = 0} #4 0x00007fb91664baa3 in thrdStarter (arg=0x7fb91716a750) at ../threads.c:212 iRet = RS_RET_OK thrdName = "in:imudp", '\000' <repeats 23 times> sigSet = {__val = {1048576, 0 <repeats 15 times>}} #5 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #6 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 7 (Thread 0x7fb90bfff700 (LWP 27419)): #0 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fb913d4193f in wrkr (myself=0x7fb913f44b40) at tcpsrv.c:632 me = 0x7fb913f44b40 #2 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 6 (Thread 0x7fb90affb700 (LWP 27423)): #0 0x00007fb915fa75bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fb91663cbc5 in doIdleProcessing (pThis=0x7fb9170f24c0) at wti.c:266 t = {tv_sec = 0, tv_nsec = 0} #2 wtiWorker (pThis=0x7fb9170f24c0) at wti.c:344 __clframe = {__cancel_routine = 0x7fb91663c920 <wtiWorkerCancelCleanup>, __cancel_arg = 0x7fb9170f24c0, __do_it = 1, __cancel_type = 0} pWtp = 0x7fb9170f2370 pAction = <value optimized out> bInactivityTOOccured = 0 localRet = <value optimized out> terminateRet = <value optimized out> ---Type <return> to continue, or q <return> to quit--- wrkrInfo = <value optimized out> iCancelStateSave = <value optimized out> i = <value optimized out> j = <value optimized out> k = <value optimized out> #3 0x00007fb91663c6a2 in wtpWorker (arg=0x7fb9170f24c0) at wtp.c:390 __clframe = {__cancel_routine = 0x7fb91663c750 <wtpWrkrExecCancelCleanup>, __cancel_arg = 0x7fb9170f24c0, __do_it = 1, __cancel_type = 0} pThis = 0x7fb9170f2370 sigSet = {__val = {1048576, 0 <repeats 15 times>}} pszDbgHdr = <value optimized out> thrdName = "rs:action 1 queue:Reg\000\000\000\000\000\000\000\000\000\000" #4 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 5 (Thread 0x7fb910d19700 (LWP 27418)): #0 0x00007fb915faa264 in __lll_lock_wait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007fb915fa5508 in _L_lock_854 () from /lib64/libpthread.so.0 No symbol table info available. #2 0x00007fb915fa53d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 No symbol table info available. #3 0x00007fb9166416fe in qqueueEnqMsg (pThis=0x7fb9170f96e0, flowCtlType=eFLOWCTL_NO_DELAY, pMsg=<value optimized out>) at queue.c:2854 iRet = RS_RET_OK iCancelStateSave = <value optimized out> #4 0x00007fb91664afcc in doSubmitToActionQ (pAction=0x7fb9170ea7d0, pWti=0x7fb917168620, pMsg=<value optimized out>) at ../action.c:1402 ttNow = {timeType = 32 ' ', month = -122 '\206', day = 22 '\026', hour = 23 '\027', minute = -71 '\271', second = 127 '\177', secfracPrecision = 0 '\000', OffsetMinute = 0 '\000', OffsetHour = 0 '\000', OffsetMode = 32 ' ', year = 5768, secfrac = 32697} iRet = RS_RET_OK #5 0x00007fb916643086 in execAct (root=<value optimized out>, pMsg=0x7fb8f4267520, pWti=0x7fb917168620) at ruleset.c:197 iRet = RS_RET_OK #6 scriptExec (root=<value optimized out>, pMsg=0x7fb8f4267520, pWti=0x7fb917168620) at ruleset.c:424 stmt = 0x7fb9170e0c60 ---Type <return> to continue, or q <return> to quit--- iRet = RS_RET_OK #7 0x00007fb9166436ac in processBatch (pBatch=0x7fb917168650, pWti=0x7fb917168620) at ruleset.c:475 i = <value optimized out> pMsg = 0x7fb8f4267520 pRuleset = <value optimized out> #8 0x00007fb916604fcc in msgConsumer (notNeeded=<value optimized out>, pBatch=0x7fb917168650, pWti=0x7fb917168620) at rsyslogd.c:342 i = <value optimized out> #9 0x00007fb916642583 in ConsumerReg (pThis=0x7fb917168120, pWti=0x7fb917168620) at queue.c:1897 iCancelStateSave = <value optimized out> bNeedReLock = 1 iRet = RS_RET_OK #10 0x00007fb91663cb52 in wtiWorker (pThis=0x7fb917168620) at wti.c:334 __clframe = {__cancel_routine = 0x7fb91663c920 <wtiWorkerCancelCleanup>, __cancel_arg = 0x7fb917168620, __do_it = 1, __cancel_type = 0} pWtp = 0x7fb9171684d0 pAction = <value optimized out> bInactivityTOOccured = 0 localRet = <value optimized out> terminateRet = RS_RET_OK wrkrInfo = <value optimized out> iCancelStateSave = <value optimized out> i = <value optimized out> j = <value optimized out> k = <value optimized out> #11 0x00007fb91663c6a2 in wtpWorker (arg=0x7fb917168620) at wtp.c:390 __clframe = {__cancel_routine = 0x7fb91663c750 <wtpWrkrExecCancelCleanup>, __cancel_arg = 0x7fb917168620, __do_it = 1, __cancel_type = 0} pThis = 0x7fb9171684d0 sigSet = {__val = {1048576, 0 <repeats 15 times>}} pszDbgHdr = <value optimized out> thrdName = "rs:main Q:Reg", '\000' <repeats 18 times> #12 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #13 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. ---Type <return> to continue, or q <return> to quit--- Thread 4 (Thread 0x7fb9165dd740 (LWP 27411)): #0 0x00007fb914eb22e3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fb916606c9c in mainloop (argc=<value optimized out>, argv=0x7fffdc0e3cc8) at rsyslogd.c:1196 tvSelectTimeout = {tv_sec = 161, tv_usec = 326665} #2 main (argc=<value optimized out>, argv=0x7fffdc0e3cc8) at rsyslogd.c:1321 No locals. Thread 3 (Thread 0x7fb90a11b700 (LWP 27414)): #0 0x00007fb914eb22e3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fb916631053 in srSleep (iSeconds=<value optimized out>, iuSeconds=<value optimized out>) at srutils.c:493 tvSelectTimeout = {tv_sec = 159, tv_usec = 30058} #2 0x00007fb9145580b2 in runInput (pThrd=<value optimized out>) at immark.c:190 No locals. #3 0x00007fb91664baa3 in thrdStarter (arg=0x7fb91716a550) at ../threads.c:212 iRet = RS_RET_OK thrdName = "in:immark", '\000' <repeats 22 times> sigSet = {__val = {1048576, 0 <repeats 15 times>}} #4 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 2 (Thread 0x7fb91351d700 (LWP 27412)): #0 0x00007fb914eb22e3 in select () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fb9149759e3 in runInput (pThrd=<value optimized out>) at imuxsock.c:1412 iRet = RS_RET_OK maxfds = <value optimized out> nfds = <value optimized out> i = <value optimized out> fd = <value optimized out> pReadfds = 0x7fb90c0008c0 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fb91664baa3 in thrdStarter (arg=0x7fb91716a150) at ../threads.c:212 iRet = RS_RET_OK thrdName = "in:imuxsock", '\000' <repeats 20 times> sigSet = {__val = {1048576, 0 <repeats 15 times>}} #3 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. Thread 1 (Thread 0x7fb908d19700 (LWP 11028)): #0 strmSetWCntr (pThis=0x0, pWCnt=0x7fb908d18c18) at stream.c:1918 No locals. #1 0x00007fb91663e418 in qAddDisk (pThis=0x7fb9171639e0, pMsg=0x7fb8f40b2ed0) at queue.c:930 iRet = RS_RET_OK nWriteCount = <value optimized out> #2 0x00007fb91663e886 in qqueueAdd (pThis=0x7fb9171639e0, flowCtlType=<value optimized out>, pMsg=0x7fb8f40b2ed0) at queue.c:1034 iRet = RS_RET_OK #3 doEnqSingleObj (pThis=0x7fb9171639e0, flowCtlType=<value optimized out>, pMsg=0x7fb8f40b2ed0) at queue.c:2770 iRet = <value optimized out> err = <value optimized out> t = {tv_sec = 0, tv_nsec = 140432933025480} #4 0x00007fb91664170d in qqueueEnqMsg (pThis=0x7fb9171639e0, flowCtlType=eFLOWCTL_NO_DELAY, pMsg=<value optimized out>) at queue.c:2857 iRet = RS_RET_OK iCancelStateSave = <value optimized out> #5 0x00007fb916642386 in ConsumerDA (pThis=0x7fb9170f96e0, pWti=0x7fb917163610) at queue.c:1955 i = <value optimized out> iCancelStateSave = <value optimized out> bNeedReLock = 1 iRet = <value optimized out> #6 0x00007fb91663cb52 in wtiWorker (pThis=0x7fb917163610) at wti.c:334 __clframe = {__cancel_routine = 0x7fb91663c920 <wtiWorkerCancelCleanup>, __cancel_arg = 0x7fb917163610, __do_it = 1, __cancel_type = 0} pWtp = 0x7fb9171634c0 pAction = <value optimized out> bInactivityTOOccured = 0 ---Type <return> to continue, or q <return> to quit--- localRet = <value optimized out> terminateRet = RS_RET_OK wrkrInfo = <value optimized out> iCancelStateSave = <value optimized out> i = <value optimized out> j = <value optimized out> k = <value optimized out> #7 0x00007fb91663c6a2 in wtpWorker (arg=0x7fb917163610) at wtp.c:390 __clframe = {__cancel_routine = 0x7fb91663c750 <wtpWrkrExecCancelCleanup>, __cancel_arg = 0x7fb917163610, __do_it = 1, __cancel_type = 0} pThis = 0x7fb9171634c0 sigSet = {__val = {1048576, 0 <repeats 15 times>}} pszDbgHdr = <value optimized out> thrdName = "rs:action 2 queue:DAwpo\000\000\000\000\000\000\000\000" #8 0x00007fb915fa39d1 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #9 0x00007fb914eb986d in clone () from /lib64/libc.so.6 No symbol table info available. (gdb) (gdb) quit
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
