Philip, the config looks relatively simple, that's good (so far I had more complex configs to work with). Can you tell me a bit about the workload you process. Just the regular stuff? Do you have the impression that this bug is triggered by some specific messages?
Thanks, Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Philip M. Gollucci > Sent: Thursday, February 03, 2011 7:56 AM > To: rsyslog-users > Subject: [rsyslog] rsyslog5 core dumps a-plenty > > Rainer (All), > > At first I thought it was something in rsyslog 5.5.6+ that changed b/c > I > couldn't get the core dumps happen with 5.5.5; however, now I'm pretty > sure it has nothing to do with that milestone. Unfortunately, from the > looks of it, I'm betting its a FreeBSD specific threading issue. Below > is the useful info, gdb back trace, rsyslog.conf, and the build > transcript. > > I've also uploaded the files here for easier viewing: > http://people.freebsd.org/~pgollucci/rsyslog-core > http://people.apache.org/~pgollucci/rsyslog-core > > Move info is available on request. > > > > $ uname -a > FreeBSD hulk 8.1-RELEASE FreeBSD 8.1-RELEASE #0 r212058: Wed Sep 1 > 23:12:18 UTC 2010 root@elektra > :/usr/obj/usr/src/sys/GENERIC amd64 > > $ rsyslog --version > rsyslogd 5.6.2, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: No > FEATURE_NETZIP (message compression): Yes > GSSAPI Kerberos 5 support: No > FEATURE_DEBUG (debug build, slow code): Yes > Atomic operations supported: Yes > Runtime Instrumentation (slow code): Yes > > $ sudo /usr/local/etc/rc.d/rsyslogd restart > rsyslogd not running? (check /var/run/syslog.pid). > Starting rsyslogd. > > $ ps -auxwww |grep rsyslogd > root 9135 6.1 0.0 25524 4388 1 S 6:37AM 0:01.29 > /usr/local/sbin/rsyslogd -c4 -i /var > /run/syslog.pid -f /usr/local/etc/rsyslog.conf > > $ cd / ; ls core.rsyslogd.8* > -rw------- 1 root wheel 10354688 Feb 3 06:25 core.rsyslogd.8742 > -rw------- 1 root wheel 8257536 Feb 3 06:26 core.rsyslogd.8782 > -rw------- 1 root wheel 8257536 Feb 3 06:27 core.rsyslogd.8826 > -rw------- 1 root wheel 8257536 Feb 3 06:28 core.rsyslogd.8844 > -rw------- 1 root wheel 10354688 Feb 3 06:29 core.rsyslogd.8875 > -rw------- 1 root wheel 8257536 Feb 3 06:29 core.rsyslogd.8900 > > $ gdb /usr/local/sbin/rsyslogd core.rsyslogd.81742 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and > you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "amd64-marcel-freebsd"... > Core was generated by `rsyslogd'. > Program terminated with signal 11, Segmentation fault. > Reading symbols from /lib/libz.so.5...done. > Loaded symbols for /lib/libz.so.5 > Reading symbols from /lib/libthr.so.3...done. > Loaded symbols for /lib/libthr.so.3 > Reading symbols from /lib/libc.so.7...done. > Loaded symbols for /lib/libc.so.7 > Reading symbols from /usr/local/lib/rsyslog/lmnet.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmnet.so > Reading symbols from /usr/local/lib/rsyslog/immark.so...done. > Loaded symbols for /usr/local/lib/rsyslog/immark.so > Reading symbols from /usr/local/lib/rsyslog/imudp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imudp.so > Reading symbols from /usr/local/lib/rsyslog/imtcp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imtcp.so > Reading symbols from /usr/local/lib/rsyslog/lmnetstrms.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmnetstrms.so > Reading symbols from /usr/local/lib/rsyslog/lmtcpsrv.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmtcpsrv.so > Reading symbols from /usr/local/lib/rsyslog/imuxsock.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imuxsock.so > Reading symbols from /usr/local/lib/rsyslog/imklog.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imklog.so > Reading symbols from /usr/local/lib/rsyslog/imrelp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imrelp.so > Reading symbols from /usr/local/lib/librelp.so.0...done. > Loaded symbols for /usr/local/lib/librelp.so.0 > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x00000008007e3876 in pthread_mutex_destroy () from > /lib/libthr.so.3 > [New Thread 801a3ac80 (LWP 100568)] > [New Thread 800c87fc0 (LWP 100567)] > [New Thread 800c88180 (LWP 100541)] > [New Thread 800c88340 (LWP 100540)] > [New Thread 800c88500 (LWP 100539)] > [New Thread 800c886c0 (LWP 100538)] > [New Thread 800c88880 (LWP 100379)] > [New Thread 800c041c0 (LWP 100548)] > (gdb) bt full > #0 0x00000008007e3876 in pthread_mutex_destroy () from > /lib/libthr.so.3 > No symbol table info available. > #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) > at msg.c:624 > __func__ = "MsgDeleteMutexLockingCase" > #2 0x00000000004218d1 in msgDestruct (ppThis=0x7ffffeff8d70) at > msg.c:839 > dbgCALLStaCK_POP_POINT = 7 > iRet = RS_RET_OK > iCancelStateSave = 1 > pThis = (msg_t *) 0x801aaea00 > currRefCount = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a12500 > __func__ = "msgDestruct" > #3 0x000000000044fbe1 in DeleteProcessedBatch (pThis=0x800c04380, > pBatch=0x800c2fe70) at queue.c:1404 > i = 0 > pUsr = (void *) 0x801aaea00 > nEnqueued = 0 > localRet = 8 > dbgCALLStaCK_POP_POINT = 6 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13500 > __func__ = "DeleteProcessedBatch" > #4 0x000000000044f828 in DequeueConsumableElements (pThis=0x800c04380, > pWti=0x800c2fe40, piRemainingQueueSize=0x7ffffeff8e14) at queue.c:1441 > nDequeued = 8 > nDiscarded = 4678208 > nDeleted = 2 > iQueueSize = 4673262 > pUsr = (void *) 0x40042df73 > localRet = RS_RET_OK > dbgCALLStaCK_POP_POINT = 5 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13600 > __func__ = "DequeueConsumableElements" > #5 0x000000000044f711 in DequeueConsumable (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1489 > dbgCALLStaCK_POP_POINT = 4 > iRet = RS_RET_OK > iQueueSize = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13700 > __func__ = "DequeueConsumable" > #6 0x0000000000450fca in DequeueForConsumer (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1626 > dbgCALLStaCK_POP_POINT = 3 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13800 > __func__ = "DequeueForConsumer" > #7 0x0000000000450c6b in ConsumerReg (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1679 > iCancelStateSave = 8 > dbgCALLStaCK_POP_POINT = 2 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13900 > __func__ = "ConsumerReg" > #8 0x0000000000443b21 in wtiWorker (pThis=0x800c2fe40) at wti.c:315 > __cleanup_info__ = {pthread_cleanup_pad = {140737471549248, > 4470256, 34372517440, 0, 4667374, 5824688, 0, 34368002717}} > pWtp = (wtp_t *) 0x800c1c380 > bInactivityTOOccured = 0 > localRet = RS_RET_OK > terminateRet = RS_RET_OK > iCancelStateSave = 0 > dbgCALLStaCK_POP_POINT = 1 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13d00 > __func__ = "wtiWorker" > #9 0x0000000000442019 in wtpWorker (arg=0x800c2fe40) at wtp.c:381 > __cleanup_info__ = {pthread_cleanup_pad = {0, 4462880, > 34372517440, 0, 0, 0, 0, 0}} > pWti = (wti_t *) 0x800c2fe40 > pThis = (wtp_t *) 0x800c1c380 > sigSet = {__bits = {1048576, 0, 0, 0}} > dbgCALLStaCK_POP_POINT = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13e00 > __func__ = "wtpWorker" > #10 0x00000008007de511 in pthread_getprio () from /lib/libthr.so.3 > No symbol table info available. > #11 0x0000000000000000 in ?? () > No symbol table info available. > Cannot access memory at address 0x7ffffeff9000 > (gdb) frame 1 > #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) > at msg.c:624 > 624 msg.c: No such file or directory. > in msg.c > (gdb) info locals > __func__ = "MsgDeleteMutexLockingCase" > (gdb) p *pThis > $1 = {objData = {pObjInfo = 0x800c16400, iObjCooCKiE = 195948526, > pszName = 0x0}, flowCtlType = eFLOWCTL_NO_DELAY, mut = > 0xffffffffffffffff, iRefCount = 0, bDoLock = 0 '\0', iSeverity = 3, > iFacility = 22, of > fAfterPRI = 0, offMSG = 91, iProtocolVersion = 0, msgFlags = 32, > iLenRawMsg = 121, iLenMSG = 30, iLenTAG = 31, iLenHOSTNAME = 21, > pszRawMsg = 0x801ab5600 "<179>2011-02-03T06:20:37.681910+00:00 phoenix > app001[99856]: last message repeated 5 times", pszHOSTNAME = 0x801aae > b75 "phoenix", > pszRcvdAt3164 = 0x0, pszRcvdAt3339 = 0x0, pszRcvdAt_MySQL = 0x0, > pszRcvdAt_PgSQL = 0x0, pszTIMESTAMP3164 = 0x0, pszTIMESTAMP3339 = > 0x801aaebc8 "2011-02-03T06:25:05.776392+00:00", pszTIMESTAMP_MySQL = > 0x0, > pszTIMESTAMP_PgSQL = 0x0, pCSProgName = 0x0, pCSStrucData = 0x0, > pCSAPPNAME = 0x0, pCSPROCID = 0x0, pCSMSGID = 0x0, pInputName = 0x0, > pRcvFromIP = 0x0, rcvFrom = {pRcvFrom = 0x0, pfrominet = 0x0}, pRuleset > = 0x0, ttGenTime = 1296714305, tRcvdAt = {timeType = 2 '\002', m > onth = 2 '\002', day = 3 '\003', hour = 6 '\006', > minute = 25 '\031', second = 5 '\005', secfracPrecision = 6 '\006', > OffsetMinute = 0 '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = > 2011, secfrac = 776392}, tTIMESTAMP = {timeType = 2 '\002', mon > th = 2 '\002', day = 3 '\003', hour = 6 '\006', minute = 25 '\031', > second = 5 '\005', secfracPrecision = 6 '\006', OffsetMinute = 0 > '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = 2011, secfrac = > 776392}, szRawMsg = '\xff' <repeats 101 times>, szHOSTNAME = "phoen > ix\000\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff", TAG = { > pszTAG = 0x722e313030707061 <Address 0x722e313030707061 out of > bounds>, szBuf = "app001[99856]:\xff"}, pszTimestamp3164 = "\000", > '\xff' <repeats 15 times>, pszTimestamp3339 = "2011-02- > 03T06:25:05.776392 > +00:00", pszTIMESTAMP_SecFrac = "\000\xff\xff\xff\xff\xff\xff", > pszRcvdAt_SecFrac = "\000\xff\xff\xff\xff\xff\xff"} > (gdb) quit > > rsyslog.conf: > ------------- > #### Modules > ## Input > $ModLoad immark > $ModLoad imudp > $ModLoad imtcp > $ModLoad imuxsock > $ModLoad imklog > $ModLoad imrelp > > ## Output > #$ModLoad omsnmp.so # Send SNMP traps > > #### Globals > $umask 0000 > $DirCreateMode 0640 > $FileCreateMode 0640 > > $RepeatedMsgReduction on > #$RepeatedMsgContainsOrigionalMsg on > > $DefaultNetstreamDriver ptcp > > # location for work (spool) file > $WorkDirectory /var/spool/rsyslog > > # Queue Types > $MainMsgQueueType LinkedList > $ActionQueueType LinkedList > > # Become DA Queues > $MainMsgQueueFileName mainq > $ActionQueueFileNAme actionq > > # Disable Rate Throttling > $MainMsgQueueTimeoutEnqueue 0 > $ActionQueueTimeoutEnqueue 0 > > # Discard any message at all if there double the queue we allow in > memory > # 10000 in ram + 10000 on disk > $MainMsgQueueDiscardMark 20000 > $ActionQueueDiscardMark 20000 > > 0 Emergency: system is unusable > # 1 Alert: action must be taken immediately > # 2 Critical: critical conditions > # 3 Error: error conditions > # 4 Warning: warning conditions > # 5 Notice: normal but significant condition > # 6 Informational: informational messages > # 7 Debug: debug-level messages > > # When discarding kicks in (above) any msg > # with priority >= this is out the door > $MainMsgQueueDiscardSeverity 0 > $ActionQueueDiscardSeverity 0 > > # default templates > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > $UDPServerRun 514 > $InputRELPServerRun 2514 > > # Templates > $template logsaggd,"%TIMESTAMP:::date-rfc3339% %syslogfacility-text% > %syslogpriority-text% %HOSTNAME% %syslogtag% %msg:::drop-last-lf%" > > #### Logging Facilities > *.* @localhost:18712;logsaggd > > +hulk > *.err;kern.warning;auth.notice;mail.crit > /var/log/console.log > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > /var/log/messages > security.* /var/log/security > auth.info;authpriv.info /var/log/auth.log > mail.info /var/log/maillog > lpr.info /var/log/lpd-errs > ftp.info /var/log/xferlog > cron.* /var/log/cron > *.=debug /var/log/debug.log > *.emerg * > > -- > ----------------------------------------------------------------------- > - > 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C > Philip M. Gollucci ([email protected]) c: 703.336.9354 > VP Apache Infrastructure; Member, Apache Software Foundation > Committer, FreeBSD Foundation > Consultant, P6M7G8 Inc. > Sr. System Admin, Ridecharge Inc. > > Work like you don't need the money, > love like you'll never get hurt, > and dance like nobody's watching. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
