Rainer (All), At first I thought it was something in rsyslog 5.5.6+ that changed b/c I couldn't get the core dumps happen with 5.5.5; however, now I'm pretty sure it has nothing to do with that milestone. Unfortunately, from the looks of it, I'm betting its a FreeBSD specific threading issue. Below is the useful info, gdb back trace, rsyslog.conf, and the build transcript.
I've also uploaded the files here for easier viewing: http://people.freebsd.org/~pgollucci/rsyslog-core http://people.apache.org/~pgollucci/rsyslog-core Move info is available on request. $ uname -a FreeBSD hulk 8.1-RELEASE FreeBSD 8.1-RELEASE #0 r212058: Wed Sep 1 23:12:18 UTC 2010 root@elektra :/usr/obj/usr/src/sys/GENERIC amd64 $ rsyslog --version rsyslogd 5.6.2, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: No FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): Yes Atomic operations supported: Yes Runtime Instrumentation (slow code): Yes $ sudo /usr/local/etc/rc.d/rsyslogd restart rsyslogd not running? (check /var/run/syslog.pid). Starting rsyslogd. $ ps -auxwww |grep rsyslogd root 9135 6.1 0.0 25524 4388 1 S 6:37AM 0:01.29 /usr/local/sbin/rsyslogd -c4 -i /var /run/syslog.pid -f /usr/local/etc/rsyslog.conf $ cd / ; ls core.rsyslogd.8* -rw------- 1 root wheel 10354688 Feb 3 06:25 core.rsyslogd.8742 -rw------- 1 root wheel 8257536 Feb 3 06:26 core.rsyslogd.8782 -rw------- 1 root wheel 8257536 Feb 3 06:27 core.rsyslogd.8826 -rw------- 1 root wheel 8257536 Feb 3 06:28 core.rsyslogd.8844 -rw------- 1 root wheel 10354688 Feb 3 06:29 core.rsyslogd.8875 -rw------- 1 root wheel 8257536 Feb 3 06:29 core.rsyslogd.8900 $ gdb /usr/local/sbin/rsyslogd core.rsyslogd.81742 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Core was generated by `rsyslogd'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libz.so.5...done. Loaded symbols for /lib/libz.so.5 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/local/lib/rsyslog/lmnet.so...done. Loaded symbols for /usr/local/lib/rsyslog/lmnet.so Reading symbols from /usr/local/lib/rsyslog/immark.so...done. Loaded symbols for /usr/local/lib/rsyslog/immark.so Reading symbols from /usr/local/lib/rsyslog/imudp.so...done. Loaded symbols for /usr/local/lib/rsyslog/imudp.so Reading symbols from /usr/local/lib/rsyslog/imtcp.so...done. Loaded symbols for /usr/local/lib/rsyslog/imtcp.so Reading symbols from /usr/local/lib/rsyslog/lmnetstrms.so...done. Loaded symbols for /usr/local/lib/rsyslog/lmnetstrms.so Reading symbols from /usr/local/lib/rsyslog/lmtcpsrv.so...done. Loaded symbols for /usr/local/lib/rsyslog/lmtcpsrv.so Reading symbols from /usr/local/lib/rsyslog/imuxsock.so...done. Loaded symbols for /usr/local/lib/rsyslog/imuxsock.so Reading symbols from /usr/local/lib/rsyslog/imklog.so...done. Loaded symbols for /usr/local/lib/rsyslog/imklog.so Reading symbols from /usr/local/lib/rsyslog/imrelp.so...done. Loaded symbols for /usr/local/lib/rsyslog/imrelp.so Reading symbols from /usr/local/lib/librelp.so.0...done. Loaded symbols for /usr/local/lib/librelp.so.0 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x00000008007e3876 in pthread_mutex_destroy () from /lib/libthr.so.3 [New Thread 801a3ac80 (LWP 100568)] [New Thread 800c87fc0 (LWP 100567)] [New Thread 800c88180 (LWP 100541)] [New Thread 800c88340 (LWP 100540)] [New Thread 800c88500 (LWP 100539)] [New Thread 800c886c0 (LWP 100538)] [New Thread 800c88880 (LWP 100379)] [New Thread 800c041c0 (LWP 100548)] (gdb) bt full #0 0x00000008007e3876 in pthread_mutex_destroy () from /lib/libthr.so.3 No symbol table info available. #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) at msg.c:624 __func__ = "MsgDeleteMutexLockingCase" #2 0x00000000004218d1 in msgDestruct (ppThis=0x7ffffeff8d70) at msg.c:839 dbgCALLStaCK_POP_POINT = 7 iRet = RS_RET_OK iCancelStateSave = 1 pThis = (msg_t *) 0x801aaea00 currRefCount = 0 pdbgFuncDB = (dbgFuncDB_t *) 0x801a12500 __func__ = "msgDestruct" #3 0x000000000044fbe1 in DeleteProcessedBatch (pThis=0x800c04380, pBatch=0x800c2fe70) at queue.c:1404 i = 0 pUsr = (void *) 0x801aaea00 nEnqueued = 0 localRet = 8 dbgCALLStaCK_POP_POINT = 6 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13500 __func__ = "DeleteProcessedBatch" #4 0x000000000044f828 in DequeueConsumableElements (pThis=0x800c04380, pWti=0x800c2fe40, piRemainingQueueSize=0x7ffffeff8e14) at queue.c:1441 nDequeued = 8 nDiscarded = 4678208 nDeleted = 2 iQueueSize = 4673262 pUsr = (void *) 0x40042df73 localRet = RS_RET_OK dbgCALLStaCK_POP_POINT = 5 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13600 __func__ = "DequeueConsumableElements" #5 0x000000000044f711 in DequeueConsumable (pThis=0x800c04380, pWti=0x800c2fe40) at queue.c:1489 dbgCALLStaCK_POP_POINT = 4 iRet = RS_RET_OK iQueueSize = 0 pdbgFuncDB = (dbgFuncDB_t *) 0x801a13700 __func__ = "DequeueConsumable" #6 0x0000000000450fca in DequeueForConsumer (pThis=0x800c04380, pWti=0x800c2fe40) at queue.c:1626 dbgCALLStaCK_POP_POINT = 3 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13800 __func__ = "DequeueForConsumer" #7 0x0000000000450c6b in ConsumerReg (pThis=0x800c04380, pWti=0x800c2fe40) at queue.c:1679 iCancelStateSave = 8 dbgCALLStaCK_POP_POINT = 2 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13900 __func__ = "ConsumerReg" #8 0x0000000000443b21 in wtiWorker (pThis=0x800c2fe40) at wti.c:315 __cleanup_info__ = {pthread_cleanup_pad = {140737471549248, 4470256, 34372517440, 0, 4667374, 5824688, 0, 34368002717}} pWtp = (wtp_t *) 0x800c1c380 bInactivityTOOccured = 0 localRet = RS_RET_OK terminateRet = RS_RET_OK iCancelStateSave = 0 dbgCALLStaCK_POP_POINT = 1 iRet = RS_RET_OK pdbgFuncDB = (dbgFuncDB_t *) 0x801a13d00 __func__ = "wtiWorker" #9 0x0000000000442019 in wtpWorker (arg=0x800c2fe40) at wtp.c:381 __cleanup_info__ = {pthread_cleanup_pad = {0, 4462880, 34372517440, 0, 0, 0, 0, 0}} pWti = (wti_t *) 0x800c2fe40 pThis = (wtp_t *) 0x800c1c380 sigSet = {__bits = {1048576, 0, 0, 0}} dbgCALLStaCK_POP_POINT = 0 pdbgFuncDB = (dbgFuncDB_t *) 0x801a13e00 __func__ = "wtpWorker" #10 0x00000008007de511 in pthread_getprio () from /lib/libthr.so.3 No symbol table info available. #11 0x0000000000000000 in ?? () No symbol table info available. Cannot access memory at address 0x7ffffeff9000 (gdb) frame 1 #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) at msg.c:624 624 msg.c: No such file or directory. in msg.c (gdb) info locals __func__ = "MsgDeleteMutexLockingCase" (gdb) p *pThis $1 = {objData = {pObjInfo = 0x800c16400, iObjCooCKiE = 195948526, pszName = 0x0}, flowCtlType = eFLOWCTL_NO_DELAY, mut = 0xffffffffffffffff, iRefCount = 0, bDoLock = 0 '\0', iSeverity = 3, iFacility = 22, of fAfterPRI = 0, offMSG = 91, iProtocolVersion = 0, msgFlags = 32, iLenRawMsg = 121, iLenMSG = 30, iLenTAG = 31, iLenHOSTNAME = 21, pszRawMsg = 0x801ab5600 "<179>2011-02-03T06:20:37.681910+00:00 phoenix app001[99856]: last message repeated 5 times", pszHOSTNAME = 0x801aae b75 "phoenix", pszRcvdAt3164 = 0x0, pszRcvdAt3339 = 0x0, pszRcvdAt_MySQL = 0x0, pszRcvdAt_PgSQL = 0x0, pszTIMESTAMP3164 = 0x0, pszTIMESTAMP3339 = 0x801aaebc8 "2011-02-03T06:25:05.776392+00:00", pszTIMESTAMP_MySQL = 0x0, pszTIMESTAMP_PgSQL = 0x0, pCSProgName = 0x0, pCSStrucData = 0x0, pCSAPPNAME = 0x0, pCSPROCID = 0x0, pCSMSGID = 0x0, pInputName = 0x0, pRcvFromIP = 0x0, rcvFrom = {pRcvFrom = 0x0, pfrominet = 0x0}, pRuleset = 0x0, ttGenTime = 1296714305, tRcvdAt = {timeType = 2 '\002', m onth = 2 '\002', day = 3 '\003', hour = 6 '\006', minute = 25 '\031', second = 5 '\005', secfracPrecision = 6 '\006', OffsetMinute = 0 '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = 2011, secfrac = 776392}, tTIMESTAMP = {timeType = 2 '\002', mon th = 2 '\002', day = 3 '\003', hour = 6 '\006', minute = 25 '\031', second = 5 '\005', secfracPrecision = 6 '\006', OffsetMinute = 0 '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = 2011, secfrac = 776392}, szRawMsg = '\xff' <repeats 101 times>, szHOSTNAME = "phoen ix\000\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff", TAG = { pszTAG = 0x722e313030707061 <Address 0x722e313030707061 out of bounds>, szBuf = "app001[99856]:\xff"}, pszTimestamp3164 = "\000", '\xff' <repeats 15 times>, pszTimestamp3339 = "2011-02-03T06:25:05.776392 +00:00", pszTIMESTAMP_SecFrac = "\000\xff\xff\xff\xff\xff\xff", pszRcvdAt_SecFrac = "\000\xff\xff\xff\xff\xff\xff"} (gdb) quit rsyslog.conf: ------------- #### Modules ## Input $ModLoad immark $ModLoad imudp $ModLoad imtcp $ModLoad imuxsock $ModLoad imklog $ModLoad imrelp ## Output #$ModLoad omsnmp.so # Send SNMP traps #### Globals $umask 0000 $DirCreateMode 0640 $FileCreateMode 0640 $RepeatedMsgReduction on #$RepeatedMsgContainsOrigionalMsg on $DefaultNetstreamDriver ptcp # location for work (spool) file $WorkDirectory /var/spool/rsyslog # Queue Types $MainMsgQueueType LinkedList $ActionQueueType LinkedList # Become DA Queues $MainMsgQueueFileName mainq $ActionQueueFileNAme actionq # Disable Rate Throttling $MainMsgQueueTimeoutEnqueue 0 $ActionQueueTimeoutEnqueue 0 # Discard any message at all if there double the queue we allow in memory # 10000 in ram + 10000 on disk $MainMsgQueueDiscardMark 20000 $ActionQueueDiscardMark 20000 0 Emergency: system is unusable # 1 Alert: action must be taken immediately # 2 Critical: critical conditions # 3 Error: error conditions # 4 Warning: warning conditions # 5 Notice: normal but significant condition # 6 Informational: informational messages # 7 Debug: debug-level messages # When discarding kicks in (above) any msg # with priority >= this is out the door $MainMsgQueueDiscardSeverity 0 $ActionQueueDiscardSeverity 0 # default templates $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $UDPServerRun 514 $InputRELPServerRun 2514 # Templates $template logsaggd,"%TIMESTAMP:::date-rfc3339% %syslogfacility-text% %syslogpriority-text% %HOSTNAME% %syslogtag% %msg:::drop-last-lf%" #### Logging Facilities *.* @localhost:18712;logsaggd +hulk *.err;kern.warning;auth.notice;mail.crit /var/log/console.log *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages security.* /var/log/security auth.info;authpriv.info /var/log/auth.log mail.info /var/log/maillog lpr.info /var/log/lpd-errs ftp.info /var/log/xferlog cron.* /var/log/cron *.=debug /var/log/debug.log *.emerg * -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci ([email protected]) c: 703.336.9354 VP Apache Infrastructure; Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
