Will be testing 5.8.0 friday or next week, and will report back.
On 02/03/11 06:56, Philip M. Gollucci wrote: > Rainer (All), > > At first I thought it was something in rsyslog 5.5.6+ that changed b/c I > couldn't get the core dumps happen with 5.5.5; however, now I'm pretty > sure it has nothing to do with that milestone. Unfortunately, from the > looks of it, I'm betting its a FreeBSD specific threading issue. Below > is the useful info, gdb back trace, rsyslog.conf, and the build transcript. > > I've also uploaded the files here for easier viewing: > http://people.freebsd.org/~pgollucci/rsyslog-core > http://people.apache.org/~pgollucci/rsyslog-core > > Move info is available on request. > > > > $ uname -a > FreeBSD hulk 8.1-RELEASE FreeBSD 8.1-RELEASE #0 r212058: Wed Sep 1 > 23:12:18 UTC 2010 root@elektra > :/usr/obj/usr/src/sys/GENERIC amd64 > > $ rsyslog --version > rsyslogd 5.6.2, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: No > FEATURE_NETZIP (message compression): Yes > GSSAPI Kerberos 5 support: No > FEATURE_DEBUG (debug build, slow code): Yes > Atomic operations supported: Yes > Runtime Instrumentation (slow code): Yes > > $ sudo /usr/local/etc/rc.d/rsyslogd restart > rsyslogd not running? (check /var/run/syslog.pid). > Starting rsyslogd. > > $ ps -auxwww |grep rsyslogd > root 9135 6.1 0.0 25524 4388 1 S 6:37AM 0:01.29 > /usr/local/sbin/rsyslogd -c4 -i /var > /run/syslog.pid -f /usr/local/etc/rsyslog.conf > > $ cd / ; ls core.rsyslogd.8* > -rw------- 1 root wheel 10354688 Feb 3 06:25 core.rsyslogd.8742 > -rw------- 1 root wheel 8257536 Feb 3 06:26 core.rsyslogd.8782 > -rw------- 1 root wheel 8257536 Feb 3 06:27 core.rsyslogd.8826 > -rw------- 1 root wheel 8257536 Feb 3 06:28 core.rsyslogd.8844 > -rw------- 1 root wheel 10354688 Feb 3 06:29 core.rsyslogd.8875 > -rw------- 1 root wheel 8257536 Feb 3 06:29 core.rsyslogd.8900 > > $ gdb /usr/local/sbin/rsyslogd core.rsyslogd.81742 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > Core was generated by `rsyslogd'. > Program terminated with signal 11, Segmentation fault. > Reading symbols from /lib/libz.so.5...done. > Loaded symbols for /lib/libz.so.5 > Reading symbols from /lib/libthr.so.3...done. > Loaded symbols for /lib/libthr.so.3 > Reading symbols from /lib/libc.so.7...done. > Loaded symbols for /lib/libc.so.7 > Reading symbols from /usr/local/lib/rsyslog/lmnet.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmnet.so > Reading symbols from /usr/local/lib/rsyslog/immark.so...done. > Loaded symbols for /usr/local/lib/rsyslog/immark.so > Reading symbols from /usr/local/lib/rsyslog/imudp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imudp.so > Reading symbols from /usr/local/lib/rsyslog/imtcp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imtcp.so > Reading symbols from /usr/local/lib/rsyslog/lmnetstrms.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmnetstrms.so > Reading symbols from /usr/local/lib/rsyslog/lmtcpsrv.so...done. > Loaded symbols for /usr/local/lib/rsyslog/lmtcpsrv.so > Reading symbols from /usr/local/lib/rsyslog/imuxsock.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imuxsock.so > Reading symbols from /usr/local/lib/rsyslog/imklog.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imklog.so > Reading symbols from /usr/local/lib/rsyslog/imrelp.so...done. > Loaded symbols for /usr/local/lib/rsyslog/imrelp.so > Reading symbols from /usr/local/lib/librelp.so.0...done. > Loaded symbols for /usr/local/lib/librelp.so.0 > Reading symbols from /libexec/ld-elf.so.1...done. > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x00000008007e3876 in pthread_mutex_destroy () from /lib/libthr.so.3 > [New Thread 801a3ac80 (LWP 100568)] > [New Thread 800c87fc0 (LWP 100567)] > [New Thread 800c88180 (LWP 100541)] > [New Thread 800c88340 (LWP 100540)] > [New Thread 800c88500 (LWP 100539)] > [New Thread 800c886c0 (LWP 100538)] > [New Thread 800c88880 (LWP 100379)] > [New Thread 800c041c0 (LWP 100548)] > (gdb) bt full > #0 0x00000008007e3876 in pthread_mutex_destroy () from /lib/libthr.so.3 > No symbol table info available. > #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) > at msg.c:624 > __func__ = "MsgDeleteMutexLockingCase" > #2 0x00000000004218d1 in msgDestruct (ppThis=0x7ffffeff8d70) at msg.c:839 > dbgCALLStaCK_POP_POINT = 7 > iRet = RS_RET_OK > iCancelStateSave = 1 > pThis = (msg_t *) 0x801aaea00 > currRefCount = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a12500 > __func__ = "msgDestruct" > #3 0x000000000044fbe1 in DeleteProcessedBatch (pThis=0x800c04380, > pBatch=0x800c2fe70) at queue.c:1404 > i = 0 > pUsr = (void *) 0x801aaea00 > nEnqueued = 0 > localRet = 8 > dbgCALLStaCK_POP_POINT = 6 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13500 > __func__ = "DeleteProcessedBatch" > #4 0x000000000044f828 in DequeueConsumableElements (pThis=0x800c04380, > pWti=0x800c2fe40, piRemainingQueueSize=0x7ffffeff8e14) at queue.c:1441 > nDequeued = 8 > nDiscarded = 4678208 > nDeleted = 2 > iQueueSize = 4673262 > pUsr = (void *) 0x40042df73 > localRet = RS_RET_OK > dbgCALLStaCK_POP_POINT = 5 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13600 > __func__ = "DequeueConsumableElements" > #5 0x000000000044f711 in DequeueConsumable (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1489 > dbgCALLStaCK_POP_POINT = 4 > iRet = RS_RET_OK > iQueueSize = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13700 > __func__ = "DequeueConsumable" > #6 0x0000000000450fca in DequeueForConsumer (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1626 > dbgCALLStaCK_POP_POINT = 3 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13800 > __func__ = "DequeueForConsumer" > #7 0x0000000000450c6b in ConsumerReg (pThis=0x800c04380, > pWti=0x800c2fe40) at queue.c:1679 > iCancelStateSave = 8 > dbgCALLStaCK_POP_POINT = 2 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13900 > __func__ = "ConsumerReg" > #8 0x0000000000443b21 in wtiWorker (pThis=0x800c2fe40) at wti.c:315 > __cleanup_info__ = {pthread_cleanup_pad = {140737471549248, > 4470256, 34372517440, 0, 4667374, 5824688, 0, 34368002717}} > pWtp = (wtp_t *) 0x800c1c380 > bInactivityTOOccured = 0 > localRet = RS_RET_OK > terminateRet = RS_RET_OK > iCancelStateSave = 0 > dbgCALLStaCK_POP_POINT = 1 > iRet = RS_RET_OK > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13d00 > __func__ = "wtiWorker" > #9 0x0000000000442019 in wtpWorker (arg=0x800c2fe40) at wtp.c:381 > __cleanup_info__ = {pthread_cleanup_pad = {0, 4462880, > 34372517440, 0, 0, 0, 0, 0}} > pWti = (wti_t *) 0x800c2fe40 > pThis = (wtp_t *) 0x800c1c380 > sigSet = {__bits = {1048576, 0, 0, 0}} > dbgCALLStaCK_POP_POINT = 0 > pdbgFuncDB = (dbgFuncDB_t *) 0x801a13e00 > __func__ = "wtpWorker" > #10 0x00000008007de511 in pthread_getprio () from /lib/libthr.so.3 > No symbol table info available. > #11 0x0000000000000000 in ?? () > No symbol table info available. > Cannot access memory at address 0x7ffffeff9000 > (gdb) frame 1 > #1 0x00000000004210cf in MsgDeleteMutexLockingCase (pThis=0x801aaea00) > at msg.c:624 > 624 msg.c: No such file or directory. > in msg.c > (gdb) info locals > __func__ = "MsgDeleteMutexLockingCase" > (gdb) p *pThis > $1 = {objData = {pObjInfo = 0x800c16400, iObjCooCKiE = 195948526, > pszName = 0x0}, flowCtlType = eFLOWCTL_NO_DELAY, mut = > 0xffffffffffffffff, iRefCount = 0, bDoLock = 0 '\0', iSeverity = 3, > iFacility = 22, of > fAfterPRI = 0, offMSG = 91, iProtocolVersion = 0, msgFlags = 32, > iLenRawMsg = 121, iLenMSG = 30, iLenTAG = 31, iLenHOSTNAME = 21, > pszRawMsg = 0x801ab5600 "<179>2011-02-03T06:20:37.681910+00:00 phoenix > app001[99856]: last message repeated 5 times", pszHOSTNAME = 0x801aae > b75 "phoenix", > pszRcvdAt3164 = 0x0, pszRcvdAt3339 = 0x0, pszRcvdAt_MySQL = 0x0, > pszRcvdAt_PgSQL = 0x0, pszTIMESTAMP3164 = 0x0, pszTIMESTAMP3339 = > 0x801aaebc8 "2011-02-03T06:25:05.776392+00:00", pszTIMESTAMP_MySQL = 0x0, > pszTIMESTAMP_PgSQL = 0x0, pCSProgName = 0x0, pCSStrucData = 0x0, > pCSAPPNAME = 0x0, pCSPROCID = 0x0, pCSMSGID = 0x0, pInputName = 0x0, > pRcvFromIP = 0x0, rcvFrom = {pRcvFrom = 0x0, pfrominet = 0x0}, pRuleset > = 0x0, ttGenTime = 1296714305, tRcvdAt = {timeType = 2 '\002', m > onth = 2 '\002', day = 3 '\003', hour = 6 '\006', > minute = 25 '\031', second = 5 '\005', secfracPrecision = 6 '\006', > OffsetMinute = 0 '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = > 2011, secfrac = 776392}, tTIMESTAMP = {timeType = 2 '\002', mon > th = 2 '\002', day = 3 '\003', hour = 6 '\006', minute = 25 '\031', > second = 5 '\005', secfracPrecision = 6 '\006', OffsetMinute = 0 > '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = 2011, secfrac = > 776392}, szRawMsg = '\xff' <repeats 101 times>, szHOSTNAME = "phoen > ix\000\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff", TAG = { > pszTAG = 0x722e313030707061 <Address 0x722e313030707061 out of > bounds>, szBuf = "app001[99856]:\xff"}, pszTimestamp3164 = "\000", > '\xff' <repeats 15 times>, pszTimestamp3339 = "2011-02-03T06:25:05.776392 > +00:00", pszTIMESTAMP_SecFrac = "\000\xff\xff\xff\xff\xff\xff", > pszRcvdAt_SecFrac = "\000\xff\xff\xff\xff\xff\xff"} > (gdb) quit > > rsyslog.conf: > ------------- > #### Modules > ## Input > $ModLoad immark > $ModLoad imudp > $ModLoad imtcp > $ModLoad imuxsock > $ModLoad imklog > $ModLoad imrelp > > ## Output > #$ModLoad omsnmp.so # Send SNMP traps > > #### Globals > $umask 0000 > $DirCreateMode 0640 > $FileCreateMode 0640 > > $RepeatedMsgReduction on > #$RepeatedMsgContainsOrigionalMsg on > > $DefaultNetstreamDriver ptcp > > # location for work (spool) file > $WorkDirectory /var/spool/rsyslog > > # Queue Types > $MainMsgQueueType LinkedList > $ActionQueueType LinkedList > > # Become DA Queues > $MainMsgQueueFileName mainq > $ActionQueueFileNAme actionq > > # Disable Rate Throttling > $MainMsgQueueTimeoutEnqueue 0 > $ActionQueueTimeoutEnqueue 0 > > # Discard any message at all if there double the queue we allow in memory > # 10000 in ram + 10000 on disk > $MainMsgQueueDiscardMark 20000 > $ActionQueueDiscardMark 20000 > > 0 Emergency: system is unusable > # 1 Alert: action must be taken immediately > # 2 Critical: critical conditions > # 3 Error: error conditions > # 4 Warning: warning conditions > # 5 Notice: normal but significant condition > # 6 Informational: informational messages > # 7 Debug: debug-level messages > > # When discarding kicks in (above) any msg > # with priority >= this is out the door > $MainMsgQueueDiscardSeverity 0 > $ActionQueueDiscardSeverity 0 > > # default templates > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > $UDPServerRun 514 > $InputRELPServerRun 2514 > > # Templates > $template logsaggd,"%TIMESTAMP:::date-rfc3339% %syslogfacility-text% > %syslogpriority-text% %HOSTNAME% %syslogtag% %msg:::drop-last-lf%" > > #### Logging Facilities > *.* @localhost:18712;logsaggd > > +hulk > *.err;kern.warning;auth.notice;mail.crit > /var/log/console.log > *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > /var/log/messages > security.* /var/log/security > auth.info;authpriv.info /var/log/auth.log > mail.info /var/log/maillog > lpr.info /var/log/lpd-errs > ftp.info /var/log/xferlog > cron.* /var/log/cron > *.=debug /var/log/debug.log > *.emerg * > -- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci ([email protected]) c: 703.336.9354 Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Sr. System Admin, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
