> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Philip M. Gollucci > Sent: Monday, April 18, 2011 3:33 PM > To: rsyslog-users > Subject: Re: [rsyslog] rsyslog5 core dumps a-plenty > > On 04/14/11 22:20, Philip M. Gollucci wrote: > > Will be testing 5.8.0 friday or next week, and will report back. > Whatever this bug was 5.8.0 seems to have fixed it. Its been up since > ~Saturday on both the central, and a subset of the clients.
Thanks, I guess it was the one related to name resolution. Rainer > > Thanks! > > > > > > > > > > > On 02/03/11 06:56, Philip M. Gollucci wrote: > >> Rainer (All), > >> > >> At first I thought it was something in rsyslog 5.5.6+ that changed > >> b/c I couldn't get the core dumps happen with 5.5.5; however, now I'm > >> pretty sure it has nothing to do with that milestone. Unfortunately, > >> from the looks of it, I'm betting its a FreeBSD specific threading > >> issue. Below is the useful info, gdb back trace, rsyslog.conf, and the build > transcript. > >> > >> I've also uploaded the files here for easier viewing: > >> http://people.freebsd.org/~pgollucci/rsyslog-core > >> http://people.apache.org/~pgollucci/rsyslog-core > >> > >> Move info is available on request. > >> > >> > >> > >> $ uname -a > >> FreeBSD hulk 8.1-RELEASE FreeBSD 8.1-RELEASE #0 r212058: Wed Sep 1 > >> 23:12:18 UTC 2010 root@elektra > >> :/usr/obj/usr/src/sys/GENERIC amd64 > >> > >> $ rsyslog --version > >> rsyslogd 5.6.2, compiled with: > >> FEATURE_REGEXP: Yes > >> FEATURE_LARGEFILE: No > >> FEATURE_NETZIP (message compression): Yes > >> GSSAPI Kerberos 5 support: No > >> FEATURE_DEBUG (debug build, slow code): Yes > >> Atomic operations supported: Yes > >> Runtime Instrumentation (slow code): Yes > >> > >> $ sudo /usr/local/etc/rc.d/rsyslogd restart rsyslogd not running? > >> (check /var/run/syslog.pid). > >> Starting rsyslogd. > >> > >> $ ps -auxwww |grep rsyslogd > >> root 9135 6.1 0.0 25524 4388 1 S 6:37AM 0:01.29 > >> /usr/local/sbin/rsyslogd -c4 -i /var > >> /run/syslog.pid -f /usr/local/etc/rsyslog.conf > >> > >> $ cd / ; ls core.rsyslogd.8* > >> -rw------- 1 root wheel 10354688 Feb 3 06:25 core.rsyslogd.8742 > >> -rw------- 1 root wheel 8257536 Feb 3 06:26 core.rsyslogd.8782 > >> -rw------- 1 root wheel 8257536 Feb 3 06:27 core.rsyslogd.8826 > >> -rw------- 1 root wheel 8257536 Feb 3 06:28 core.rsyslogd.8844 > >> -rw------- 1 root wheel 10354688 Feb 3 06:29 core.rsyslogd.8875 > >> -rw------- 1 root wheel 8257536 Feb 3 06:29 core.rsyslogd.8900 > >> > >> $ gdb /usr/local/sbin/rsyslogd core.rsyslogd.81742 GNU gdb 6.1.1 > >> [FreeBSD] Copyright 2004 Free Software Foundation, Inc. > >> GDB is free software, covered by the GNU General Public License, and > >> you are welcome to change it and/or distribute copies of it under > >> certain conditions. > >> Type "show copying" to see the conditions. > >> There is absolutely no warranty for GDB. Type "show warranty" for > details. > >> This GDB was configured as "amd64-marcel-freebsd"... > >> Core was generated by `rsyslogd'. > >> Program terminated with signal 11, Segmentation fault. > >> Reading symbols from /lib/libz.so.5...done. > >> Loaded symbols for /lib/libz.so.5 > >> Reading symbols from /lib/libthr.so.3...done. > >> Loaded symbols for /lib/libthr.so.3 > >> Reading symbols from /lib/libc.so.7...done. > >> Loaded symbols for /lib/libc.so.7 > >> Reading symbols from /usr/local/lib/rsyslog/lmnet.so...done. > >> Loaded symbols for /usr/local/lib/rsyslog/lmnet.so Reading symbols > >> from /usr/local/lib/rsyslog/immark.so...done. > >> Loaded symbols for /usr/local/lib/rsyslog/immark.so Reading symbols > >> from /usr/local/lib/rsyslog/imudp.so...done. > >> Loaded symbols for /usr/local/lib/rsyslog/imudp.so Reading symbols > >> from /usr/local/lib/rsyslog/imtcp.so...done. > >> Loaded symbols for /usr/local/lib/rsyslog/imtcp.so Reading symbols > >> from /usr/local/lib/rsyslog/lmnetstrms.so...done. > >> Loaded symbols for /usr/local/lib/rsyslog/lmnetstrms.so > >> Reading symbols from /usr/local/lib/rsyslog/lmtcpsrv.so...done. > >> Loaded symbols for /usr/local/lib/rsyslog/lmtcpsrv.so > >> Reading symbols from /usr/local/lib/rsyslog/imuxsock.so...done. > >> Loaded symbols for /usr/local/lib/rsyslog/imuxsock.so > >> Reading symbols from /usr/local/lib/rsyslog/imklog.so...done. > >> Loaded symbols for /usr/local/lib/rsyslog/imklog.so Reading symbols > >> from /usr/local/lib/rsyslog/imrelp.so...done. > >> Loaded symbols for /usr/local/lib/rsyslog/imrelp.so Reading symbols > >> from /usr/local/lib/librelp.so.0...done. > >> Loaded symbols for /usr/local/lib/librelp.so.0 Reading symbols from > >> /libexec/ld-elf.so.1...done. > >> Loaded symbols for /libexec/ld-elf.so.1 > >> #0 0x00000008007e3876 in pthread_mutex_destroy () from > >> /lib/libthr.so.3 [New Thread 801a3ac80 (LWP 100568)] [New Thread > >> 800c87fc0 (LWP 100567)] [New Thread 800c88180 (LWP 100541)] [New > >> Thread 800c88340 (LWP 100540)] [New Thread 800c88500 (LWP 100539)] > >> [New Thread 800c886c0 (LWP 100538)] [New Thread 800c88880 (LWP > >> 100379)] [New Thread 800c041c0 (LWP 100548)] > >> (gdb) bt full > >> #0 0x00000008007e3876 in pthread_mutex_destroy () from > >> /lib/libthr.so.3 No symbol table info available. > >> #1 0x00000000004210cf in MsgDeleteMutexLockingCase > >> (pThis=0x801aaea00) at msg.c:624 > >> __func__ = "MsgDeleteMutexLockingCase" > >> #2 0x00000000004218d1 in msgDestruct (ppThis=0x7ffffeff8d70) at > msg.c:839 > >> dbgCALLStaCK_POP_POINT = 7 > >> iRet = RS_RET_OK > >> iCancelStateSave = 1 > >> pThis = (msg_t *) 0x801aaea00 > >> currRefCount = 0 > >> pdbgFuncDB = (dbgFuncDB_t *) 0x801a12500 > >> __func__ = "msgDestruct" > >> #3 0x000000000044fbe1 in DeleteProcessedBatch (pThis=0x800c04380, > >> pBatch=0x800c2fe70) at queue.c:1404 > >> i = 0 > >> pUsr = (void *) 0x801aaea00 > >> nEnqueued = 0 > >> localRet = 8 > >> dbgCALLStaCK_POP_POINT = 6 > >> iRet = RS_RET_OK > >> pdbgFuncDB = (dbgFuncDB_t *) 0x801a13500 > >> __func__ = "DeleteProcessedBatch" > >> #4 0x000000000044f828 in DequeueConsumableElements > >> (pThis=0x800c04380, pWti=0x800c2fe40, > piRemainingQueueSize=0x7ffffeff8e14) at queue.c:1441 > >> nDequeued = 8 > >> nDiscarded = 4678208 > >> nDeleted = 2 > >> iQueueSize = 4673262 > >> pUsr = (void *) 0x40042df73 > >> localRet = RS_RET_OK > >> dbgCALLStaCK_POP_POINT = 5 > >> iRet = RS_RET_OK > >> pdbgFuncDB = (dbgFuncDB_t *) 0x801a13600 > >> __func__ = "DequeueConsumableElements" > >> #5 0x000000000044f711 in DequeueConsumable (pThis=0x800c04380, > >> pWti=0x800c2fe40) at queue.c:1489 > >> dbgCALLStaCK_POP_POINT = 4 > >> iRet = RS_RET_OK > >> iQueueSize = 0 > >> pdbgFuncDB = (dbgFuncDB_t *) 0x801a13700 > >> __func__ = "DequeueConsumable" > >> #6 0x0000000000450fca in DequeueForConsumer (pThis=0x800c04380, > >> pWti=0x800c2fe40) at queue.c:1626 > >> dbgCALLStaCK_POP_POINT = 3 > >> iRet = RS_RET_OK > >> pdbgFuncDB = (dbgFuncDB_t *) 0x801a13800 > >> __func__ = "DequeueForConsumer" > >> #7 0x0000000000450c6b in ConsumerReg (pThis=0x800c04380, > >> pWti=0x800c2fe40) at queue.c:1679 > >> iCancelStateSave = 8 > >> dbgCALLStaCK_POP_POINT = 2 > >> iRet = RS_RET_OK > >> pdbgFuncDB = (dbgFuncDB_t *) 0x801a13900 > >> __func__ = "ConsumerReg" > >> #8 0x0000000000443b21 in wtiWorker (pThis=0x800c2fe40) at wti.c:315 > >> __cleanup_info__ = {pthread_cleanup_pad = {140737471549248, > >> 4470256, 34372517440, 0, 4667374, 5824688, 0, 34368002717}} > >> pWtp = (wtp_t *) 0x800c1c380 > >> bInactivityTOOccured = 0 > >> localRet = RS_RET_OK > >> terminateRet = RS_RET_OK > >> iCancelStateSave = 0 > >> dbgCALLStaCK_POP_POINT = 1 > >> iRet = RS_RET_OK > >> pdbgFuncDB = (dbgFuncDB_t *) 0x801a13d00 > >> __func__ = "wtiWorker" > >> #9 0x0000000000442019 in wtpWorker (arg=0x800c2fe40) at wtp.c:381 > >> __cleanup_info__ = {pthread_cleanup_pad = {0, 4462880, > >> 34372517440, 0, 0, 0, 0, 0}} > >> pWti = (wti_t *) 0x800c2fe40 > >> pThis = (wtp_t *) 0x800c1c380 > >> sigSet = {__bits = {1048576, 0, 0, 0}} > >> dbgCALLStaCK_POP_POINT = 0 > >> pdbgFuncDB = (dbgFuncDB_t *) 0x801a13e00 > >> __func__ = "wtpWorker" > >> #10 0x00000008007de511 in pthread_getprio () from /lib/libthr.so.3 No > >> symbol table info available. > >> #11 0x0000000000000000 in ?? () > >> No symbol table info available. > >> Cannot access memory at address 0x7ffffeff9000 > >> (gdb) frame 1 > >> #1 0x00000000004210cf in MsgDeleteMutexLockingCase > >> (pThis=0x801aaea00) at msg.c:624 > >> 624 msg.c: No such file or directory. > >> in msg.c > >> (gdb) info locals > >> __func__ = "MsgDeleteMutexLockingCase" > >> (gdb) p *pThis > >> $1 = {objData = {pObjInfo = 0x800c16400, iObjCooCKiE = 195948526, > >> pszName = 0x0}, flowCtlType = eFLOWCTL_NO_DELAY, mut = > >> 0xffffffffffffffff, iRefCount = 0, bDoLock = 0 '\0', iSeverity = 3, > >> iFacility = 22, of fAfterPRI = 0, offMSG = 91, iProtocolVersion = 0, > >> msgFlags = 32, > >> iLenRawMsg = 121, iLenMSG = 30, iLenTAG = 31, iLenHOSTNAME = 21, > >> pszRawMsg = 0x801ab5600 "<179>2011-02-03T06:20:37.681910+00:00 > >> phoenix > >> app001[99856]: last message repeated 5 times", pszHOSTNAME = > 0x801aae > >> b75 "phoenix", > >> pszRcvdAt3164 = 0x0, pszRcvdAt3339 = 0x0, pszRcvdAt_MySQL = 0x0, > >> pszRcvdAt_PgSQL = 0x0, pszTIMESTAMP3164 = 0x0, pszTIMESTAMP3339 = > >> 0x801aaebc8 "2011-02-03T06:25:05.776392+00:00", > pszTIMESTAMP_MySQL = > >> 0x0, pszTIMESTAMP_PgSQL = 0x0, pCSProgName = 0x0, pCSStrucData = > 0x0, > >> pCSAPPNAME = 0x0, pCSPROCID = 0x0, pCSMSGID = 0x0, pInputName = > >> 0x0, pRcvFromIP = 0x0, rcvFrom = {pRcvFrom = 0x0, pfrominet = 0x0}, > >> pRuleset = 0x0, ttGenTime = 1296714305, tRcvdAt = {timeType = 2 > >> '\002', m onth = 2 '\002', day = 3 '\003', hour = 6 '\006', > >> minute = 25 '\031', second = 5 '\005', secfracPrecision = 6 > >> '\006', OffsetMinute = 0 '\0', OffsetHour = 0 '\0', OffsetMode = 43 > >> '+', year = 2011, secfrac = 776392}, tTIMESTAMP = {timeType = 2 > >> '\002', mon th = 2 '\002', day = 3 '\003', hour = 6 '\006', minute = 25 '\031', > >> second = 5 '\005', secfracPrecision = 6 '\006', OffsetMinute = 0 > >> '\0', OffsetHour = 0 '\0', OffsetMode = 43 '+', year = 2011, secfrac > >> = 776392}, szRawMsg = '\xff' <repeats 101 times>, szHOSTNAME = "phoen > >> ix\000\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff", TAG = { > >> pszTAG = 0x722e313030707061 <Address 0x722e313030707061 out of > >> bounds>, szBuf = "app001[99856]:\xff"}, pszTimestamp3164 = "\000", > >> '\xff' <repeats 15 times>, pszTimestamp3339 = > >> "2011-02-03T06:25:05.776392 > >> +00:00", pszTIMESTAMP_SecFrac = "\000\xff\xff\xff\xff\xff\xff", > >> pszRcvdAt_SecFrac = "\000\xff\xff\xff\xff\xff\xff"} > >> (gdb) quit > >> > >> rsyslog.conf: > >> ------------- > >> #### Modules > >> ## Input > >> $ModLoad immark > >> $ModLoad imudp > >> $ModLoad imtcp > >> $ModLoad imuxsock > >> $ModLoad imklog > >> $ModLoad imrelp > >> > >> ## Output > >> #$ModLoad omsnmp.so # Send SNMP traps > >> > >> #### Globals > >> $umask 0000 > >> $DirCreateMode 0640 > >> $FileCreateMode 0640 > >> > >> $RepeatedMsgReduction on > >> #$RepeatedMsgContainsOrigionalMsg on > >> > >> $DefaultNetstreamDriver ptcp > >> > >> # location for work (spool) file > >> $WorkDirectory /var/spool/rsyslog > >> > >> # Queue Types > >> $MainMsgQueueType LinkedList > >> $ActionQueueType LinkedList > >> > >> # Become DA Queues > >> $MainMsgQueueFileName mainq > >> $ActionQueueFileNAme actionq > >> > >> # Disable Rate Throttling > >> $MainMsgQueueTimeoutEnqueue 0 > >> $ActionQueueTimeoutEnqueue 0 > >> > >> # Discard any message at all if there double the queue we allow in > >> memory # 10000 in ram + 10000 on disk $MainMsgQueueDiscardMark > 20000 > >> $ActionQueueDiscardMark 20000 > >> > >> 0 Emergency: system is unusable > >> # 1 Alert: action must be taken immediately > >> # 2 Critical: critical conditions > >> # 3 Error: error conditions > >> # 4 Warning: warning conditions > >> # 5 Notice: normal but significant condition > >> # 6 Informational: informational messages > >> # 7 Debug: debug-level messages > >> > >> # When discarding kicks in (above) any msg # with priority >= this is > >> out the door $MainMsgQueueDiscardSeverity 0 > >> $ActionQueueDiscardSeverity 0 > >> > >> # default templates > >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > >> > >> $UDPServerRun 514 > >> $InputRELPServerRun 2514 > >> > >> # Templates > >> $template logsaggd,"%TIMESTAMP:::date-rfc3339% %syslogfacility-text% > >> %syslogpriority-text% %HOSTNAME% %syslogtag% %msg:::drop-last-lf%" > >> > >> #### Logging Facilities > >> *.* @localhost:18712;logsaggd > >> > >> +hulk > >> *.err;kern.warning;auth.notice;mail.crit > >> /var/log/console.log > >> *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err > >> /var/log/messages > >> security.* /var/log/security > >> auth.info;authpriv.info /var/log/auth.log > >> mail.info /var/log/maillog > >> lpr.info /var/log/lpd-errs > >> ftp.info /var/log/xferlog > >> cron.* /var/log/cron > >> *.=debug /var/log/debug.log > >> *.emerg * > >> > > > > > > > -- > ------------------------------------------------------------------------ > 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C > Philip M. Gollucci ([email protected]) c: 703.336.9354 > Member, Apache Software Foundation > Committer, FreeBSD Foundation > Consultant, P6M7G8 Inc. > Sr. System Admin, Ridecharge Inc. > > Work like you don't need the money, > love like you'll never get hurt, > and dance like nobody's watching. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
