Thx, this all is very useful. I'll try to see if something immediately catches my eye or if I can reproduce.
Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Matt Wise > Sent: Friday, May 04, 2012 12:11 AM > To: rsyslog-users > Subject: Re: [rsyslog] Rsyslog 5.8.10 segfault... > > Here are the last 100 lines with the valgrind failure: > > > 0806.842884190:766b700: prop repl 4, pRes='2012', len -1 > > 0806.843084207:766b700: end prop repl, pRes='2012', len 4 > > 0806.843400424:766b700: prop repl 4, pRes='05', len -1 > > 0806.843621232:766b700: end prop repl, pRes='05', len 2 > > 0806.843881651:766b700: prop repl 4, pRes='03', len -1 > > 0806.844104549:766b700: end prop repl, pRes='03', len 2 > > 0806.844349716:766b700: XXXXX: tryDoAction 0x5a2e6d0, pnElem 1, nElem 1 > > 0806.844572553:766b700: Action 0x5a2e6d0 transitioned to state: itx > > 0806.844802574:766b700: entering actionCalldoAction(), state: itx > > 0806.844991662:966f700: hasRcvInBuffer on nsd 0xb21b530: pszRcvBuf > 0xb6ae500, lenRcvBuf -1 > > 0806.845201781:966f700: hasRcvInBuffer on nsd 0xb25ab60: pszRcvBuf > 0xb6a2620, lenRcvBuf -1 > > 0806.845409857:966f700: hasRcvInBuffer on nsd 0xb66a760: pszRcvBuf > 0x5b32220, lenRcvBuf -1 > > 0806.845612309:966f700: hasRcvInBuffer on nsd 0xb6853c0: pszRcvBuf > 0xb0a6de0, lenRcvBuf -1 > > 0806.845816142:966f700: hasRcvInBuffer on nsd 0xb6fd9a0: pszRcvBuf > 0xb3cfa80, lenRcvBuf -1 > > 0806.846031471:966f700: hasRcvInBuffer on nsd 0x5dccd70: pszRcvBuf > 0xb7c52d0, lenRcvBuf -1 > > 0806.846245322:966f700: hasRcvInBuffer on nsd 0xa3b8dc0: pszRcvBuf > 0xb7d4c50, lenRcvBuf -1 > > 0806.846459545:966f700: hasRcvInBuffer on nsd 0xa61f620: pszRcvBuf > 0xaec5ad0, lenRcvBuf -1 > > 0806.851832340:966f700: hasRcvInBuffer on nsd 0xa672a90: pszRcvBuf > 0xaf55b00, lenRcvBuf -1 > > 0806.857035160:966f700: hasRcvInBuffer on nsd 0xa6d1380: pszRcvBuf > 0xaf87050, lenRcvBuf -1 > > 0806.857161795:966f700: hasRcvInBuffer on nsd 0xaf32440: pszRcvBuf > 0xb3facd0, lenRcvBuf -1 > > 0806.857288730:966f700: hasRcvInBuffer on nsd 0xad84170: pszRcvBuf > 0xb455560, lenRcvBuf -1 > > 0806.857415826:966f700: hasRcvInBuffer on nsd 0xaff3ec0: pszRcvBuf > 0xb363fb0, lenRcvBuf -1 > > 0806.857538625:966f700: hasRcvInBuffer on nsd 0xb21c290: pszRcvBuf > 0xb377410, lenRcvBuf -1 > > 0806.857661441:966f700: hasRcvInBuffer on nsd 0xb3fcd10: pszRcvBuf > 0x5c62250, lenRcvBuf -1 > > 0806.857783775:966f700: hasRcvInBuffer on nsd 0xb4f1250: pszRcvBuf > 0xb42a6a0, lenRcvBuf -1 > > 0806.857906248:966f700: hasRcvInBuffer on nsd 0x5b7e170: pszRcvBuf > 0x5cb11e0, lenRcvBuf -1 > > 0806.858028461:966f700: hasRcvInBuffer on nsd 0xa61afa0: pszRcvBuf > 0x5b4b140, lenRcvBuf -1 > > 0806.858150704:966f700: hasRcvInBuffer on nsd 0x5c82fa0: pszRcvBuf > 0xb66c460, lenRcvBuf -1 > > 0806.858276530:966f700: hasRcvInBuffer on nsd 0x5b52e30: pszRcvBuf > 0xaf74510, lenRcvBuf -1 > > 0806.858399229:966f700: hasRcvInBuffer on nsd 0xa3c4bd0: pszRcvBuf > 0xb187ac0, lenRcvBuf -1 > > 0806.858525189:966f700: --------<NSDSEL_PTCP> calling select, active fds > (max 52): 6 7 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 > 31 32 33 34 35 36 38 40 41 47 50 52 > > 0806.861597391:966f700: hasRcvInBuffer on nsd 0xa5649e0: pszRcvBuf (nil), > lenRcvBuf 0 > > 0806.861729623:966f700: hasRcvInBuffer on nsd 0xa564d40: pszRcvBuf (nil), > lenRcvBuf 0 > > 0806.861853810:966f700: hasRcvInBuffer on nsd 0xa565bb0: pszRcvBuf > 0xae14080, lenRcvBuf -1 > > 0806.861977507:966f700: hasRcvInBuffer on nsd 0xa583f40: pszRcvBuf > 0xae1b220, lenRcvBuf -1 > > 0806.862099608:966f700: hasRcvInBuffer on nsd 0xa5a0ef0: pszRcvBuf > 0xae425b0, lenRcvBuf -1 > > 0806.862222182:966f700: hasRcvInBuffer on nsd 0xadf7440: pszRcvBuf > 0xb172f30, lenRcvBuf -1 > > 0806.862356388:966f700: hasRcvInBuffer on nsd 0xae226a0: pszRcvBuf > 0xb17e550, lenRcvBuf -1 > > 0806.862482567:966f700: hasRcvInBuffer on nsd 0xae49ae0: pszRcvBuf > 0xb30c870, lenRcvBuf -1 > > 0806.862653944:966f700: netstream 0xae66470 with new data > > 0806.863446427:966f700: gtlsRecordRecv return. nsd 0xae49ae0, iRet 0, > lenRcvd 102, lenRcvBuf 102, ptrRcvBuf 0 > > 0806.863592609:966f700: gtlsRcv return. nsd 0xae49ae0, iRet 0, lenRcvBuf -1, > ptrRcvBuf 102 > > 0806.863794844:966f700: main Q: entry added, size now log 1, phys 2 entries > > 0806.863942651:966f700: main Q: MultiEnqObj advised worker start > > 0806.864247480:966f700: hasRcvInBuffer on nsd 0xa5649e0: pszRcvBuf (nil), > lenRcvBuf 0 > > 0806.864388314:966f700: hasRcvInBuffer on nsd 0xa564d40: pszRcvBuf (nil), > lenRcvBuf 0 > > 0806.864514931:966f700: hasRcvInBuffer on nsd 0xa565bb0: pszRcvBuf > 0xae14080, lenRcvBuf -1 > > 0806.864637925:966f700: hasRcvInBuffer on nsd 0xa583f40: pszRcvBuf > 0xae1b220, lenRcvBuf -1 > > 0806.864770948:966f700: hasRcvInBuffer on nsd 0xa5a0ef0: pszRcvBuf > 0xae425b0, lenRcvBuf -1 > > 0806.864894490:966f700: hasRcvInBuffer on nsd 0xadf7440: pszRcvBuf > 0xb172f30, lenRcvBuf -1 > > 0806.865016270:966f700: hasRcvInBuffer on nsd 0xae226a0: pszRcvBuf > 0xb17e550, lenRcvBuf -1 > > 0806.865137759:966f700: hasRcvInBuffer on nsd 0xae49ae0: pszRcvBuf > 0xb30c870, lenRcvBuf -1 > > 0806.865259187:966f700: hasRcvInBuffer on nsd 0xb152eb0: pszRcvBuf > 0xb5c8620, lenRcvBuf -1 > > 0806.865381030:966f700: hasRcvInBuffer on nsd 0xb2e6170: pszRcvBuf > 0xb5d6430, lenRcvBuf -1 > > 0806.865502527:966f700: hasRcvInBuffer on nsd 0xb5a5a30: pszRcvBuf > 0xb77e290, lenRcvBuf -1 > > 0806.865624236:966f700: hasRcvInBuffer on nsd 0xacf3760: pszRcvBuf > 0xb1f5ed0, lenRcvBuf -1 > > 0806.865745916:966f700: hasRcvInBuffer on nsd 0xad3c320: pszRcvBuf > 0xb214930, lenRcvBuf -1 > > 0806.865867237:966f700: hasRcvInBuffer on nsd 0xad9e300: pszRcvBuf > 0xb21dea0, lenRcvBuf -1 > > 0806.865992393:966f700: hasRcvInBuffer on nsd 0xb1ce440: pszRcvBuf > 0xb68dfa0, lenRcvBuf -1 > > 0806.866124142:966f700: hasRcvInBuffer on nsd 0xb21b530: pszRcvBuf > 0xb6ae500, lenRcvBuf -1 > > 0806.866249334:966f700: hasRcvInBuffer on nsd 0xb25ab60: pszRcvBuf > 0xb6a2620, lenRcvBuf -1 > > 0806.866370740:966f700: hasRcvInBuffer on nsd 0xb66a760: pszRcvBuf > 0x5b32220, lenRcvBuf -1 > > 0806.866493330:966f700: hasRcvInBuffer on nsd 0xb6853c0: pszRcvBuf > 0xb0a6de0, lenRcvBuf -1 > > 0806.866620898:966f700: hasRcvInBuffer on nsd 0xb6fd9a0: pszRcvBuf > 0xb3cfa80, lenRcvBuf -1 > > 0806.867328571:==3366== Thread 3: > > ==3366== Jump to the invalid address stated on the next line > > ==3366== at 0x0: ??? > > ==3366== by 0x6215C3C: MsgSetInputName (in /usr/lib/rsyslog/imuxsock.so) > > ==3366== by 0x6210BD8: ??? (in /usr/lib/rsyslog/imuxsock.so) > > ==3366== by 0x6211392: ??? (in /usr/lib/rsyslog/imuxsock.so) > > ==3366== by 0x43B809: ??? (in /usr/sbin/rsyslogd) > > ==3366== by 0x504A9C9: start_thread (pthread_create.c:300) > > ==3366== by 0x7E6C6FF: ??? > > ==3366== Address 0x0 is not stack'd, malloc'd or (recently) free'd > > ==3366== > > ==3366== > > ==3366== Process terminating with default action of signal 11 (SIGSEGV) > > ==3366== Bad permissions for mapped region at address 0x0 > > ==3366== at 0x0: ??? > > ==3366== by 0x6215C3C: MsgSetInputName (in /usr/lib/rsyslog/imuxsock.so) > > ==3366== by 0x6210BD8: ??? (in /usr/lib/rsyslog/imuxsock.so) > > ==3366== by 0x6211392: ??? (in /usr/lib/rsyslog/imuxsock.so) > > ==3366== by 0x43B809: ??? (in /usr/sbin/rsyslogd) > > ==3366== by 0x504A9C9: start_thread (pthread_create.c:300) > > ==3366== by 0x7E6C6FF: ??? > > ==3366== > > ==3366== HEAP SUMMARY: > > ==3366== in use at exit: 1,682,711 bytes in 4,422 blocks > > ==3366== total heap usage: 635,445 allocs, 631,023 frees, 89,783,678 bytes > allocated > > ==3366== > > ==3366== LEAK SUMMARY: > > ==3366== definitely lost: 0 bytes in 0 blocks > > ==3366== indirectly lost: 0 bytes in 0 blocks > > ==3366== possibly lost: 1,416 bytes in 8 blocks > > ==3366== still reachable: 1,681,295 bytes in 4,414 blocks > > ==3366== suppressed: 0 bytes in 0 blocks > > ==3366== Rerun with --leak-check=full to see details of leaked memory > > ==3366== > > ==3366== For counts of detected and suppressed errors, rerun with: -v > > ==3366== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 43 from 7) > > On May 3, 2012, at 1:02 PM, Rainer Gerhards wrote: > > > Can you run it under valgrind? > > > > Rainer > > > > Matt Wise <[email protected]> hat geschrieben:We're using Rsyslog in our > inftrastructure on a bunch of Ubuntu 10.04 hosts. Due to memory and CPU leaks > in the 4.xx rsyslog code (we're using TLS on everything), we've had to upgrade to > the 5.xx series of code. > > > > Our clients are running the Evax-built Rsyslog 5.6.3 and they generally seem to > work fine. Our rsyslog 'receiver' hosts were running 5.6.3, but we ran into a > serious memory leak that caused Rsyslog to bloat after about a day. We have > upgraded them to hand-built Rsyslog 5.8.10 code (and also tried 5.9.3 at one > point). This worked fine for about 2 days.. and now we are getting near non- > stop Seg Faults on the receiver host running Rsyslog. > > > > I've tried everything I can think of, but we still have the problem.. here are our > configs: > > > > /etc/rsyslog.d/custom.conf: > >> $ModLoad imtcp # load TCP listener > >> $DefaultNetstreamDriverCAFile /etc/rsyslog.d/certs/nextdoor/logd-ca.pem > >> $DefaultNetstreamDriverCertFile > /etc/rsyslog.d/certs/nextdoor/nextdoor.pem > >> $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/certs/nextdoor/nextdoor.key > >> $DefaultNetstreamDriver gtls > >> $InputTCPServerStreamDriverMode 1 > >> $InputTCPServerStreamDriverAuthMode x509/certvalid > >> $CreateDirs on > >> $template FILENAME,"/mnt/logs/%fromhost-ip%/%syslogfacility-text%- > %$year%-%$month%-%$day%.log" > >> *.* ?FILENAME > >> # Create an additional socket in postfix's chroot in order not to break > >> # mail logging when rsyslog is restarted. If the directory is missing, > >> # rsyslog will silently skip creating the socket. > >> $AddUnixListenSocket /var/spool/postfix/dev/log > > > > > > /etc/rsyslog.conf > >> $ModLoad imuxsock # provides support for local system logging > >> $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > >> $ModLoad immark # provides --MARK-- message capability > >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > >> $EscapeControlCharactersOnReceive off > >> $RepeatedMsgReduction off > >> $FileOwner sysadm > >> $FileGroup adm > >> $FileCreateMode 0640 > >> $DirOwner sysadm > >> $DirGroup adm > >> $DirCreateMode 0755 > >> $Umask 0022 > >> $PreserveFQDN on > >> $IncludeConfig /etc/rsyslog.d/*.conf > > > > > >> rsyslogd 5.8.10, compiled with: > >> FEATURE_REGEXP: Yes > >> FEATURE_LARGEFILE: No > >> GSSAPI Kerberos 5 support: Yes > >> FEATURE_DEBUG (debug build, slow code): No > >> 32bit Atomic operations supported: Yes > >> 64bit Atomic operations supported: Yes > >> Runtime Instrumentation (slow code): No > > > > > >> 2.6.32-317-ec2 #36-Ubuntu SMP Fri Jul 8 18:12:30 UTC 2011 x86_64 > GNU/Linux > > > > > > And here's the log output of the failure.. > >> 8901.295674722:7f04c029a700: New connect on NSD 0x1db0ee0 > >> . > >> > >> 8901.359429713:7f04c1a9d700: Message from UNIX socket: #3 > >> 8901.359450981:7f04c1a9d700: XXX: pre CM loop, length of control message > 32 > >> 8901.359461759:7f04c1a9d700: XXX: in CM loop, 1, 2 > >> 8901.359471828:7f04c1a9d700: XXX: got credentials pid 24640 > >> 8901.359481718:7f04c1a9d700: XXX: post CM loop > >> 8901.359493088:7f04c1a9d700: imuxsock: no ratelimiter for pid 24640, > creating one > >> > >> Segmentation fault > > > > The failure looks the same every single time... some message comes in, the > 'CM Loop' comments start, the 'rate limit' comment comes, and then the > segfault. > > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
