Here are the last 100 lines with the valgrind failure: > 0806.842884190:766b700: prop repl 4, pRes='2012', len -1 > 0806.843084207:766b700: end prop repl, pRes='2012', len 4 > 0806.843400424:766b700: prop repl 4, pRes='05', len -1 > 0806.843621232:766b700: end prop repl, pRes='05', len 2 > 0806.843881651:766b700: prop repl 4, pRes='03', len -1 > 0806.844104549:766b700: end prop repl, pRes='03', len 2 > 0806.844349716:766b700: XXXXX: tryDoAction 0x5a2e6d0, pnElem 1, nElem 1 > 0806.844572553:766b700: Action 0x5a2e6d0 transitioned to state: itx > 0806.844802574:766b700: entering actionCalldoAction(), state: itx > 0806.844991662:966f700: hasRcvInBuffer on nsd 0xb21b530: pszRcvBuf 0xb6ae500, > lenRcvBuf -1 > 0806.845201781:966f700: hasRcvInBuffer on nsd 0xb25ab60: pszRcvBuf 0xb6a2620, > lenRcvBuf -1 > 0806.845409857:966f700: hasRcvInBuffer on nsd 0xb66a760: pszRcvBuf 0x5b32220, > lenRcvBuf -1 > 0806.845612309:966f700: hasRcvInBuffer on nsd 0xb6853c0: pszRcvBuf 0xb0a6de0, > lenRcvBuf -1 > 0806.845816142:966f700: hasRcvInBuffer on nsd 0xb6fd9a0: pszRcvBuf 0xb3cfa80, > lenRcvBuf -1 > 0806.846031471:966f700: hasRcvInBuffer on nsd 0x5dccd70: pszRcvBuf 0xb7c52d0, > lenRcvBuf -1 > 0806.846245322:966f700: hasRcvInBuffer on nsd 0xa3b8dc0: pszRcvBuf 0xb7d4c50, > lenRcvBuf -1 > 0806.846459545:966f700: hasRcvInBuffer on nsd 0xa61f620: pszRcvBuf 0xaec5ad0, > lenRcvBuf -1 > 0806.851832340:966f700: hasRcvInBuffer on nsd 0xa672a90: pszRcvBuf 0xaf55b00, > lenRcvBuf -1 > 0806.857035160:966f700: hasRcvInBuffer on nsd 0xa6d1380: pszRcvBuf 0xaf87050, > lenRcvBuf -1 > 0806.857161795:966f700: hasRcvInBuffer on nsd 0xaf32440: pszRcvBuf 0xb3facd0, > lenRcvBuf -1 > 0806.857288730:966f700: hasRcvInBuffer on nsd 0xad84170: pszRcvBuf 0xb455560, > lenRcvBuf -1 > 0806.857415826:966f700: hasRcvInBuffer on nsd 0xaff3ec0: pszRcvBuf 0xb363fb0, > lenRcvBuf -1 > 0806.857538625:966f700: hasRcvInBuffer on nsd 0xb21c290: pszRcvBuf 0xb377410, > lenRcvBuf -1 > 0806.857661441:966f700: hasRcvInBuffer on nsd 0xb3fcd10: pszRcvBuf 0x5c62250, > lenRcvBuf -1 > 0806.857783775:966f700: hasRcvInBuffer on nsd 0xb4f1250: pszRcvBuf 0xb42a6a0, > lenRcvBuf -1 > 0806.857906248:966f700: hasRcvInBuffer on nsd 0x5b7e170: pszRcvBuf 0x5cb11e0, > lenRcvBuf -1 > 0806.858028461:966f700: hasRcvInBuffer on nsd 0xa61afa0: pszRcvBuf 0x5b4b140, > lenRcvBuf -1 > 0806.858150704:966f700: hasRcvInBuffer on nsd 0x5c82fa0: pszRcvBuf 0xb66c460, > lenRcvBuf -1 > 0806.858276530:966f700: hasRcvInBuffer on nsd 0x5b52e30: pszRcvBuf 0xaf74510, > lenRcvBuf -1 > 0806.858399229:966f700: hasRcvInBuffer on nsd 0xa3c4bd0: pszRcvBuf 0xb187ac0, > lenRcvBuf -1 > 0806.858525189:966f700: --------<NSDSEL_PTCP> calling select, active fds (max > 52): 6 7 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 > 32 33 34 35 36 38 40 41 47 50 52 > 0806.861597391:966f700: hasRcvInBuffer on nsd 0xa5649e0: pszRcvBuf (nil), > lenRcvBuf 0 > 0806.861729623:966f700: hasRcvInBuffer on nsd 0xa564d40: pszRcvBuf (nil), > lenRcvBuf 0 > 0806.861853810:966f700: hasRcvInBuffer on nsd 0xa565bb0: pszRcvBuf 0xae14080, > lenRcvBuf -1 > 0806.861977507:966f700: hasRcvInBuffer on nsd 0xa583f40: pszRcvBuf 0xae1b220, > lenRcvBuf -1 > 0806.862099608:966f700: hasRcvInBuffer on nsd 0xa5a0ef0: pszRcvBuf 0xae425b0, > lenRcvBuf -1 > 0806.862222182:966f700: hasRcvInBuffer on nsd 0xadf7440: pszRcvBuf 0xb172f30, > lenRcvBuf -1 > 0806.862356388:966f700: hasRcvInBuffer on nsd 0xae226a0: pszRcvBuf 0xb17e550, > lenRcvBuf -1 > 0806.862482567:966f700: hasRcvInBuffer on nsd 0xae49ae0: pszRcvBuf 0xb30c870, > lenRcvBuf -1 > 0806.862653944:966f700: netstream 0xae66470 with new data > 0806.863446427:966f700: gtlsRecordRecv return. nsd 0xae49ae0, iRet 0, lenRcvd > 102, lenRcvBuf 102, ptrRcvBuf 0 > 0806.863592609:966f700: gtlsRcv return. nsd 0xae49ae0, iRet 0, lenRcvBuf -1, > ptrRcvBuf 102 > 0806.863794844:966f700: main Q: entry added, size now log 1, phys 2 entries > 0806.863942651:966f700: main Q: MultiEnqObj advised worker start > 0806.864247480:966f700: hasRcvInBuffer on nsd 0xa5649e0: pszRcvBuf (nil), > lenRcvBuf 0 > 0806.864388314:966f700: hasRcvInBuffer on nsd 0xa564d40: pszRcvBuf (nil), > lenRcvBuf 0 > 0806.864514931:966f700: hasRcvInBuffer on nsd 0xa565bb0: pszRcvBuf 0xae14080, > lenRcvBuf -1 > 0806.864637925:966f700: hasRcvInBuffer on nsd 0xa583f40: pszRcvBuf 0xae1b220, > lenRcvBuf -1 > 0806.864770948:966f700: hasRcvInBuffer on nsd 0xa5a0ef0: pszRcvBuf 0xae425b0, > lenRcvBuf -1 > 0806.864894490:966f700: hasRcvInBuffer on nsd 0xadf7440: pszRcvBuf 0xb172f30, > lenRcvBuf -1 > 0806.865016270:966f700: hasRcvInBuffer on nsd 0xae226a0: pszRcvBuf 0xb17e550, > lenRcvBuf -1 > 0806.865137759:966f700: hasRcvInBuffer on nsd 0xae49ae0: pszRcvBuf 0xb30c870, > lenRcvBuf -1 > 0806.865259187:966f700: hasRcvInBuffer on nsd 0xb152eb0: pszRcvBuf 0xb5c8620, > lenRcvBuf -1 > 0806.865381030:966f700: hasRcvInBuffer on nsd 0xb2e6170: pszRcvBuf 0xb5d6430, > lenRcvBuf -1 > 0806.865502527:966f700: hasRcvInBuffer on nsd 0xb5a5a30: pszRcvBuf 0xb77e290, > lenRcvBuf -1 > 0806.865624236:966f700: hasRcvInBuffer on nsd 0xacf3760: pszRcvBuf 0xb1f5ed0, > lenRcvBuf -1 > 0806.865745916:966f700: hasRcvInBuffer on nsd 0xad3c320: pszRcvBuf 0xb214930, > lenRcvBuf -1 > 0806.865867237:966f700: hasRcvInBuffer on nsd 0xad9e300: pszRcvBuf 0xb21dea0, > lenRcvBuf -1 > 0806.865992393:966f700: hasRcvInBuffer on nsd 0xb1ce440: pszRcvBuf 0xb68dfa0, > lenRcvBuf -1 > 0806.866124142:966f700: hasRcvInBuffer on nsd 0xb21b530: pszRcvBuf 0xb6ae500, > lenRcvBuf -1 > 0806.866249334:966f700: hasRcvInBuffer on nsd 0xb25ab60: pszRcvBuf 0xb6a2620, > lenRcvBuf -1 > 0806.866370740:966f700: hasRcvInBuffer on nsd 0xb66a760: pszRcvBuf 0x5b32220, > lenRcvBuf -1 > 0806.866493330:966f700: hasRcvInBuffer on nsd 0xb6853c0: pszRcvBuf 0xb0a6de0, > lenRcvBuf -1 > 0806.866620898:966f700: hasRcvInBuffer on nsd 0xb6fd9a0: pszRcvBuf 0xb3cfa80, > lenRcvBuf -1 > 0806.867328571:==3366== Thread 3: > ==3366== Jump to the invalid address stated on the next line > ==3366== at 0x0: ??? > ==3366== by 0x6215C3C: MsgSetInputName (in /usr/lib/rsyslog/imuxsock.so) > ==3366== by 0x6210BD8: ??? (in /usr/lib/rsyslog/imuxsock.so) > ==3366== by 0x6211392: ??? (in /usr/lib/rsyslog/imuxsock.so) > ==3366== by 0x43B809: ??? (in /usr/sbin/rsyslogd) > ==3366== by 0x504A9C9: start_thread (pthread_create.c:300) > ==3366== by 0x7E6C6FF: ??? > ==3366== Address 0x0 is not stack'd, malloc'd or (recently) free'd > ==3366== > ==3366== > ==3366== Process terminating with default action of signal 11 (SIGSEGV) > ==3366== Bad permissions for mapped region at address 0x0 > ==3366== at 0x0: ??? > ==3366== by 0x6215C3C: MsgSetInputName (in /usr/lib/rsyslog/imuxsock.so) > ==3366== by 0x6210BD8: ??? (in /usr/lib/rsyslog/imuxsock.so) > ==3366== by 0x6211392: ??? (in /usr/lib/rsyslog/imuxsock.so) > ==3366== by 0x43B809: ??? (in /usr/sbin/rsyslogd) > ==3366== by 0x504A9C9: start_thread (pthread_create.c:300) > ==3366== by 0x7E6C6FF: ??? > ==3366== > ==3366== HEAP SUMMARY: > ==3366== in use at exit: 1,682,711 bytes in 4,422 blocks > ==3366== total heap usage: 635,445 allocs, 631,023 frees, 89,783,678 bytes > allocated > ==3366== > ==3366== LEAK SUMMARY: > ==3366== definitely lost: 0 bytes in 0 blocks > ==3366== indirectly lost: 0 bytes in 0 blocks > ==3366== possibly lost: 1,416 bytes in 8 blocks > ==3366== still reachable: 1,681,295 bytes in 4,414 blocks > ==3366== suppressed: 0 bytes in 0 blocks > ==3366== Rerun with --leak-check=full to see details of leaked memory > ==3366== > ==3366== For counts of detected and suppressed errors, rerun with: -v > ==3366== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 43 from 7)
On May 3, 2012, at 1:02 PM, Rainer Gerhards wrote: > Can you run it under valgrind? > > Rainer > > Matt Wise <[email protected]> hat geschrieben:We're using Rsyslog in our > inftrastructure on a bunch of Ubuntu 10.04 hosts. Due to memory and CPU leaks > in the 4.xx rsyslog code (we're using TLS on everything), we've had to > upgrade to the 5.xx series of code. > > Our clients are running the Evax-built Rsyslog 5.6.3 and they generally seem > to work fine. Our rsyslog 'receiver' hosts were running 5.6.3, but we ran > into a serious memory leak that caused Rsyslog to bloat after about a day. We > have upgraded them to hand-built Rsyslog 5.8.10 code (and also tried 5.9.3 at > one point). This worked fine for about 2 days.. and now we are getting near > non-stop Seg Faults on the receiver host running Rsyslog. > > I've tried everything I can think of, but we still have the problem.. here > are our configs: > > /etc/rsyslog.d/custom.conf: >> $ModLoad imtcp # load TCP listener >> $DefaultNetstreamDriverCAFile /etc/rsyslog.d/certs/nextdoor/logd-ca.pem >> $DefaultNetstreamDriverCertFile /etc/rsyslog.d/certs/nextdoor/nextdoor.pem >> $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/certs/nextdoor/nextdoor.key >> $DefaultNetstreamDriver gtls >> $InputTCPServerStreamDriverMode 1 >> $InputTCPServerStreamDriverAuthMode x509/certvalid >> $CreateDirs on >> $template >> FILENAME,"/mnt/logs/%fromhost-ip%/%syslogfacility-text%-%$year%-%$month%-%$day%.log" >> *.* ?FILENAME >> # Create an additional socket in postfix's chroot in order not to break >> # mail logging when rsyslog is restarted. If the directory is missing, >> # rsyslog will silently skip creating the socket. >> $AddUnixListenSocket /var/spool/postfix/dev/log > > > /etc/rsyslog.conf >> $ModLoad imuxsock # provides support for local system logging >> $ModLoad imklog # provides kernel logging support (previously done by >> rklogd) >> $ModLoad immark # provides --MARK-- message capability >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> $EscapeControlCharactersOnReceive off >> $RepeatedMsgReduction off >> $FileOwner sysadm >> $FileGroup adm >> $FileCreateMode 0640 >> $DirOwner sysadm >> $DirGroup adm >> $DirCreateMode 0755 >> $Umask 0022 >> $PreserveFQDN on >> $IncludeConfig /etc/rsyslog.d/*.conf > > >> rsyslogd 5.8.10, compiled with: >> FEATURE_REGEXP: Yes >> FEATURE_LARGEFILE: No >> GSSAPI Kerberos 5 support: Yes >> FEATURE_DEBUG (debug build, slow code): No >> 32bit Atomic operations supported: Yes >> 64bit Atomic operations supported: Yes >> Runtime Instrumentation (slow code): No > > >> 2.6.32-317-ec2 #36-Ubuntu SMP Fri Jul 8 18:12:30 UTC 2011 x86_64 GNU/Linux > > > And here's the log output of the failure.. >> 8901.295674722:7f04c029a700: New connect on NSD 0x1db0ee0 >> . >> >> 8901.359429713:7f04c1a9d700: Message from UNIX socket: #3 >> 8901.359450981:7f04c1a9d700: XXX: pre CM loop, length of control message 32 >> 8901.359461759:7f04c1a9d700: XXX: in CM loop, 1, 2 >> 8901.359471828:7f04c1a9d700: XXX: got credentials pid 24640 >> 8901.359481718:7f04c1a9d700: XXX: post CM loop >> 8901.359493088:7f04c1a9d700: imuxsock: no ratelimiter for pid 24640, >> creating one >> >> Segmentation fault > > The failure looks the same every single time... some message comes in, the > 'CM Loop' comments start, the 'rate limit' comment comes, and then the > segfault. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
