I'm running: valgrind /usr/sbin/rsyslogd -c4 -n -d 2>&1 | tee /tmp/valgrind.log. Once I have the output of the failure, I'll get you the last few thousand lines, sanitize them of any important data of ours, and then get you the log.
—Matt On May 3, 2012, at 1:02 PM, Rainer Gerhards wrote: > Can you run it under valgrind? > > Rainer > > Matt Wise <[email protected]> hat geschrieben:We're using Rsyslog in our > inftrastructure on a bunch of Ubuntu 10.04 hosts. Due to memory and CPU leaks > in the 4.xx rsyslog code (we're using TLS on everything), we've had to > upgrade to the 5.xx series of code. > > Our clients are running the Evax-built Rsyslog 5.6.3 and they generally seem > to work fine. Our rsyslog 'receiver' hosts were running 5.6.3, but we ran > into a serious memory leak that caused Rsyslog to bloat after about a day. We > have upgraded them to hand-built Rsyslog 5.8.10 code (and also tried 5.9.3 at > one point). This worked fine for about 2 days.. and now we are getting near > non-stop Seg Faults on the receiver host running Rsyslog. > > I've tried everything I can think of, but we still have the problem.. here > are our configs: > > /etc/rsyslog.d/custom.conf: >> $ModLoad imtcp # load TCP listener >> $DefaultNetstreamDriverCAFile /etc/rsyslog.d/certs/nextdoor/logd-ca.pem >> $DefaultNetstreamDriverCertFile /etc/rsyslog.d/certs/nextdoor/nextdoor.pem >> $DefaultNetstreamDriverKeyFile /etc/rsyslog.d/certs/nextdoor/nextdoor.key >> $DefaultNetstreamDriver gtls >> $InputTCPServerStreamDriverMode 1 >> $InputTCPServerStreamDriverAuthMode x509/certvalid >> $CreateDirs on >> $template >> FILENAME,"/mnt/logs/%fromhost-ip%/%syslogfacility-text%-%$year%-%$month%-%$day%.log" >> *.* ?FILENAME >> # Create an additional socket in postfix's chroot in order not to break >> # mail logging when rsyslog is restarted. If the directory is missing, >> # rsyslog will silently skip creating the socket. >> $AddUnixListenSocket /var/spool/postfix/dev/log > > > /etc/rsyslog.conf >> $ModLoad imuxsock # provides support for local system logging >> $ModLoad imklog # provides kernel logging support (previously done by >> rklogd) >> $ModLoad immark # provides --MARK-- message capability >> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat >> $EscapeControlCharactersOnReceive off >> $RepeatedMsgReduction off >> $FileOwner sysadm >> $FileGroup adm >> $FileCreateMode 0640 >> $DirOwner sysadm >> $DirGroup adm >> $DirCreateMode 0755 >> $Umask 0022 >> $PreserveFQDN on >> $IncludeConfig /etc/rsyslog.d/*.conf > > >> rsyslogd 5.8.10, compiled with: >> FEATURE_REGEXP: Yes >> FEATURE_LARGEFILE: No >> GSSAPI Kerberos 5 support: Yes >> FEATURE_DEBUG (debug build, slow code): No >> 32bit Atomic operations supported: Yes >> 64bit Atomic operations supported: Yes >> Runtime Instrumentation (slow code): No > > >> 2.6.32-317-ec2 #36-Ubuntu SMP Fri Jul 8 18:12:30 UTC 2011 x86_64 GNU/Linux > > > And here's the log output of the failure.. >> 8901.295674722:7f04c029a700: New connect on NSD 0x1db0ee0 >> . >> >> 8901.359429713:7f04c1a9d700: Message from UNIX socket: #3 >> 8901.359450981:7f04c1a9d700: XXX: pre CM loop, length of control message 32 >> 8901.359461759:7f04c1a9d700: XXX: in CM loop, 1, 2 >> 8901.359471828:7f04c1a9d700: XXX: got credentials pid 24640 >> 8901.359481718:7f04c1a9d700: XXX: post CM loop >> 8901.359493088:7f04c1a9d700: imuxsock: no ratelimiter for pid 24640, >> creating one >> >> Segmentation fault > > The failure looks the same every single time... some message comes in, the > 'CM Loop' comments start, the 'rate limit' comment comes, and then the > segfault. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
