Hi David, Thank you for your reply aswell. I don't see any increased load on the server if it's having issues, cpu utilization is about 5%.
I parse the apache logs like this: CustomLog "|/usr/bin/logger -t apache -p local0.info" combined So the sum up your advice: - run udp - upgrade rsyslog (style filters is better) - increment $DynaFileCacheSize The only problem is is that the client wants to use tls but using udp this has to be out of the question? Regards, Erik On Mon, 9 Sep 2013 07:36:57 -0700 (PDT) David Lang <[email protected]> wrote: > when you use reliable logging (tcp) to send logs, then you can run into the > problem that if there is a problem delivering logs, your system that is > generating the logs will stop and wait for the logs to be delivered. > > This is probably what is going on here. > > on your central box, does it look like it is falling behind? what sort of CPU > utilization are you seeing for rsyslog there? > > rsyslog 7 is significantly faster than rsyslog 5, so that shold help (but it > may > not be enough, depending on how bad the problem is) > > the fact that you use if..then style filters is a very large performance hit > on > rsyslog 5, but not on 7, so you will get a huge speedup just from that. > > One issue that I see is that you are using the filename templates, the > default > number of files that rsyslog keeps open for this is _way_ too small for > anything > serious. you will need to set $DynaFileCacheSize to something large enough to > handle all the open files that you will have at any one time. try setting it > to > 1000 to get you started and see if that makes a difference. > > how are you getting the logs from apache to rsyslog? > > David Lang > > > On Mon, 9 > Sep 2013, Erik van Dam wrote: > > > Date: Mon, 9 Sep 2013 15:45:32 +0200 > > From: Erik van Dam <[email protected]> > > Reply-To: rsyslog-users <[email protected]> > > To: [email protected] > > Subject: [rsyslog] rsyslog bringing machines down due amount of messages (?) > > > > Hi everybody, > > > > I have a 8 client machines who send tcp syslog messages to an syslog > > server. On two machines i'm logging more than the others; apache access > > logs. This results in two completely unresponsive machines. Now if i > > restart rsyslog on the machine that becomes unresponsive everything is > > fine, i can't notice anything like load or memory consumption while they > > are unresponsive? Previously i used certificates but in the process of > > finding the problem i disabled this. Further i tried using disk queue. > > IMUXSock set to 0 as of > > http://www.rsyslog.com/tag/imuxsockratelimitinterval/ > > > > Private information has been renamed, i'm grateful if anybody has some > > pointers for me. Sofar i tried: > > > > - disk queue = not really an affect > > - restart rsyslog = relieves (a buffer within rsyslog that gets full?) > > - remove tls on client side = not really an affect > > > > What might resolve some issues: > > > > - change tcp to udp > > - goto rsyslog 7.4.(4) ? > > > > =====================================================CLIENT========================================================== > > Rsyslog version: > > > > rsyslog-5.8.10-2.el6.x86_64 > > rsyslog-gnutls-5.8.10-2.el6.x86_64 > > > > > > $ModLoad imuxsock.so # provides support for local system logging > > (e.g. via logger command) > > $ModLoad imklog.so # provides kernel logging support (previously done by > > rklogd) > > $IMUXSockRateLimitInterval 0 > > $SystemLogRateLimitInterval 0 > > $SystemLogRateLimitBurst 0 > > $WorkDirectory /var/log > > $MainMsgQueueFileName /var/log/rsyslog.main.q > > $ActionQueueFileName /var/log/rsyslog.action.q > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > > > > > local5.* @@syslogserver:514 > > & ~ > > > > local0.* @@syslogserver:514 # > > log the access logs > > & ~ > > > > local1.* /var/log/httpd/error_log > > local1.* @@syslogserver:514 > > # log the error logs > > & ~ > > > > *.* @@syslogserver:514 # forward everything to remote server > > > > *.info;mail.none;authpriv.none;cron.none /var/log/messages > > authpriv.* /var/log/secure > > mail.* -/var/log/maillog > > cron.* /var/log/cron > > *.emerg * > > uucp,news.crit /var/log/spooler > > local7.* /var/log/boot.log > > > > =====================================================SERVER========================================================== > > rsyslog-gnutls-5.8.10-2.el6.x86_64 > > rsyslog-5.8.10-2.el6.x86_64 > > > > > > $ModLoad imuxsock.so # provides support for local system logging > > (e.g. via logger command) > > $ModLoad imklog.so # provides kernel logging support (previously done by > > rklogd) > > $IMUXSockRateLimitInterval 0 > > $ModLoad imudp.so > > $UDPServerRun 514 > > $ModLoad imtcp.so > > $PreserveFQDN on > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > > > $DefaultNetstreamDriver gtls > > $DefaultNetstreamDriverCAFile /etc/rsyslog/protected/ca.pem > > $DefaultNetstreamDriverCertFile /etc/rsyslog/protected/cert.pem > > $DefaultNetstreamDriverKeyFile /etc/rsyslog/protected/key.pem > > > > $InputTCPServerStreamDriverPermittedPeer machine1 > > $InputTCPServerStreamDriverPermittedPeer machine2 > > $InputTCPServerStreamDriverPermittedPeer machine3 > > $InputTCPServerStreamDriverPermittedPeer machine4 > > $InputTCPServerStreamDriverPermittedPeer machine5 > > $InputTCPServerStreamDriverPermittedPeer machine6 > > $InputTCPServerStreamDriverPermittedPeer machine7 > > $InputTCPServerStreamDriverMode 1 > > $InputTCPServerRun 514 > > > > > > $template > > DailyPerHostLogs,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_messages.log" > > $template > > DailyrootshPerHostLogs,"/bigdisk/syslog/rootsh/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_messages.log" > > local5.info -?DailyrootshPerHostLogs > > & ~ > > > > $template > > cactilog,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_cacti-access.log" > > if $syslogfacility-text == 'local0' and $msg contains '/cacti' then > > -?cactilog > > & ~ > > > > $template > > nagioslog,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_nagios-access.log" > > if $syslogfacility-text == 'local0' and $msg contains '/nagios' then > > -?nagioslog > > & ~ > > > > $template > > somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log" > > if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then > > -?somedomainname > > & ~ > > > > $template > > somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log" > > if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then > > -?somedomainname > > & ~ > > > > $template > > somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log" > > if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then > > -?somedomainname > > & ~ > > > > $template > > somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log" > > if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then > > -?somedomainname > > & ~ > > > > $template > > nagiosandcactierror,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_nagiosandcactierror.log" > > if $syslogfacility-text == 'local1' then -?nagiosandcactierror > > & ~ > > > > local0.* ~ > > > > *.* -?DailyPerHostLogs > > > > > > > > > > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > -- Met vriendelijke groet, Erik van Dam RedBee / FortyTwo _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
