On Thu, 12 Sep 2013 08:07:49 -0700 (PDT) David Lang <[email protected]> wrote:
> On Thu, 12 Sep 2013, Erik van Dam wrote: > > > Hi David, > > > > Thank you for your reply aswell. I don't see any increased load on the > > server if it's having issues, cpu utilization is about 5%. > > > > I parse the apache logs like this: > > > > CustomLog "|/usr/bin/logger -t apache -p local0.info" combined > > > > So the sum up your advice: > > - run udp > > - upgrade rsyslog (style filters is better) > > - increment $DynaFileCacheSize > > > > The only problem is is that the client wants to use tls but using udp this > > has to be out of the question? > > The problem that you have when using a TCP based transport (which is needed > to > use encryption) is that your entire system can back up if the server gets > behind > in processing the messages. > > you can address this on the client by setting high/low watermarks to tell > rsyslog that if it gets behind and the queue gets close to filling up that it > should throw away some messages instead of letting the queue fill up, which > is > what causes things to stop. > > When the queue fills up, the proceses writing the logs are going to stall, > but > they will not be eating any CPU, they will be blocked. > > On the syslog server, the system may have it's CPU pegged, or it may be > stalled > waiting for disk I/O (which should show up as a high CPU wait time) > > are you sure the syslog server is not showing high utilization when you get > into > trouble? > > even before you get into trouble, could you please run top on the syslog > server > and then hit 'H' to show the individual threads? if you can cut-n-paste the > upper part of the report (the header info and the rsyslog lines, which I > expect > to be the top processes showing)? I expect that it will help us see what's > going > on. > > David Lang > > David Lang > Hi David, Thanks for your reply. No i'm not sure about the syslog server itself, i can only talk about the clients, but i will monitor the syslog server on the next run. Regards, Erik > > Regards, > > Erik > > > > > > > > On Mon, 9 Sep 2013 07:36:57 -0700 (PDT) > > David Lang <[email protected]> wrote: > > > >> when you use reliable logging (tcp) to send logs, then you can run into the > >> problem that if there is a problem delivering logs, your system that is > >> generating the logs will stop and wait for the logs to be delivered. > >> > >> This is probably what is going on here. > >> > >> on your central box, does it look like it is falling behind? what sort of > >> CPU > >> utilization are you seeing for rsyslog there? > >> > >> rsyslog 7 is significantly faster than rsyslog 5, so that shold help (but > >> it may > >> not be enough, depending on how bad the problem is) > >> > >> the fact that you use if..then style filters is a very large performance > >> hit on > >> rsyslog 5, but not on 7, so you will get a huge speedup just from that. > >> > >> One issue that I see is that you are using the filename templates, the > >> default > >> number of files that rsyslog keeps open for this is _way_ too small for > >> anything > >> serious. you will need to set $DynaFileCacheSize to something large enough > >> to > >> handle all the open files that you will have at any one time. try setting > >> it to > >> 1000 to get you started and see if that makes a difference. > >> > >> how are you getting the logs from apache to rsyslog? > >> > >> David Lang > >> > >> > >> On Mon, 9 > >> Sep 2013, Erik van Dam wrote: > >> > >>> Date: Mon, 9 Sep 2013 15:45:32 +0200 > >>> From: Erik van Dam <[email protected]> > >>> Reply-To: rsyslog-users <[email protected]> > >>> To: [email protected] > >>> Subject: [rsyslog] rsyslog bringing machines down due amount of messages > >>> (?) > >>> > >>> Hi everybody, > >>> > >>> I have a 8 client machines who send tcp syslog messages to an syslog > >>> server. On two machines i'm logging more than the others; apache access > >>> logs. This results in two completely unresponsive machines. Now if i > >>> restart rsyslog on the machine that becomes unresponsive everything is > >>> fine, i can't notice anything like load or memory consumption while they > >>> are unresponsive? Previously i used certificates but in the process of > >>> finding the problem i disabled this. Further i tried using disk queue. > >>> IMUXSock set to 0 as of > >>> http://www.rsyslog.com/tag/imuxsockratelimitinterval/ > >>> > >>> Private information has been renamed, i'm grateful if anybody has some > >>> pointers for me. Sofar i tried: > >>> > >>> - disk queue = not really an affect > >>> - restart rsyslog = relieves (a buffer within rsyslog that gets full?) > >>> - remove tls on client side = not really an affect > >>> > >>> What might resolve some issues: > >>> > >>> - change tcp to udp > >>> - goto rsyslog 7.4.(4) ? > >>> > >>> =====================================================CLIENT========================================================== > >>> Rsyslog version: > >>> > >>> rsyslog-5.8.10-2.el6.x86_64 > >>> rsyslog-gnutls-5.8.10-2.el6.x86_64 > >>> > >>> > >>> $ModLoad imuxsock.so # provides support for local system logging > >>> (e.g. via logger command) > >>> $ModLoad imklog.so # provides kernel logging support (previously > >>> done by rklogd) > >>> $IMUXSockRateLimitInterval 0 > >>> $SystemLogRateLimitInterval 0 > >>> $SystemLogRateLimitBurst 0 > >>> $WorkDirectory /var/log > >>> $MainMsgQueueFileName /var/log/rsyslog.main.q > >>> $ActionQueueFileName /var/log/rsyslog.action.q > >>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > >>> > >>> > >>> local5.* @@syslogserver:514 > >>> & ~ > >>> > >>> local0.* @@syslogserver:514 # > >>> log the access logs > >>> & ~ > >>> > >>> local1.* /var/log/httpd/error_log > >>> local1.* > >>> @@syslogserver:514 # log the error logs > >>> & ~ > >>> > >>> *.* @@syslogserver:514 # forward everything to remote server > >>> > >>> *.info;mail.none;authpriv.none;cron.none /var/log/messages > >>> authpriv.* /var/log/secure > >>> mail.* -/var/log/maillog > >>> cron.* /var/log/cron > >>> *.emerg * > >>> uucp,news.crit /var/log/spooler > >>> local7.* /var/log/boot.log > >>> > >>> =====================================================SERVER========================================================== > >>> rsyslog-gnutls-5.8.10-2.el6.x86_64 > >>> rsyslog-5.8.10-2.el6.x86_64 > >>> > >>> > >>> $ModLoad imuxsock.so # provides support for local system logging > >>> (e.g. via logger command) > >>> $ModLoad imklog.so # provides kernel logging support (previously > >>> done by rklogd) > >>> $IMUXSockRateLimitInterval 0 > >>> $ModLoad imudp.so > >>> $UDPServerRun 514 > >>> $ModLoad imtcp.so > >>> $PreserveFQDN on > >>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > >>> > >>> $DefaultNetstreamDriver gtls > >>> $DefaultNetstreamDriverCAFile /etc/rsyslog/protected/ca.pem > >>> $DefaultNetstreamDriverCertFile /etc/rsyslog/protected/cert.pem > >>> $DefaultNetstreamDriverKeyFile /etc/rsyslog/protected/key.pem > >>> > >>> $InputTCPServerStreamDriverPermittedPeer machine1 > >>> $InputTCPServerStreamDriverPermittedPeer machine2 > >>> $InputTCPServerStreamDriverPermittedPeer machine3 > >>> $InputTCPServerStreamDriverPermittedPeer machine4 > >>> $InputTCPServerStreamDriverPermittedPeer machine5 > >>> $InputTCPServerStreamDriverPermittedPeer machine6 > >>> $InputTCPServerStreamDriverPermittedPeer machine7 > >>> $InputTCPServerStreamDriverMode 1 > >>> $InputTCPServerRun 514 > >>> > >>> > >>> $template > >>> DailyPerHostLogs,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_messages.log" > >>> $template > >>> DailyrootshPerHostLogs,"/bigdisk/syslog/rootsh/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_messages.log" > >>> local5.info > >>> -?DailyrootshPerHostLogs > >>> & ~ > >>> > >>> $template > >>> cactilog,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_cacti-access.log" > >>> if $syslogfacility-text == 'local0' and $msg contains '/cacti' then > >>> -?cactilog > >>> & ~ > >>> > >>> $template > >>> nagioslog,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_nagios-access.log" > >>> if $syslogfacility-text == 'local0' and $msg contains '/nagios' then > >>> -?nagioslog > >>> & ~ > >>> > >>> $template > >>> somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log" > >>> if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' > >>> then -?somedomainname > >>> & ~ > >>> > >>> $template > >>> somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log" > >>> if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' > >>> then -?somedomainname > >>> & ~ > >>> > >>> $template > >>> somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log" > >>> if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' > >>> then -?somedomainname > >>> & ~ > >>> > >>> $template > >>> somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log" > >>> if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' > >>> then -?somedomainname > >>> & ~ > >>> > >>> $template > >>> nagiosandcactierror,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_nagiosandcactierror.log" > >>> if $syslogfacility-text == 'local1' then -?nagiosandcactierror > >>> & ~ > >>> > >>> local0.* ~ > >>> > >>> *.* -?DailyPerHostLogs > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > > > > > > -- Met vriendelijke groet, Erik van Dam RedBee / FortyTwo _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
