On Thu, 12 Sep 2013, Erik van Dam wrote:
Hi David,
Thank you for your reply aswell. I don't see any increased load on the server
if it's having issues, cpu utilization is about 5%.
I parse the apache logs like this:
CustomLog "|/usr/bin/logger -t apache -p local0.info" combined
So the sum up your advice:
- run udp
- upgrade rsyslog (style filters is better)
- increment $DynaFileCacheSize
The only problem is is that the client wants to use tls but using udp this has
to be out of the question?
The problem that you have when using a TCP based transport (which is needed to
use encryption) is that your entire system can back up if the server gets behind
in processing the messages.
you can address this on the client by setting high/low watermarks to tell
rsyslog that if it gets behind and the queue gets close to filling up that it
should throw away some messages instead of letting the queue fill up, which is
what causes things to stop.
When the queue fills up, the proceses writing the logs are going to stall, but
they will not be eating any CPU, they will be blocked.
On the syslog server, the system may have it's CPU pegged, or it may be stalled
waiting for disk I/O (which should show up as a high CPU wait time)
are you sure the syslog server is not showing high utilization when you get into
trouble?
even before you get into trouble, could you please run top on the syslog server
and then hit 'H' to show the individual threads? if you can cut-n-paste the
upper part of the report (the header info and the rsyslog lines, which I expect
to be the top processes showing)? I expect that it will help us see what's going
on.
David Lang
David Lang
Regards,
Erik
On Mon, 9 Sep 2013 07:36:57 -0700 (PDT)
David Lang <[email protected]> wrote:
when you use reliable logging (tcp) to send logs, then you can run into the
problem that if there is a problem delivering logs, your system that is
generating the logs will stop and wait for the logs to be delivered.
This is probably what is going on here.
on your central box, does it look like it is falling behind? what sort of CPU
utilization are you seeing for rsyslog there?
rsyslog 7 is significantly faster than rsyslog 5, so that shold help (but it may
not be enough, depending on how bad the problem is)
the fact that you use if..then style filters is a very large performance hit on
rsyslog 5, but not on 7, so you will get a huge speedup just from that.
One issue that I see is that you are using the filename templates, the default
number of files that rsyslog keeps open for this is _way_ too small for anything
serious. you will need to set $DynaFileCacheSize to something large enough to
handle all the open files that you will have at any one time. try setting it to
1000 to get you started and see if that makes a difference.
how are you getting the logs from apache to rsyslog?
David Lang
On Mon, 9
Sep 2013, Erik van Dam wrote:
Date: Mon, 9 Sep 2013 15:45:32 +0200
From: Erik van Dam <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: [email protected]
Subject: [rsyslog] rsyslog bringing machines down due amount of messages (?)
Hi everybody,
I have a 8 client machines who send tcp syslog messages to an syslog server. On
two machines i'm logging more than the others; apache access logs. This results
in two completely unresponsive machines. Now if i restart rsyslog on the
machine that becomes unresponsive everything is fine, i can't notice anything
like load or memory consumption while they are unresponsive? Previously i used
certificates but in the process of finding the problem i disabled this. Further
i tried using disk queue. IMUXSock set to 0 as of
http://www.rsyslog.com/tag/imuxsockratelimitinterval/
Private information has been renamed, i'm grateful if anybody has some pointers
for me. Sofar i tried:
- disk queue = not really an affect
- restart rsyslog = relieves (a buffer within rsyslog that gets full?)
- remove tls on client side = not really an affect
What might resolve some issues:
- change tcp to udp
- goto rsyslog 7.4.(4) ?
=====================================================CLIENT==========================================================
Rsyslog version:
rsyslog-5.8.10-2.el6.x86_64
rsyslog-gnutls-5.8.10-2.el6.x86_64
$ModLoad imuxsock.so # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by
rklogd)
$IMUXSockRateLimitInterval 0
$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0
$WorkDirectory /var/log
$MainMsgQueueFileName /var/log/rsyslog.main.q
$ActionQueueFileName /var/log/rsyslog.action.q
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
local5.* @@syslogserver:514
& ~
local0.* @@syslogserver:514 #
log the access logs
& ~
local1.* /var/log/httpd/error_log
local1.* @@syslogserver:514 #
log the error logs
& ~
*.* @@syslogserver:514 # forward everything to remote server
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg *
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
=====================================================SERVER==========================================================
rsyslog-gnutls-5.8.10-2.el6.x86_64
rsyslog-5.8.10-2.el6.x86_64
$ModLoad imuxsock.so # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by
rklogd)
$IMUXSockRateLimitInterval 0
$ModLoad imudp.so
$UDPServerRun 514
$ModLoad imtcp.so
$PreserveFQDN on
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/rsyslog/protected/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog/protected/cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog/protected/key.pem
$InputTCPServerStreamDriverPermittedPeer machine1
$InputTCPServerStreamDriverPermittedPeer machine2
$InputTCPServerStreamDriverPermittedPeer machine3
$InputTCPServerStreamDriverPermittedPeer machine4
$InputTCPServerStreamDriverPermittedPeer machine5
$InputTCPServerStreamDriverPermittedPeer machine6
$InputTCPServerStreamDriverPermittedPeer machine7
$InputTCPServerStreamDriverMode 1
$InputTCPServerRun 514
$template
DailyPerHostLogs,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_messages.log"
$template
DailyrootshPerHostLogs,"/bigdisk/syslog/rootsh/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_messages.log"
local5.info -?DailyrootshPerHostLogs
& ~
$template
cactilog,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_cacti-access.log"
if $syslogfacility-text == 'local0' and $msg contains '/cacti' then -?cactilog
& ~
$template
nagioslog,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_nagios-access.log"
if $syslogfacility-text == 'local0' and $msg contains '/nagios' then -?nagioslog
& ~
$template
somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log"
if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then
-?somedomainname
& ~
$template
somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log"
if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then
-?somedomainname
& ~
$template
somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log"
if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then
-?somedomainname
& ~
$template
somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log"
if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then
-?somedomainname
& ~
$template
nagiosandcactierror,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_nagiosandcactierror.log"
if $syslogfacility-text == 'local1' then -?nagiosandcactierror
& ~
local0.* ~
*.* -?DailyPerHostLogs
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
