when you use reliable logging (tcp) to send logs, then you can run into the
problem that if there is a problem delivering logs, your system that is
generating the logs will stop and wait for the logs to be delivered.
This is probably what is going on here.
on your central box, does it look like it is falling behind? what sort of CPU
utilization are you seeing for rsyslog there?
rsyslog 7 is significantly faster than rsyslog 5, so that shold help (but it may
not be enough, depending on how bad the problem is)
the fact that you use if..then style filters is a very large performance hit on
rsyslog 5, but not on 7, so you will get a huge speedup just from that.
One issue that I see is that you are using the filename templates, the default
number of files that rsyslog keeps open for this is _way_ too small for anything
serious. you will need to set $DynaFileCacheSize to something large enough to
handle all the open files that you will have at any one time. try setting it to
1000 to get you started and see if that makes a difference.
how are you getting the logs from apache to rsyslog?
David Lang
On Mon, 9
Sep 2013, Erik van Dam wrote:
Date: Mon, 9 Sep 2013 15:45:32 +0200
From: Erik van Dam <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: [email protected]
Subject: [rsyslog] rsyslog bringing machines down due amount of messages (?)
Hi everybody,
I have a 8 client machines who send tcp syslog messages to an syslog server. On
two machines i'm logging more than the others; apache access logs. This results
in two completely unresponsive machines. Now if i restart rsyslog on the
machine that becomes unresponsive everything is fine, i can't notice anything
like load or memory consumption while they are unresponsive? Previously i used
certificates but in the process of finding the problem i disabled this. Further
i tried using disk queue. IMUXSock set to 0 as of
http://www.rsyslog.com/tag/imuxsockratelimitinterval/
Private information has been renamed, i'm grateful if anybody has some pointers
for me. Sofar i tried:
- disk queue = not really an affect
- restart rsyslog = relieves (a buffer within rsyslog that gets full?)
- remove tls on client side = not really an affect
What might resolve some issues:
- change tcp to udp
- goto rsyslog 7.4.(4) ?
=====================================================CLIENT==========================================================
Rsyslog version:
rsyslog-5.8.10-2.el6.x86_64
rsyslog-gnutls-5.8.10-2.el6.x86_64
$ModLoad imuxsock.so # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by
rklogd)
$IMUXSockRateLimitInterval 0
$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0
$WorkDirectory /var/log
$MainMsgQueueFileName /var/log/rsyslog.main.q
$ActionQueueFileName /var/log/rsyslog.action.q
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
local5.* @@syslogserver:514
& ~
local0.* @@syslogserver:514 #
log the access logs
& ~
local1.* /var/log/httpd/error_log
local1.* @@syslogserver:514 #
log the error logs
& ~
*.* @@syslogserver:514 # forward everything to remote server
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg *
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
=====================================================SERVER==========================================================
rsyslog-gnutls-5.8.10-2.el6.x86_64
rsyslog-5.8.10-2.el6.x86_64
$ModLoad imuxsock.so # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by
rklogd)
$IMUXSockRateLimitInterval 0
$ModLoad imudp.so
$UDPServerRun 514
$ModLoad imtcp.so
$PreserveFQDN on
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/rsyslog/protected/ca.pem
$DefaultNetstreamDriverCertFile /etc/rsyslog/protected/cert.pem
$DefaultNetstreamDriverKeyFile /etc/rsyslog/protected/key.pem
$InputTCPServerStreamDriverPermittedPeer machine1
$InputTCPServerStreamDriverPermittedPeer machine2
$InputTCPServerStreamDriverPermittedPeer machine3
$InputTCPServerStreamDriverPermittedPeer machine4
$InputTCPServerStreamDriverPermittedPeer machine5
$InputTCPServerStreamDriverPermittedPeer machine6
$InputTCPServerStreamDriverPermittedPeer machine7
$InputTCPServerStreamDriverMode 1
$InputTCPServerRun 514
$template
DailyPerHostLogs,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_messages.log"
$template
DailyrootshPerHostLogs,"/bigdisk/syslog/rootsh/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_messages.log"
local5.info -?DailyrootshPerHostLogs
& ~
$template
cactilog,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_cacti-access.log"
if $syslogfacility-text == 'local0' and $msg contains '/cacti' then -?cactilog
& ~
$template
nagioslog,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_nagios-access.log"
if $syslogfacility-text == 'local0' and $msg contains '/nagios' then -?nagioslog
& ~
$template
somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log"
if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then
-?somedomainname
& ~
$template
somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log"
if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then
-?somedomainname
& ~
$template
somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log"
if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then
-?somedomainname
& ~
$template
somedomainname,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_somedomainname.log"
if $syslogfacility-text == 'local0' and $msg contains 'somedomainname' then
-?somedomainname
& ~
$template
nagiosandcactierror,"/bigdisk/syslog/%$YEAR%/%$MONTH%/%$DAY%/%FROMHOST-IP%_nagiosandcactierror.log"
if $syslogfacility-text == 'local1' then -?nagiosandcactierror
& ~
local0.* ~
*.* -?DailyPerHostLogs
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
