it sounds as if an output is getting stuck and then the queue is filling
up.
re all your rules writing to local files, or are some of them writing to
something that could block?
in top, use the 'H' option to turn on per-thread reporting, I suspect that
there is one thread that is getting stuck. It may beworth dong an strace
on the threads to see what they are doing (especially if there is one
stuck at 100% cpu)
David Lang
On Fri, 8 Oct 2010, Karsten Heymann wrote:
Date: Fri, 08 Oct 2010 10:50:02 +0200
From: Karsten Heymann <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: [email protected]
Subject: [rsyslog] 5.4.0 hangs after several hours
Hello,
I'm currently implementing a new central log server with 5.4.0 on Debian
Linux for our company and am running into severe stability problems. I
hope this list is the right place to report and discuss these, if not,
please point me to the right direction.
Our logserver receives logs via udp and tcp on several ports and handles
them with different rulesets (this is why we upgraded to 5.4.0):
%- /etc/rsyslog.conf
$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$IncludeConfig /etc/rsyslog.d/*.conf
auth,authpriv.* /var/log/auth.log
[... more file rules omitted]
%-
%- /etc/rsyslog.d/remote.conf
$RuleSet udp514
local0.* -/var/log/cisco/local0.log
[... more file rules omitted]
$RuleSet tcp514
$RuleSet tcp10514
auth,authpriv.* /var/log/server/auth.log
[... more file rules omitted]
$RuleSet tcp20514
$ModLoad imudp
$InputUDPServerBindRuleset udp514
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerBindRuleset tcp514
$InputTCPServerRun 514
$InputTCPServerBindRuleset tcp10514
$INPUTTCPServerRun 10514
$InputTCPServerBindRuleset tcp20514
$INPUTTCPServerRun 20514
$RuleSet RSYSLOG_DefaultRuleset
%-
rsyslog is started with "/usr/sbin/rsyslogd -c5".
The Problem:
After several hours, one rsyslogd process starts running at 100% cpu and
uses more and more memory, also it completely stops writing to the
logfiles (hence no rsyslog error messages too). If I run
strace -p <PID of 100% CPU rsyslogd>
i get a constant stream of
write(3, "Oct 8 09:40:42 loghost1-01 kerne"..., 266) = -1 EAGAIN
(Resource temporarily unavailable)
system calls.
Can you give me any hints how to debug this further?
Yours
Karsten
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
