On Tue, 12 Jul 2011, Mike Forbes wrote:
On 12 July 2011 15:13, <[email protected]> wrote:
Ok, this does sound like you aren't able to keep up with the incoming log
volume, what sort of volume do you get?
14 hosts, a total of about 70Mb of logs for today, logfile sizes are
between 1-7Mb
I was thinking in terms of logs per min or logs per second
70MB of logs for a day is a fair bit (and depending on how bunched up they
are when they arrive, I could see the data rate being pretty high)
as an experiment, can you try disabeling the per-hosts dynafile and
replace it with something just writing all the logs into one file? I am
wondering how much that ends up costing in CPU time.
also, unless you have a raid card with a good amount of cach on it, you
could be having problems with the disk keeping up as you seek back and
forth to the different files.
so I would suggest doing the following tests
remove the dynafile option and just have everything in one file
change the dynafile option to remove the hostname but keep the date/time
portions.
I think you will see the peak load go up as you enable dynafiles at all,
but UI suspect that you will have significantly lower peaks with your I/O
useage, and that should keep the memory consumption down (as rsyslog never
backs up waiting for the disk)
In your place, I would be rotating the files periodically (with a script
run out of cron), and having the log rotation script break them down to
the different servers if you need that.
David Lang
On Tue, 12 Jul 2011, Mike Forbes wrote:
Subject: [rsyslog] High memory use by rsyslogd - 5.6.3
Rsyslogd is using 1.4G of resident memory.
Here's my config snippets: http://paste.nothing.net.nz/806cba
There's 14 hosts sending data to this rsyslog server.
I've tried setting a ulimit
(syslog soft stack 1024)
via pam & /etc/security/limits.conf
and ulimit -a as the syslog user shows
stack size (kbytes, -s) 1024
Yet top clearly shows using 1.4G of resident memory:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16414 syslog 20 0 2006m 1.4g 1252 S 0 72.1 1:28.83 rsyslogd
-c4
my rsyslogd version is rsyslogd 5.6.3
Any advice is welcome.
--
// Mike Forbes
GPG: BFC7 3F32 2CCF D91F 53E1 DF88 1578 B2E4 1399 6844
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
--
// Mike Forbes
Operations Engineer
Koordinates Ltd.
PO Box 1604, Shortland St, Auckland 1140, New Zealand
Cell +64-21-999416
Phone +64-9-966 0433
Fax +64-9-969 0045
Web http://koordinates.com/
GPG: BFC7 3F32 2CCF D91F 53E1 DF88 1578 B2E4 1399 6844
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
--
// Mike Forbes
Operations Engineer
Koordinates Ltd.
PO Box 1604, Shortland St, Auckland 1140, New Zealand
Cell +64-21-999416
Phone +64-9-966 0433
Fax +64-9-969 0045
Web http://koordinates.com/
GPG: BFC7 3F32 2CCF D91F 53E1 DF88 1578 B2E4 1399 6844
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
