On Mon, 20 Sep 2010, Joe Williams wrote:
Are there commonly used tuning options for rsyslog, networking stack and
kernel? I have found a bit of information (links below) and have made a
few changes but I am curious if there are any low hanging fruit to
increase my message rate. To that end what are "normal" message rates
for the various versions of rsyslog? I know a lot of this is subjective
depending on various factors unique to each system but I am curious of
commonly hit performance issues that can be tuned around.
version 3 didn't have much in the way of performance tuning knobs (you
could adjust the message size and count limits)
it's been a long time since I tested v3, but I think it's performance was
below 10k messages/sec
version 4 is where the performance changes really started, and there are a
lot of knobs that were created
version 4 could receive messages up to 1Gb wire speed with the correct
tuning, I could get it to forward or write messages at ~80K messages/sec
or do both at ~30K messages/sec
version 5 gained a lot more performance and dropped some of the tuning
knobs, I haven't tested the most recent versions, but reports are that it
can do >250K messages/sec
the key tuning knobs will vary a bit depending on your source of logs. I
deal mostly with UDP logs.
one key thing is to reduce the frequency of gettimeofday() calls
$UDPServerTimeRequery 10
lets you say that if you get a continuous stream of messages (i.e., ever
time rsyslog finishes processing one message there is already another
waiting) instead of checking the time every message it checks it every 10
messages and uses the same local timestamp for the messages in between.
you get most of the benifit even with a small value like 10
another thing to look at (and I don't remember the config option at the
moment) is the batch size for processing messages (especially if you are
doing something like inserting them into a database, but even for much
simpler configs)
If you can disable DNS lookups, that will make a huge difference (not that
even with DNS lookups disabled you will have hostnames if the sending
server puts their name in the message like they are supposed to.
however, for most hardware and uses, it really is going to be fast enough
out of the box to not need much, if any tuning.
remember 'premature optimization is the root of all evil', get rsyslog
running, look at the CPU that it's taking (per-thread, not just total
CPU), and then look at what the threads that are busy are doing. In V3 and
early V4 it was the thread that was receiving messages that was the
bottleneck, by the end of V4 it was the threads outputting the messages
that was the bottleneck (mostly by thrashing the queue locks), in the very
recent V5 versions most of this locking went away and performance is _way_
up, but it means that whatever bottlenecks are left are in different
places.
one thing you should do is to tune your OS. make sure you have pleanty of
network buffers (tcp or udp as appropriate for your system) and check your
disk I/O capibilities (especially if you are doing something other than
simple buffered writes to files)
the high database link you point to is obsolete now in the face of the
batch mode of inserts. the key is to tune your batch sizes to be fairly
large (but you need to watch your database to make sure they don't get so
large that your database chokes on them)
rsyslog has changes so much and so rapidly that I don't think there are
really any good documents on tuning. The current version with it's ability
to ahve subsets of the rules, each with their own queue can be configured
to spread itself across all the processor cores in your system (although
the configuration gets very messy) and with that you can do a huge amount
of processing on the log messages.
what does your environment look like? how many messages of what size do
you expect to receive, and what do you want to do with them?
David Lang
Thanks.
-Joe
http://mperedim.wordpress.com/2010/01/21/rsyslog-evaluation/
http://www.rsyslog.com/doc/queues.html
http://www.rsyslog.com/doc/rsyslog_high_database_rate.html
http://www.gossamer-threads.com/lists/rsyslog/users/4029
Name: Joseph A. Williams
Email: [email protected]
Blog: http://www.joeandmotorboat.com/
Twitter: http://twitter.com/williamsjoe
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
