----- Original Message -----
From: Rainer Gerhards
Sent: 09/06/13 05:35 AM
To: rsyslog-users
Subject: Re: [rsyslog] perfomance tweaking
On Thu, Sep 5, 2013 at 11:02 PM, David Lang <[email protected]> wrote:
On Thu, 5 Sep 2013, Robert Ortiz wrote:
Hello everyone,
I upgraded to 7.4.3, and as everyone mentioned (David) it improved
significantly, I am at 110k mps, which is great, but I think it can do
better, currently rsyslog nice level is at its max, and I am curious if
there was anything that I could do to bring it up to 200k mps? this is my
current config file, I've tweaked it around and removed unnecessary things,
but I am wondering if anyone can see if there is something that I am
missing or something I need to unmark to improve the performance,
As Rainer says, we need to find out where the bottleneck is, if you can
run top, hit 'H' so that it shows threads, and then load it down and tell
us which thread is hitting max CPU it would tell us where to look
everything below is specualtion and places that can possibly be improved,
but if that is not the bottleneck, it won't make any difference unless the
system as a whole is running out of CPU.
setting the timerequery for UDP will help if the bottleneck is there.
are you disabling DNS lookups at startup (the -x flag on rsyslogd), this
is a very significant win if you can do it. If you have not done this, you
need to check and see if your DNS server is the bottleneck.
With v7, this should not affect performance very much. Do you have other
experience?
I suggest using queue type FixedArray instead of LinkedList. It will use
the max amount of memory all the time, but my testing showed that this was
faster than LinkedList (which makes sense, since rsyslog doesn't need to
manipulate the list and bounce all over memory to access the queue)
you have the zip compression set to 9, which is the most CPU intensive
setting, if the output ends up being your bottleneck, try reducing this and
see if things improve.
not knowing your system names, if you can change any of the tests from
'contains' to 'startswith', that could improve things.
if your output is the bottleneck, you can split the outputs to use
multiple queues instead of all being part of one queue (add another set of
queue parameters about midway through the tests)
The action queue parameter that Rainier suggested may help with this as
well, if you cna use multiple threads to read from the queue and output the
messages this can be as good or better than using multiple queues.
Be careful: I was talking about the main message queue. If I don't overlook
anything, the action queue settings are not very relevant here.
Rainer, am I correct in reading this tht the actionqueue will be used for
all of the output? I know that using the v7 config you cannot set an action
queue to be used for more than one output (without resorting to rulesets),
but I think that in the earlier config language the queue settings remain
until they are set to something else.
I think these are all auto-reset parameters, so they only apply to the
first output. I have not checked the code, mostly because I really think
the action queue parameters make a big difference here.
I have written an in-depth blog about those tuning parameters some time
ago. It can be found here:
http://blog.gerhards.net/2013/06/rsyslog-performance-main-and-action.html
Also, I think the $OMfile parameters also apply only to the first file -
but I may be wrong. My strong recommendation is to use new-style config in
complex cases, as it is then much easier to see what is actually going on
(after all, that's why the new style was implemented ;)).
looking through the config, I strongly suspect that your bottleneck is in
writing the files. You have all of the writing being done with a single
thread, and that thread is compressing the logs, so it has a lot of work to
do.
yup, I guess it is the filters and running the main queue on multiple
threads will take care of that.
one thing that may help is if you turn on asyncwriting, that will decouple
the filtering and creation of the string to be written from the process of
writing it. Since you are doing compression as you are writing it, this
should move the compression out to a separate thread (or possibly a
separate thread per output??? Rainer?
the background writer thread is per action instance. Compression is done on
the background writer threads - in other words: "yes". But again, I doubt
the parameters apply to all actions.
Rainer
) instead of it all being done in the worker thread.
David Lang
Thanks in advance
# For more information see /usr/share/doc/rsyslog-*/**rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/**
troubleshoot.html <http://www.rsyslog.com/doc/troubleshoot.html>
#### MODULES ####
module(load="imuxsock") # needs to be done just once Roberto 8-9-13
SysSock.FlowControl=(:"on") # enable flow control (use if needed) Roberto
8-9-13
#$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
module(load="imklog")
#$ModLoad imklog # provides kernel logging support (previously done by
rklogd)
$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
#$ModLoad imudp
#$UDPServerRun 514
#$UDPServerTimeRequery 1000000
#$UDPServerTimeRequery 10 - Gil 06/06/13
# Provides TCP syslog reception
#$ModLoad imptcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually
not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Set Buffer Size - default is 4k
#$OMFileIOBufferSize 128k # - Gil 06/06/13
$OMFileAsyncWriting on
$OMFileFlushOnTXEnd on
$OMFileFlushInterval 1
#$OMFileZipLevel 9
$OMFileIOBufferSize 1000k
#Turn on Main Ruleset Roberto 8-20-13
#$RulesetCreateMainQueue on
# Set Main Message Queue Size - default is 10000
$MainMsgQueueSize 20000000 # Roberto 8-9-13
$InputUDPMaxSessions 40000000
$MainMsgQueueDequeueBatchSize 500000
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*
# /dev/console
#Specific ruleset for remote messages
#$Ruleset <name>
#*.* /var/log/test/f_all #Roberto 8-21-13
#Module (load="builtin:omfile")
#*.* action(type="omfile"
# DirCreateMode="0700"
# FileCreateMode="0644"
# File="/var/log/test/alllogs")
#switch back to default ruleset
#$Ruleset RSYSLOG DefaultRuleset
# Begin action Roberto 8-20-13
# $ActionOmrulesetRulesetName somename
$ActionQueueWorkerThreads 5000000
$ActionQueueSize 10000000
#$ActionQueueType LinkedList # use asynchronous processing
#$ActionQueueFileName aaaaafwd # set file name, also enables disk mode
#$ActionQueueMaxFileSize 400m # default: 1m, should be 1% of MaxDiskSpace
#$ActionQueueMaxDiskSpace 40g # space limit (use as much as possible)
#$ActionQueueTimeoutEnqueue 0 # throtteling, 0 disables throttling and
discard immediately if queue is full
#$ActionQueueDequeueBatchSize 500000
#$ActionResumeRetryCount -1 # infinite retries on insert failure
#$ActionResumeInterval 1 # faster than default 30 second delay
#$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
:hostname, contains, "fdfw" /var/log/test/f_fw
:hostname, contains, "mail" /var/log/test/f_mail
:hostname, contains, "shib" /var/log/test/f_shib
:hostname, contains, "pdc" /var/log/test/f_ad
:hostname, contains, "networks" /var/log/test/f_networks
:hostname, contains, "rout" /var/log/test/f_router
:hostname, contains, "vm" /var/log/test/f_vm
:hostname, contains, "pix" /var/log/test/f_pix
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.**none;cron.none /var/log/messages
#*.debug /var/log/messages
# The authpriv file has restricted access.
#authpriv.* /var/log/secure
# Log all the mail messages in one place.
#mail.* -/var/log/maillog
# Log cron stuff
#cron.* /var/log/cron
# Everybody gets emergency messages
#*.emerg *
# Save news errors of level crit and higher in a special file.
#uucp,news.crit /var/log/spooler
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
(END)
Robert.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
