On Thu, 5 Sep 2013, Robert Ortiz wrote:
Hello everyone,
I upgraded to 7.4.3, and as everyone mentioned (David) it improved
significantly, I am at 110k mps, which is great, but I think it can do better,
currently rsyslog nice level is at its max, and I am curious if there was
anything that I could do to bring it up to 200k mps? this is my current config
file, I've tweaked it around and removed unnecessary things, but I am
wondering if anyone can see if there is something that I am missing or
something I need to unmark to improve the performance,
As Rainer says, we need to find out where the bottleneck is, if you can run top,
hit 'H' so that it shows threads, and then load it down and tell us which thread
is hitting max CPU it would tell us where to look
everything below is specualtion and places that can possibly be improved, but if
that is not the bottleneck, it won't make any difference unless the system as a
whole is running out of CPU.
setting the timerequery for UDP will help if the bottleneck is there.
are you disabling DNS lookups at startup (the -x flag on rsyslogd), this is a
very significant win if you can do it. If you have not done this, you need to
check and see if your DNS server is the bottleneck.
I suggest using queue type FixedArray instead of LinkedList. It will use the
max amount of memory all the time, but my testing showed that this was faster
than LinkedList (which makes sense, since rsyslog doesn't need to manipulate the
list and bounce all over memory to access the queue)
you have the zip compression set to 9, which is the most CPU intensive setting,
if the output ends up being your bottleneck, try reducing this and see if things
improve.
not knowing your system names, if you can change any of the tests from
'contains' to 'startswith', that could improve things.
if your output is the bottleneck, you can split the outputs to use multiple
queues instead of all being part of one queue (add another set of queue
parameters about midway through the tests)
The action queue parameter that Rainier suggested may help with this as well, if
you cna use multiple threads to read from the queue and output the messages this
can be as good or better than using multiple queues.
Rainer, am I correct in reading this tht the actionqueue will be used for all of
the output? I know that using the v7 config you cannot set an action queue to be
used for more than one output (without resorting to rulesets), but I think that
in the earlier config language the queue settings remain until they are set to
something else.
looking through the config, I strongly suspect that your bottleneck is in
writing the files. You have all of the writing being done with a single thread,
and that thread is compressing the logs, so it has a lot of work to do.
one thing that may help is if you turn on asyncwriting, that will decouple the
filtering and creation of the string to be written from the process of writing
it. Since you are doing compression as you are writing it, this should move the
compression out to a separate thread (or possibly a separate thread per
output??? Rainer?) instead of it all being done in the worker thread.
David Lang
Thanks in advance
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
module(load="imuxsock") # needs to be done just once Roberto 8-9-13
SysSock.FlowControl=(:"on") # enable flow control (use if needed) Roberto 8-9-13
#$ModLoad imuxsock # provides support for local system logging (e.g. via logger
command)
module(load="imklog")
#$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
#$ModLoad imudp
#$UDPServerRun 514
#$UDPServerTimeRequery 1000000
#$UDPServerTimeRequery 10 - Gil 06/06/13
# Provides TCP syslog reception
#$ModLoad imptcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not
required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Set Buffer Size - default is 4k
#$OMFileIOBufferSize 128k # - Gil 06/06/13
$OMFileAsyncWriting on
$OMFileFlushOnTXEnd on
$OMFileFlushInterval 1
#$OMFileZipLevel 9
$OMFileIOBufferSize 1000k
#Turn on Main Ruleset Roberto 8-20-13
#$RulesetCreateMainQueue on
# Set Main Message Queue Size - default is 10000
$MainMsgQueueSize 20000000 # Roberto 8-9-13
$InputUDPMaxSessions 40000000
$MainMsgQueueDequeueBatchSize 500000
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*
# /dev/console
#Specific ruleset for remote messages
#$Ruleset <name>
#*.* /var/log/test/f_all #Roberto 8-21-13
#Module (load="builtin:omfile")
#*.* action(type="omfile"
# DirCreateMode="0700"
# FileCreateMode="0644"
# File="/var/log/test/alllogs")
#switch back to default ruleset
#$Ruleset RSYSLOG DefaultRuleset
# Begin action Roberto 8-20-13
# $ActionOmrulesetRulesetName somename
$ActionQueueWorkerThreads 5000000
$ActionQueueSize 10000000
#$ActionQueueType LinkedList # use asynchronous processing
#$ActionQueueFileName aaaaafwd # set file name, also enables disk mode
#$ActionQueueMaxFileSize 400m # default: 1m, should be 1% of MaxDiskSpace
#$ActionQueueMaxDiskSpace 40g # space limit (use as much as possible)
#$ActionQueueTimeoutEnqueue 0 # throtteling, 0 disables throttling and discard
immediately if queue is full
#$ActionQueueDequeueBatchSize 500000
#$ActionResumeRetryCount -1 # infinite retries on insert failure
#$ActionResumeInterval 1 # faster than default 30 second delay
#$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
:hostname, contains, "fdfw" /var/log/test/f_fw
:hostname, contains, "mail" /var/log/test/f_mail
:hostname, contains, "shib" /var/log/test/f_shib
:hostname, contains, "pdc" /var/log/test/f_ad
:hostname, contains, "networks" /var/log/test/f_networks
:hostname, contains, "rout" /var/log/test/f_router
:hostname, contains, "vm" /var/log/test/f_vm
:hostname, contains, "pix" /var/log/test/f_pix
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none /var/log/messages
#*.debug /var/log/messages
# The authpriv file has restricted access.
#authpriv.* /var/log/secure
# Log all the mail messages in one place.
#mail.* -/var/log/maillog
# Log cron stuff
#cron.* /var/log/cron
# Everybody gets emergency messages
#*.emerg *
# Save news errors of level crit and higher in a special file.
#uucp,news.crit /var/log/spooler
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
(END)
Robert.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
