Date: Wed, 18 Sep 2013 11:26:46 -0400
From: Robert <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] performance tweaking
Thanks for looking at the isue, it is a little messy , but below is my
current config file, the impstats module is not working on either legacy or
current syntax, also the rules seem to perform better with the legacy "if
then", if you see something that's out of place or something to improve it
please let me know thanks!
#############################################################################################################
#########################################################################################################
# Run this in debugger mode to troubleshoot make sure to place it on top of
this file
module(load="impstats" interval="600" severity="7")
# # to actually gather the data:
syslog.=debug /var/log/rsyslog-stats
#
# #Legacy style impstats module
# #$ModLoad impstats
# #$PStatInterval 600
# #$PStatSeverity 7
#
# #syslog.=debug /var/log/rsyslog-stats
#
# rsyslog v7 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see
http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
module(load="imuxsock") # needs to be done just once Robert 8-9-13
#SysSock.FlowControl=(:"on") # enable flow control (use if needed) Robert
8-9-13
#$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
module(load="imklog")
#$ModLoad imklog # provides kernel logging support (previously done by
rklogd)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
module(load="imudp"
TimeRequery="10") # needs to be done just once Robert 8-9-13
input(type="imudp" port="514") # Robert 8-9-13
#$ModLoad imudp
#$UDPServerRun 514
#$UDPServerTimeRequery 10
# Provides TCP syslog reception
#$ModLoad imptcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually
not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Set Buffer Size - default is 4k
$OMFileAsyncWriting on
#$OMFileFlushOnTXEnd on
#$OMFileFlushInterval 1
#$OMFileZipLevel 9
$OMFileIOBufferSize 1000k # modified 9-18-13
#Turn on Main Ruleset Robert 8-20-13
#$RulesetCreateMainQueue on
# Set Main Message Queue Size - default is 10000
$MainMsgQueueType FixedArray #LinkedList
$MainMsgQueueSize 200000000
$MainMsgQueueWorkerThreads 8
#$MainMsgQueueWorkerTimeoutThreadShutdown -1
$MainMsgQueueDequeueBatchSize 1000
#$MainMsgQueueSaveOnShutdown on
$InputUDPMaxSessions 40000000
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*
# /dev/console
#Specific ruleset for remote messages
#$Ruleset <name>
#Module (load="builtin:omfile")
#*.* action(type="omfile"
# DirCreateMode="0700"
# FileCreateMode="0644"
# File="/var/log/test/alllogs")
#switch back to default ruleset
#$Ruleset RSYSLOG DefaultRuleset
#^Fx Begin action Robert 8-20-13
# $ActionOmrulesetRulesetName somename
$ActionQueueWorkerThreads 8
$ActionQueueSize 10000000
$ActionQueueType FixedArray #LinkedList - use asynchronous processing
#$ActionQueueFileName aaaaafwd # set file name, also enables disk mode
#$ActionQueueMaxFileSize 400m # default: 1m, should be 1% of MaxDiskSpace
#$ActionQueueMaxDiskSpace 40g # space limit (use as much as possible)
#$ActionQueueTimeoutEnqueue 0 # throtteling, 0 disables throttling and
discard immediately if queue is full
#$ActionQueueDequeueBatchSize 500000
#$ActionResumeRetryCount -1 # infinite retries on insert failure
#$ActionResumeInterval 1 # faster than default 30 second delay
#$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
#*.* /var/log/test/f_all #Robert 8-21-13
#:hostname, contains, "fdfw" /var/log/test/f_fw
#:hostname, contains, "mail" /var/log/test/f_mail
#:hostname, contains, "shib" /var/log/test/f_shib
#:hostname, contains, "pdc" /var/log/test/f_ad
#:hostname, contains, "networks" /var/log/test/f_networks
#:hostname, contains, "rout" /var/log/test/f_router
#:hostname, contains, "vm" /var/log/test/f_vm
#:hostname, contains, "pix" /var/log/test/f_pix
if $hostname contains 'pdc' then /var/log/test/f_ad
& ~
if $hostname contains 'fdfw' then /var/log/test/f_fw
& ~
if $hostname contains 'networks' then /var/log/test/f_networks
& ~
if $hostname contains 'shib' then /var/log/test/f_shib
& ~
if $hostname contains 'mail' then /var/log/test/f_mail
& ~
if $hostname contains 'vm' then /var/log/test/f_vm
& ~
if $hostname contains 'pix' then /var/log/test/f_pix
& ~
if $hostname contains 'rout' then /var/log/test/f_router
& ~
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none /var/log/messages
#*.debug /var/log/messages
#
# The authpriv file has restricted access.
#authpriv.* /var/log/secure
#
# Log all the mail messages in one place.
#mail.* -/var/log/maillog
#
#
# Log cron stuff
#cron.* /var/log/cron
#
# Everybody gets emergency messages
#*.emerg *
#
# Save news errors of level crit and higher in a special file.
#uucp,news.crit /var/log/spooler
#
# Save boot messages also to boot.log
#local7.* /var/log/boot.log
#
#
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
################################################################################################
#########################################################################################################
----- Original Message -----
From: Rainer Gerhards
Sent: 09/18/13 03:00 AM
To: rsyslog-users
Subject: Re: [rsyslog] performance tweaking
On Wed, Sep 18, 2013 at 1:42 AM, David Lang <[email protected]> wrote: > At
this point, it's probably not the input thread then, which thread is >
running out of CPU? You mention running top with the H flag, what threads >
is using most of the CPU? > > if one of the outputs is not able to keep up,
rsyslog will end up loosing > messages > Usually an excellent way to spot
the culprit is also to have a look at rsyslog's internal stats (the
usefulness for performance tweaking depends a bit on the config,
though...). I highly recommend gathering them. It's done via the impstats
module: http://www.rsyslog.com/doc/impstats.html Also, it would be good to
see the full rsyslog configuration, not just a snippet. Sorry if you
already posted it, but I couldn't get hold of it in the large volume of
mail ;) Rainer _______________________________________________ rsyslog
mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/ What's up with !
