The first thing I would do is make sure that you start rsyslog without DNS
lookups (add the -x flag to startup), the overhead of doing a DNS lookup on each
message that comes in is very significant. The newest versions of rsyslog (7.4)
include some caching of DNS data, but it can still be significant. With 5.x I
think this change by itself will probably get you over 100K logs/sec
The next thing is the main message queue size, your configuration leaves it at
the default of 10K, if you are looking to receive 100K messages/sec, that's not
very big, I would set it large enough to handle at least a couple seconds worth
of logs, and if this box is a dedicated syslog server, set it so that it will
use the majority of RAM on your system. with 32G of ram on the system, and a
default 2k message size, setting this well above 1M is very reasonable.
As noted by someone else, setting larger buffers in /etc/sysctl.conf may help
If you can disable connection tracking in the iptables stack, it will
significantly reduce the kernel overhead (how many systems are you recieving
logs from?) Setting net.ipv4.netfilter.ip_conntrack_max large may help
As far as your rules go:
'contains' is significantly more expensive than 'startswith'
on version 5.x, the if..then structure is significantly slower than the
properties filter like:
:hostname, contains, 'pdc' /var/log/test/f_ad
rsyslog 7.x contains a ruleset optimizer that eliminates this performance
problem.
what do you have in your included files?
It's worth checking to see where your bottleneck is, simplify your rules to
write everything to one file and see what the resulting performance is like.
That way you know if your problem is on the input side or the output side.
if you run top, and hit 'H' to show the different threads, you can see what
threads are running out of CPU time. My guess is that it will be a thread
labeled "Main Q", which is the output side of things (due to the use of the
inefficient if..then filters), and that's causing the too-small queue to fill
up, causing UDP messages to be lost.
rsyslog 7.4 combined with a recent Linux kernel also has the ability to recieve
multiple UDP packets in a single system call, this would significantly improve
performance. I don't know if RHEL 6.4 includes a recent enough kernel. This is
the batchSize parameter.
Another useful parameter for UDP input is TimeRequery. If you have a lot of
messages arriving at the same time, doing a gettimeofday() call to the system
can be slow, and many consecutive calls will return the same value, so rsyslog
lets you say that as long as the incoming buffer from the OS has more logs
ready, only do a time lookup every N messages instead of every message. Setting
this to something like 100 or 1000 will virtually eliminate the overhead of
doing this lookup, and the worst that can happen is that the time received
timestamp may be off by 1 second for messages that arrive in a batch right at
the end of one second and the beginning of the next second (i.e. you will almost
certinly never notice this, this does not affect the timestamp generated by the
host system in any case)
back in the rsyslog 4.x days, I was able to get rsyslog to handle gig-e wire
speed (~380K logs/sec), and rsyslog has only gotten faster since.
David Lang
On Thu, 8 Aug 2013, Robert Ortiz wrote:
Hey Guys,
I am new to this mailing list and I wanted to see about getting some pointers
if possible regarding tweakin rsyslog:
I am pretty new to rsyslog, and I've been given a pretty fun task... to test
rsyslog vs syslog-ng and pick the best one, I am having a problem with rsyslog
where im at 25K/mps and im dropping logs, I need to get it at 100k mps with
and I'm not sure where the misconfiguration is if anyone could take a look I
would really appreciatte it,
my current setup:
rhel 6.4 x86_64
rsyslog-5.8.10-2.el6.x86_64
Dual Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz
32GB RAM
500GB 15k raid 0
# rsyslog v5 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger
command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# $UDPServerTimeRequery 10
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not
required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Set Buffer Size - default is 4k
# $OMFileIOBufferSize 128k
# Set Main Message Queue Size - default is 10000
# $MainMsgQueueSize 50000
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
if $hostname contains 'pdc' then /var/log/test/f_ad
& ~
if $hostname contains 'fdfw' then /var/log/test/f_fw
& ~
if $hostname contains 'mail' then /var/log/test/f_mail
& ~
if $hostname contains 'pix' then /var/log/test/ix
& ~
if $hostname contains 'rout' then /var/log/test/rout
& ~
if $hostname contains 'networks' then /var/log/test/net
& ~
#if $fromhost-ip == '10.0.0.10' then /var/log/test/thost
#& ~
#if $hostname startswith 'virtserv' then /var/log/test/test_virtserv
#&~
#if $fromhost-ip startswith '10.0.6' then /var/log/test/test_10.0.6
#& ~
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none /var/log/messages
*.debug /var/log/messages
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
Robert.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
