Hello guys,
So i was able to get the logs to come in at 25k mps and not drop a single
one, I changed the ctl file to increase the mem to 200000, I also installed
nscd and was able to get this to work, unfortunately when i went up to 50k
mps i dropped about 20k mps, is there a way I can see something that can
tell me where I might have a problem?
Robert
----- Original Message -----
From: David Lang
Sent: 08/08/13 02:14 PM
To: rsyslog-users
Subject: Re: [rsyslog] performance tweaking
The first thing I would do is make sure that you start rsyslog without
DNS lookups (add the -x flag to startup), the overhead of doing a DNS
lookup on each message that comes in is very significant. The newest
versions of rsyslog (7.4) include some caching of DNS data, but it can
still be significant. With 5.x I think this change by itself will probably
get you over 100K logs/sec The next thing is the main message queue size,
your configuration leaves it at the default of 10K, if you are looking to
receive 100K messages/sec, that's not very big, I would set it large enough
to handle at least a couple seconds worth of logs, and if this box is a
dedicated syslog server, set it so that it will use the majority of RAM on
your system. with 32G of ram on the system, and a default 2k message size,
setting this well above 1M is very reasonable. As noted by someone else,
setting larger buffers in /etc/sysctl.conf may help If you can disable
connection tracking in the iptables stack, i
t will significantly reduce the kernel overhead (how many systems are
you recieving logs from?) Setting net.ipv4.netfilter.ip_conntrack_max large
may help As far as your rules go: 'contains' is significantly more
expensive than 'startswith' on version 5.x, the if..then structure is
significantly slower than the properties filter like: :hostname, contains,
'pdc' /var/log/test/f_ad rsyslog 7.x contains a ruleset optimizer that
eliminates this performance problem. what do you have in your included
files? It's worth checking to see where your bottleneck is, simplify your
rules to write everything to one file and see what the resulting
performance is like. That way you know if your problem is on the input side
or the output side. if you run top, and hit 'H' to show the different
threads, you can see what threads are running out of CPU time. My guess is
that it will be a thread labeled "Main Q", which is the output side of
things (due to the use of the inefficient if..then filters
), and that's causing the too-small queue to fill up, causing UDP
messages to be lost. rsyslog 7.4 combined with a recent Linux kernel also
has the ability to recieve multiple UDP packets in a single system call,
this would significantly improve performance. I don't know if RHEL 6.4
includes a recent enough kernel. This is the batchSize parameter. Another
useful parameter for UDP input is TimeRequery. If you have a lot of
messages arriving at the same time, doing a gettimeofday() call to the
system can be slow, and many consecutive calls will return the same value,
so rsyslog lets you say that as long as the incoming buffer from the OS has
more logs ready, only do a time lookup every N messages instead of every
message. Setting this to something like 100 or 1000 will virtually
eliminate the overhead of doing this lookup, and the worst that can happen
is that the time received timestamp may be off by 1 second for messages
that arrive in a batch right at the end of one second
and the beginning of the next second (i.e. you will almost certinly
never notice this, this does not affect the timestamp generated by the host
system in any case) back in the rsyslog 4.x days, I was able to get rsyslog
to handle gig-e wire speed (~380K logs/sec), and rsyslog has only gotten
faster since. David Lang On Thu, 8 Aug 2013, Robert Ortiz wrote: > Hey
Guys, > > I am new to this mailing list and I wanted to see about getting
some pointers > if possible regarding tweakin rsyslog: > > I am pretty new
to rsyslog, and I've been given a pretty fun task... to test > rsyslog vs
syslog-ng and pick the best one, I am having a problem with rsyslog > where
im at 25K/mps and im dropping logs, I need to get it at 100k mps with > and
I'm not sure where the misconfiguration is if anyone could take a look I >
would really appreciatte it, > > my current setup: > > rhel 6.4 x86_64 >
rsyslog-5.8.10-2.el6.x86_64 > Dual Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz
32GB RAM > 500GB 15k rai
d 0 > > > # rsyslog v5 configuration file > > # For more information see
/usr/share/doc/rsyslog-*/rsyslog_conf.html > # If you experience problems,
see http://www.rsyslog.com/doc/troubleshoot.html > > #### MODULES #### >
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command) > $ModLoad imklog # provides kernel logging support
(previously done by rklogd) > #$ModLoad immark # provides --MARK-- message
capability > > # Provides UDP syslog reception > $ModLoad imudp >
$UDPServerRun 514 > # $UDPServerTimeRequery 10 > > # Provides TCP syslog
reception > #$ModLoad imtcp > #$InputTCPServerRun 514 > > > #### GLOBAL
DIRECTIVES #### > > # Use default timestamp format >
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # File syncing
capability is disabled by default. This feature is usually not required, >
# not useful and an extreme performance hit > #$ActionFileEnableSync on > >
# Include all config files in /etc/rsyslog.d/ > $IncludeConf
ig /etc/rsyslog.d/*.conf > > # Set Buffer Size - default is 4k > #
$OMFileIOBufferSize 128k > # Set Main Message Queue Size - default is 10000
# $MainMsgQueueSize 50000 > > #### RULES #### > > # Log all kernel
messages to the console. > # Logging much else clutters up the screen. >
#kern.* /dev/console > > if $hostname contains 'pdc' then
/var/log/test/f_ad > & ~ > if $hostname contains 'fdfw' then
/var/log/test/f_fw > & ~ > if $hostname contains 'mail' then
/var/log/test/f_mail > & ~ > if $hostname contains 'pix' then
/var/log/test/ix > & ~ > if $hostname contains 'rout' then
/var/log/test/rout > & ~ > if $hostname contains 'networks' then
/var/log/test/net > & ~ > #if $fromhost-ip == '10.0.0.10' then
/var/log/test/thost > #& ~ > #if $hostname startswith 'virtserv' then
/var/log/test/test_virtserv > #&~ > #if $fromhost-ip startswith '10.0.6'
then /var/log/test/test_10.0.6 > #& ~ > > > # Log anything (except mail) of
level info or higher. > # Don't log private authenticati
on messages! > #*.info;mail.none;authpriv.none;cron.none
/var/log/messages > *.debug /var/log/messages > > # Log all the mail
messages in one place. > mail.* -/var/log/maillog > > > # Log cron stuff >
cron.* /var/log/cron > > # Everybody gets emergency messages > *.emerg * >
# Save news errors of level crit and higher in a special file. >
uucp,news.crit /var/log/spooler > > # Save boot messages also to boot.log >
local7.* /var/log/boot.log > > > # ### begin forwarding rule ### > # The
statement between the begin ... end define a SINGLE forwarding > # rule.
They belong together, do NOT split them. If you create multiple > # The
statement between the begin ... end define a SINGLE forwarding > # rule.
They belong together, do NOT split them. If you create multiple > #
forwarding rules, duplicate the whole block! > # Remote Logging (we use TCP
for reliable delivery) > # > # An on-disk queue is created for this action.
If the remote host is > # down, messages are spooled to dis
k and sent when it is up again. > #$WorkDirectory /var/lib/rsyslog #
where to place spool files > #$ActionQueueFileName fwdRule1 # unique name
prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit
(use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages
to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously >
#$ActionResumeRetryCount -1 # infinite retries if host is down > # remote
host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.*
@@remote-host:514 > # ### end of the forwarding rule ### > > > > Robert. >
_______________________________________________ > rsyslog mailing list >
http://lists.adiscon.net/mailman/listinfo/rsyslog >
http://www.rsyslog.com/professional-services/ > What's up with rsyslog?
Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC
mailing list, posts are ARCHIVED by a myriad of sites beyond our control.
PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > ___________
____________________________________ rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/ What's up with rsyslog?
Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing
list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE
UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Robert.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
