So as it turns out I cannot disable DNS lookup beause we have a host file that
rsyslog is supposed to look at so it knows who are the messages from and where
to dump them according to the filters, currently I am running a load tests at
25K/mps and I am dropping messages considerably {see below} this is my current
base with the rsyslog.conf that I currently have, once this test is done, I
will modify the filters to what David suggested and will try it again.
24994 09:05:30
24971 09:05:31
24972 09:05:32
24952 09:05:33
24975 09:05:34
24953 09:05:35
24919 09:05:36
24806 09:05:37
25003 09:05:38
24987 09:05:39
24980 09:05:40
24881 09:05:41
----- Original Message -----
From: Robert Ortiz
Sent: 08/08/13 04:53 PM
To: rsyslog-users
Subject: Re: [rsyslog] performance tweaking
Im sorry to sound so new, but when you say start rsyslog without dns lookups,
you mean modify rc.d/init.d/rsyslog? ----- Original Message ----- From: David
Lang Sent: 08/08/13 02:14 PM To: rsyslog-users Subject: Re: [rsyslog]
performance tweaking The first thing I would do is make sure that you start
rsyslog without DNS lookups (add the -x flag to startup), the overhead of doing
a DNS lookup on each message that comes in is very significant. The newest
versions of rsyslog (7.4) include some caching of DNS data, but it can still be
significant. With 5.x I think this change by itself will probably get you over
100K logs/sec The next thing is the main message queue size, your configuration
leaves it at the default of 10K, if you are looking to receive 100K
messages/sec, that's not very big, I would set it large enough to handle at
least a couple seconds worth of logs, and if this box is a dedicated syslog
server, set it so that it will use the majority of RAM on your system. wit
h 32G of ram on the system, and a default 2k message size, setting this well
above 1M is very reasonable. As noted by someone else, setting larger buffers
in /etc/sysctl.conf may help If you can disable connection tracking in the
iptables stack, i t will significantly reduce the kernel overhead (how many
systems are you recieving logs from?) Setting
net.ipv4.netfilter.ip_conntrack_max large may help As far as your rules go:
'contains' is significantly more expensive than 'startswith' on version 5.x,
the if..then structure is significantly slower than the properties filter like:
:hostname, contains, 'pdc' /var/log/test/f_ad rsyslog 7.x contains a ruleset
optimizer that eliminates this performance problem. what do you have in your
included files? It's worth checking to see where your bottleneck is, simplify
your rules to write everything to one file and see what the resulting
performance is like. That way you know if your problem is on the input side or
the output side. if you
run top, and hit 'H' to show the different threads, you can see what threads
are running out of CPU time. My guess is that it will be a thread labeled "Main
Q", which is the output side of things (due to the use of the inefficient
if..then filters ), and that's causing the too-small queue to fill up, causing
UDP messages to be lost. rsyslog 7.4 combined with a recent Linux kernel also
has the ability to recieve multiple UDP packets in a single system call, this
would significantly improve performance. I don't know if RHEL 6.4 includes a
recent enough kernel. This is the batchSize parameter. Another useful parameter
for UDP input is TimeRequery. If you have a lot of messages arriving at the
same time, doing a gettimeofday() call to the system can be slow, and many
consecutive calls will return the same value, so rsyslog lets you say that as
long as the incoming buffer from the OS has more logs ready, only do a time
lookup every N messages instead of every message. Setting th
is to something like 100 or 1000 will virtually eliminate the overhead of
doing this lookup, and the worst that can happen is that the time received
timestamp may be off by 1 second for messages that arrive in a batch right at
the end of one second and the beginning of the next second (i.e. you will
almost certinly never notice this, this does not affect the timestamp generated
by the host system in any case) back in the rsyslog 4.x days, I was able to get
rsyslog to handle gig-e wire speed (~380K logs/sec), and rsyslog has only
gotten faster since. David Lang On Thu, 8 Aug 2013, Robert Ortiz wrote: > Hey
Guys, > > I am new to this mailing list and I wanted to see about getting some
pointers > if possible regarding tweakin rsyslog: > > I am pretty new to
rsyslog, and I've been given a pretty fun task... to test > rsyslog vs
syslog-ng and pick the best one, I am having a problem with rsyslog > where im
at 25K/mps and im dropping logs, I need to get it at 100k mps with > and I
'm not sure where the misconfiguration is if anyone could take a look I >
would really appreciatte it, > > my current setup: > > rhel 6.4 x86_64 >
rsyslog-5.8.10-2.el6.x86_64 > Dual Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz >
32GB RAM > 500GB 15k rai d 0 > > > # rsyslog v5 configuration file > > # For
more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html > # If you
experience problems, see http://www.rsyslog.com/doc/troubleshoot.html > > ####
MODULES #### > > $ModLoad imuxsock # provides support for local system logging
(e.g. via logger command) > $ModLoad imklog # provides kernel logging support
(previously done by rklogd) > #$ModLoad immark # provides --MARK-- message
capability > > # Provides UDP syslog reception > $ModLoad imudp > $UDPServerRun
514 > # $UDPServerTimeRequery 10 > > # Provides TCP syslog reception >
#$ModLoad imtcp > #$InputTCPServerRun 514 > > > #### GLOBAL DIRECTIVES #### > >
# Use default timestamp format > $ActionFileDefaultTemplate RSYSLOG_Tr
aditionalFileFormat > > # File syncing capability is disabled by default. This
feature is usually not required, > # not useful and an extreme performance hit
> #$ActionFileEnableSync on > > # Include all config files in /etc/rsyslog.d/ >
$IncludeConf ig /etc/rsyslog.d/*.conf > > # Set Buffer Size - default is 4k > #
$OMFileIOBufferSize 128k > # Set Main Message Queue Size - default is 10000 > #
$MainMsgQueueSize 50000 > > #### RULES #### > > # Log all kernel messages to
the console. > # Logging much else clutters up the screen. > #kern.*
/dev/console > > if $hostname contains 'pdc' then /var/log/test/f_ad > & ~ > if
$hostname contains 'fdfw' then /var/log/test/f_fw > & ~ > if $hostname contains
'mail' then /var/log/test/f_mail > & ~ > if $hostname contains 'pix' then
/var/log/test/ix > & ~ > if $hostname contains 'rout' then /var/log/test/rout >
& ~ > if $hostname contains 'networks' then /var/log/test/net > & ~ > #if
$fromhost-ip == '10.0.0.10' then /var/log/test/thost > #&
~ > #if $hostname startswith 'virtserv' then /var/log/test/test_virtserv >
#&~ > #if $fromhost-ip startswith '10.0.6' then /var/log/test/test_10.0.6 > #&
~ > > > # Log anything (except mail) of level info or higher. > # Don't log
private authenticati on messages! > #*.info;mail.none;authpriv.none;cron.none
/var/log/messages > *.debug /var/log/messages > > # Log all the mail messages
in one place. > mail.* -/var/log/maillog > > > # Log cron stuff > cron.*
/var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save
news errors of level crit and higher in a special file. > uucp,news.crit
/var/log/spooler > > # Save boot messages also to boot.log > local7.*
/var/log/boot.log > > > # ### begin forwarding rule ### > # The statement
between the begin ... end define a SINGLE forwarding > # rule. They belong
together, do NOT split them. If you create multiple > # The statement between
the begin ... end define a SINGLE forwarding > # rule. They belong together, do
NO
T split them. If you create multiple > # forwarding rules, duplicate the whole
block! > # Remote Logging (we use TCP for reliable delivery) > # > # An on-disk
queue is created for this action. If the remote host is > # down, messages are
spooled to dis k and sent when it is up again. > #$WorkDirectory
/var/lib/rsyslog # where to place spool files > #$ActionQueueFileName fwdRule1
# unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space
limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save
messages to disk on shutdown > #$ActionQueueType LinkedList # run
asynchronously > #$ActionResumeRetryCount -1 # infinite retries if host is down
> # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.*
@@remote-host:514 > # ### end of the forwarding rule ### > > > > Robert. >
_______________________________________________ > rsyslog mailing list >
http://lists.adiscon.net/mailman/listinfo/rsyslog >
http://www.rsyslog.com/professio
nal-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards >
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT. > ___________ ____________________________________ rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts
are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
NOT POST if you DON'T LIKE THAT. Robert.
_______________________________________________ rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts
are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and
DO NOT POST if you DON'T LIKE THAT.
Robert.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
