David Thank you very much for the input, I will be running some baseload tests tomorrow, and I will also make changes according to your suggestions and will test again. I will let you guys know how it went, thank you! ----- Original Message ----- From: David Lang Sent: 08/08/13 02:14 PM To: rsyslog-users Subject: Re: [rsyslog] performance tweaking
The first thing I would do is make sure that you start rsyslog without DNS lookups (add the -x flag to startup), the overhead of doing a DNS lookup on each message that comes in is very significant. The newest versions of rsyslog (7.4) include some caching of DNS data, but it can still be significant. With 5.x I think this change by itself will probably get you over 100K logs/sec The next thing is the main message queue size, your configuration leaves it at the default of 10K, if you are looking to receive 100K messages/sec, that's not very big, I would set it large enough to handle at least a couple seconds worth of logs, and if this box is a dedicated syslog server, set it so that it will use the majority of RAM on your system. with 32G of ram on the system, and a default 2k message size, setting this well above 1M is very reasonable. As noted by someone else, setting larger buffers in /etc/sysctl.conf may help If you can disable connection tracking in the iptables stack, i t will significantly reduce the kernel overhead (how many systems are you recieving logs from?) Setting net.ipv4.netfilter.ip_conntrack_max large may help As far as your rules go: 'contains' is significantly more expensive than 'startswith' on version 5.x, the if..then structure is significantly slower than the properties filter like: :hostname, contains, 'pdc' /var/log/test/f_ad rsyslog 7.x contains a ruleset optimizer that eliminates this performance problem. what do you have in your included files? It's worth checking to see where your bottleneck is, simplify your rules to write everything to one file and see what the resulting performance is like. That way you know if your problem is on the input side or the output side. if you run top, and hit 'H' to show the different threads, you can see what threads are running out of CPU time. My guess is that it will be a thread labeled "Main Q", which is the output side of things (due to the use of the inefficient if..then filters ), and that's causing the too-small queue to fill up, causing UDP messages to be lost. rsyslog 7.4 combined with a recent Linux kernel also has the ability to recieve multiple UDP packets in a single system call, this would significantly improve performance. I don't know if RHEL 6.4 includes a recent enough kernel. This is the batchSize parameter. Another useful parameter for UDP input is TimeRequery. If you have a lot of messages arriving at the same time, doing a gettimeofday() call to the system can be slow, and many consecutive calls will return the same value, so rsyslog lets you say that as long as the incoming buffer from the OS has more logs ready, only do a time lookup every N messages instead of every message. Setting this to something like 100 or 1000 will virtually eliminate the overhead of doing this lookup, and the worst that can happen is that the time received timestamp may be off by 1 second for messages that arrive in a batch right at the end of one second and the beginning of the next second (i.e. you will almost certinly never notice this, this does not affect the timestamp generated by the host system in any case) back in the rsyslog 4.x days, I was able to get rsyslog to handle gig-e wire speed (~380K logs/sec), and rsyslog has only gotten faster since. David Lang On Thu, 8 Aug 2013, Robert Ortiz wrote: > Hey Guys, > > I am new to this mailing list and I wanted to see about getting some pointers > if possible regarding tweakin rsyslog: > > I am pretty new to rsyslog, and I've been given a pretty fun task... to test > rsyslog vs syslog-ng and pick the best one, I am having a problem with rsyslog > where im at 25K/mps and im dropping logs, I need to get it at 100k mps with > and I'm not sure where the misconfiguration is if anyone could take a look I > would really appreciatte it, > > my current setup: > > rhel 6.4 x86_64 > rsyslog-5.8.10-2.el6.x86_64 > Dual Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz > 32GB RAM > 500GB 15k rai d 0 > > > # rsyslog v5 configuration file > > # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html > # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html > > #### MODULES #### > > $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) > $ModLoad imklog # provides kernel logging support (previously done by rklogd) > #$ModLoad immark # provides --MARK-- message capability > > # Provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > # $UDPServerTimeRequery 10 > > # Provides TCP syslog reception > #$ModLoad imtcp > #$InputTCPServerRun 514 > > > #### GLOBAL DIRECTIVES #### > > # Use default timestamp format > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # File syncing capability is disabled by default. This feature is usually not required, > # not useful and an extreme performance hit > #$ActionFileEnableSync on > > # Include all config files in /etc/rsyslog.d/ > $IncludeConf ig /etc/rsyslog.d/*.conf > > # Set Buffer Size - default is 4k > # $OMFileIOBufferSize 128k > # Set Main Message Queue Size - default is 10000 > # $MainMsgQueueSize 50000 > > #### RULES #### > > # Log all kernel messages to the console. > # Logging much else clutters up the screen. > #kern.* /dev/console > > if $hostname contains 'pdc' then /var/log/test/f_ad > & ~ > if $hostname contains 'fdfw' then /var/log/test/f_fw > & ~ > if $hostname contains 'mail' then /var/log/test/f_mail > & ~ > if $hostname contains 'pix' then /var/log/test/ix > & ~ > if $hostname contains 'rout' then /var/log/test/rout > & ~ > if $hostname contains 'networks' then /var/log/test/net > & ~ > #if $fromhost-ip == '10.0.0.10' then /var/log/test/thost > #& ~ > #if $hostname startswith 'virtserv' then /var/log/test/test_virtserv > #&~ > #if $fromhost-ip startswith '10.0.6' then /var/log/test/test_10.0.6 > #& ~ > > > # Log anything (except mail) of level info or higher. > # Don't log private authenticati on messages! > #*.info;mail.none;authpriv.none;cron.none /var/log/messages > *.debug /var/log/messages > > # Log all the mail messages in one place. > mail.* -/var/log/maillog > > > # Log cron stuff > cron.* /var/log/cron > > # Everybody gets emergency messages > *.emerg * > > # Save news errors of level crit and higher in a special file. > uucp,news.crit /var/log/spooler > > # Save boot messages also to boot.log > local7.* /var/log/boot.log > > > # ### begin forwarding rule ### > # The statement between the begin ... end define a SINGLE forwarding > # rule. They belong together, do NOT split them. If you create multiple > # The statement between the begin ... end define a SINGLE forwarding > # rule. They belong together, do NOT split them. If you create multiple > # forwarding rules, duplicate the whole block! > # Remote Logging (we use TCP for reliable delivery) > # > # An on-disk queue is created for this action. If the remote host is > # down, messages are spooled to dis k and sent when it is up again. > #$WorkDirectory /var/lib/rsyslog # where to place spool files > #$ActionQueueFileName fwdRule1 # unique name prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > #$ActionResumeRetryCount -1 # infinite retries if host is down > # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* @@remote-host:514 > # ### end of the forwarding rule ### > > > > Robert. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > ___________ ____________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. Robert. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
