Hello Robert, Some pointers can be: - use the impstats module to see the state of your queues - use htop or something like that to see which threads consume the most CPU (maybe you can start more of those threads to solve the problem). I didn't do this myself but David and Rainer keep saying that threads are labeled so you can understand what uses more resources - netstat -su should also help with some information on packet loss and stuff
Best regards, Radu 2013/8/21 Robert Ortiz <[email protected]> > Hello guys, > > So i was able to get the logs to come in at 25k mps and not drop a single > one, I changed the ctl file to increase the mem to 200000, I also installed > nscd and was able to get this to work, unfortunately when i went up to 50k > mps i dropped about 20k mps, is there a way I can see something that can > tell me where I might have a problem? > > Robert > ----- Original Message ----- > From: David Lang > Sent: 08/08/13 02:14 PM > To: rsyslog-users > Subject: Re: [rsyslog] performance tweaking > > The first thing I would do is make sure that you start rsyslog without DNS > lookups (add the -x flag to startup), the overhead of doing a DNS lookup on > each message that comes in is very significant. The newest versions of > rsyslog (7.4) include some caching of DNS data, but it can still be > significant. With 5.x I think this change by itself will probably get you > over 100K logs/sec The next thing is the main message queue size, your > configuration leaves it at the default of 10K, if you are looking to > receive 100K messages/sec, that's not very big, I would set it large enough > to handle at least a couple seconds worth of logs, and if this box is a > dedicated syslog server, set it so that it will use the majority of RAM on > your system. with 32G of ram on the system, and a default 2k message size, > setting this well above 1M is very reasonable. As noted by someone else, > setting larger buffers in /etc/sysctl.conf may help If you can disable > connection tracking in the iptables stack, i > t will significantly reduce the kernel overhead (how many systems are you > recieving logs from?) Setting net.ipv4.netfilter.ip_conntrack_max large may > help As far as your rules go: 'contains' is significantly more expensive > than 'startswith' on version 5.x, the if..then structure is significantly > slower than the properties filter like: :hostname, contains, 'pdc' > /var/log/test/f_ad rsyslog 7.x contains a ruleset optimizer that eliminates > this performance problem. what do you have in your included files? It's > worth checking to see where your bottleneck is, simplify your rules to > write everything to one file and see what the resulting performance is > like. That way you know if your problem is on the input side or the output > side. if you run top, and hit 'H' to show the different threads, you can > see what threads are running out of CPU time. My guess is that it will be a > thread labeled "Main Q", which is the output side of things (due to the use > of the inefficient if..then filters > ), and that's causing the too-small queue to fill up, causing UDP > messages to be lost. rsyslog 7.4 combined with a recent Linux kernel also > has the ability to recieve multiple UDP packets in a single system call, > this would significantly improve performance. I don't know if RHEL 6.4 > includes a recent enough kernel. This is the batchSize parameter. Another > useful parameter for UDP input is TimeRequery. If you have a lot of > messages arriving at the same time, doing a gettimeofday() call to the > system can be slow, and many consecutive calls will return the same value, > so rsyslog lets you say that as long as the incoming buffer from the OS has > more logs ready, only do a time lookup every N messages instead of every > message. Setting this to something like 100 or 1000 will virtually > eliminate the overhead of doing this lookup, and the worst that can happen > is that the time received timestamp may be off by 1 second for messages > that arrive in a batch right at the end of one second > and the beginning of the next second (i.e. you will almost certinly never > notice this, this does not affect the timestamp generated by the host > system in any case) back in the rsyslog 4.x days, I was able to get rsyslog > to handle gig-e wire speed (~380K logs/sec), and rsyslog has only gotten > faster since. David Lang On Thu, 8 Aug 2013, Robert Ortiz wrote: > Hey > Guys, > > I am new to this mailing list and I wanted to see about getting > some pointers > if possible regarding tweakin rsyslog: > > I am pretty new > to rsyslog, and I've been given a pretty fun task... to test > rsyslog vs > syslog-ng and pick the best one, I am having a problem with rsyslog > where > im at 25K/mps and im dropping logs, I need to get it at 100k mps with > and > I'm not sure where the misconfiguration is if anyone could take a look I > > would really appreciatte it, > > my current setup: > > rhel 6.4 x86_64 > > rsyslog-5.8.10-2.el6.x86_64 > Dual Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz > > 32GB RAM > 500GB 15k rai > d 0 > > > # rsyslog v5 configuration file > > # For more information see > /usr/share/doc/rsyslog-*/rsyslog_conf.html > # If you experience problems, > see http://www.rsyslog.com/doc/troubleshoot.html > > #### MODULES #### > > > $ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > $ModLoad imklog # provides kernel logging support > (previously done by rklogd) > #$ModLoad immark # provides --MARK-- message > capability > > # Provides UDP syslog reception > $ModLoad imudp > > $UDPServerRun 514 > # $UDPServerTimeRequery 10 > > # Provides TCP syslog > reception > #$ModLoad imtcp > #$InputTCPServerRun 514 > > > #### GLOBAL > DIRECTIVES #### > > # Use default timestamp format > > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # File syncing > capability is disabled by default. This feature is usually not required, > > # not useful and an extreme performance hit > #$ActionFileEnableSync on > > > # Include all config files in /etc/rsyslog.d/ > $IncludeConf > ig /etc/rsyslog.d/*.conf > > # Set Buffer Size - default is 4k > # > $OMFileIOBufferSize 128k > # Set Main Message Queue Size - default is 10000 > > # $MainMsgQueueSize 50000 > > #### RULES #### > > # Log all kernel > messages to the console. > # Logging much else clutters up the screen. > > #kern.* /dev/console > > if $hostname contains 'pdc' then > /var/log/test/f_ad > & ~ > if $hostname contains 'fdfw' then > /var/log/test/f_fw > & ~ > if $hostname contains 'mail' then > /var/log/test/f_mail > & ~ > if $hostname contains 'pix' then > /var/log/test/ix > & ~ > if $hostname contains 'rout' then > /var/log/test/rout > & ~ > if $hostname contains 'networks' then > /var/log/test/net > & ~ > #if $fromhost-ip == '10.0.0.10' then > /var/log/test/thost > #& ~ > #if $hostname startswith 'virtserv' then > /var/log/test/test_virtserv > #&~ > #if $fromhost-ip startswith '10.0.6' > then /var/log/test/test_10.0.6 > #& ~ > > > # Log anything (except mail) of > level info or higher. > # Don't log private authenticati > on messages! > #*.info;mail.none;authpriv.none;cron.none > /var/log/messages > *.debug /var/log/messages > > # Log all the mail > messages in one place. > mail.* -/var/log/maillog > > > # Log cron stuff > > cron.* /var/log/cron > > # Everybody gets emergency messages > *.emerg * > > > # Save news errors of level crit and higher in a special file. > > uucp,news.crit /var/log/spooler > > # Save boot messages also to boot.log > > local7.* /var/log/boot.log > > > # ### begin forwarding rule ### > # The > statement between the begin ... end define a SINGLE forwarding > # rule. > They belong together, do NOT split them. If you create multiple > # The > statement between the begin ... end define a SINGLE forwarding > # rule. > They belong together, do NOT split them. If you create multiple > # > forwarding rules, duplicate the whole block! > # Remote Logging (we use TCP > for reliable delivery) > # > # An on-disk queue is created for this action. > If the remote host is > # down, messages are spooled to dis > k and sent when it is up again. > #$WorkDirectory /var/lib/rsyslog # > where to place spool files > #$ActionQueueFileName fwdRule1 # unique name > prefix for spool files > #$ActionQueueMaxDiskSpace 1g # 1gb space limit > (use as much as possible) > #$ActionQueueSaveOnShutdown on # save messages > to disk on shutdown > #$ActionQueueType LinkedList # run asynchronously > > #$ActionResumeRetryCount -1 # infinite retries if host is down > # remote > host is: name/ip:port, e.g. 192.168.0.1:514, port optional > #*.* > @@remote-host:514 > # ### end of the forwarding rule ### > > > > Robert. > > _______________________________________________ > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? > Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC > mailing list, posts are ARCHIVED by a myriad of sites beyond our control. > PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > ___________ > ____________________________________ rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ What's up with rsyslog? > Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing > list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE > UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > > > > > > Robert. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
