with modern versions and simple configs, rsyslog can do 1m messages/sec
your setup involved some fairly nasty regex matches, those can eat up a fair bit of time. If you can simplify the regex it would help a lot.
If you can create a template for mmnormalize to use to create the variables that you need, it will make a huge difference.
But overall, you first need to make sure that the bottleneck is where you expect it to be.
depending on how bursty your inbound traffic is, you may find that just increasing the size of the main queue will help (and possibly switching to different type of main queue, I find fixedarray to be faster than linked list for example)
run top, and hit 'H' to show the different threads. Then you can see when one thread is using all of the CPU.
a strace of the thread eating up all the CPU will give you a very good window of that that thread is doing (what the thread is labeled as in top may be enough and avoid the need to do this.)
David Lang On Tue, 30 Jul 2013, Esmq wrote:
hi,all i have about 20 servers running as rsyslog client that forward logs to a rsyslog server via TCP, each client forwards at a rate of 200 messages per second, that's to say my rsyslog server should process 5k messages per second. though i really know that process 5k messages per seconds should not a problem of rsyslog ~ when i use "netstat" command, it some times show there a non-zero value of Recv-Q on rsyslog-server, which means rsyslog unable to accept the incoming message quickly... i thought there's a problem of my rsyslog configuration, especially for i using some complex template to generate dynamic file name, and some regex to filter the logs~ here is the config sample ( i use rsyslog 4.6.4 on debian squeeze) ################################################################################################# $template relayLogFormat, "%msg:2:$%\n" #using dynamic file name, according to some specific message field $template valySplitLogFile,"/home/data/logs/%msg:R,ERE,2,ZERO:.*(chl|channel)=([^&? ]+)\.(test\.com|example\.com).*--end%-valy-%$YEAR%%$MONTH%%$DAY%.access.log" #filtering the log and put it to appropriate file :msg, ereregex, "http://valy\.nie\.test\.com/query\?.*(chl|channel)=([^&? ]+)\.(test\.com|example\.com).*" -?valySplitLogFile;relayLogFormat #following is some other config... i guess the above config is an extreme performance hit, but i don't konw how to make a improvement~ any suggestion for improvement will be appreciate, _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
