We are running 100,000 to 600,000 messages per second of messages ranging from 4KB to 2MB in size. (obviously done using imptcp)
I wasn't aware of the potential batching issue. That is extremely good to know. -- James --- Sent from my mobile phone --- ----- Reply message ----- From: "David Lang" <[email protected]> To: "rsyslog-users" <[email protected]> Subject: [rsyslog] Question about maximum queue.workerthreads count Date: Tue, Mar 10, 2015 5:30 PM On Tue, 10 Mar 2015, Boylan, James wrote: > Greetings everyone! > > > Is there a hardcoded maximum number of worker threads for queues and threads > for imtcp/imudp? Also what are the concerns with using a higher number of > threads for handling of these. There is probably a max number of threads, but it is almost certinly much higher than you should need to go before the limiting factor will be the speed of your connection. a couple thread can handle a saturated gig-E network (at least without encryption or compression), at ~400k logs/second The problem you run into with too many threads is that message batching doesn't take place, each thread grabs one message and puts it on the queue. Doing this one message at a time results in a lot of locking overhead and can end up drastically slowing down all work. The threads that try to retrieve the messages from the queue end up having trouble getting the lock to be able to do work because they are competing against the thread trying to add messages. The batching process drastically reduces the amount of time the main queue is locked to process a given set of messages. We've seen cases where people set the number of threads in the 4-8 range and found they were getting much worse throughput than when they had it set to 1. This is really something that you need to measure, both under full load and under partial load > I find myself in a position to have to scale central logging servers > vertically and I'm attempting to locate the risks and issues of increasing > these numbers. what sort of log numbers and traffic volume are you looking at? David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
