unsubscribe 2015-03-11 4:15 GMT-03:00 David Lang <[email protected]>:
> How fat a pipe do you have? that sounds like it's pushing the limit for > 10G interfaces. > > Also, remember that using multiple threads on the input module only helps > if you have multiple connections to you. A given connection can only really > be handled by one thread (if you think about it, the possibility that > different threads could get different parts of the same message when > reading from one connection would be a nightmare to deal with) > > I would guess that you can probably handle this load with a handful of > threads (low to mid single digits), but at those volumes, you are going to > be running into all sorts of bottlenecks. Just getting the OS TCP stack to > keep up at 10Gb is rough, and you then have to do something with the log > messages. It takes quite an impressive I/O system to be able to write data > at that speed, and if you are sending the data out at that speed, I hope > it's through another interface. > > Around the late v5 and early v6 days I was doing testing that was pushing > ~400K logs/sec at 1/4KB per message (256 bytes) and I found that Gig-E was > my limiting factor, so going 1.5x faster with messages 16x-8000x larger on > a pipe only 10x faster seems like it would be pushing it. If you are doing > compression you may be able to get away with it, but it will take > significantly more CPU, and again, you have to be able to output the logs. > > I'm pretty sure that you're going to need to scale horizontally to do > this. What is it that's driving you to try scaling vertically? > > David Lang > > > On Tue, 10 Mar 2015, Boylan, James wrote: > > We are running 100,000 to 600,000 messages per second of messages ranging >> from 4KB to 2MB in size. (obviously done using imptcp) >> >> I wasn't aware of the potential batching issue. That is extremely good to >> know. >> >> -- James >> --- Sent from my mobile phone --- >> >> ----- Reply message ----- >> From: "David Lang" <[email protected]> >> To: "rsyslog-users" <[email protected]> >> Subject: [rsyslog] Question about maximum queue.workerthreads count >> Date: Tue, Mar 10, 2015 5:30 PM >> >> On Tue, 10 Mar 2015, Boylan, James wrote: >> >> Greetings everyone! >>> >>> >>> Is there a hardcoded maximum number of worker threads for queues and >>> threads for imtcp/imudp? Also what are the concerns with using a higher >>> number of threads for handling of these. >>> >> >> There is probably a max number of threads, but it is almost certinly much >> higher >> than you should need to go before the limiting factor will be the speed >> of your >> connection. a couple thread can handle a saturated gig-E network (at least >> without encryption or compression), at ~400k logs/second >> >> The problem you run into with too many threads is that message batching >> doesn't >> take place, each thread grabs one message and puts it on the queue. Doing >> this >> one message at a time results in a lot of locking overhead and can end up >> drastically slowing down all work. The threads that try to retrieve the >> messages >> from the queue end up having trouble getting the lock to be able to do >> work >> because they are competing against the thread trying to add messages. The >> batching process drastically reduces the amount of time the main queue is >> locked >> to process a given set of messages. >> >> We've seen cases where people set the number of threads in the 4-8 range >> and >> found they were getting much worse throughput than when they had it set >> to 1. >> >> This is really something that you need to measure, both under full load >> and >> under partial load >> >> >> I find myself in a position to have to scale central logging servers >>> vertically and I'm attempting to locate the risks and issues of >>> increasing >>> these numbers. >>> >> >> what sort of log numbers and traffic volume are you looking at? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- Paulo Ricardo Bruck consultor tel 011 3596-4881/4882 011 98140-9184 (TIM) http://www.contatogs.com.br http://www.protejasuarede.com.br gpg AAA59989 at wwwkeys.us.pgp.net _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
