On Tue, 22 Oct 2013, Aaron Wiebe wrote:
On instances: rsyslog will top out around 2-3 cores. Run 5-10 instances on the same machine using different ports if possible, on modern hardware.
this largely depends on your config, we've seen configs that will max out more cores. you should be able to use approximatly one core per output plus one or more cores for your input. so the more different things you are doing, the more cores you will use.
It doesn't sound like your volume is that high. You just need to segment your ingest a bit. One machine should comfortably be able to handle 100-200k messages per second - but the threading model (as much as it's improved recently) still can't quite max out modern hardware. Look at multiple instances on the same machine to see if you can't bring the concurrency up.
what parts are you seeing that you are running into problems with? Getting reports of specific bottlenecks helps a lot.
right now we know we have limits with message modification modules, and many of the output modules are single threaded.
the discussion over the last couple of days started with talking about ways to have multiple action clauses and split the load across them so that you would have more concurrent connections to external services (hadoop, elasticsearch, etc)
a week or so ago we found that the imudp module could be overwelmed on a single core, so it gained the ability to use multiple threads.
imptcp allows for multiple threads receiving tcp syslog. so the input side should be able to keep up with just about any load. more info on the bottlenecks is always useful. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
