On Wed, 8 Apr 2015, Radu Gheorghe wrote:
Thanks Rainer! So let me see if I get this straight. Say I want to
mmnormalize some logs and then omelasticsearch them. I would (please
correct me where I'm wrong):
- define a beefy main queue for buffering and everything
- add a mmnormalize action that will change events from the main queue. Can
I use queue.workerthreads to parallelize this? What about processing in
batches - does queue.dequeuebatchsize on the main queue govern that or it's
impossible?
- add a omelasticsearch action with no action queue (unless I want to copy
events from the main queue and process them async - say write them to ES
but also to a file). The ES bulk size would be queue.dequeuebatchsize of
the main queue and I can use multiple threads via queue.workerthreads
Am I getting it right?
I'm pretty sure that you understand this, but for others who see this thread:
The reason to have a separate queue for an action or ruleset is so that if that
blocks, it doesn't immediatly block all other processing of logs on the main
queue. The other things happening can be writing to local files, or delivering
to other remote destinations.
giving everything it's own queue and creating multiple worker threads will
actually significantly HURT performance, because one of the big bottlenecks at
high performance is the locking of the queues as messages get added and removed
from a queue. When everything is a queue and you allow many worker threads, you
can get a batch of messages arriving (enough to trigger additional worker thread
creation), but then the worker threads get in each other's way as they are all
trying to lock the main queue at the same time to move a batch of messages to
the other queues.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
