Thanks Rainer! So let me see if I get this straight. Say I want to mmnormalize some logs and then omelasticsearch them. I would (please correct me where I'm wrong):
- define a beefy main queue for buffering and everything - add a mmnormalize action that will change events from the main queue. Can I use queue.workerthreads to parallelize this? What about processing in batches - does queue.dequeuebatchsize on the main queue govern that or it's impossible? - add a omelasticsearch action with no action queue (unless I want to copy events from the main queue and process them async - say write them to ES but also to a file). The ES bulk size would be queue.dequeuebatchsize of the main queue and I can use multiple threads via queue.workerthreads Am I getting it right? -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Apr 8, 2015 at 12:28 PM, Rainer Gerhards <[email protected]> wrote: > 2015-04-08 9:34 GMT+02:00 Radu Gheorghe <[email protected]>: > > Hello, > > > > I have three questions about action queues: > > 1) if I have a message modifier action (e.g. mmnormalize), should I add > an > > action queue for better performance? > > no, simply because that wouldn't work as expected. Remember that a > queue runs async, so the modification would not get back to the > original message object. > > > 2) what's the flow of messages if I have both mmnormalize and, say, > > omelasticsearch? Messages flow from the main queue to mmnormalize's queue > > (if any) and then back to the main queue to be processed by > omelasticsearch? > > I guess I already answered this ;) With a queue, message objects are > duplicated and never be injected back. that's the prime reason you > want a queue. > > > 3) how does queue.dequeuebatchsize work if I have an action queue. For > > example: > > > > * I have a main queue with dequeuebatchsize=100 > > * I have an omelasticsearch action with no queue (direct) and > > dequeuebatchsize=1000 and BulkMode="on" > > dequeuebatchsize has no effect in direct mode. > > > > > How many messages are sent to ES in a single bulk? 100? 1000? 1? > > at most 100 (fewer if the batch doesn't get full). > > > I can do a test to answer this last question, but I'd like to understand > > how multiple actions and direct queues work. For example, I didn't notice > > any difference in performance (during a benchmark) with and without an > > action queue with mmnormalize. But maybe the bottleneck was somewhere > else. > > queues always *cost* performance. If you need more concurrency, you > should adjust the main or ruleset queue accordingly. > > HTH > Rainer > > > > Best regards, > > Radu > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
