2015-04-08 12:43 GMT+02:00 Radu Gheorghe <[email protected]>: > Thanks Rainer! So let me see if I get this straight. Say I want to > mmnormalize some logs and then omelasticsearch them. I would (please > correct me where I'm wrong): > > - define a beefy main queue for buffering and everything > - add a mmnormalize action that will change events from the main queue. Can > I use queue.workerthreads to parallelize this? What about processing in > batches - does queue.dequeuebatchsize on the main queue govern that or it's > impossible? > - add a omelasticsearch action with no action queue (unless I want to copy > events from the main queue and process them async - say write them to ES > but also to a file). The ES bulk size would be queue.dequeuebatchsize of > the main queue and I can use multiple threads via queue.workerthreads > > Am I getting it right?
sounds good, assuming this is your whole config (sounds so). Regarding the queue settings: they affect every action that's not runing on a separate queue. So if you define workers and batch sizes, they also apply to mmnormalize. Note that you define an *upper limit* on worker threads. Rsyslog will only spawn them if required (what's required is defined via other queue settings). Rainer > > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > On Wed, Apr 8, 2015 at 12:28 PM, Rainer Gerhards <[email protected]> > wrote: > >> 2015-04-08 9:34 GMT+02:00 Radu Gheorghe <[email protected]>: >> > Hello, >> > >> > I have three questions about action queues: >> > 1) if I have a message modifier action (e.g. mmnormalize), should I add >> an >> > action queue for better performance? >> >> no, simply because that wouldn't work as expected. Remember that a >> queue runs async, so the modification would not get back to the >> original message object. >> >> > 2) what's the flow of messages if I have both mmnormalize and, say, >> > omelasticsearch? Messages flow from the main queue to mmnormalize's queue >> > (if any) and then back to the main queue to be processed by >> omelasticsearch? >> >> I guess I already answered this ;) With a queue, message objects are >> duplicated and never be injected back. that's the prime reason you >> want a queue. >> >> > 3) how does queue.dequeuebatchsize work if I have an action queue. For >> > example: >> > >> > * I have a main queue with dequeuebatchsize=100 >> > * I have an omelasticsearch action with no queue (direct) and >> > dequeuebatchsize=1000 and BulkMode="on" >> >> dequeuebatchsize has no effect in direct mode. >> >> > >> > How many messages are sent to ES in a single bulk? 100? 1000? 1? >> >> at most 100 (fewer if the batch doesn't get full). >> >> > I can do a test to answer this last question, but I'd like to understand >> > how multiple actions and direct queues work. For example, I didn't notice >> > any difference in performance (during a benchmark) with and without an >> > action queue with mmnormalize. But maybe the bottleneck was somewhere >> else. >> >> queues always *cost* performance. If you need more concurrency, you >> should adjust the main or ruleset queue accordingly. >> >> HTH >> Rainer >> > >> > Best regards, >> > Radu >> > -- >> > Performance Monitoring * Log Analytics * Search Analytics >> > Solr & Elasticsearch Support * http://sematext.com/ >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
