Thanks for your answers, Rainer. That clarifies a lot! Regarding upper limits, I'm aware that both workers and batch sizes defined are maximums.
-- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Apr 8, 2015 at 1:51 PM, Rainer Gerhards <[email protected]> wrote: > 2015-04-08 12:43 GMT+02:00 Radu Gheorghe <[email protected]>: > > Thanks Rainer! So let me see if I get this straight. Say I want to > > mmnormalize some logs and then omelasticsearch them. I would (please > > correct me where I'm wrong): > > > > - define a beefy main queue for buffering and everything > > - add a mmnormalize action that will change events from the main queue. > Can > > I use queue.workerthreads to parallelize this? What about processing in > > batches - does queue.dequeuebatchsize on the main queue govern that or > it's > > impossible? > > - add a omelasticsearch action with no action queue (unless I want to > copy > > events from the main queue and process them async - say write them to ES > > but also to a file). The ES bulk size would be queue.dequeuebatchsize of > > the main queue and I can use multiple threads via queue.workerthreads > > > > Am I getting it right? > > sounds good, assuming this is your whole config (sounds so). Regarding > the queue settings: they affect every action that's not runing on a > separate queue. So if you define workers and batch sizes, they also > apply to mmnormalize. Note that you define an *upper limit* on worker > threads. Rsyslog will only spawn them if required (what's required is > defined via other queue settings). > > Rainer > > > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > On Wed, Apr 8, 2015 at 12:28 PM, Rainer Gerhards < > [email protected]> > > wrote: > > > >> 2015-04-08 9:34 GMT+02:00 Radu Gheorghe <[email protected]>: > >> > Hello, > >> > > >> > I have three questions about action queues: > >> > 1) if I have a message modifier action (e.g. mmnormalize), should I > add > >> an > >> > action queue for better performance? > >> > >> no, simply because that wouldn't work as expected. Remember that a > >> queue runs async, so the modification would not get back to the > >> original message object. > >> > >> > 2) what's the flow of messages if I have both mmnormalize and, say, > >> > omelasticsearch? Messages flow from the main queue to mmnormalize's > queue > >> > (if any) and then back to the main queue to be processed by > >> omelasticsearch? > >> > >> I guess I already answered this ;) With a queue, message objects are > >> duplicated and never be injected back. that's the prime reason you > >> want a queue. > >> > >> > 3) how does queue.dequeuebatchsize work if I have an action queue. For > >> > example: > >> > > >> > * I have a main queue with dequeuebatchsize=100 > >> > * I have an omelasticsearch action with no queue (direct) and > >> > dequeuebatchsize=1000 and BulkMode="on" > >> > >> dequeuebatchsize has no effect in direct mode. > >> > >> > > >> > How many messages are sent to ES in a single bulk? 100? 1000? 1? > >> > >> at most 100 (fewer if the batch doesn't get full). > >> > >> > I can do a test to answer this last question, but I'd like to > understand > >> > how multiple actions and direct queues work. For example, I didn't > notice > >> > any difference in performance (during a benchmark) with and without an > >> > action queue with mmnormalize. But maybe the bottleneck was somewhere > >> else. > >> > >> queues always *cost* performance. If you need more concurrency, you > >> should adjust the main or ruleset queue accordingly. > >> > >> HTH > >> Rainer > >> > > >> > Best regards, > >> > Radu > >> > -- > >> > Performance Monitoring * Log Analytics * Search Analytics > >> > Solr & Elasticsearch Support * http://sematext.com/ > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
