Hi Dave, Thanks for the answer. Ouch. This sounds suboptimal :( and makes me feel like I have no control :(. Now that you said this, I have a feeling Radu asked about this exact same thing at one point and had a similar reaction.....
.... aha, yes, found it: http://search-devops.com/m/PamuZftUHylTXGc1&subj=+rsyslog+Can+we+have+a+minimum+bulk+size+for+omelasticsearch+ + issue from that thread: https://github.com/rsyslog/rsyslog/issues/495 Thanks and sorry for the noise. Otis -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ On Wed, Nov 18, 2015 at 9:15 PM, David Lang <[email protected]> wrote: > in rsyslog, batches don't wait to be filled. the algorithm is designed for > low-latency, in that each time a batch is processed, it takes from > 1-batchsize messages (how ever many are available) and processes them. > > so if you only have one message arrive per minute, each message will be > processed individually. You have to have messages arriving faster than they > can be individually processed before the batches will contain more than one > message. > > David Lang > > > On Wed, 18 Nov 2015, Otis Gospodnetić wrote: > > Date: Wed, 18 Nov 2015 17:05:02 -0500 >> From: Otis Gospodnetić <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: [rsyslog] omelasticsearch: send every N seconds even if bulk >> not full? >> >> >> Hi, >> >> omelasticseearch has options like: >> >> queue.type="linkedlist" >> queue.size="5000" >> queue.dequeuebatchsize="300" >> action.resumeretrycount="-1") >> >> >> What happens when the volume of logs is very low. For example, imagine a >> log file where a new log event shows up only once a minute. >> >> Looking at >> >> http://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html >> I don't see a mention of "timeout" or "send every N seconds even if the >> specified bulk size has not been reached yet". >> >> Is there an option for that? >> >> Thanks, >> Otis >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
