hello, everyone. Is there any process about how to force a larger batch? I remember there was an email that rainer said he would implement some options to queue.c in the future.
2015-08-31 16:06 GMT+08:00 Radu Gheorghe <[email protected]>: > Hi David, > > This sounds interesting, I especially like the idea of spawning threads > based on load. > > The problem, really, is that an ES cluster would have an "optimal" batch > size. If we send batches too small or too large, ingestion will slow down. > Fortunately, that optimum isn't really that precise. For example, on one > test I did, sending 500, 1000 or 10000 messages at once didn't > significantly impact ingestion. There was a problem when going under 100, > though. > > To me, any solution that makes rsyslog send batches that are within a > certain range (and within a certain timeframe - since I wouldn't want to > see today's log indexed tomorrow) would work. > > Best regards, > Radu > > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > On Sat, Aug 29, 2015 at 7:21 PM, David Lang <[email protected]> wrote: > > > On Mon, 24 Aug 2015, Rainer Gerhards wrote: > > > > I am mostly with Radu on this topic. I think there are some use cases > >> where it really would be advantageous to submit a larger batch, even > >> if this means waiting. True, these use cases were very seldom in the > >> early days of rsyslog and may still be, but I think it's something one > >> might validly want. > >> > > > > The thought hit me that we are loking at this wrong. > > > > The problem is overloading the receiver with too many small batches. > > > > rather than trying to define batch size, isn't what we really want to > have > > is a limit on how many batches we send in a given timeframe? possibly > with > > a 'escape clause' that says tht if we are sending maxbatch size messages > > for the entire timeframe we do something (spawn a new sending thread, > > temporarily allow higher sending rates, or just let the backlog > accumulate > > are all valid choices under different conditions) > > > > thoughts? > > > > David Lang > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
