Hi David, This sounds interesting, I especially like the idea of spawning threads based on load.
The problem, really, is that an ES cluster would have an "optimal" batch size. If we send batches too small or too large, ingestion will slow down. Fortunately, that optimum isn't really that precise. For example, on one test I did, sending 500, 1000 or 10000 messages at once didn't significantly impact ingestion. There was a problem when going under 100, though. To me, any solution that makes rsyslog send batches that are within a certain range (and within a certain timeframe - since I wouldn't want to see today's log indexed tomorrow) would work. Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Sat, Aug 29, 2015 at 7:21 PM, David Lang <[email protected]> wrote: > On Mon, 24 Aug 2015, Rainer Gerhards wrote: > > I am mostly with Radu on this topic. I think there are some use cases >> where it really would be advantageous to submit a larger batch, even >> if this means waiting. True, these use cases were very seldom in the >> early days of rsyslog and may still be, but I think it's something one >> might validly want. >> > > The thought hit me that we are loking at this wrong. > > The problem is overloading the receiver with too many small batches. > > rather than trying to define batch size, isn't what we really want to have > is a limit on how many batches we send in a given timeframe? possibly with > a 'escape clause' that says tht if we are sending maxbatch size messages > for the entire timeframe we do something (spawn a new sending thread, > temporarily allow higher sending rates, or just let the backlog accumulate > are all valid choices under different conditions) > > thoughts? > > David Lang > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
