nothing has been done yet.
David Lang
On Sun, 20 Mar 2016, chenlin rao wrote:
hello, everyone. Is there any process about how to force a larger batch? I
remember there was an email that rainer said he would implement some
options to queue.c in the future.
2015-08-31 16:06 GMT+08:00 Radu Gheorghe <[email protected]>:
Hi David,
This sounds interesting, I especially like the idea of spawning threads
based on load.
The problem, really, is that an ES cluster would have an "optimal" batch
size. If we send batches too small or too large, ingestion will slow down.
Fortunately, that optimum isn't really that precise. For example, on one
test I did, sending 500, 1000 or 10000 messages at once didn't
significantly impact ingestion. There was a problem when going under 100,
though.
To me, any solution that makes rsyslog send batches that are within a
certain range (and within a certain timeframe - since I wouldn't want to
see today's log indexed tomorrow) would work.
Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Sat, Aug 29, 2015 at 7:21 PM, David Lang <[email protected]> wrote:
On Mon, 24 Aug 2015, Rainer Gerhards wrote:
I am mostly with Radu on this topic. I think there are some use cases
where it really would be advantageous to submit a larger batch, even
if this means waiting. True, these use cases were very seldom in the
early days of rsyslog and may still be, but I think it's something one
might validly want.
The thought hit me that we are loking at this wrong.
The problem is overloading the receiver with too many small batches.
rather than trying to define batch size, isn't what we really want to
have
is a limit on how many batches we send in a given timeframe? possibly
with
a 'escape clause' that says tht if we are sending maxbatch size messages
for the entire timeframe we do something (spawn a new sending thread,
temporarily allow higher sending rates, or just let the backlog
accumulate
are all valid choices under different conditions)
thoughts?
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
