On Tue, 5 May 2015, chenlin rao wrote:
I'm using rsyslog-elasticsearch to writing nginx accesslog into
Elasticsearch cluster. I found the document told that the plugin would use
queue.dequeuesize as the bulk size.But my tcpdump show that every POST only
has 8-9 events in the bulk body while my input flow is nearly 10k per
second.
How can I force a larger bulk size?
Rsyslog adapts the size to the number of messages waiting to be delivered, so if
it's keeping up at that size, it won't increase it.
are you running impstats? if so, please look at the queue size. If it's staying
low, then you just have a nice, fast ES instance that is able to do >1k
inserts/sec (which is not unreasonable), so each insert would be <10 messages.
Trying to force a larger bulk size would mean not inserting messages as fast as
we can, and instead pausing and waiting for enough messages to accumulate to
fill the bulk size. We never delay messages intentionally, each pass through the
loop we grab all pending messages, up to the max dequeue size, and deliver them.
If more messages arrive than we deliver, the next pass through the queue is
larger, so we grab more messages (this quickly stabilizes to inserting messages
as fast as they are arriving)
there is a dequeue delay that forces rsyslog to sit and do nothing between one
batch of messages and the next. It's use is discouraged, but delaying like this
would allow more messages to accumulate.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
