So how can I define the output queue configuration? I found the omelasticsearch action process 60000/min, and the queue.discarded.nf was 600000. I run `tcpdump -i eth1 -s0 -A 'tcp dst port 9200' | grep Content-Length` and saw the length is 1.6k. As my msgline size is 0.1k, the bulk size is only 10. Too small.
Sometimes when I restart rsyslogd, the Content-Length grows to 8MB. Why~~ 2015-05-06 1:39 GMT+08:00 David Lang <[email protected]>: > On Tue, 5 May 2015, chenlin rao wrote: > > I'm using rsyslog-elasticsearch to writing nginx accesslog into >> Elasticsearch cluster. I found the document told that the plugin would use >> queue.dequeuesize as the bulk size.But my tcpdump show that every POST >> only >> has 8-9 events in the bulk body while my input flow is nearly 10k per >> second. >> >> How can I force a larger bulk size? >> > > Rsyslog adapts the size to the number of messages waiting to be delivered, > so if it's keeping up at that size, it won't increase it. > > are you running impstats? if so, please look at the queue size. If it's > staying low, then you just have a nice, fast ES instance that is able to do > >1k inserts/sec (which is not unreasonable), so each insert would be <10 > messages. > > Trying to force a larger bulk size would mean not inserting messages as > fast as we can, and instead pausing and waiting for enough messages to > accumulate to fill the bulk size. We never delay messages intentionally, > each pass through the loop we grab all pending messages, up to the max > dequeue size, and deliver them. If more messages arrive than we deliver, > the next pass through the queue is larger, so we grab more messages (this > quickly stabilizes to inserting messages as fast as they are arriving) > > there is a dequeue delay that forces rsyslog to sit and do nothing between > one batch of messages and the next. It's use is discouraged, but delaying > like this would allow more messages to accumulate. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
