One thing is that Rsyslog has very smart batching.
Instead of delaying processing messages so that they can be combined with others
that arrive later, rsyslog processes all messages as fast as it can, and only
batches messages when it falls behind.
So instead of
message arrives, logstash waits until either X messages arrive or Y time passes
and then does the insert, repeat
we have
thread 1 message arrives
thread 2 rsyslog inserts 1 message
thread 1 while rsyslog is doing the 'inefficient' insert of a single message,
multiple messages arrive
thread 2 rsyslog inserts all pending messages (up to the max batch size)
repeat.
This means that the number of inserts/sec will ramp up faster with rsyslog, with
each insert being a single message, then when it can't process them any faster
as individual messages, rsyslog will start batching them and the batch size will
grow.
But as long as the ElasticSerch instance can keep up with the rate that rsyslog
is feeding it the messages, the batch size will stay small.
This minimizes the latency of getting messages into ES and avoids many nasty
classes of failure modes that relate to timeout handling.
So if you want to see the batch size increase, load down ES so that it can't
keep up or throw more data at rsyslog.
David Lang
On Wed, 17 Jun 2015, chenlin rao wrote:
So how can I define the output queue configuration?
I found the omelasticsearch action process 60000/min, and the
queue.discarded.nf was 600000.
I run `tcpdump -i eth1 -s0 -A 'tcp dst port 9200' | grep Content-Length`
and saw the length is 1.6k. As my msgline size is 0.1k, the bulk size is
only 10. Too small.
Sometimes when I restart rsyslogd, the Content-Length grows to 8MB. Why~~
2015-05-06 1:39 GMT+08:00 David Lang <[email protected]>:
On Tue, 5 May 2015, chenlin rao wrote:
I'm using rsyslog-elasticsearch to writing nginx accesslog into
Elasticsearch cluster. I found the document told that the plugin would use
queue.dequeuesize as the bulk size.But my tcpdump show that every POST
only
has 8-9 events in the bulk body while my input flow is nearly 10k per
second.
How can I force a larger bulk size?
Rsyslog adapts the size to the number of messages waiting to be delivered,
so if it's keeping up at that size, it won't increase it.
are you running impstats? if so, please look at the queue size. If it's
staying low, then you just have a nice, fast ES instance that is able to do
1k inserts/sec (which is not unreasonable), so each insert would be <10
messages.
Trying to force a larger bulk size would mean not inserting messages as
fast as we can, and instead pausing and waiting for enough messages to
accumulate to fill the bulk size. We never delay messages intentionally,
each pass through the loop we grab all pending messages, up to the max
dequeue size, and deliver them. If more messages arrive than we deliver,
the next pass through the queue is larger, so we grab more messages (this
quickly stabilizes to inserting messages as fast as they are arriving)
there is a dequeue delay that forces rsyslog to sit and do nothing between
one batch of messages and the next. It's use is discouraged, but delaying
like this would allow more messages to accumulate.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
