ES uses worker-pool for indexing(there is a worker-pool for
bulk-indexing too). Prioritizing approach may not be easy, and
possibly a little dangerous too, but sizing that thread-pool is
definitely easy. Just size it to your need and it'll shape the
batch-size optimally when under pressure (like David explained).
On Wed, Jun 17, 2015 at 6:14 PM, chenlin rao <[email protected]>
wrote:
well, there is something I can't understand: If rsyslog use 10msg per
bulk
because Elasticsearch keep up the sending speed, why the output queue
has a
size reached maxsize and discarded.nf/enqueued = 90%.
here is my configuration:
```
action (
type="omelasticsearch"
template="videotmpl"
server="10.13.244.214"
dynSearchIndex="on"
searchIndex="videoIndexName"
searchType="videoaccess"
bulkmode="on"
name="action_videoaccess-es1003"
queue.size="1000000"
queue.dequeuebatchsize="40000"
queue.discardmark="950000"
queue.highwatermark="600000"
queue.lowwatermark="400000"
queue.discardseverity="3"
queue.dequeueslowdown="10000"
queue.type="linkedlist"
queue.maxdiskspace="15G"
queue.maxfilesize="500M"
queue.filename="action_videoaccess-es1003"
queue.checkpointinterval="10000"
queue.saveonshutdown="on"
)
```
and pstats.log:
```
2015-06-17T12:17:48.708364+08:00 localhost rsyslogd-pstats:
{"name":"action_videoaccess-es1003
queue[DA]","origin":"core.queue","size":27838434,"enqueued":9,"full":735,"discarded.full":9,"
discarded.nf":0,"maxqsize":28153530}
2015-06-17T12:17:48.708370+08:00 localhost rsyslogd-pstats:
{"name":"action_videoaccess-es1003
queue","origin":"core.queue","size":950000,"enqueued":522298,"full":0,"discarded.full":0,"
discarded.nf":442298,"maxqsize":950000}
```
btw: I had try slowdown setting from 10 to 10000, no change to 10 msg per
bulk.
2015-06-17 19:54 GMT+08:00 Radu Gheorghe <[email protected]>:
That might work, thanks for the feedback and the interesting article!
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Jun 17, 2015 at 12:58 PM, David Lang <[email protected]> wrote:
Probably a risk, something to keep an eye on (or watch the pstats from
rsyslog and tweak the priority if the queue too large)
I also believe that the vast majority of searches that are typically
done
are done wrong (see my dashboards/reports article at
https://www.usenix.org/publications/login/feb14/logging-reports-dashboards
)
David Lang
On Wed, 17 Jun 2015, Radu Gheorghe wrote:
This sounds interesting, David. I guess it's possible to renice just
some
threads from an app and make it "nicer", right? Googling a bit it
seems
it
is possible.
The only problem I see with this approach is that searches (and other
kinds
of requests from other threadpools
<
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html
)
would automatically have higher priority so, with heavy searches,
indexing
might fall behind more than usual. Am I getting it right?
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Jun 17, 2015 at 11:53 AM, David Lang <[email protected]> wrote:
Thinking about it, probably the best thing to do is to renice the ES
threads that accept the messages from rsyslog. That way if nothing
else
needs the capacity, everything works at the fastest insert speed
(even
if
less optimized than if there were larger batches) But if anything
else
on
the system need the resources, the indexing threads work slower,
which
will
result in larger batches.
all self tuning.
David Lang
On Wed, 17 Jun 2015, Radu Gheorghe wrote:
Date: Wed, 17 Jun 2015 10:20:46 +0300
From: Radu Gheorghe <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] how to force a larger omelasticsearch bulk
size?
Maybe this went overlooked, but David suggested earlier that you
can
slowdown the queue to let more messages arrive before sending a
bulk.
queue.dequeueslowdown
<
http://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html
is the option and it's in microseconds.
I think you have a valid point in that if batches are too small
then
Elasticsearch will do more work than necessary (as indexing in very
small
batches is more expensive). Plus, since the refresh rate (i.e. how
long
it
may take for an indexed doc to be visible to searches, because
Searchers
reopen their view in the index at a certain interval) is typically
a
few
seconds
<
http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/
,
waiting a bit before submitting a batch will have no impact on the
user
experience.
On the other hand, in my experience you'll be sending small
batches if
the
indexing rate is low - which means the load on ES is low anyway. So
I'm
not
sure if optimizing this will actually give significant results. You
could
introduce that slowdown, but then rsyslog may have trouble keeping
up
when
the load is high. You can compensate by raising the limit of
maximum
worker
threads for the queue (queue.workerthreads) and play with
queue.workerthreadminimummessages and
queue.timeoutworkerthreadshutdown
to
make rsyslog spawn new threads when there are at least N messages
in
the
queue (that's what min messages does) and kill them when the queue
is
smaller than that for a while (that's the timeout option). If the
load
is
low, you'd have just one thread that works with that slowdown.
I hope this helps.
Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Jun 17, 2015 at 6:23 AM, chenlin rao <
[email protected]>
wrote:
So how can I define the output queue configuration?
I found the omelasticsearch action process 60000/min, and the
queue.discarded.nf was 600000.
I run `tcpdump -i eth1 -s0 -A 'tcp dst port 9200' | grep
Content-Length`
and saw the length is 1.6k. As my msgline size is 0.1k, the bulk
size
is
only 10. Too small.
Sometimes when I restart rsyslogd, the Content-Length grows to
8MB.
Why~~
2015-05-06 1:39 GMT+08:00 David Lang <[email protected]>:
On Tue, 5 May 2015, chenlin rao wrote:
I'm using rsyslog-elasticsearch to writing nginx accesslog into
Elasticsearch cluster. I found the document told that the plugin
would
use
queue.dequeuesize as the bulk size.But my tcpdump show that every
POST
only
has 8-9 events in the bulk body while my input flow is nearly
10k
per
second.
How can I force a larger bulk size?
Rsyslog adapts the size to the number of messages waiting to be
delivered,
so if it's keeping up at that size, it won't increase it.
are you running impstats? if so, please look at the queue size.
If
it's
staying low, then you just have a nice, fast ES instance that is
able
to
do
1k inserts/sec (which is not unreasonable), so each insert would
be
<10
messages.
Trying to force a larger bulk size would mean not inserting
messages
as
fast as we can, and instead pausing and waiting for enough
messages
to
accumulate to fill the bulk size. We never delay messages
intentionally,
each pass through the loop we grab all pending messages, up to
the
max
dequeue size, and deliver them. If more messages arrive than we
deliver,
the next pass through the queue is larger, so we grab more
messages
(this
quickly stabilizes to inserting messages as fast as they are
arriving)
there is a dequeue delay that forces rsyslog to sit and do
nothing
between
one batch of messages and the next. It's use is discouraged, but
delaying
like this would allow more messages to accumulate.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
Regards,
Janmejay
http://codehunk.wordpress.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
