On 05/16/2018 05:58 PM, David Lang wrote:
there's no need to add this extra complexity (multiple rulesets and
queues)
What should be happening (on any output module) is:
submit a batch.
If rejected with a soft error, retry/suspend the output
retry of the entire batch? see below
if batch-size=1 and a hard error, send to errorfile
if rejected with a hard error resubmit half of the batch
But what if 90% of the batch was successfully added? Then you are
needlessly resubmitting many of the records in the batch.
If using the "index" (default) bulk type, this causes duplicate records
to be added.
If using the "create" type (and you have assigned a unique _id), you
will get back many 409 Duplicate errors.
This causes problems - we know because this is how the fluentd plugin
used to work, which is why we had to change it.
https://www.elastic.co/guide/en/elasticsearch/guide/2.x/_monitoring_individual_nodes.html#_threadpool_section
"Bulk Rejections"
"It is much better to handle queuing in your application by gracefully
handling the back pressure from a full queue. When you receive bulk
rejections, you should take these steps:
Pause the import thread for 3–5 seconds.
Extract the rejected actions from the bulk response, since it is
probable that many of the actions were successful. The bulk response
will tell you which succeeded and which were rejected.
Send a new bulk request with just the rejected actions.
Repeat from step 1 if rejections are encountered again.
Using this procedure, your code naturally adapts to the load of your
cluster and naturally backs off.
"
repeat
all that should be needed is to add tests into omelasticsearch to
detect the soft errors and turn them into retries (or suspend the
output as appropriate)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
