In many cases, when adding a record to Elasticsearch, an http status not
200 or 201 does not necessarily indicate that the record cannot be
added. One case is bulk index rejection - in this case, the http status
for the record in the response is 429, and it may be that a short pause
is required before resubmitting the record.
omelasticsearch has support for an errorfile, but this requires the
operator to examine the file and resubmit it manually. The fluentd
elasticsearch plugin recently got support for better error handling:
https://github.com/richm/docs/blob/master/fluent-plugin-elasticsearch-retry.md
I would like to do the same with omelasticsearch. It would work
something like this:
- a record that fails with a "soft" error would be sent to a "retry queue"
- a record that fails with a "hard" error would be sent to an "error queue"
In fluentd this is best accomplished with judicious use of tagging and
labeling.
I think for rsyslog, this would best be handled by rulesets - a retry_es
ruleset, and an error ruleset.
We have the original request in JSON string form, and the response JSON
contains 1 response object for each request object, in the exact same order.
pseudo code:
if response is an error
for each item in response
get the corresponding request string
convert request json string to json object
MsgNew - set Msg fields
if is soft error
MsgSetRuleset retry_es ruleset
else
MsgSetRuleset error ruleset
ratelimitAddMsg
Note that status 200 and 201 are success, and status 409 when using the
"create" operation (I'm also working on adding support for this) is a
duplicate and is considered successful.
Without using a ruleset, I assume a message would be submitted at the
"top" of the processing pipeline, and would require a lot of work to
make that pipeline idempotent in most cases.
The config would look something like this:
ruleset(name="error_es") {
action(type="omfile" ... write hard failures to error file ...)
}
ruleset(name="try_es") {
action(type="omelasticsearch" .... retryRuleset="try_es"
errorRuleset="error_es" ...)
}
... normal pipeline ...
call try_es # for both the normal case and the retry case
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
