On 08/02/2018 01:18 AM, eSX via rsyslog wrote:
Hello rsyslog
omes is the retry information when omelasticsearch is in failure and retry.
The contents of omes are response message from Elasticsearch mainly.
But the response didn't give a full information for retring.
What information is missing from the response?
https://www.elastic.co/guide/en/elasticsearch/guide/current/bulk.html
$.omes should have everything in the bulk response.
If you mean "missing local variables" then see below.
And when retry is happened, the message(smsg_t) is reset.
It means any local variables is invalided, so that people can't get extra
information from it.
Correct. I'm not sure how to handle this. Please file an issue at
https://github.com/rsyslog/rsyslog/issues
omelasticsearch doesn't keep the smsg_t - it just constructs the bulk
index request string JSON, then sends the request once the length has
reached the threshold.
When I wrote the retry logic, I was expecting everything that was needed
to be in the bulk retry would be in $!, or at least in whatever was
returned by "template", since that is the only thing kept and available
when omelasticsearch processes the bulk index response for retry. That
allow using the same action for both the initial request and retries,
but that would involve adding extra fields to the outgoing message, and
would cause users to rewrite their templates.
If retry is active (if retryfailures="on"), omelasticsearch could
somehow "cache" the evaluation of any templates, and make them available
in the retry record, but that would definitely mean having different
actions for the "initial" request and the retries.
I suppose the only way to ensure that you could use the exact same
action for retry as for the initial request would be to cache all of the
smsg_t, or at least all of the variables - built in and local.
Is it acceptable to require a different action for retries, if retries
are being used? Or should the goal be that there should be retry
support, with _no_ changes required to any existing templates, and _one_
action for both the initial request and retries?
For example, If I set pipelineName and set dynPipelineName "on" in
omelasticsearch.
Which the pipelineName was generated by template, like:
template (
name = "es_idx_tpl"
type = "string"
string = "%syslogseverity-text%"
)
When a "status 429" failure is happened, and retry.
Because message variables is reset, there isn't $syslogseverity-text
anymore.
I can't regenerate a valid pipelineName, so retry would be failed or
mistaken.
The searchIndex option with dynSearchIndex="on" in omelasticsearch has same
problem.
Fortunately, I can retrieve it again from $.omes!_index.
A workaround for above is:
I have to make an retryFailure ruleset for each pipelineName.
Like:
ruleset (
name = "es_retry_for_pipeline1"
) {
if strlen($.omes!status) > 0 then {
...
}
action (
name = "es_retry_with_pipeline1"
type = "omelasticsearch"
....
pipelineName = "pipeline1"
)
}
action (
name = "es_for_pipeline"
type = "omelasticsearch"
...
pipelineName = "pipeline1"
retryFailures="on"
retryRuleset="es_retry_for_pipeline1"
)
ruleset (
name = "es_retry_for_pipeline2"
) {
if strlen($.omes!status) > 0 then {
...
}
action (
name = "es_retry_with_pipeline2"
type = "omelasticsearch"
....
pipelineName = "pipeline2"
)
}
action (
name = "es_for_pipeline2"
type = "omelasticsearch"
...
pipelineName = "pipeline2"
retryFailures="on"
retryRuleset="es_retry_for_pipeline2"
)
......
ruleset (
name = "es_retry_for_pipelineN"
) {
if strlen($.omes!status) > 0 then {
...
}
action (
name = "es_retry_with_pipelineN"
type = "omelasticsearch"
....
pipelineName = "pipelineN"
)
}
action (
name = "es_for_pipelineN"
type = "omelasticsearch"
...
pipelineName = "pipelineN"
retryFailures="on"
retryRuleset="es_retry_for_pipelineN"
)
It's so...ugly, isn't it?
So, like pipelineName or any others, it should in omes also?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
