Just to set context in little depth.
Currently rsyslog dumps entire data in form of json containing
{
"request": {
"url": "http://localhost:9200/_bulk";,
"postdata": "[new line seperated entire post data]"
},
"reply": {
"took": ,
"errors": ,
"items": [JSON array with all the response]
}
}
In bulk insert mode where we can easily have 1000+ request. If there is a
failure it becomes very difficult to figure out what went wrong. Also if
only one request failed, current system still inserts entire info; which
just not readable and makes disk file very large.
Wanted to introduce two format flags in *action* to format data in error
file (Both of the flags can be used independently as well as together).
1. *erroronly* => This flag checks the response stream and logs only
failed requests and response.
2. *interleaved* => This flag interleaves the request to its response
and dumps it as pair.
When both flags are used we get only failed request in an interleaved
format.
*Format for erroronly*
{
"request": {
"url": "http://localhost:9200/_bulk";,
"postdata": "[new line seperated only failed requests]"
},
"reply": {
"took": ,
"errors": ,
"items": [JSON array with only failed response]
}
}
*Format for interleaved*
{
"url": "http://localhost:9200/_bulk";,
"response": [{
"request": "Failed request",
"reply": "Corresponding response"
}...]
}
*sample:* action(server="localhost"
XXX
type="omelasticsearch"
XXX
errorfile="bad_message.log"
erroronly="on"
interleaved="on")
Let me know if I have not cleared the intentions for the change introduced.
Regards
Harsh
On Wed, Feb 18, 2015 at 6:13 PM, Harshvardhan Shrivastava <
[email protected]> wrote:
> Hi,
>
> Just submitted a pull request to have different formats in error file for
> omelasticsearch bulk insert mode. (
> https://github.com/rsyslog/rsyslog/pull/246 )
>
> Introducing them to make error file more readable. I wanted get forum's
> view on it.
>
> Regards
> Harsh
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
