When I turn bulkmode=on for my omelasticsearch setup, I get
"MapperParsingException" errors. Not sure why this is happening other than
the fact that the JSON sent to elasticsearch doesn't look right. Maybe
someone can help me spot something that I'm not seeing. First here's my
elasticsearch config:
$template elasticsearchSchema,"%$!all-json%"
action(type="mmjsonparse")
if $parsesuccess == "OK" then {
if $!es-service != "" then {
$template
elasticsearchIndex,"%$!es-service%-%timereported:1:10:date-rfc3339%"
action(type="omelasticsearch"
template="elasticsearchSchema"
searchIndex="elasticsearchIndex"
dynSearchIndex="on"
server="es-host"
serverport="9200"
errorFile="/var/log/rsyslog-elasticsearch.log"
bulkmode="on"
queue.dequeuebatchsize="200"
queue.type="linkedlist"
queue.filename="dbq"
queue.highwatermark="500000"
queue.lowwatermark="400000"
queue.discardmark="5000000"
queue.timeoutenqueue="0"
queue.maxdiskspace="5g"
queue.size="2000000"
queue.saveonshutdown="on"
action.resumeretrycount="-1"
)
}
}
I'm creating the index based on a JSON variable so that the developers can
control which index logs what. From another machine, if I run this command
a few times, it will trigger a error:
# logger '@cee:
{"transfer_from_guid":"1111","item_type":30404,"quantity":840,"auth_code":"0000","metric_basename":"ledger","es-service":"ledger","logged_at":1372805031}'
The error:
{
"request": {
"url": "http://search-v01-ew1.r5internal.com:9200/_bulk?";,
"postdata": "{\"index\":{\"_index\":
\"ledger-2013-07-03\",\"_type\":\"events\"}}\n1372805031 }{
\"transfer_from_guid\": \"1111\", \"item_type\": 30404, \"quantity\": 840,
\"auth_code\": \"0000\", \"metric_basename\": \"ledger\", \"es-service\":
\"ledger\", \"logged_at\": 1372805031 }\n"
},
"reply": {
"took": 235,
"items": [{
"create": {
"_index": "ledger-2013-07-03",
"_type": "events",
"_id": "z2MbQE5IQV6EHyqwiu8ejA",
"error":
"MapperParsingException[Malformed content, must start with an object]"
}
}]
}
}
I've seen other elasticsearch errors like
MapperParsingException[failed to parse]; nested:
JsonParseException[Unexpected end-of-input in field name\n at [Source:
[B@360f5d87; line: 1, column: 261]];
and
MapperParsingException[failed to parse]; nested:
ElasticSearchParseException[Failed to derive xcontent from (offset=59,
length=196): [123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 34, 95, 105,
110, 100, 101, 120, 34, 58, 32, 34, 108, 101, 100, 103, 101, 114, 45, 50,
48, 49, 51, 45, 48, 55, 45, 48, 51, 34, 44, 34, 95, 116, 121, 112, 101, 34,
58, 34, 101, 118, 101, 110, 116, 115, 34, 125, 125, 10, 34, 44, 32, 34,
108, 111, 103, 103, 101, 100, 95, 97, 116, 34, 58, 32, 49, 51, 55, 50, 56,
48, 53, 48, 51, 49, 32, 125, 123, 32, 34, 116, 114, 97, 110, 115, 102, 101,
114, 95, 102, 114, 111, 109, 95, 103, 117, 105, 100, 34, 58, 32, 34, 49,
49, 49, 49, 34, 44, 32, 34, 105, 116, 101, 109, 95, 116, 121, 112, 101, 34,
58, 32, 51, 48, 52, 48, 52, 44, 32, 34, 113, 117, 97, 110, 116, 105, 116,
121, 34, 58, 32, 56, 52, 48, 44, 32, 34, 97, 117, 116, 104, 95, 99, 111,
100, 101, 34, 58, 32, 34, 48, 48, 48, 48, 34, 44, 32, 34, 109, 101, 116,
114, 105, 99, 95, 98, 97, 115, 101, 110, 97, 109, 101, 34, 58, 32, 34, 108,
101, 100, 103, 101, 114, 34, 44, 32, 34, 101, 115, 45, 115, 101, 114, 118,
105, 99, 101, 34, 58, 32, 34, 108, 101, 100, 103, 101, 114, 34, 44, 32, 34,
108, 111, 103, 103, 101, 100, 95, 97, 116, 34, 58, 32, 49, 51, 55, 50, 56,
48, 53, 48, 51, 49, 32, 125, 10]];
I did file a bug and included the debug output as an attachment:
http://bugzilla.adiscon.com/show_bug.cgi?id=462
If someone can provide any insight, I would really appreciate it!!
Thanks,
Ajay
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
