Hi Ajay, The request ends up being wrong. I'm bolding the stuff that shouldn't be there:
{\"index\":{\"_index\":
\"ledger-2013-07-03\",\"_type\
":\"events\"}*}*\n*1372805031 }*{
\"transfer_from_guid\": \"1111\", \"item_type\": 30404, \"quantity\": 840,
\"auth_code\": \"0000\", \"metric_basename\": \"ledger\", \"es-service\":
\"ledger\", \"logged_at\": 1372805031 }\n"
A bulk request <http://www.elasticsearch.org/guide/reference/api/bulk/>should
have the following structure:
{$JSON-METADATA1}\n
{$JSON-DOCUMENT1}\n
{$JSON-METADATA2}\n
{$JSON-DOCUMENT2}\n
And so on.
I've tested with (roughly) what you sent here and it works for me. I'm
attaching my conf and in there you can see the command I've used. Do you
see any significant differences?
BTW, I've tried this with 7.4.0 from the RPM repository.
If it still doesn't work for you, can you paste me a full conf and rsyslog
version? I'd appreciate if you can take stuff such as queue settings out
from the conf you send, to have the bare minimum for reproducing the
problem and minimize the chance of replies like "it works for me, but there
might be a minor change that is actually the key" :)
Best regards,
Radu
2013/7/3 Ajay Sharma <[email protected]>
> When I turn bulkmode=on for my omelasticsearch setup, I get
> "MapperParsingException" errors. Not sure why this is happening other than
> the fact that the JSON sent to elasticsearch doesn't look right. Maybe
> someone can help me spot something that I'm not seeing. First here's my
> elasticsearch config:
>
> $template elasticsearchSchema,"%$!all-json%"
>
> action(type="mmjsonparse")
> if $parsesuccess == "OK" then {
> if $!es-service != "" then {
> $template
> elasticsearchIndex,"%$!es-service%-%timereported:1:10:date-rfc3339%"
> action(type="omelasticsearch"
> template="elasticsearchSchema"
> searchIndex="elasticsearchIndex"
> dynSearchIndex="on"
> server="es-host"
> serverport="9200"
> errorFile="/var/log/rsyslog-elasticsearch.log"
> bulkmode="on"
> queue.dequeuebatchsize="200"
> queue.type="linkedlist"
> queue.filename="dbq"
> queue.highwatermark="500000"
> queue.lowwatermark="400000"
> queue.discardmark="5000000"
> queue.timeoutenqueue="0"
> queue.maxdiskspace="5g"
> queue.size="2000000"
> queue.saveonshutdown="on"
> action.resumeretrycount="-1"
> )
> }
> }
>
> I'm creating the index based on a JSON variable so that the developers can
> control which index logs what. From another machine, if I run this command
> a few times, it will trigger a error:
>
> # logger '@cee:
>
> {"transfer_from_guid":"1111","item_type":30404,"quantity":840,"auth_code":"0000","metric_basename":"ledger","es-service":"ledger","logged_at":1372805031}'
>
> The error:
>
> {
> "request": {
> "url": "http://search-v01-ew1.r5internal.com:9200/_bulk
> ?",
> "postdata": "{\"index\":{\"_index\":
> \"ledger-2013-07-03\",\"_type\":\"events\"}}\n1372805031 }{
> \"transfer_from_guid\": \"1111\", \"item_type\": 30404, \"quantity\": 840,
> \"auth_code\": \"0000\", \"metric_basename\": \"ledger\", \"es-service\":
> \"ledger\", \"logged_at\": 1372805031 }\n"
> },
> "reply": {
> "took": 235,
> "items": [{
> "create": {
> "_index":
> "ledger-2013-07-03",
> "_type": "events",
> "_id": "z2MbQE5IQV6EHyqwiu8ejA",
> "error":
> "MapperParsingException[Malformed content, must start with an object]"
> }
> }]
> }
> }
>
> I've seen other elasticsearch errors like
>
> MapperParsingException[failed to parse]; nested:
> JsonParseException[Unexpected end-of-input in field name\n at [Source:
> [B@360f5d87; line: 1, column: 261]];
>
> and
>
> MapperParsingException[failed to parse]; nested:
> ElasticSearchParseException[Failed to derive xcontent from (offset=59,
> length=196): [123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 34, 95, 105,
> 110, 100, 101, 120, 34, 58, 32, 34, 108, 101, 100, 103, 101, 114, 45, 50,
> 48, 49, 51, 45, 48, 55, 45, 48, 51, 34, 44, 34, 95, 116, 121, 112, 101, 34,
> 58, 34, 101, 118, 101, 110, 116, 115, 34, 125, 125, 10, 34, 44, 32, 34,
> 108, 111, 103, 103, 101, 100, 95, 97, 116, 34, 58, 32, 49, 51, 55, 50, 56,
> 48, 53, 48, 51, 49, 32, 125, 123, 32, 34, 116, 114, 97, 110, 115, 102, 101,
> 114, 95, 102, 114, 111, 109, 95, 103, 117, 105, 100, 34, 58, 32, 34, 49,
> 49, 49, 49, 34, 44, 32, 34, 105, 116, 101, 109, 95, 116, 121, 112, 101, 34,
> 58, 32, 51, 48, 52, 48, 52, 44, 32, 34, 113, 117, 97, 110, 116, 105, 116,
> 121, 34, 58, 32, 56, 52, 48, 44, 32, 34, 97, 117, 116, 104, 95, 99, 111,
> 100, 101, 34, 58, 32, 34, 48, 48, 48, 48, 34, 44, 32, 34, 109, 101, 116,
> 114, 105, 99, 95, 98, 97, 115, 101, 110, 97, 109, 101, 34, 58, 32, 34, 108,
> 101, 100, 103, 101, 114, 34, 44, 32, 34, 101, 115, 45, 115, 101, 114, 118,
> 105, 99, 101, 34, 58, 32, 34, 108, 101, 100, 103, 101, 114, 34, 44, 32, 34,
> 108, 111, 103, 103, 101, 100, 95, 97, 116, 34, 58, 32, 49, 51, 55, 50, 56,
> 48, 53, 48, 51, 49, 32, 125, 10]];
>
> I did file a bug and included the debug output as an attachment:
>
> http://bugzilla.adiscon.com/show_bug.cgi?id=462
>
> If someone can provide any insight, I would really appreciate it!!
>
> Thanks,
> Ajay
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
rsyslog.conf
Description: Binary data
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
