On Apr 7, 2016, at 12:04 PM, Matt Ford <[email protected]> wrote: > > Thanks for the help thus far I'm able to parse arbitrary json logs and > get them into kafka very nicely. > However, due to the many different systems in use there is key > namespace clashes in the final destination (Elasticsearch) > > I have some JSON logs like this from one app > > { "login": 234343,... } > > and some JSON logs like this from another app > > { "login": "matt",... } > > Is it possible to parse and change the key space to look like this > > { "app1_login": 234343, "app1_XX:": ....} > { "app2_login": "matt", "app2_XX:":...}
I'm not sure how deep into ElasticSearch you've gotten, but it sounds like maybe you're seeing the result of automatic type mapping where the first field called "login" happens to be interpreted as a number, and later on a string value shows up and fails to be indexed because ElasticSearch now expects only numeric values. You can solve this at ElasticSearch directly by having an explicit mapping (for example, "login" is a string), which in this case would force the numeric login value to be inserted as a string instead. (See: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html ) So you don't have to change the upstream JSON sources if you don't want to (though you certainly could do that instead). - Dave Caplinger _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
