Hi, yes I've gotten fairly deep into ES. We would like to be able to search over and perhaps even calculate based on fields with the same name and different types. What's more we don't know when developers will add services and new json logs. Without the rsyslog change - as far as I can see the only way to do this to have per app indexes (as per the tip at the very bottom of this page https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html).
This is all cool stuff! Please keep the ideas coming :-) One thing that makes me nervous is the overhead of doing the transformation via normalize on rsyslog. Some the applications generate a lot of logs per second. On 7 April 2016 at 18:38, Dave Caplinger <[email protected]> wrote: > On Apr 7, 2016, at 12:04 PM, Matt Ford <[email protected]> wrote: >> >> Thanks for the help thus far I'm able to parse arbitrary json logs and >> get them into kafka very nicely. >> However, due to the many different systems in use there is key >> namespace clashes in the final destination (Elasticsearch) >> >> I have some JSON logs like this from one app >> >> { "login": 234343,... } >> >> and some JSON logs like this from another app >> >> { "login": "matt",... } >> >> Is it possible to parse and change the key space to look like this >> >> { "app1_login": 234343, "app1_XX:": ....} >> { "app2_login": "matt", "app2_XX:":...} > > I'm not sure how deep into ElasticSearch you've gotten, but it sounds like > maybe you're seeing the result of automatic type mapping where the first > field called "login" happens to be interpreted as a number, and later on a > string value shows up and fails to be indexed because ElasticSearch now > expects only numeric values. You can solve this at ElasticSearch directly by > having an explicit mapping (for example, "login" is a string), which in this > case would force the numeric login value to be inserted as a string instead. > > (See: > https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html ) > > So you don't have to change the upstream JSON sources if you don't want to > (though you certainly could do that instead). > > - Dave Caplinger > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
