I have a convention of making loggers slap a special tag "obj@<ns>"
where <ns> = a key based namespace.
So obj@w3 would have website logs, obj@oms would have
order-management-system logs etc.
I extract the ns, and generate objects in the form: {host: ...,
severity: ...., obj.<ns>.x : ..., obj.<ns>.y: ... }So <ns> allows users to not conflict across un-expected contexts. Turns out, when users want to retrieve logs, in almost all scenarios they want logs from a specific <ns>. Infact fields that have similar names, have very different semantics across contexts, eg. status field for web-server access logs is http-status, whereas in a stage-machine log event is current-state, so pushing them in the same field makes little sense. On Thu, Apr 7, 2016 at 11:26 PM, Matt Ford <[email protected]> wrote: > Hi, yes I've gotten fairly deep into ES. We would like to be able to > search over and perhaps even calculate based on fields with the same > name and different types. What's more we don't know when developers > will add services and new json logs. Without the rsyslog change - as > far as I can see the only way to do this to have per app indexes (as > per the tip at the very bottom of this page > https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html). > > This is all cool stuff! Please keep the ideas coming :-) > > One thing that makes me nervous is the overhead of doing the > transformation via normalize on rsyslog. Some the applications > generate a lot of logs per second. > > On 7 April 2016 at 18:38, Dave Caplinger <[email protected]> > wrote: >> On Apr 7, 2016, at 12:04 PM, Matt Ford <[email protected]> wrote: >>> >>> Thanks for the help thus far I'm able to parse arbitrary json logs and >>> get them into kafka very nicely. >>> However, due to the many different systems in use there is key >>> namespace clashes in the final destination (Elasticsearch) >>> >>> I have some JSON logs like this from one app >>> >>> { "login": 234343,... } >>> >>> and some JSON logs like this from another app >>> >>> { "login": "matt",... } >>> >>> Is it possible to parse and change the key space to look like this >>> >>> { "app1_login": 234343, "app1_XX:": ....} >>> { "app2_login": "matt", "app2_XX:":...} >> >> I'm not sure how deep into ElasticSearch you've gotten, but it sounds like >> maybe you're seeing the result of automatic type mapping where the first >> field called "login" happens to be interpreted as a number, and later on a >> string value shows up and fails to be indexed because ElasticSearch now >> expects only numeric values. You can solve this at ElasticSearch directly >> by having an explicit mapping (for example, "login" is a string), which in >> this case would force the numeric login value to be inserted as a string >> instead. >> >> (See: >> https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html >> ) >> >> So you don't have to change the upstream JSON sources if you don't want to >> (though you certainly could do that instead). >> >> - Dave Caplinger >> >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
