David, Disclaimer: I am a member of the Apache NiFi PMC
Regarding metadata from the edge: Solutions like apache-minifi, filebeats allow you to pack metadata, either "inline" (i.e. filebeats JSON keys) or "aside" (NiFi attributes and provenance vs. contents). The key point being: As long you treat content as content and metadata as metadata, (like for example, two distinct JSON elements or different objects) you should be ok to ship data downstream without significant prejudice to parsers. Kind regards On Wed, Nov 23, 2016 at 3:27 AM, David Lang <[email protected]> wrote: > On Tue, 22 Nov 2016, Rich Megginson wrote: > > my problem has been that I find you tend to loose metadata that you need >>> to have to properly do the additional processing (exactly what directory >>> did this file come from, what is the timezone on the sending machine, etc) >>> >> >> Sure. The trick is to minimize the processing per node, but still have >> all of the log metadata you need, and I'm not saying that is an easy >> tradeoff. >> > > the trick is figuring out what metadata to send and in what format :-) the > lack of standards really hurts here. > > If you include metadata that the destination isn't expecting, you will > break it's parsers. This is especially bad when you are using the > proprietary log analysis tools, they expect the files to be unmodified. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
