Hey we'd like to set up a default format for all of our logging systems... perhaps looking like this:
"key1=value1;key2=value2;key3=value3...." With this pattern, we'd allow developers to define any key/value pairs they want to log, and separate them with a common separator. If we did this, what do we need to do in Flume to get Flume to parse out the key=value pairs into dynamic headers? We pass our data from Flume into both HDFS and ElasticSearch sinks. We would really like to have these fields dynamically sent to the sinks for much easier parsing and analysis later. Any thoughts on this? I know that we can define a unique interceptor for each service that looks for explicit field names ... but thats a nightmare to manage. I really want something truly dynamic. Matt Wise Sr. Systems Architect Nextdoor.com
