Is there an easy way to "flatten" arbitrary JSON within NiFi?
For input data like that shown below from Yahoo [1]
{
"query": {
"count": 1,
"created": "2017-09-15T11:20:26Z",
"lang": "en-US",
"results": {
"channel": {
"item": {
"condition": {
"code": "33",
"date": "Fri, 15 Sep 2017 06:00 AM EDT",
"temp": "63",
"text": "Mostly Clear"
}
}
}
}
}
}
...I'd like to end up with output something like this:
{
"query.count": 1,
"query.created": "2017-09-15T11:20:26Z",
"query.lang": "en-US",
"query.results.channel.item.condition.code": "33",
"query.results.channel.item.condition.date": "Fri, 15 Sep 2017 06:00 AM EDT",
"query.results.channel.item.condition.temp": "63",
"query.results.channel.item.condition.text": "Mostly Clear"
}
I checked out the JoltTransformJSON processor and some examples, such as
the nested data to "prefix soup" demo [2], but it seems as though I need to
enter information about the schema for the incoming data in order to
transform it. Ideally, I'd like to have a processor "just figure it out"
without explicit entry of a schema.
Is there any way to accomplish this in a generic way with JoltTransformJSON
(or another native processor)?
If not, would a ticket requesting a "Field Flattener" processor much like
the one included in StreamSets Data Collector [3] be worthwhile?
Thanks in advance!
-Nick
[1]
https://query.yahooapis.com/v1/public/yql?q=select%20item.condition%20from%20weather.forecast%20where%20woeid%20%3D%202383558&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys
[2] http://jolt-demo.appspot.com/#bucketToPrefixSoup
[3]
https://github.com/streamsets/datacollector/tree/master/basic-lib/src/main/java/com/streamsets/pipeline/stage/processor/fieldflattener
