Josiah Johnston created NIFI-9462:
-------------------------------------
Summary: JSON Tree Reader mangles nested portions of schema
Key: NIFI-9462
URL: https://issues.apache.org/jira/browse/NIFI-9462
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Josiah Johnston
Attachments: JSON record set writer.png, JSON tree reader.png,
flow.png, updateRecord.png
I generate a flow file with this JSON line content
{{{"_source": \{"name": "battery-voltage-changed", "metadata": {"voltage":
2.8}}}}}
{{{"_source": \{"name": "temperature-changed", "foo": 3, "metadata":
{"temperature": 19.54}}}}}
Note that the data fields are nested under metadata (3 levels deep), and differ
between records.
I send it through an UpdateRecord processor that adds the file name as a
top-level key of each record.
The output appears to build a schema from the first record and applies it to
all subsequent records (ex: temperature in 2nd record is clobbered).
{{{"_source":\{"name":"battery-voltage-changed","metadata":{"voltage":2.8},"foo":null},"s3file":"2ec085a0-bbdf-4bc3-88da-de9cd85e44e9"}}}
{{{"_source":\{"name":"temperature-changed","metadata":{"voltage":null},"foo":3},"s3file":"2ec085a0-bbdf-4bc3-88da-de9cd85e44e9"}}}
Note that the `foo` field which appears as a 2nd level key in the 2nd record is
preserved, but the `temperature` field in the 3rd level is clobbered.
This is a simplified version of a real data source, and I have no control over
the schema. I've validated this happens to all records in each file in my real
use case.
I've attached screenshots of a minimal example and settings of JSON reader &
writer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
