[
https://issues.apache.org/jira/browse/NIFI-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Stieglitz updated NIFI-14331:
------------------------------------
Description:
NIFI-13843 was aimed to eliminate any fields found in the JSON which were
defined in a specifed Avro schema. While that fix seems to have solved the
issue for top level items it did not solve the issue for an undefined key
within a defined object and for an undefined key in a defined object for an
array. Attached are the person.avsc Avro schema and the person_dropfield.json
which includes undefined top level fields such as single key value pair
("undefinedKey"), array ("undefinedScalarArray"), object ("undefinedObject")
and object array ("undefinedObjectArray"). It also includes undefined field
("undefinedKeyInObject") inside the defined "name" top level object and an
undefined field ("undefinedKeyInObject") in a "job" object found in the "jobs"
array. The result after calling ConvertRecord can be seen in the attached
convertRecordResults.json. Note fields "undefinedKey", "undefinedScalarArray",
"undefinedObject" and "undefinedObjectArray" all get dropped while fields
"undefinedKeyInObject" still exist in the "name" object and the "job" object
inside the "jobs" array.
Currently this behavior is seen in both ConvertRecord and MergeRecord when both
are configured with a JsonTreeReader and JsonRecordSetWriter.
It is interesting to note this behavior is seen in NIFI 1.28.1 only for
MergeRecord while ConvertRecord drops all unknown fields.
was:
NIFI-13843 was aimed to eliminate any fields found in the JSON which were
defined in a specifed Avro schema. While that fix seems to have solved the
issue for top level items it did not solve the issue for an undefined key
within a defined object and for an undefined key in a defined object for an
array. Attached are the person.avsc Avro schema and the person_dropfield.json
which includes undefined top level fields such as single key value pair
("undefinedKey"), array ("undefinedScalarArray"), object ("undefinedObject")
and object array ("undefinedObjectArray"). It also includes undefined field
("undefinedKeyInObject") inside the defined "name" top level object and an
undefined field ("undefinedKeyInObject") in a "job" object found in the "jobs"
array. The result after calling ConvertRecord can be seen in the attached
convertRecordResults.json. Note fields "undefinedKey", "undefinedScalarArray",
"undefinedObject" and "undefinedObjectArray" all get dropped while fields
"undefinedKeyInObject" still exist in the "name" object and the "job" object
inside the "jobs" array.
Currently this behavior is seen in both ConvertRecord and MergeRecord when both
are configured with a JsonTreeReader and JsonRecordSetWriter.
It is interesting to note this behavior is also seen in NIFI 1.28.1 for
MergeRecord but not ConvertRecord as there ConvertRecord seems to drop all
unknown fields.
> Unknown embedded fields not dropped by JSON Writer as expected by specified
> schema
> ----------------------------------------------------------------------------------
>
> Key: NIFI-14331
> URL: https://issues.apache.org/jira/browse/NIFI-14331
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.2.0
> Reporter: Daniel Stieglitz
> Priority: Major
> Attachments: convertRecordResults.json, person.avsc,
> person_dropfield.json
>
>
> NIFI-13843 was aimed to eliminate any fields found in the JSON which were
> defined in a specifed Avro schema. While that fix seems to have solved the
> issue for top level items it did not solve the issue for an undefined key
> within a defined object and for an undefined key in a defined object for an
> array. Attached are the person.avsc Avro schema and the person_dropfield.json
> which includes undefined top level fields such as single key value pair
> ("undefinedKey"), array ("undefinedScalarArray"), object ("undefinedObject")
> and object array ("undefinedObjectArray"). It also includes undefined field
> ("undefinedKeyInObject") inside the defined "name" top level object and an
> undefined field ("undefinedKeyInObject") in a "job" object found in the
> "jobs" array. The result after calling ConvertRecord can be seen in the
> attached convertRecordResults.json. Note fields "undefinedKey",
> "undefinedScalarArray", "undefinedObject" and "undefinedObjectArray" all get
> dropped while fields "undefinedKeyInObject" still exist in the "name" object
> and the "job" object inside the "jobs" array.
> Currently this behavior is seen in both ConvertRecord and MergeRecord when
> both are configured with a JsonTreeReader and JsonRecordSetWriter.
> It is interesting to note this behavior is seen in NIFI 1.28.1 only for
> MergeRecord while ConvertRecord drops all unknown fields.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
