[
https://issues.apache.org/jira/browse/NIFI-4551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009746#comment-17009746
]
Mark Payne commented on NIFI-4551:
----------------------------------
[~cemeyer2] as far as I can tell, this is working as expected. It looks like
the attached JSON payload does not conform to the schema. The JSON is as
follows:
{code:java}
{
"id" : "de4181c4-287c-4d4d-8413-3204dc109950",
"payload" : {
"com.example.SubtypeB" : {
"firstName" : "John",
"lastName" : "Smith"
}
}
}{code}
However, the structure of your data should not incorporate the names of 'record
types' in your schema, but rather only the fields. For example, this would be
valid json that adheres to the schema:
{code:java}
{
"id" : "de4181c4-287c-4d4d-8413-3204dc109950",
"payload" : {
"firstName" : "John",
"lastName" : "Smith"
}
}{code}
Because the 'payload' type has no field named "com.example.SubtypeB". The type
of the 'payload' element is either a record with a 'firstName' and a 'lastName'
field or a record with a 'name' field. So the following would also be valid
according to the schema:
{code:java}
{
"id" : "de4181c4-287c-4d4d-8413-3204dc109950",
"payload" : {
"name" : "John Smith"
}
} {code}
> JSON to Avro conversion fails for records which have nested records
> -------------------------------------------------------------------
>
> Key: NIFI-4551
> URL: https://issues.apache.org/jira/browse/NIFI-4551
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.4.0
> Reporter: Charlie Meyer
> Priority: Major
> Attachments: ExampleObject.avsc, examplePayload.avro,
> examplePayload.json, example_object.avdl, nifi_json_avro_bug.xml,
> schema_registry_payload.json
>
>
> JSON to Avro conversion fails for records which have nested records.
> Given a confluent schema registry exists at some accessible address
> Steps to recreate:
> # register the schema:
> {{$ curl -H "Content-Type: application/vnd.schemaregistry.v1+json" -d
> @schema_registry_payload.json 4.3.2.1:8081/subjects/nifiBug/versions | jq}}
> # verify that we can use that schema to convert json to and from avro
> {{$ avro-tools fromjson --schema-file ExampleObject.avsc examplePayload.json
> > examplePayload.avro
> $ avro-tools tojson examplePayload.avro | jq}}
> # apply the attached template to nifi: nifi_avro_bug.xml
> # start up the components that the template created in nifi
> run the following command:
> {{$ curl -X POST -d @examplePayload.json http://localhost:5001/ | jq}}
> The serialization to avro fails with the following stack trace:
> {{
> 2017-10-30 11:41:02,199 ERROR [Timer-Driven Process Thread-5]
> o.a.n.p.k.pubsub.PublishKafkaRecord_0_10
> PublishKafkaRecord_0_10[id=19a933c0-f766-1221-4373-21c102ff71ab] Failed to
> send all message for
> StandardFlowFileRecord[uuid=4834f5cb-f513-49ee-8c3e-305a3acc64b6,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1509378326140-1, container=default,
> section=1], offset=4297, length=156],offset=0,name=75094273920075,size=156]
> to Kafka; routing to failure due to
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of string in field name of
> com.example.SubtypeA of union in field payload of com.example.ExampleObject:
> {}
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of string in field name of
> com.example.SubtypeA of union in field payload of com.example.ExampleObject
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at
> org.apache.nifi.avro.WriteAvroResultWithSchema.writeRecord(WriteAvroResultWithSchema.java:61)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
> at
> org.apache.nifi.processors.kafka.pubsub.PublisherLease.publish(PublisherLease.java:114)
> at
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_0_10$1.process(PublishKafkaRecord_0_10.java:339)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2174)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2144)
> at
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_0_10.onTrigger(PublishKafkaRecord_0_10.java:331)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: null of string in field name of
> com.example.SubtypeA of union in field payload of com.example.ExampleObject
> at
> org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:132)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:126)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> ... 19 common frames omitted
> Caused by: java.lang.NullPointerException: null
> at org.apache.avro.io.Encoder.writeString(Encoder.java:121)
> at
> org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:254)
> at
> org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:249)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:115)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:112)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> ... 22 common frames omitted}}
> I did a bit of digging on this one and had a few observations:
> When writing to avro, the following code is run to generate the avro record:
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java#L558]
> Here it iterates over all the fields of the object. This same code appears to
> be excuted on thee nested record. When run on the nested record, the schema
> on it has an empty list of fields. Thus, when the avro is generated, it has
> null values for all fields on the nested record.
> This appears to be being set here:
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/json/AbstractJsonRowRecordReader.java#L162]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
