Bryan Bende created NIFI-4030:
---------------------------------
Summary: Avro Schema with default values can cause NPE
Key: NIFI-4030
URL: https://issues.apache.org/jira/browse/NIFI-4030
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.2.0, 1.3.0
Reporter: Bryan Bende
Priority: Minor
Attachments: Convert_Avro_Short_to_Full.xml
While coming up with an example to convert between Avro schemas, I ran into the
following scenario...
I created two schemas like the following:
{code}
{
"name": "shortSchema",
"namespace": "nifi",
"type": "record",
"fields": [
{ "name": "a", "type": "string" },
{ "name": "b", "type": "string" }
]
}
{code}
{code}
{
"name": "fullSchema",
"namespace": "nifi",
"type": "record",
"fields": [
{ "name": "c", "type": "string", "default" : "default value for field c" },
{ "name": "d", "type": "string", "default" : "default value for field d" },
{ "name": "a", "type": "string" },
{ "name": "b", "type": "string" }
]
}
{code}
Then I created a flow with a ConvertRecord processor that read a flow file
which had Avro created with the shortSchema, and used a writer configured to
write with the fullSchema.
This produced the following exception, even though the fullSchema has default
values for additional fields:
{code}
ERROR [Timer-Driven Process Thread-1] o.a.n.processors.standard.ConvertRecord
ConvertRecord[id=ec373b3a-5704-1c6b-f501-0977a332d513] Failed to process
StandardFlowFileRecord[uuid=a8e604a9-1e15-4b9b-a4b7-fdbd441d3f2a,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1496842800576-10, container=default,
section=10], offset=40141, length=196],offset=0,name=128518448076937,size=196]:
org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: null of string in field c of nifi.fullSchema
org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: null of string in field c of nifi.fullSchema
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
at
org.apache.nifi.avro.WriteAvroResultWithSchema.writeRecord(WriteAvroResultWithSchema.java:59)
at
org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
at
org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:130)
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2785)
at
org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:118)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: null of string in field c of
nifi.fullSchema
at
org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:132)
at
org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:126)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
... 17 common frames omitted
Caused by: java.lang.NullPointerException: null
{code}
I worked around this by changing the type of the new fields from "string" to
["null", "string"] and then Avro seems to fill in the default value.
We should be able to work around this and give the default value to Avro before
it gets to the point of a NPE.
Template attached.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
