Hi,
I'm trying to manipulate record-based avro data in Nifi, and I'm getting
consistent errors. Here's a simple schema that illustrates the problem for
me: https://gist.github.com/dmsolow/13992482534eb0b23de94a385fe999e8
The schema has a root record, with a single field. The field's type is an
array with union type items. The union type contains two different record
types as sub-types, with different fields. My understanding is that this is
perfectly acceptable in an avro schema, and that unions can support any
number of named members.
However when I try to process a record written using this schema with
AvroReader and AvroRecordSetWriter, I see an error on the write side.
Here's the log message:
2018-10-20 16:26:07,091 ERROR [Timer-Driven Process Thread-3]
> o.a.n.processors.standard.ValidateRecord
> ValidateRecord[id=01651155-d17b-1b8a-56a7-d4dc64b64499] Failed to process
> StandardFlowFileRecord[uuid=9b7e7b3b-286a-4f91-b6f1-da3ca355ffcd,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1540052765488-169441,
> container=default, section=481], offset=277006,
> length=345],offset=0,name=2513241609759835,size=345]; will route to
> failure: org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of string in field f1 of left of union
> of array in field children of root
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of string in field f1 of left of union
> of array in field children of root
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at
> org.apache.nifi.avro.WriteAvroResultWithSchema.writeRecord(WriteAvroResultWithSchema.java:61)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
> at
> org.apache.nifi.processors.standard.ValidateRecord.onTrigger(ValidateRecord.java:344)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: null of string in field f1 of
> left of union of array in field children of root
> at
> org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:132)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:126)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> ... 14 common frames omitted
> Caused by: java.lang.NullPointerException: null
> at org.apache.avro.io.Encoder.writeString(Encoder.java:121)
> at
> org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:254)
> at
> org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:249)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:115)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:112)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:179)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:107)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> ... 17 common frames omitted
Here's the JSON representation of the record that's failing:
{
"children" : [ {
"left" : {
"f1" : "a"
}
}, {
"right" : {
"f2" : 2
}
} ]
}
- The schema access method does not matter (I've tried several different
methods and they all result in the same error).
- Notably I'm able to convert the record to JSON using a
JSONRecordSetWriter. I'm also able to read it using third party tools (like
avro-cli), so I think it's valid avro.
- I realize that this is likely an issue with the Apache Avro library,
but I thought I'd check with the Nifi community first since I'm using Nifi.
- In the github gist at the top I've included a base64 encoded avro
record (with embedded schema) if anyone wants to try to reproduce the
problem.
I realize from the stack trace that this is probably caused by the writer
being unable to resolve which record type is being written (it seems like
it's trying to write "left" when it should be writing "right"). My
understanding is that the generic avro classes should support this kind of
resolution however (the generic avro record object has a method to get the
schema).
Any guidance is appreciated.
Thanks,
Daniel
