[
https://issues.apache.org/jira/browse/NIFI-6000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16761383#comment-16761383
]
Sujesh Menon commented on NIFI-6000:
------------------------------------
When using InferAvroSchema for a JSON Input with null value or empty array, the
inferred.avro.schema could be as the following,
*JSON input 1*
{"key1":"value1","key2":null}
*Inferred Avro Schema*
{ "type" : "record", "name" : "test", "fields" : [ { "name" : "key1", "type" :
"string", "doc" : "Type inferred from '\"value1\"'" }, { "name" : "key2",
"type" : "null", "doc" : "Type inferred from 'null'" } ] }
*JSON input 2*
{"key1":"value1","key2":[]}
*Inferred Avro Schema*
{ "type" : "record", "name" : "test", "fields" : [ { "name" : "key1", "type" :
"string", "doc" : "Type inferred from '\"value1\"'" }, { "name" : "key2",
"type" : { "type" : "array", "items" : "null" }, "doc" : "Type inferred from
'[]'" } ] }
This could be fixed by handling IllegalArgumentException in
org.apache.nifi.processors.hiveorg.apache.nifi.processors.hive.ConvertAvroToORC.java
or by dealing the Avro type null to a valid hive TypeInfo object in
org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.java
> ConvertAvroToORC processor fails to process Avro type null and rollback
> instead of transferring the flowfile to failure.
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-6000
> URL: https://issues.apache.org/jira/browse/NIFI-6000
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.8.0, 1.9.0
> Reporter: Sujesh Menon
> Priority: Major
> Labels: easyfix
>
> The ConvertAvroToORC processor throws an IllegalArgumentException when the
> input avro data has null types or empty arrays.
> ConvertAvroToORC[id=9d22f79d-4ead-3924-df40-2bac4a672055]
> ConvertAvroToORC[id=9d22f79d-4ead-3924-df40-2bac4a672055] failed to process
> session due to java.lang.IllegalArgumentException: Did not recognize Avro
> type null; Processor Administratively Yielded for 1 sec:
> java.lang.IllegalArgumentException: Did not recognize Avro type null
> java.lang.IllegalArgumentException: Did not recognize Avro type null
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.getOrcField(NiFiOrcUtils.java:295)
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.lambda$getOrcField$11(NiFiOrcUtils.java:284)
> at java.util.ArrayList.forEach(Unknown Source)
> at
> org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.getOrcField(NiFiOrcUtils.java:281)
> at
> org.apache.nifi.processors.hive.ConvertAvroToORC.lambda$onTrigger$0(ConvertAvroToORC.java:217)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2910)
> at
> org.apache.nifi.processors.hive.ConvertAvroToORC.onTrigger(ConvertAvroToORC.java:209)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:205)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
>
> The flowfile is not transfered to failure as the
> https://github.com/apache/nifi/blob/412c4908e2c5d79d958b09403c816db57c828179/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java#L286
> only catches ProcessException but
> https://github.com/apache/nifi/blob/412c4908e2c5d79d958b09403c816db57c828179/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java#L217
> Throws IllegalArgumentException when the fieldSchema is anything other than
> hive primitive types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
