Re: FetchParquet Type Conversion Error

[Joe Trite]

will do, thx

On Thu, Jun 28, 2018 at 8:25 PM, Bryan Bende  wrote:

> Joe,
>
> There was a similar issue I worked on recently that had to do with array
> handling:
> https://issues.apache.org/jira/browse/NIFI-5316
>
> From a quick glance this may be a different variation of a similar issue.
>
> We are using Parquet’s Java API that allows you to read Parquet into Avro,
> and some of the resulting conversions they perform are not lining up with
> what NiFi is expecting in terms of Avro types.
>
> Feel free to create a JIRA with the stack trace.
>
> Thanks,
>
> Bryan
>
> On Jun 28, 2018, at 7:49 PM, Joe Trite  wrote:
>
> Greetings, I'm using NiFi 1.6 trying to ListHDFS -> FetchParquet.  I'm
> seeing the below error on the fetch side.  Are there any limitations with
> FetchParquet data type conversions?  I am able to cat/dump the file with
> parquet-tools and load/query the data in Spark so believe the files are
> solid/not corrupt.  Any help would be appreciated.
>
> Here is the data type of the field throwing the error:  optional
> fixed_len_byte_array(16) amt (DECIMAL(38,10));
>
> Here is the error:
> 2018-06-28 22:56:57,030 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.parquet.FetchParquet 
> FetchParquet[id=9280c518-6a92-363f-893a-5a03feb45ebc]
> Failed to retrieve content from /data/000157_0 for
> StandardFlowFileRecord[uuid=112b6e45-011a-44ac-b258-97f52a50
> 5597,claim=,offset=0,name=000157_0,size=0] due to
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64,
> 0] of type class org.apache.avro.generic.GenericData$Fixed because no
> compatible types exist in the UNION for field amt; routing to failure:
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64,
> 0] of type class org.apache.avro.generic.GenericData$Fixed because no
> compatible types exist in the UNION for field amt
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64,
> 0] of type class org.apache.avro.generic.GenericData$Fixed because no
> compatible types exist in the UNION for field amt
> at org.apache.nifi.avro.AvroTypeUtil.convertUnionFieldValue(Avr
> oTypeUtil.java:759)
> at org.apache.nifi.avro.AvroTypeUtil.normalizeValue(AvroTypeUti
> l.java:844)
> at org.apache.nifi.avro.AvroTypeUtil.convertAvroRecordToMap(Avr
> oTypeUtil.java:710)
> at org.apache.nifi.processors.parquet.record.AvroParquetHDFSRec
> ordReader.nextRecord(AvroParquetHDFSRecordReader.java:62)
> at org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.la
> mbda$null$0(AbstractFetchHDFSRecord.java:197)
> at org.apache.nifi.controller.repository.StandardProcessSession
> .write(StandardProcessSession.java:2621)
> at org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.la
> mbda$onTrigger$1(AbstractFetchHDFSRecord.java:193)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
> upInformation.java:1678)
> at org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.on
> Trigger(AbstractFetchHDFSRecord.java:177)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
> ctProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
> tandardProcessorNode.java:1147)
> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(Conn
> ectableTask.java:175)
> at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
> gent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(
> Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:
> 308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> tureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> tureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> Thanks
> Joe
>
>
>

Re: FetchParquet Type Conversion Error

[Bryan Bende]

Joe,

There was a similar issue I worked on recently that had to do with array 
handling:
https://issues.apache.org/jira/browse/NIFI-5316 


From a quick glance this may be a different variation of a similar issue.

We are using Parquet’s Java API that allows you to read Parquet into Avro, and 
some of the resulting conversions they perform are not lining up with what NiFi 
is expecting in terms of Avro types.

Feel free to create a JIRA with the stack trace.

Thanks,

Bryan

> On Jun 28, 2018, at 7:49 PM, Joe Trite  wrote:
> 
> Greetings, I'm using NiFi 1.6 trying to ListHDFS -> FetchParquet.  I'm seeing 
> the below error on the fetch side.  Are there any limitations with 
> FetchParquet data type conversions?  I am able to cat/dump the file with 
> parquet-tools and load/query the data in Spark so believe the files are 
> solid/not corrupt.  Any help would be appreciated.
> 
> Here is the data type of the field throwing the error:  optional 
> fixed_len_byte_array(16) amt (DECIMAL(38,10));
> 
> Here is the error:
> 2018-06-28 22:56:57,030 ERROR [Timer-Driven Process Thread-3] 
> o.a.nifi.processors.parquet.FetchParquet 
> FetchParquet[id=9280c518-6a92-363f-893a-5a03feb45ebc] Failed to retrieve 
> content from /data/000157_0 for 
> StandardFlowFileRecord[uuid=112b6e45-011a-44ac-b258-97f52a505597,claim=,offset=0,name=000157_0,size=0]
>  due to 
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException: 
> Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64, 0] 
> of type class org.apache.avro.generic.GenericData$Fixed because no compatible 
> types exist in the UNION for field amt; routing to failure: 
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException: 
> Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64, 0] 
> of type class org.apache.avro.generic.GenericData$Fixed because no compatible 
> types exist in the UNION for field amt
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException: 
> Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64, 0] 
> of type class org.apache.avro.generic.GenericData$Fixed because no compatible 
> types exist in the UNION for field amt
> at 
> org.apache.nifi.avro.AvroTypeUtil.convertUnionFieldValue(AvroTypeUtil.java:759)
> at 
> org.apache.nifi.avro.AvroTypeUtil.normalizeValue(AvroTypeUtil.java:844)
> at 
> org.apache.nifi.avro.AvroTypeUtil.convertAvroRecordToMap(AvroTypeUtil.java:710)
> at 
> org.apache.nifi.processors.parquet.record.AvroParquetHDFSRecordReader.nextRecord(AvroParquetHDFSRecordReader.java:62)
> at 
> org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.lambda$null$0(AbstractFetchHDFSRecord.java:197)
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2621)
> at 
> org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.lambda$onTrigger$1(AbstractFetchHDFSRecord.java:193)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
> at 
> org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.onTrigger(AbstractFetchHDFSRecord.java:177)
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
> at 
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 
> Thanks
> Joe



signature.asc
Description: Message signed with OpenPGP

Re: FetchParquet Type Conversion Error

[Michael Xu]

 *list*-*un*subscr...@apache.org

On Thu, Jun 28, 2018 at 7:49 PM Joe Trite  wrote:

> Greetings, I'm using NiFi 1.6 trying to ListHDFS -> FetchParquet.  I'm
> seeing the below error on the fetch side.  Are there any limitations with
> FetchParquet data type conversions?  I am able to cat/dump the file with
> parquet-tools and load/query the data in Spark so believe the files are
> solid/not corrupt.  Any help would be appreciated.
>
> Here is the data type of the field throwing the error:  optional
> fixed_len_byte_array(16) amt (DECIMAL(38,10));
>
>
>
> Here is the error:
>
> 2018-06-28 22:56:57,030 ERROR [Timer-Driven Process Thread-3]
> o.a.nifi.processors.parquet.FetchParquet
> FetchParquet[id=9280c518-6a92-363f-893a-5a03feb45ebc] Failed to retrieve
> content from /data/000157_0 for
> StandardFlowFileRecord[uuid=112b6e45-011a-44ac-b258-97f52a505597,claim=,offset=0,name=000157_0,size=0]
> due to
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64,
> 0] of type class org.apache.avro.generic.GenericData$Fixed because no
> compatible types exist in the UNION for field amt; routing to failure:
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64,
> 0] of type class org.apache.avro.generic.GenericData$Fixed because no
> compatible types exist in the UNION for field amt
>
> org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
> Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64,
> 0] of type class org.apache.avro.generic.GenericData$Fixed because no
> compatible types exist in the UNION for field amt
>
> at
> org.apache.nifi.avro.AvroTypeUtil.convertUnionFieldValue(AvroTypeUtil.java:759)
>
> at
> org.apache.nifi.avro.AvroTypeUtil.normalizeValue(AvroTypeUtil.java:844)
>
> at
> org.apache.nifi.avro.AvroTypeUtil.convertAvroRecordToMap(AvroTypeUtil.java:710)
>
> at
> org.apache.nifi.processors.parquet.record.AvroParquetHDFSRecordReader.nextRecord(AvroParquetHDFSRecordReader.java:62)
>
> at
> org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.lambda$null$0(AbstractFetchHDFSRecord.java:197)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2621)
>
> at
> org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.lambda$onTrigger$1(AbstractFetchHDFSRecord.java:193)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:360)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1678)
>
> at
> org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.onTrigger(AbstractFetchHDFSRecord.java:177)
>
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
>
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
>
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Thanks
> Joe
>

FetchParquet Type Conversion Error

[Joe Trite]

Greetings, I'm using NiFi 1.6 trying to ListHDFS -> FetchParquet.  I'm
seeing the below error on the fetch side.  Are there any limitations with
FetchParquet data type conversions?  I am able to cat/dump the file with
parquet-tools and load/query the data in Spark so believe the files are
solid/not corrupt.  Any help would be appreciated.

Here is the data type of the field throwing the error:  optional
fixed_len_byte_array(16) amt (DECIMAL(38,10));



Here is the error:

2018-06-28 22:56:57,030 ERROR [Timer-Driven Process Thread-3]
o.a.nifi.processors.parquet.FetchParquet
FetchParquet[id=9280c518-6a92-363f-893a-5a03feb45ebc]
Failed to retrieve content from /data/000157_0 for
StandardFlowFileRecord[uuid=112b6e45-011a-44ac-b258-
97f52a505597,claim=,offset=0,name=000157_0,size=0] due to
org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64,
0] of type class org.apache.avro.generic.GenericData$Fixed because no
compatible types exist in the UNION for field amt; routing to failure:
org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64,
0] of type class org.apache.avro.generic.GenericData$Fixed because no
compatible types exist in the UNION for field amt

org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
Cannot convert value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 90, -13, 16, 122, 64,
0] of type class org.apache.avro.generic.GenericData$Fixed because no
compatible types exist in the UNION for field amt

at org.apache.nifi.avro.AvroTypeUtil.convertUnionFieldValue(
AvroTypeUtil.java:759)

at org.apache.nifi.avro.AvroTypeUtil.normalizeValue(
AvroTypeUtil.java:844)

at org.apache.nifi.avro.AvroTypeUtil.convertAvroRecordToMap(
AvroTypeUtil.java:710)

at org.apache.nifi.processors.parquet.record.
AvroParquetHDFSRecordReader.nextRecord(AvroParquetHDFSRecordReader.java:62)

at org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.
lambda$null$0(AbstractFetchHDFSRecord.java:197)

at org.apache.nifi.controller.repository.
StandardProcessSession.write(StandardProcessSession.java:2621)

at org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.
lambda$onTrigger$1(AbstractFetchHDFSRecord.java:193)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:360)

at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGroupInformation.java:1678)

at org.apache.nifi.processors.hadoop.AbstractFetchHDFSRecord.
onTrigger(AbstractFetchHDFSRecord.java:177)

at org.apache.nifi.processor.AbstractProcessor.onTrigger(
AbstractProcessor.java:27)

at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
StandardProcessorNode.java:1147)

at org.apache.nifi.controller.tasks.ConnectableTask.invoke(
ConnectableTask.java:175)

at org.apache.nifi.controller.scheduling.
TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)

at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$
ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$
ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Thanks
Joe