[
https://issues.apache.org/jira/browse/PARQUET-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541623#comment-17541623
]
ASF GitHub Bot commented on PARQUET-2069:
-----------------------------------------
islamismailov commented on PR #957:
URL: https://github.com/apache/parquet-mr/pull/957#issuecomment-1136197459
@theosib-amazon I've hit a similar issue. Tried your fix and I see this
error unfortunately. Can you test on Map<string, struct<bigint, string>>?
java.lang.RuntimeException: Failed on record 0
at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:86)
at org.apache.parquet.cli.Main.run(Main.java:157)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.parquet.cli.Main.main(Main.java:187)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassCastException: required binary element (STRING) is
not a group
at org.apache.parquet.schema.Type.asGroupType(Type.java:248)
at
org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:284)
at
org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:228)
at
org.apache.parquet.avro.AvroRecordConverter.access$100(AvroRecordConverter.java:74)
at
org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter$ElementConverter.<init>(AvroRecordConverter.java:539)
at
org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.<init>(AvroRecordConverter.java:489)
at
org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:293)
at
org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:137)
at
org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:91)
at
org.apache.parquet.avro.AvroRecordMaterializer.<init>(AvroRecordMaterializer.java:33)
at
org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:142)
at
org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:195)
at
org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:156)
at
org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
at
org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:363)
at
org.apache.parquet.cli.BaseCommand$1$1.<init>(BaseCommand.java:344)
at
org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:342)
at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:73)
> Parquet file containing arrays, written by Parquet-MR, cannot be read again
> by Parquet-MR
> -----------------------------------------------------------------------------------------
>
> Key: PARQUET-2069
> URL: https://issues.apache.org/jira/browse/PARQUET-2069
> Project: Parquet
> Issue Type: Bug
> Components: parquet-avro
> Affects Versions: 1.12.0
> Environment: Windows 10
> Reporter: Devon Kozenieski
> Priority: Blocker
> Attachments: modified.parquet, original.parquet, parquet-diff.png
>
>
> In the attached files, there is one original file, and one written modified
> file that results after reading the original file and writing it back with
> Parquet-MR, with a few values modified. The schema should not be modified,
> since the schema of the input file is used as the schema to write the output
> file. However, the output file has a slightly modified schema that then
> cannot be read back the same way again with Parquet-MR, resulting in the
> exception message: java.lang.ClassCastException: optional binary element
> (STRING) is not a group
> My guess is that the issue lies in the Avro schema conversion.
> The Parquet files attached have some arrays and some nested fields.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
