[
https://issues.apache.org/jira/browse/PARQUET-364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Blue resolved PARQUET-364.
-------------------------------
Resolution: Fixed
Assignee: Ryan Blue
This is now fixed in master.
> Parquet-avro cannot decode Avro/Thrift array of primitive array (e.g.
> array<array<int>>)
> ----------------------------------------------------------------------------------------
>
> Key: PARQUET-364
> URL: https://issues.apache.org/jira/browse/PARQUET-364
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0
> Reporter: Cheng Lian
> Assignee: Ryan Blue
> Attachments: bad-avro.parquet, bad-thrift.parquet
>
>
> The problematic Avro and Thrift schemas are:
> {noformat}
> record AvroArrayOfArray {
> array<array<int>> int_arrays_column;
> }
> {noformat}
> and
> {noformat}
> struct ThriftListOfList {
> 1: list<list<i32>> intArraysColumn;
> }
> {noformat}
> They are converted to the following structurally equivalent Parquet schemas
> by parquet-avro 1.7.0 and parquet-thrift 1.7.0 respectively:
> {noformat}
> message AvroArrayOfArray {
> required group int_arrays_column (LIST) {
> repeated group array (LIST) {
> repeated int32 array;
> }
> }
> }
> {noformat}
> and
> {noformat}
> message ParquetSchema {
> required group intListsColumn (LIST) {
> repeated group intListsColumn_tuple (LIST) {
> repeated int32 intListsColumn_tuple_tuple;
> }
> }
> }
> {noformat}
> {{AvroIndexedRecordConverter}} cannot decode such records correctly. The
> reason is that the 2nd level repeated group {{array}} doesn't pass
> {{AvroIndexedRecordConverter.isElementType()}} check. We should check for
> field name "array" and field name suffix "_thrift" in {{isElementType()}} to
> fix this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
