[
https://issues.apache.org/jira/browse/PARQUET-364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian updated PARQUET-364:
-------------------------------
Description:
The problematic Avro and Thrift schemas are:
{noformat}
record AvroArrayOfArray {
array<array<int>> int_arrays_column;
}
{noformat}
and
{noformat}
struct ThriftListOfList {
1: list<list<i32>> intArraysColumn;
}
{noformat}
They are converted to the following structurally equivalent Parquet schemas by
parquet-avro 1.7.0 and parquet-thrift 1.7.0 respectively:
{noformat}
message AvroArrayOfArray {
required group int_arrays_column (LIST) {
repeated group array (LIST) {
repeated int32 array;
}
}
}
{noformat}
and
{noformat}
message ParquetSchema {
required group intListsColumn (LIST) {
repeated group intListsColumn_tuple (LIST) {
repeated int32 intListsColumn_tuple_tuple;
}
}
}
{noformat}
{{AvroIndexedRecordConverter}} cannot decode such records correctly. The reason
is that the 2nd level repeated group {{array}} doesn't pass
{{AvroIndexedRecordConverter.isElementType()}} check. We should check for field
name "array" and field name suffix "_thrift" in {{isElementType()}} to fix this
issue.
was:
The problematic Avro and Thrift schemas are:
{noformat}
record AvroArrayOfArray {
array<array<int>> int_arrays_column;
}
{noformat}
and
{noformat}
struct ThriftListOfList {
1: list<list<i32>> intArraysColumn;
}
{noformat}
They are converted to the following Parquet schemas by parquet-avro 1.7.0 and
parquet-thrift 1.7.0 respectively:
{noformat}
message AvroArrayOfArray {
required group int_arrays_column (LIST) {
repeated group array (LIST) {
repeated int32 array;
}
}
}
{noformat}
and
{noformat}
message ParquetSchema {
required group intListsColumn (LIST) {
repeated group intListsColumn_tuple (LIST) {
repeated int32 intListsColumn_tuple_tuple;
}
}
}
{noformat}
{{AvroIndexedRecordConverter}} cannot decode such records correctly. The reason
is that the 2nd level repeated group {{array}} doesn't pass
{{AvroIndexedRecordConverter.isElementType()}} check. We should check for field
name "array" and field name suffix "_thrift" in {{isElementType()}} to fix this
issue.
> Parque-avro cannot decode Avro/Thrift array of primitive array (e.g.
> array<array<int>>)
> ---------------------------------------------------------------------------------------
>
> Key: PARQUET-364
> URL: https://issues.apache.org/jira/browse/PARQUET-364
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0
> Reporter: Cheng Lian
>
> The problematic Avro and Thrift schemas are:
> {noformat}
> record AvroArrayOfArray {
> array<array<int>> int_arrays_column;
> }
> {noformat}
> and
> {noformat}
> struct ThriftListOfList {
> 1: list<list<i32>> intArraysColumn;
> }
> {noformat}
> They are converted to the following structurally equivalent Parquet schemas
> by parquet-avro 1.7.0 and parquet-thrift 1.7.0 respectively:
> {noformat}
> message AvroArrayOfArray {
> required group int_arrays_column (LIST) {
> repeated group array (LIST) {
> repeated int32 array;
> }
> }
> }
> {noformat}
> and
> {noformat}
> message ParquetSchema {
> required group intListsColumn (LIST) {
> repeated group intListsColumn_tuple (LIST) {
> repeated int32 intListsColumn_tuple_tuple;
> }
> }
> }
> {noformat}
> {{AvroIndexedRecordConverter}} cannot decode such records correctly. The
> reason is that the 2nd level repeated group {{array}} doesn't pass
> {{AvroIndexedRecordConverter.isElementType()}} check. We should check for
> field name "array" and field name suffix "_thrift" in {{isElementType()}} to
> fix this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
