[
https://issues.apache.org/jira/browse/SPARK-10136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian updated SPARK-10136:
-------------------------------
Description:
The following Avro schema
{noformat}
record AvroNonNullableArrays {
array<array<int>> int_arrays_column;
}
{noformat}
is translated into the following Parquet schema by parquet-avro 1.7.0:
{noformat}
message root {
required group int_arrays_column (LIST) {
repeated group array (LIST) {
repeated int32 array;
}
}
}
{noformat}
When making converters, the inner most {{array}} field is covered by
{{RepeatedPrimitiveConverter}}, which is introduced in SPARK-9340 and [PR
#8070|https://github.com/apache/spark/pull/8070]. However, PR #8070 assumes
that repeated converters can only be created for struct fields, and doesn't
take array elements into account.
Verified that 1.4 doesn't suffer this issue. This is a 1.5 regression.
was:
The following Avro schema
{noformat}
record AvroNonNullableArrays {
array<array<int>> int_arrays_column;
}
{noformat}
is translated into the following Parquet schema by parquet-avro 1.7.0:
{noformat}
message root {
required group int_arrays_column (LIST) {
repeated group array (LIST) {
repeated int32 array;
}
}
}
{noformat}
When making converters, the inner most {{array}} field is covered by
{{RepeatedPrimitiveConverter}}, which is introduced in SPARK-9340 and [PR
#8070|https://github.com/apache/spark/pull/8070]. However, PR #8070 assumes
that repeated converters can only be created for struct fields, and doesn't
take array elements into account.
> Parquet support fail to decode Avro arrays of primitive array (e.g.
> array<array<int>>)
> --------------------------------------------------------------------------------------
>
> Key: SPARK-10136
> URL: https://issues.apache.org/jira/browse/SPARK-10136
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Cheng Lian
> Assignee: Cheng Lian
>
> The following Avro schema
> {noformat}
> record AvroNonNullableArrays {
> array<array<int>> int_arrays_column;
> }
> {noformat}
> is translated into the following Parquet schema by parquet-avro 1.7.0:
> {noformat}
> message root {
> required group int_arrays_column (LIST) {
> repeated group array (LIST) {
> repeated int32 array;
> }
> }
> }
> {noformat}
> When making converters, the inner most {{array}} field is covered by
> {{RepeatedPrimitiveConverter}}, which is introduced in SPARK-9340 and [PR
> #8070|https://github.com/apache/spark/pull/8070]. However, PR #8070 assumes
> that repeated converters can only be created for struct fields, and doesn't
> take array elements into account.
> Verified that 1.4 doesn't suffer this issue. This is a 1.5 regression.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
