Cheng Lian created SPARK-10136:
----------------------------------
Summary: Parquet support fail to decode Avro arrays of primitive
array (e.g. array<array<int>>)
Key: SPARK-10136
URL: https://issues.apache.org/jira/browse/SPARK-10136
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.0
Reporter: Cheng Lian
Assignee: Cheng Lian
The following Avro schema
{noformat}
record AvroNonNullableArrays {
array<array<int>> int_arrays_column;
}
{noformat}
is translated into the following Parquet schema by parquet-avro 1.7.0:
{noformat}
message root {
required group int_arrays_column (LIST) {
repeated group array (LIST) {
repeated int32 array;
}
}
}
{noformat}
When making converters, the inner most {{array}} field is covered by
{{RepeatedPrimitiveConverter}}, which is introduced in SPARK-9340 and [PR
#8070|https://github.com/apache/spark/pull/8070]. However, PR #8070 assumes
that repeated converters can only be created for struct fields, and doesn't
take array elements into account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
