[
https://issues.apache.org/jira/browse/SPARK-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14680277#comment-14680277
]
Cheng Lian commented on SPARK-9340:
-----------------------------------
Ah, thanks a lot! I see the problem now. {{parquet-avro}} doesn't allow
{{repeated}} fields outside {{LIST}} or {{MAP}}, and I was following
{{parquet-avro}} when implementing all the compatibility rules.
So I think the real problematic position is
[here|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystSchemaConverter.scala#L102-L104]
(and
[here|https://github.com/apache/parquet-mr/blob/apache-parquet-1.8.1/parquet-avro/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java#L217]
in {{parquet-avro}}).
This issue could have a simpler solution, especially the schema conversion
part. Row converter needs bigger changes though. I'm working on a simplified
version of PR #8063. Will attribute this issue to you since you spot this
issue and #8063 inspired me a lot!
> ParquetTypeConverter incorrectly handling of repeated types results in schema
> mismatch
> --------------------------------------------------------------------------------------
>
> Key: SPARK-9340
> URL: https://issues.apache.org/jira/browse/SPARK-9340
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0
> Reporter: Damian Guy
> Attachments: ParquetTypesConverterTest.scala
>
>
> The way ParquetTypesConverter handles primitive repeated types results in an
> incompatible schema being used for querying data. For example, given a schema
> like so:
> message root {
> repeated int32 repeated_field;
> }
> Spark produces a read schema like:
> message root {
> optional int32 repeated_field;
> }
> These are incompatible and all attempts to read fail.
> In ParquetTypesConverter.toDataType:
> if (parquetType.isPrimitive) {
> toPrimitiveDataType(parquetType.asPrimitiveType, isBinaryAsString,
> isInt96AsTimestamp)
> } else {...}
> The if condition should also have
> !parquetType.isRepetition(Repetition.REPEATED)
>
> And then this case will need to be handled in the else
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
