Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/7209#issuecomment-118548365
After rethinking about this PR, I think it does spot another issue: the
current master breaks backwards-compatibility of reading Parquet files created
by parquet-avro. When converting a Spark SQL schema to a Parquet schema, Spark
1.4.0 and prior versions mostly follow parquet-avro, and convert arrays which
may contain null values into something like this:
```
message root {
optional group _c0 (LIST) {
repeated group bag {
optional group array {
<element-type>
}
}
}
}
```
Please note the field name `array`. However, current master changes this to
`element` even when we are using compatible mode.
@Sephiroth-Lin Would you mind to fix this issue by changing the
`array_element` string to `array`? My motivation is that, we should behave
exactly the same as Spark 1.4.0- and then fix SPARK-8811 in the work I
mentioned in my previous comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
