[
https://issues.apache.org/jira/browse/PARQUET-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15019022#comment-15019022
]
Ryan Blue commented on PARQUET-359:
-----------------------------------
It looks like this was [fixed last month by
Alex|https://github.com/apache/parquet-mr/commit/b1ea059a66c7d6d6bb4cb53d2005a9b7bb599ada#diff-d5b10649340d9819495f082b4ceb63c4R99]
as part of PARQUET-381. [~lian cheng], do you think we can close this?
> Existing _common_metadata should be deleted when ParquetOutputCommitter fails
> to write summary files
> ----------------------------------------------------------------------------------------------------
>
> Key: PARQUET-359
> URL: https://issues.apache.org/jira/browse/PARQUET-359
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.6.0, 1.7.0, 1.8.0
> Reporter: Cheng Lian
>
> {{ParquetOutputCommitter}} only deletes {{_metadata}} when fails to write
> summary files. This may leave inconsistent existing {{_common_metadata}} out
> there.
> This issue can be reproduced via the following Spark shell snippet:
> {noformat}
> import sqlContext.implicits._
> val path = "file:///tmp/foo"
> (0 until 3).map(i => Tuple1((s"a_$i",
> s"b_$i"))).toDF().coalesce(1).write.mode("overwrite").parquet(path)
> (0 until 3).map(i => Tuple1((s"a_$i", s"b_$i",
> s"c_$i"))).toDF().coalesce(1).write.mode("append").parquet(path)
> {noformat}
> The 2nd write job fails to write the summary file because two written Parquet
> files contain different user-defined metadata (Spark SQL schema). We can find
> out that there is an {{_common_metadata}} left there:
> {noformat}
> $ tree /tmp/foo
> /tmp/foo
> ├── _SUCCESS
> ├── _common_metadata
> ├── part-r-00000-1c8bcb7f-84cf-43e3-9cd6-04d371322d95.gz.parquet
> └── part-r-00000-d759c53f-d12f-4555-9b27-8b03a8343b17.gz.parquet
> {noformat}
> Check its schema, the nested group contains only 2 fields, which is wrong:
> {noformat}
> $ parquet-schema /tmp/foo/_common_metadata
> message root {
> optional group _1 {
> optional binary _1 (UTF8);
> optional binary _2 (UTF8);
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
