[ https://issues.apache.org/jira/browse/SPARK-8604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Lian resolved SPARK-8604. ------------------------------- Resolution: Fixed Fix Version/s: 1.5.0 1.4.1 Issue resolved by pull request 6998 [https://github.com/apache/spark/pull/6998] > Parquet data source doesn't write summary file while doing appending > -------------------------------------------------------------------- > > Key: SPARK-8604 > URL: https://issues.apache.org/jira/browse/SPARK-8604 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.4.0 > Reporter: Cheng Lian > Assignee: Cheng Lian > Fix For: 1.4.1, 1.5.0 > > > Currently, Parquet and ORC data sources don't set their output format class, > as we override the output committer in Spark SQL. However, SPARK-8678 ignores > user defined output committer class while doing appending to avoid potential > issues brought by direct output committers (e.g. > {{DirectParquetOutputCommitter}}). This makes both of these data sources > fallback to the default output committer retrieved from {{TextOutputFormat}}, > which is {{FileOutputCommitter}}. For ORC, it's totally fine since ORC itself > just uses {{FileOutputCommitter}}. But for Parquet, > {{ParquetOutputCommitter}} also writes the summary files while committing the > job. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org