Github user fjh100456 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20087#discussion_r160086482
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala
---
@@ -68,6 +68,10 @@ private[hive] trait SaveAsHiveFile extends
DataWritingCommand {
.get("mapreduce.output.fileoutputformat.compress.type"))
}
+ // Set compression by priority
+ HiveOptions.getHiveWriteCompression(fileSinkConf.getTableInfo,
sparkSession.sessionState.conf)
+ .foreach { case (compression, codec) => hadoopConf.set(compression,
codec) }
--- End diff --
For parquet, without the changes of this pr, the precedence is table-level
compression > `mapreduce.output.fileoutputformat.compress`.
`spark.sql.parquet.compression` never takes effect. But now with this pr,
`mapreduce.output.fileoutputformat.compress` will not take effect. As an
alternative, `spark.sql.parquet.compression` will always take effect if there
is no table level compression.
For ORC, `hive.exec.compress.output` does not take effect, as explained in
the comments of the Code.
Shall we keep this precedence for parquet? If so, how to deal with ORC?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
