Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19218#discussion_r157478781
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala
---
@@ -86,6 +110,19 @@ private[hive] trait SaveAsHiveFile extends
DataWritingCommand {
options = Map.empty)
}
+ // Because compression configurations can come in a variety of ways,
+ // we choose the compression configuration in this order:
+ // For parquet: `compression` > `parquet.compression` >
`spark.sql.parquet.compression.codec`
+ // For orc: `compression` > `orc.compress` >
`spark.sql.orc.compression.codec`
--- End diff --
Is it okay to leave this priority in the spark document or somewhere?
https://spark.apache.org/docs/latest/sql-programming-guide.html#configuration
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
