Repository: spark
Updated Branches:
  refs/heads/master f5aba6573 -> 4b11d909f


[MINOR][DOC] Add missing compression codec .

## What changes were proposed in this pull request?

Parquet file provides six codecs: "snappy", "gzip", "lzo", "lz4", "brotli", 
"zstd".
This pr add missing compression codec :"lz4", "brotli", "zstd" .
## How was this patch tested?
N/A

Closes #22068 from 10110346/nosupportlz4.

Authored-by: liuxian <liu.xi...@zte.com.cn>
Signed-off-by: hyukjinkwon <gurwls...@apache.org>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4b11d909
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4b11d909
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4b11d909

Branch: refs/heads/master
Commit: 4b11d909fd9e0f55ecb1f51af64cb4ff4dbd615b
Parents: f5aba65
Author: liuxian <liu.xi...@zte.com.cn>
Authored: Sat Aug 11 20:49:52 2018 +0800
Committer: hyukjinkwon <gurwls...@apache.org>
Committed: Sat Aug 11 20:49:52 2018 +0800

----------------------------------------------------------------------
 python/pyspark/sql/readwriter.py                             | 8 ++++----
 .../main/scala/org/apache/spark/sql/internal/SQLConf.scala   | 2 +-
 .../main/scala/org/apache/spark/sql/DataFrameWriter.scala    | 4 ++--
 3 files changed, 7 insertions(+), 7 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/4b11d909/python/pyspark/sql/readwriter.py
----------------------------------------------------------------------
diff --git a/python/pyspark/sql/readwriter.py b/python/pyspark/sql/readwriter.py
index abf878a..49f4e6b 100644
--- a/python/pyspark/sql/readwriter.py
+++ b/python/pyspark/sql/readwriter.py
@@ -825,10 +825,10 @@ class DataFrameWriter(OptionUtils):
                 exists.
         :param partitionBy: names of partitioning columns
         :param compression: compression codec to use when saving to file. This 
can be one of the
-                            known case-insensitive shorten names (none, 
snappy, gzip, and lzo).
-                            This will override 
``spark.sql.parquet.compression.codec``. If None
-                            is set, it uses the value specified in
-                            ``spark.sql.parquet.compression.codec``.
+                            known case-insensitive shorten names (none, 
uncompressed, snappy, gzip,
+                            lzo, brotli, lz4, and zstd). This will override
+                            ``spark.sql.parquet.compression.codec``. If None 
is set, it uses the
+                            value specified in 
``spark.sql.parquet.compression.codec``.
 
         >>> df.write.parquet(os.path.join(tempfile.mkdtemp(), 'data'))
         """

http://git-wip-us.apache.org/repos/asf/spark/blob/4b11d909/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
----------------------------------------------------------------------
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 603c070..594952e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -377,7 +377,7 @@ object SQLConf {
       "`parquet.compression` is specified in the table-specific 
options/properties, the " +
       "precedence would be `compression`, `parquet.compression`, " +
       "`spark.sql.parquet.compression.codec`. Acceptable values include: none, 
uncompressed, " +
-      "snappy, gzip, lzo.")
+      "snappy, gzip, lzo, brotli, lz4, zstd.")
     .stringConf
     .transform(_.toLowerCase(Locale.ROOT))
     .checkValues(Set("none", "uncompressed", "snappy", "gzip", "lzo", "lz4", 
"brotli", "zstd"))

http://git-wip-us.apache.org/repos/asf/spark/blob/4b11d909/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
index db2a1e7..650c917 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -548,8 +548,8 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) 
{
    * <ul>
    * <li>`compression` (default is the value specified in 
`spark.sql.parquet.compression.codec`):
    * compression codec to use when saving to file. This can be one of the 
known case-insensitive
-   * shorten names(`none`, `snappy`, `gzip`, and `lzo`). This will override
-   * `spark.sql.parquet.compression.codec`.</li>
+   * shorten names(`none`, `uncompressed`, `snappy`, `gzip`, `lzo`, `brotli`, 
`lz4`, and `zstd`).
+   * This will override `spark.sql.parquet.compression.codec`.</li>
    * </ul>
    *
    * @since 1.4.0


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

Reply via email to