Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/18805#discussion_r147629043
--- Diff: core/src/main/scala/org/apache/spark/io/CompressionCodec.scala ---
@@ -216,3 +218,33 @@ private final class SnappyOutputStreamWrapper(os:
SnappyOutputStream) extends Ou
}
}
}
+
+/**
+ * :: DeveloperApi ::
+ * ZStandard implementation of [[org.apache.spark.io.CompressionCodec]].
For more
+ * details see - http://facebook.github.io/zstd/
+ *
+ * @note The wire protocol for this codec is not guaranteed to be
compatible across versions
+ * of Spark. This is intended for use as an internal compression utility
within a single Spark
+ * application.
+ */
+@DeveloperApi
+class ZStdCompressionCodec(conf: SparkConf) extends CompressionCodec {
+
+ val bufferSize =
conf.getSizeAsBytes("spark.io.compression.zstd.bufferSize", "32k").toInt
--- End diff --
This should be private. The intent was to lift both config values out of
the method, so level can do here too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
