This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new aa393cd [MINOR][TEST] GzipCodec should be set with Configuration
before using
aa393cd is described below
commit aa393cdafff5c9f20d41133631a4efcdee3ccdc7
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Thu Oct 7 15:56:25 2021 +0900
[MINOR][TEST] GzipCodec should be set with Configuration before using
### What changes were proposed in this pull request?
This is a minor fix to the usage of `GzipCodec` in
`WholeTextFileRecordReaderSuite` by setting `Configuration` on it.
### Why are the changes needed?
As `Configurable` class, `GzipCodec` should be set with `Configuration`
before using as an initialization. It has a `conf` member variable and many
methods use it to access hadoop configurations. This is how it is used in
Hadoop codebase.
We added an internal Hadoop configuration recently and found
`WholeTextFileRecordReaderSuite` failed due to this issue in the test suite.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes #34209 from viirya/fix-test.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
b/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
index f1d7053..5c31d7f 100644
---
a/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
+++
b/core/src/test/scala/org/apache/spark/input/WholeTextFileRecordReaderSuite.scala
@@ -23,6 +23,7 @@ import java.io.FileOutputStream
import scala.collection.immutable.IndexedSeq
+import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.io.Text
import org.apache.hadoop.io.compress.{CompressionCodecFactory, GzipCodec}
import org.scalatest.BeforeAndAfterAll
@@ -70,6 +71,7 @@ class WholeTextFileRecordReaderSuite extends SparkFunSuite
with BeforeAndAfterAl
compress: Boolean) = {
val out = if (compress) {
val codec = new GzipCodec
+ codec.setConf(new Configuration())
val path = s"${inputDir.toString}/$fileName${codec.getDefaultExtension}"
codec.createOutputStream(new DataOutputStream(new
FileOutputStream(path)))
} else {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
