This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new bf19a5e1918 [SPARK-36681][CORE][TESTS][FOLLOW-UP] Tests only when
Snappy native library is available in low Hadoop versions
bf19a5e1918 is described below
commit bf19a5e1918bd2aec52a98428ccfe184102ea464
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Apr 11 12:08:54 2022 +0900
[SPARK-36681][CORE][TESTS][FOLLOW-UP] Tests only when Snappy native library
is available in low Hadoop versions
### What changes were proposed in this pull request?
This PR is a minor followup to only test when Snappy native library is
available in low Hadoop versions.
From Hadoop 3.3.1 with `HADOOP-17125`, the tests should pass but it fails
in lower versions of Hadoop when Snappy native library is unavailable (see also
https://github.com/apache/spark/pull/35784#issuecomment-1081290978).
### Why are the changes needed?
To make the tests robust.
### Does this PR introduce _any_ user-facing change?
Nope, this is test-only.
### How was this patch tested?
Should monitor CI
Closes #36136 from HyukjinKwon/SPARK-36681.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
core/src/test/scala/org/apache/spark/FileSuite.scala | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/core/src/test/scala/org/apache/spark/FileSuite.scala
b/core/src/test/scala/org/apache/spark/FileSuite.scala
index ac7670014eb..97795c55c82 100644
--- a/core/src/test/scala/org/apache/spark/FileSuite.scala
+++ b/core/src/test/scala/org/apache/spark/FileSuite.scala
@@ -136,10 +136,18 @@ class FileSuite extends SparkFunSuite with
LocalSparkContext {
}
// Hadoop "gzip" and "zstd" codecs require native library installed for
sequence files
- val codecs = Seq((new DefaultCodec(), "default"), (new BZip2Codec(),
"bzip2"),
- (new SnappyCodec(), "snappy")) ++ {
- if (VersionUtils.isHadoop3) Seq((new Lz4Codec(), "lz4")) else Seq()
+ private val codecs = Seq((new DefaultCodec(), "default"), (new BZip2Codec(),
"bzip2")) ++ {
+ scala.util.Try {
+ // See HADOOP-17125. Hadoop lower than 3.3.1 can throw an exception when
its native
+ // library for Snappy is unavailable. Here it calls
`SnappyCodec.getCompressorType`
+ // to indirectly test if the Snappy native library is available in lower
Hadoop versions.
+ new SnappyCodec().getCompressorType
+ (new SnappyCodec(), "snappy")
+ }.toOption
+ } ++ {
+ if (VersionUtils.isHadoop3) Seq((new Lz4Codec(), "lz4")) else Seq.empty
}
+
codecs.foreach { case (codec, codecName) =>
runSequenceFileCodecTest(codec, codecName)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
