This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 2fe20ef [SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing
core-site.xml
2fe20ef is described below
commit 2fe20ef060a80aff1a20ef30f5e61992efd0fe50
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Feb 8 11:32:23 2021 +0900
[SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
### What changes were proposed in this pull request?
This is a follow-up for SPARK-34346 which causes a flakiness due to
`core-site.xml` test resource file addition. This PR aims to remove the test
resource `core/src/test/resources/core-site.xml` from `core` module.
### Why are the changes needed?
Due to the test resource `core-site.xml`, YARN UT becomes flaky in GitHub
Action and Jenkins.
```
$ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SPARK-16414" -Pyarn
...
[info] YarnClusterSuite:
[info] - yarn-cluster should respect conf overrides in SparkHadoopUtil
(SPARK-16414, SPARK-23630) *** FAILED *** (20 seconds, 209 milliseconds)
[info] FAILED did not equal FINISHED (stdout/stderr was not captured)
(BaseYarnClusterSuite.scala:210)
```
To isolate more, we may use `SPARK_TEST_HADOOP_CONF_DIR` like `yarn`
module's `yarn/Client`, but it seems an overkill in `core` module.
```
// SPARK-23630: during testing, Spark scripts filter out hadoop conf dirs
so that user's
// environments do not interfere with tests. This allows a special env
variable during
// tests so that custom conf dirs can be used by unit tests.
val confDirs = Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR") ++
(if (Utils.isTesting) Seq("SPARK_TEST_HADOOP_CONF_DIR") else Nil)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #31515 from dongjoon-hyun/SPARK-34346-2.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
(cherry picked from commit dcaf62afea8791e49a44c2062fe14bafdcc0e92f)
Signed-off-by: HyukjinKwon <[email protected]>
---
core/src/test/resources/core-site.xml | 24 ----------------------
.../scala/org/apache/spark/SparkContextSuite.scala | 3 ++-
2 files changed, 2 insertions(+), 25 deletions(-)
diff --git a/core/src/test/resources/core-site.xml
b/core/src/test/resources/core-site.xml
deleted file mode 100644
index 84eddf8..0000000
--- a/core/src/test/resources/core-site.xml
+++ /dev/null
@@ -1,24 +0,0 @@
-<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one or more
- ~ contributor license agreements. See the NOTICE file distributed with
- ~ this work for additional information regarding copyright ownership.
- ~ The ASF licenses this file to You under the Apache License, Version 2.0
- ~ (the "License"); you may not use this file except in compliance with
- ~ the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License.
- -->
-
-<configuration>
- <property>
- <name>hadoop.tmp.dir</name>
- <value>/tmp/hive_zero</value>
- <description>default is /tmp/hadoop-${user.name} and will be
overridden</description>
- </property>
-</configuration>
diff --git a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
index 7751918..0c0a9b8 100644
--- a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
@@ -1040,11 +1040,12 @@ class SparkContextSuite extends SparkFunSuite with
LocalSparkContext with Eventu
val testKey = "hadoop.tmp.dir"
val bufferKey = "io.file.buffer.size"
val hadoopConf0 = new Configuration()
+ hadoopConf0.set(testKey, "/tmp/hive_zero")
val hiveConfFile =
Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
assert(hiveConfFile != null)
hadoopConf0.addResource(hiveConfFile)
- assert(hadoopConf0.get(testKey) === "/tmp/hive_one")
+ assert(hadoopConf0.get(testKey) === "/tmp/hive_zero")
assert(hadoopConf0.get(bufferKey) === "201811")
val sparkConf = new SparkConf()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
