This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 727dbe3 [SPARK-38022][K8S][TESTS] Use relativePath for K8s remote
file test in `BasicTestsSuite`
727dbe3 is described below
commit 727dbe35f468e6e5cf0ba6abbd83676a19046a60
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Jan 25 09:29:57 2022 -0800
[SPARK-38022][K8S][TESTS] Use relativePath for K8s remote file test in
`BasicTestsSuite`
### What changes were proposed in this pull request?
This PR aims to use `relativePath` for K8s remote file test in
`BasicTestsSuite`.
### Why are the changes needed?
To make `Run SparkRemoteFileTest using a remote data file` test pass.
**BEFORE**
```
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests
-Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17
-Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"
...
[info] KubernetesSuite:
...
[info] - Run SparkRemoteFileTest using a remote data file *** FAILED *** (3
minutes, 3 seconds)
[info] The code passed to eventually never returned normally. Attempted
190 times over 3.012265011116667 minutes. Last failure message: false was not
true. (KubernetesSuite.scala:452)
...
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
```
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests
-Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17
-Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"
...
[info] KubernetesSuite:
...
[info] - Run SparkRemoteFileTest using a remote data file (8 seconds, 608
milliseconds)
...
```
Closes #35318 from dongjoon-hyun/SPARK-38022.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 277322851f3c96f812c7da115f00f66bb6f11f6b)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
index 1c12123..8c753ea 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala
@@ -102,8 +102,8 @@ private[spark] trait BasicTestsSuite { k8sSuite:
KubernetesSuite =>
test("Run SparkRemoteFileTest using a remote data file", k8sTestTag) {
assert(sys.props.contains("spark.test.home"), "spark.test.home is not
set!")
TestUtils.withHttpServer(sys.props("spark.test.home")) { baseURL =>
- sparkAppConf
- .set("spark.files", baseURL.toString + REMOTE_PAGE_RANK_DATA_FILE)
+ sparkAppConf.set("spark.files", baseURL.toString +
+ REMOTE_PAGE_RANK_DATA_FILE.replace(sys.props("spark.test.home"),
"").substring(1))
runSparkRemoteCheckAndVerifyCompletion(appArgs =
Array(REMOTE_PAGE_RANK_FILE_NAME))
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
