This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3d400d333050 [SPARK-46780][K8S][TESTS] Improve SBT K8s IT to skip R
image build step if not needed
3d400d333050 is described below
commit 3d400d333050f25e729db15817cff6940364ccce
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jan 19 15:00:37 2024 -0800
[SPARK-46780][K8S][TESTS] Improve SBT K8s IT to skip R image build step if
not needed
### What changes were proposed in this pull request?
This PR aims to allow the users to skip R image build step in SBT if R
tests are excluded.
### Why are the changes needed?
`setup-integration-test-env.sh` has this feature already and has been used
in Maven K8s IT.
https://github.com/apache/spark/blob/39f8e1a5953b5897f893151d24dc585a80c0c8a0/resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh#L119-L123
This PR aims to improve SBT logic to match the behavior by skipping SparkR
test if it's not needed.
### Does this PR introduce _any_ user-facing change?
No. This is a developer feature.
### How was this patch tested?
Manually tests.
**COMMAND**
```
$ build/sbt \
-Pkubernetes \
-Pkubernetes-integration-tests \
-Dtest.exclude.tags=minikube,local,r \
-Dspark.kubernetes.test.deployMode=docker-desktop \
"kubernetes-integration-tests/test"
```
**BEFORE**
```
$ docker images | grep spark
kubespark/spark-r dev
443f71dfbf87 17 seconds ago 1.53GB
kubespark/spark-py dev
82e3d4a859f6 About a minute ago 1.07GB
kubespark/spark dev
2eefd2785313 2 minutes ago 716MB
```
**AFTER**
```
$ docker images | grep spark
kubespark/spark-py dev
155ed5f5dd95 10 seconds ago 1.07GB
kubespark/spark dev
3edf056c5b33 About a minute ago 716MB
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44807 from dongjoon-hyun/SPARK-46780.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
project/SparkBuild.scala | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 7d399a735888..3123f931cd88 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -995,8 +995,12 @@ object KubernetesIntegrationTests {
s"$sparkHome/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile")
val pyDockerFile =
sys.props.getOrElse("spark.kubernetes.test.pyDockerFile",
s"$bindingsDir/python/Dockerfile")
- val rDockerFile =
sys.props.getOrElse("spark.kubernetes.test.rDockerFile",
+ var rDockerFile =
sys.props.getOrElse("spark.kubernetes.test.rDockerFile",
s"$bindingsDir/R/Dockerfile")
+ val excludeTags = sys.props.getOrElse("test.exclude.tags",
"").split(",")
+ if (excludeTags.exists(_.equalsIgnoreCase("r"))) {
+ rDockerFile = ""
+ }
val extraOptions = if (javaImageTag.isDefined) {
Seq("-b", s"java_image_tag=$javaImageTag")
} else {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
