This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 928a128f13cf [SPARK-57545][K8S] Make `docker-image-tool.sh` and K8s IT
readme up-to-date
928a128f13cf is described below
commit 928a128f13cfd1c4dd383571e6261ccb0af7dc8b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Jun 18 17:44:43 2026 -0700
[SPARK-57545][K8S] Make `docker-image-tool.sh` and K8s IT readme up-to-date
### What changes were proposed in this pull request?
This PR aims to make `docker-image-tool.sh` and K8s IT `README.md`
up-to-date for Apache Spark 5.0.0 which is scheduled in 2027.
### Why are the changes needed?
To provide an up-to-date correct information for Spark 5. For example, Java
25 is the default since Apache Spark 4.2.0.
- #55829
### Does this PR introduce _any_ user-facing change?
No behavior change except the usage message and test documentation.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #56605 from dongjoon-hyun/SPARK-57545.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
bin/docker-image-tool.sh | 16 ++++++++--------
.../kubernetes/integration-tests/README.md | 18 +++++++++---------
2 files changed, 17 insertions(+), 17 deletions(-)
diff --git a/bin/docker-image-tool.sh b/bin/docker-image-tool.sh
index efbd63a3037d..ed634b989fce 100755
--- a/bin/docker-image-tool.sh
+++ b/bin/docker-image-tool.sh
@@ -261,18 +261,18 @@ Examples:
$0 -m -t testing build
- Build PySpark docker image
- $0 -r docker.io/myrepo -t v4.0.0 -p
kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
+ $0 -r docker.io/myrepo -t v5.0.0 -p
kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
- - Build and push image with tag "v4.0.0" to docker.io/myrepo
- $0 -r docker.io/myrepo -t v4.0.0 build
- $0 -r docker.io/myrepo -t v4.0.0 push
+ - Build and push image with tag "v5.0.0" to docker.io/myrepo
+ $0 -r docker.io/myrepo -t v5.0.0 build
+ $0 -r docker.io/myrepo -t v5.0.0 push
- - Build and push Java17-based image with tag "v4.0.0" to docker.io/myrepo
- $0 -r docker.io/myrepo -t v4.0.0 -b java_image_tag=17 build
- $0 -r docker.io/myrepo -t v4.0.0 push
+ - Build and push Java21-based image with tag "v5.0.0" to docker.io/myrepo
+ $0 -r docker.io/myrepo -t v5.0.0 -b java_image_tag=21 build
+ $0 -r docker.io/myrepo -t v5.0.0 push
- Build and push image for multiple archs to docker.io/myrepo
- $0 -r docker.io/myrepo -t v4.0.0 -X build
+ $0 -r docker.io/myrepo -t v5.0.0 -X build
# Note: buildx, which does cross building, needs to do the push during
build
# So there is no separate push step with -X
diff --git a/resource-managers/kubernetes/integration-tests/README.md
b/resource-managers/kubernetes/integration-tests/README.md
index fcbd7030cca5..990706caaa51 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -13,10 +13,10 @@ directory:
./dev/dev-run-integration-tests.sh
-To run tests with a specific Java version instead of Java 21, use
`--java-image-tag` to specify the
+To run tests with a specific Java version instead of Java 25, use
`--java-image-tag` to specify the
[base image](https://hub.docker.com/r/azul/zulu-openjdk/tags) accordingly.
- ./dev/dev-run-integration-tests.sh --java-image-tag 17-jre
+ ./dev/dev-run-integration-tests.sh --java-image-tag 21-jre
To run tests with a custom docker image, use `--docker-file` to specify the
Dockerfile.
Note that if both `--docker-file` and `--java-image-tag` are used,
`--docker-file` is preferred,
@@ -24,7 +24,7 @@ and the custom Dockerfile need to include a Java installation
by itself.
./dev/dev-run-integration-tests.sh --docker-file
../docker/src/main/dockerfiles/spark/Dockerfile
-The minimum tested version of Minikube is 1.28.0. The kube-dns addon must be
enabled. Minikube should
+The minimum tested version of Minikube is 1.38.0. The kube-dns addon must be
enabled. Minikube should
run with a minimum of 4 CPUs and 6G of memory:
minikube start --cpus 4 --memory 6144
@@ -43,7 +43,7 @@ default this is set to `minikube`, the available backends are
their prerequisite
### `minikube`
-Uses the local `minikube` cluster, this requires that `minikube` 1.28.0 or
greater be installed and that it be allocated
+Uses the local `minikube` cluster, this requires that `minikube` 1.38.0 or
greater be installed and that it be allocated
at least 4 CPUs and 6GB memory (some users have reported success with as few
as 3 CPUs and 4GB memory). The tests will
check if `minikube` is started and abort early if it isn't currently running.
@@ -205,7 +205,7 @@ to the wrapper scripts and using the wrapper scripts will
simply set these appro
<tr>
<td><code>spark.kubernetes.test.javaImageTag</code></td>
<td>
- A specific Azul Zulu OpenJDK base image tag to use, when set uses it
instead of 21.
+ A specific Azul Zulu OpenJDK base image tag to use, when set uses it
instead of 25.
</td>
<td><code>N/A</code></td>
</tr>
@@ -303,7 +303,7 @@ You can use SBT in the same way to build image and run all
K8s integration tests
build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
-Dtest.exclude.tags=minikube \
-Dspark.kubernetes.test.deployMode=docker-desktop \
- -Dspark.kubernetes.test.imageTag=2022-03-06 \
+ -Dspark.kubernetes.test.imageTag=2027-01-01 \
'kubernetes-integration-tests/test'
The following is an example to rerun tests with the pre-built image.
@@ -311,14 +311,14 @@ The following is an example to rerun tests with the
pre-built image.
build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
-Dtest.exclude.tags=minikube \
-Dspark.kubernetes.test.deployMode=docker-desktop \
- -Dspark.kubernetes.test.imageTag=2022-03-06 \
+ -Dspark.kubernetes.test.imageTag=2027-01-01 \
'kubernetes-integration-tests/runIts'
In addition, you can run a single test selectively.
build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
-Dspark.kubernetes.test.deployMode=docker-desktop \
- -Dspark.kubernetes.test.imageTag=2022-03-06 \
+ -Dspark.kubernetes.test.imageTag=2027-01-01 \
'kubernetes-integration-tests/testOnly -- -z "Run SparkPi with a very
long application name"'
You can also specify your specific dockerfile to build JVM/Python/R based
image to test.
@@ -326,7 +326,7 @@ You can also specify your specific dockerfile to build
JVM/Python/R based image
build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \
-Dtest.exclude.tags=minikube \
-Dspark.kubernetes.test.deployMode=docker-desktop \
- -Dspark.kubernetes.test.imageTag=2022-03-06 \
+ -Dspark.kubernetes.test.imageTag=2027-01-01 \
-Dspark.kubernetes.test.dockerFile=/path/to/Dockerfile \
-Dspark.kubernetes.test.pyDockerFile=/path/to/py/Dockerfile \
-Dspark.kubernetes.test.rDockerFile=/path/to/r/Dockerfile \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
