This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 8ad2a52a3138 [SPARK-51238][K8S][INFRA][DOCS] Upgrade Volcano to 1.11.0
8ad2a52a3138 is described below
commit 8ad2a52a31389ebcde060b20abecd83a6d801ae9
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Feb 16 20:54:03 2025 -0800
[SPARK-51238][K8S][INFRA][DOCS] Upgrade Volcano to 1.11.0
### What changes were proposed in this pull request?
This PR aims to upgrade `Volcano` to 1.11.0 in K8s integration test
document and GA job.
### Why are the changes needed?
- https://github.com/volcano-sh/volcano/releases/tag/v1.11.0
- [Supports Kubernetes
v1.31](https://github.com/volcano-sh/volcano/pull/3767)
- [Topology Aware
Scheduling](https://volcano.sh/en/docs/network_topology_aware_scheduling/)
- [Hierarchical Queue](https://volcano.sh/en/docs/hierarchical_queue/)
- [Multi-cluster
Scheduling](https://volcano.sh/en/docs/multi_cluster_scheduling/)
- [Load-aware Descheduling](https://volcano.sh/en/docs/descheduler/)
- [Fine-Grained Job Failure Recovery
Strategies](https://github.com/volcano-sh/volcano/blob/master/docs/user-guide/how_to_use_job_policy.md)
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA.
I also manually verified.
```
$ build/sbt -Pvolcano -Pkubernetes -Pkubernetes-integration-tests \
-Dtest.include.tags=volcano \
-Dtest.exclude.tags=minikube \
-Dspark.kubernetes.test.deployMode=docker-desktop \
'kubernetes-integration-tests/test'
...
[info] KubernetesSuite:
[info] VolcanoSuite:
[info] - Run SparkPi with volcano scheduler (8 seconds, 770 milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minCPU (31 seconds, 578
milliseconds)
[info] - SPARK-38187: Run SparkPi Jobs with minMemory (27 seconds, 576
milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled) (14
seconds, 267 milliseconds)
[info] - SPARK-38188: Run SparkPi jobs with 2 queues (all enabled) (16
seconds, 252 milliseconds)
[info] - SPARK-38423: Run driver job to validate priority order (18
seconds, 473 milliseconds)
[info] YuniKornSuite:
[info] Run completed in 2 minutes, 30 seconds.
[info] Total number of tests run: 6
[info] Suites: completed 3, aborted 0
[info] Tests: succeeded 6, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 441 s (07:21), completed Feb 16, 2025, 7:55:30 PM
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49977 from dongjoon-hyun/SPARK-51238.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 4c37a7a1af6b4643430bb0970674052a92771c84)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/build_and_test.yml | 2 +-
resource-managers/kubernetes/integration-tests/README.md | 6 +++---
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 67013ec46840..8f9d9b459f1a 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -1262,7 +1262,7 @@ jobs:
if [[ "${{ inputs.branch }}" == 'branch-3.5' ]]; then
kubectl apply -f
https://raw.githubusercontent.com/volcano-sh/volcano/v1.7.0/installer/volcano-development.yaml
|| true
else
- kubectl apply -f
https://raw.githubusercontent.com/volcano-sh/volcano/v1.10.0/installer/volcano-development.yaml
|| true
+ kubectl apply -f
https://raw.githubusercontent.com/volcano-sh/volcano/v1.11.0/installer/volcano-development.yaml
|| true
fi
eval $(minikube docker-env)
build/sbt -Phadoop-3 -Psparkr -Pkubernetes -Pvolcano
-Pkubernetes-integration-tests
-Dspark.kubernetes.test.volcanoMaxConcurrencyJobNum=1 -Dtest.exclude.tags=local
"kubernetes-integration-tests/test"
diff --git a/resource-managers/kubernetes/integration-tests/README.md
b/resource-managers/kubernetes/integration-tests/README.md
index 36848cba0609..2b3a5c414947 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -330,11 +330,11 @@ You can also specify your specific dockerfile to build
JVM/Python/R based image
## Requirements
- A minimum of 6 CPUs and 9G of memory is required to complete all Volcano
test cases.
-- Volcano v1.10.0.
+- Volcano v1.11.0.
## Installation
- kubectl apply -f
https://raw.githubusercontent.com/volcano-sh/volcano/v1.10.0/installer/volcano-development.yaml
+ kubectl apply -f
https://raw.githubusercontent.com/volcano-sh/volcano/v1.11.0/installer/volcano-development.yaml
## Run tests
@@ -355,5 +355,5 @@ You can also specify `volcano` tag to only run Volcano test:
## Cleanup Volcano
- kubectl delete -f
https://raw.githubusercontent.com/volcano-sh/volcano/v1.10.0/installer/volcano-development.yaml
+ kubectl delete -f
https://raw.githubusercontent.com/volcano-sh/volcano/v1.11.0/installer/volcano-development.yaml
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
