Yikun opened a new pull request #35495: URL: https://github.com/apache/spark/pull/35495
### What changes were proposed in this pull request? This patch added volcano feature step to help user integrate spark with Volcano Scheduler. - Add a VolcanoFeatureStep, it can be used in driver and executor side. After this patch, users can enable this featurestep by submiting job by using ```shell --conf spark.kubernete.driver.scheduler.name=volcano \ --conf spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.scheduler.VolcanoFeatureStep ``` A PodGroup will be created before driver started, annotations will be set to driver pod to added driver pod to this pod group. Then, Volcano scheduler will help driver pod scheduling instead of deafult kubernetes scheduler. ### Why are the changes needed? This PR help user integrate Spark with Volcano Scheduler. See also: [SPARK-36057](https://issues.apache.org/jira/browse/SPARK-36057) ### Does this PR introduce _any_ user-facing change? Yes, introduced a user feature step. These are used by `VolcanoFeatureStep`, and also will be used by `YunikornFeatureStep` in future. ### How was this patch tested? - UT - Integration test: Test all VolcanoSuite (all kubernetes test with volcano + a new podgroup test) and KubernetesSuite ```bash # Deploy Volcano (x86) kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml # Deploy Volcano (arm64) kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development-arm64.yaml # Test all VolcanoSuite (all kubernetes test with volcano + a new podgroup test) and KubernetesSuite build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
