Yikun opened a new pull request #35422: URL: https://github.com/apache/spark/pull/35422
### What changes were proposed in this pull request? This patch added volcano feature step to help user integrate spark with Volcano Scheduler. - Add a VolcanoFeatureStep, it can be used in driver and executor side. - Create a PodGroup with minResource (CPU/memory) support - Added two configurations: `spark.kubernetes.job.min.memory` and `spark.kubernetes.job.min.cpu`. After this patch, users can enable this featurestep by submiting job by using ```shell --conf spark.kubernete.driver.scheduler.name=volcano \ --conf spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.scheduler.VolcanoFeatureStep ``` A PodGroup with user specified minimum resources required will be created before driver started, annotations will be set to driver pod to added driver pod to this pod group. Then, Volcano scheduler will help driver pod scheduling instead of deafult kubernetes scheduler. ### Why are the changes needed? This PR help user integrate Spark with Volcano Scheduler to enable minResource support ### Does this PR introduce _any_ user-facing change? Yes, introduced two configurations: - `spark.kubernetes.job.min.cpu`: The minimum memory for running the job, in MiB unless otherwise specified. - `spark.kubernetes.job.min.memory`: The minimum CPU for running the job These are used by `VolcanoFeatureStep`, and also will be used by `YunikornFeatureStep` in future. ### How was this patch tested? UT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
