william-wang commented on a change in pull request #35870: URL: https://github.com/apache/spark/pull/35870#discussion_r837236032
########## File path: docs/running-on-kubernetes.md ########## @@ -1732,6 +1732,95 @@ Spark allows users to specify a custom Kubernetes schedulers. - Create additional Kubernetes custom resources for driver/executor scheduling. - Set scheduler hints according to configuration or existing Pod info dynamically. +#### Using Volcano as Customized Scheduler for Spark on Kubernetes + +**This feature is currently experimental. In future versions, there may be behavioral changes around configuration, feature step improvement.** + +##### Prerequisites +* Spark on Kubernetes with Volcano as a custom scheduler is supported since Spark v3.3.0 and Volcano v1.5.1. +* See also [Volcano installation](https://volcano.sh/en/docs/installation). + +##### Build +To create a Spark distribution along with Volcano suppport like those distributed by the Spark [Downloads page](https://spark.apache.org/downloads.html): + +``` +./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes -Pvolcano +``` + +##### Usage +Spark on Kubernetes allows using Volcano as a custom scheduler. Users can use Volcano to +support more advanced resource scheduling: queue scheduling, resource reservation, priority scheduling, and more. + +To use Volcano as a custom scheduler the user needs to specify the following configuration options: + +``` +# Specify volcano scheduler +--conf spark.kubernetes.scheduler.name=volcano +# Specify driver/executor VolcanoFeatureStep +--conf spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep +--conf spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep +# Specify PodGroup template +--conf spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/path/to/podgroup-template.yaml +``` + +##### Volcano Feature Step +Volcano feature steps help users to create a Volcano PodGroup and set driver/executor pod annotation to link with this PodGroup. + +Note that currently only driver/job level PodGroup is supported in Volcano Feature Step. + +##### Volcano PodGroup Template +Volcano defines PodGroup spec using [CRD yaml](https://volcano.sh/en/docs/podgroup/#example) + +Similar to [Pod template](#pod-template), Spark users can similarly use Volcano PodGroup Template to define the PodGroup spec configurations. + +To do so, specify the Spark properties `spark.kubernetes.scheduler.volcano.podGroupTemplateFile` to point to files accessible to the `spark-submit` process. + +Below is an example of PodGroup template, see also [PodGroup Introduction](https://volcano.sh/en/docs/podgroup/#introduction): + +``` +apiVersion: scheduling.volcano.sh/v1beta1 Review comment: @dongjoon-hyun Thanks for the reminding, we are already working on the multi-arch and will make it ready as soon as possible. For the v1beta1, I understand your concern, we have stayed in v1beta1 for a long time, it is stable and there are 50+ enterprise users use Volcano in their production environment. We have planed to upgrade it to v1 version this year and v1 will also keep complete compatibility with v1beta :) ########## File path: docs/running-on-kubernetes.md ########## @@ -1732,6 +1732,95 @@ Spark allows users to specify a custom Kubernetes schedulers. - Create additional Kubernetes custom resources for driver/executor scheduling. - Set scheduler hints according to configuration or existing Pod info dynamically. +#### Using Volcano as Customized Scheduler for Spark on Kubernetes + +**This feature is currently experimental. In future versions, there may be behavioral changes around configuration, feature step improvement.** + +##### Prerequisites +* Spark on Kubernetes with Volcano as a custom scheduler is supported since Spark v3.3.0 and Volcano v1.5.1. +* See also [Volcano installation](https://volcano.sh/en/docs/installation). + +##### Build +To create a Spark distribution along with Volcano suppport like those distributed by the Spark [Downloads page](https://spark.apache.org/downloads.html): + +``` +./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes -Pvolcano +``` + +##### Usage +Spark on Kubernetes allows using Volcano as a custom scheduler. Users can use Volcano to +support more advanced resource scheduling: queue scheduling, resource reservation, priority scheduling, and more. + +To use Volcano as a custom scheduler the user needs to specify the following configuration options: + +``` +# Specify volcano scheduler +--conf spark.kubernetes.scheduler.name=volcano +# Specify driver/executor VolcanoFeatureStep +--conf spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep +--conf spark.kubernetes.executor.pod.featureSteps=org.apache.spark.deploy.k8s.features.VolcanoFeatureStep +# Specify PodGroup template +--conf spark.kubernetes.scheduler.volcano.podGroupTemplateFile=/path/to/podgroup-template.yaml +``` + +##### Volcano Feature Step +Volcano feature steps help users to create a Volcano PodGroup and set driver/executor pod annotation to link with this PodGroup. + +Note that currently only driver/job level PodGroup is supported in Volcano Feature Step. + +##### Volcano PodGroup Template +Volcano defines PodGroup spec using [CRD yaml](https://volcano.sh/en/docs/podgroup/#example) + +Similar to [Pod template](#pod-template), Spark users can similarly use Volcano PodGroup Template to define the PodGroup spec configurations. + +To do so, specify the Spark properties `spark.kubernetes.scheduler.volcano.podGroupTemplateFile` to point to files accessible to the `spark-submit` process. + +Below is an example of PodGroup template, see also [PodGroup Introduction](https://volcano.sh/en/docs/podgroup/#introduction): + +``` +apiVersion: scheduling.volcano.sh/v1beta1 Review comment: @dongjoon-hyun Thanks for the reminding, we are already working on the multi-arch and will make it ready as soon as possible. For the v1beta1, I understand your concern, we have stayed in v1beta1 for a long time, it is stable and there are 50+ enterprise users use Volcano in their production environment. We have planed to upgrade it to v1 version this year and v1 will also keep complete compatibility with v1beta1 :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
