dongjoon-hyun commented on code in PR #37622: URL: https://github.com/apache/spark/pull/37622#discussion_r952872525
########## docs/running-on-kubernetes.md: ########## @@ -1811,6 +1811,50 @@ spec: queue: default ``` +#### Using Apache YuniKorn as Customized Scheduler for Spark on Kubernetes + +[Apache YuniKorn](https://yunikorn.apache.org/) is a resource scheduler for Kubernetes that provides advanced batch scheduling +capabilities, such as job queuing, resource fairness, min/max queue capacity and flexible job ordering policies. +For available Apache YuniKorn features, please refer to [this doc](https://yunikorn.apache.org/docs/next/get_started/core_features). + +##### Prerequisites + +Install Apache YuniKorn: + +```bash +helm repo add yunikorn https://apache.github.io/yunikorn-release +helm repo update +kubectl create namespace yunikorn +helm install yunikorn yunikorn/yunikorn --namespace yunikorn +``` + +the above steps will install the latest version of YuniKorn on an existing Kubernetes cluster. + +##### Get started + +Submit Spark jobs with the following extra options: + +```bash +--conf spark.kubernetes.scheduler.name=yunikorn +--conf spark.kubernetes.driver.annotation.yunikorn.apache.org/app-id={{APP_ID}} +--conf spark.kubernetes.executor.annotation.yunikorn.apache.org/app-id={{APP_ID}} +``` + +Note, `{{APP_ID}}` is the builtin variable that will be substituted with Spark job ID automatically. +With the above configuration, the job will be scheduled by YuniKorn scheduler instead of the default Kubernetes scheduler. + +##### Work with YuniKorn queues + +Apache YuniKorn supports 2 types of resource queues: + +- Static +- Dynamic + +The static queues are predefined in YuniKorn configmap, and the dynamic queues are automatically created by the scheduler +based on [placement rules](https://yunikorn.apache.org/docs/next/user_guide/placement_rules). Spark supports to run with +both queue setup. Refer to this [doc](https://yunikorn.apache.org/docs/next/user_guide/resource_quota_management) for more +information about how to run Spark with different queue setup. + Review Comment: Lastly, is there any limitation of YuniKorn? Can we assume that YuniKorn will work with all Spark-capable environment. Specifically, Apache Spark - Supports ARM64 arch like Graviton2 - Supports IPv6-only environment via https://kubernetes.io/docs/concepts/services-networking/dual-stack/ I'm wondering if there is some documentation on YuniKorn website about those environment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
