yuchaoran2011 commented on code in PR #474: URL: https://github.com/apache/yunikorn-site/pull/474#discussion_r1798640880
########## docs/user_guide/workloads/run_spark.md: ########## @@ -25,12 +25,86 @@ specific language governing permissions and limitations under the License. --> +## Deploy Spark job with Spark Operator and Helm Review Comment: ```suggestion ## Run a Spark job with Spark Operator ``` Helm is standard tooling. Can omit in the title for brevity ########## docs/user_guide/workloads/run_spark.md: ########## @@ -25,12 +25,86 @@ specific language governing permissions and limitations under the License. --> +## Deploy Spark job with Spark Operator and Helm + +:::note +Pre-requisites: +- This tutorial assumes YuniKorn is [installed](../../get_started/get_started.md) under the namespace `yunikorn` +- Use spark-operator version >= 2.0 to enable support for YuniKorn gang scheduling +::: + +:::warning Review Comment: Consider changing it to `note` instead of `warning`. Spending a few minutes on installation doesn't seem to warrant a warning ########## docs/user_guide/workloads/run_spark.md: ########## @@ -25,12 +25,86 @@ specific language governing permissions and limitations under the License. --> +## Deploy Spark job with Spark Operator and Helm + +:::note +Pre-requisites: +- This tutorial assumes YuniKorn is [installed](../../get_started/get_started.md) under the namespace `yunikorn` +- Use spark-operator version >= 2.0 to enable support for YuniKorn gang scheduling +::: + +:::warning +This installation involves installing YuniKorn and Spark operator, which may take a few minutes to complete. To check the status we can use `kubectl get pods -n yunikorn` and `kubectl get pods -n spark-operator` +::: + +### Install YuniKorn + +A simple script to install YuniKorn under the namespace `yunikorn`, refer to [Get Started](../../get_started/get_started.md) for more details. + +```shell script +helm repo add yunikorn https://apache.github.io/yunikorn-release +helm repo update +helm install yunikorn yunikorn/yunikorn --create-namespace --namespace yunikorn +``` + +### Install spark operator + +We should install `spark-operator` with `controller.batchScheduler.enable=true` and set `controller.batchScheduler.default=yunikorn` to enable Gang Scheduling. It's optional to set the default scheduler to YuniKorn since you can specify it later on, but it's recommended to do so. +Also, note that our total allocated memory is `Driver + Execturor`, where Driver = `memory + memoryOverhead` and Execturor = `memory + memoryOverhead + spark.executor.pyspark.memory`, which will further propagate to the `yunikorn.apache.org/task-groups` annotation. Review Comment: This part can use some rephrasing to be more clear. ```suggestion Also, note that the total requested memory for the Spark job is the sum of memory requested for the driver and that for all executors, where each is computed as below: * Driver requested memory = `spark.driver.memory` + `spark.driver.memoryOverhead` * Executor requested memory = `spark.executor.memory` + `spark.executor.memoryOverhead` + `spark.executor.pyspark.memory` ``` The requested memory for the driver and executors will propagate to the `yunikorn.apache.org/task-groups` annotation on the Spark pods. ########## docs/user_guide/workloads/run_spark.md: ########## @@ -25,12 +25,86 @@ specific language governing permissions and limitations under the License. --> +## Deploy Spark job with Spark Operator and Helm + +:::note +Pre-requisites: +- This tutorial assumes YuniKorn is [installed](../../get_started/get_started.md) under the namespace `yunikorn` +- Use spark-operator version >= 2.0 to enable support for YuniKorn gang scheduling +::: + +:::warning +This installation involves installing YuniKorn and Spark operator, which may take a few minutes to complete. To check the status we can use `kubectl get pods -n yunikorn` and `kubectl get pods -n spark-operator` +::: + +### Install YuniKorn + +A simple script to install YuniKorn under the namespace `yunikorn`, refer to [Get Started](../../get_started/get_started.md) for more details. + +```shell script +helm repo add yunikorn https://apache.github.io/yunikorn-release +helm repo update +helm install yunikorn yunikorn/yunikorn --create-namespace --namespace yunikorn +``` + +### Install spark operator + +We should install `spark-operator` with `controller.batchScheduler.enable=true` and set `controller.batchScheduler.default=yunikorn` to enable Gang Scheduling. It's optional to set the default scheduler to YuniKorn since you can specify it later on, but it's recommended to do so. +Also, note that our total allocated memory is `Driver + Execturor`, where Driver = `memory + memoryOverhead` and Execturor = `memory + memoryOverhead + spark.executor.pyspark.memory`, which will further propagate to the `yunikorn.apache.org/task-groups` annotation. + +```shell script +helm repo add spark-operator https://kubeflow.github.io/spark-operator +helm repo update +helm install spark-operator spark-operator/spark-operator \ + --create-namespace \ + --namespace spark-operator \ + --set controller.batchScheduler.enable=true \ + --set controller.batchScheduler.default=yunikorn +``` + +### Create the example application Review Comment: ```suggestion ### Create an example application ``` ########## docs/user_guide/workloads/run_spark.md: ########## @@ -25,12 +25,86 @@ specific language governing permissions and limitations under the License. --> +## Deploy Spark job with Spark Operator and Helm + +:::note +Pre-requisites: +- This tutorial assumes YuniKorn is [installed](../../get_started/get_started.md) under the namespace `yunikorn` +- Use spark-operator version >= 2.0 to enable support for YuniKorn gang scheduling +::: + +:::warning +This installation involves installing YuniKorn and Spark operator, which may take a few minutes to complete. To check the status we can use `kubectl get pods -n yunikorn` and `kubectl get pods -n spark-operator` +::: + +### Install YuniKorn + +A simple script to install YuniKorn under the namespace `yunikorn`, refer to [Get Started](../../get_started/get_started.md) for more details. + +```shell script +helm repo add yunikorn https://apache.github.io/yunikorn-release +helm repo update +helm install yunikorn yunikorn/yunikorn --create-namespace --namespace yunikorn +``` + +### Install spark operator Review Comment: ```suggestion ### Install Spark Operator ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
