This is an automated email from the ASF dual-hosted git repository.
yuchaoran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new c496925cba [YUNIKORN-2919] Document spark-operator: Account
spark.memory.offHeap.size (#508)
c496925cba is described below
commit c496925cbaf6c9d7f5dfbb935a64f9afe06fd759
Author: Hsien-Cheng(Ryan) Huang <[email protected]>
AuthorDate: Wed Dec 25 07:26:57 2024 +0800
[YUNIKORN-2919] Document spark-operator: Account spark.memory.offHeap.size
(#508)
---
docs/user_guide/workloads/run_spark.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/user_guide/workloads/run_spark.md
b/docs/user_guide/workloads/run_spark.md
index fff6e1b2c2..deca293768 100644
--- a/docs/user_guide/workloads/run_spark.md
+++ b/docs/user_guide/workloads/run_spark.md
@@ -30,7 +30,7 @@ under the License.
:::note
Pre-requisites:
- This tutorial assumes YuniKorn is
[installed](../../get_started/get_started.md) under the namespace `yunikorn`
-- Use spark-operator version >= 2.0 to enable support for YuniKorn gang
scheduling
+- Use spark-operator version >= 2.1.0 to enable support for YuniKorn gang
scheduling
:::
### Install YuniKorn
@@ -48,7 +48,7 @@ helm install yunikorn yunikorn/yunikorn --create-namespace
--namespace yunikorn
We should install `spark-operator` with
`controller.batchScheduler.enable=true` and set
`controller.batchScheduler.default=yunikorn` to enable Gang Scheduling. It's
optional to set the default scheduler to YuniKorn since you can specify it
later on, but it's recommended to do so.
Also, note that the total requested memory for the Spark job is the sum of
memory requested for the driver and that for all executors, where each is
computed as below:
* Driver requested memory = `spark.driver.memory` +
`spark.driver.memoryOverhead`
-* Executor requested memory = `spark.executor.memory` +
`spark.executor.memoryOverhead` + `spark.executor.pyspark.memory`
+* Executor requested memory = `spark.executor.memory` +
`spark.executor.memoryOverhead` + `spark.executor.pyspark.memory` +
`spark.memory.offHeap.size`
```shell script
helm repo add spark-operator https://kubeflow.github.io/spark-operator
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
