This is an automated email from the ASF dual-hosted git repository.
ethanfeng pushed a commit to branch branch-0.4
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/branch-0.4 by this push:
new 18d5c1cd7 [CELEBORN-1260] Improve Spark Configuration of Deploy Spark
client for deployment document
18d5c1cd7 is described below
commit 18d5c1cd78d8fd806f54c058d6480d7f89783ef3
Author: SteNicholas <[email protected]>
AuthorDate: Fri Feb 2 14:31:36 2024 +0800
[CELEBORN-1260] Improve Spark Configuration of Deploy Spark client for
deployment document
### What changes were proposed in this pull request?
Improve `Spark Configuration` of `Deploy Spark client` in `deploy.md`.
Fix #2270.
### Why are the changes needed?
It's recommended to improve the Spark Configuration of Deploy Spark client
for deployment document with Spark Dynamic Resource Allocation support.
```
# Support Spark Dynamic Resource Allocation
# Required Spark version >= 3.5.0
spark.shuffle.sort.io.plugin.class
org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO
# Required Spark version >= 3.4.0, highly recommended to disable
spark.dynamicAllocation.shuffleTracking.enabled false
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No.
Closes #2278 from SteNicholas/CELEBORN-1260.
Authored-by: SteNicholas <[email protected]>
Signed-off-by: mingji <[email protected]>
(cherry picked from commit 383a102ab4409071c3900f490253a171f65d4eb8)
Signed-off-by: mingji <[email protected]>
---
README.md | 2 ++
docs/deploy.md | 6 ++++++
2 files changed, 8 insertions(+)
diff --git a/README.md b/README.md
index 160180b7f..cba693169 100644
--- a/README.md
+++ b/README.md
@@ -280,6 +280,8 @@ spark.sql.adaptive.skewJoin.enabled true
# Support Spark Dynamic Resource Allocation
# Required Spark version >= 3.5.0
spark.shuffle.sort.io.plugin.class
org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO
+# Required Spark version >= 3.4.0, highly recommended to disable
+spark.dynamicAllocation.shuffleTracking.enabled false
```
### Deploy Flink client
diff --git a/docs/deploy.md b/docs/deploy.md
index 6b3ffc09c..4fdd1d84b 100644
--- a/docs/deploy.md
+++ b/docs/deploy.md
@@ -193,6 +193,12 @@ spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
# we recommend enabling aqe support to gain better performance
spark.sql.adaptive.enabled true
spark.sql.adaptive.skewJoin.enabled true
+
+# Support Spark Dynamic Resource Allocation
+# Required Spark version >= 3.5.0
+spark.shuffle.sort.io.plugin.class
org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO
+# Required Spark version >= 3.4.0, highly recommended to disable
+spark.dynamicAllocation.shuffleTracking.enabled false
```
## Deploy Flink client
