This is an automated email from the ASF dual-hosted git repository.
zuston pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle-website.git
The following commit(s) were added to refs/heads/master by this push:
new c1d4788 Update Support Spark Dynamic Allocation doc (#83)
c1d4788 is described below
commit c1d4788709fa217db401b0c1245bf967e9baf7df
Author: Zhen Wang <[email protected]>
AuthorDate: Fri Aug 2 16:41:32 2024 +0800
Update Support Spark Dynamic Allocation doc (#83)
---
docs/02-client-guide.md | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/docs/02-client-guide.md b/docs/02-client-guide.md
index 4bb2946..a6cfbed 100644
--- a/docs/02-client-guide.md
+++ b/docs/02-client-guide.md
@@ -25,13 +25,17 @@ This document will introduce how to deploy Uniffle client
plugins with Spark and
### Support Spark Dynamic Allocation
To support spark dynamic allocation with Uniffle, spark code should be updated.
-There are 2 patches for spark-2.4.6 and spark-3.1.2 in spark-patches folder
for reference.
+There are 7 patches for spark (2.3.4/2.4.6/3.0.1/3.1.2/3.2.1/3.3.1/3.4.1) in
patch/spark folder for reference.
After apply the patch and rebuild spark, add following configuration in spark
conf to enable dynamic allocation:
```
spark.shuffle.service.enabled false
spark.dynamicAllocation.enabled true
```
+For spark3.5 or above just add one more configuration:
+ ```
+ spark.shuffle.sort.io.plugin.class org.apache.spark.shuffle.RssShuffleDataIo
+ ```
### Deploy MapReduce Client Plugin
