This is an automated email from the ASF dual-hosted git repository.
zhouky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 95c9ccfc3 [CELEBORN-1010] Update docs about
`spark.shuffle.service.enabled`
95c9ccfc3 is described below
commit 95c9ccfc3e13c68f7bcdb1d3c06859b915501e2e
Author: mingji <[email protected]>
AuthorDate: Sun Oct 8 09:15:42 2023 +0800
[CELEBORN-1010] Update docs about `spark.shuffle.service.enabled`
### What changes were proposed in this pull request?
To clarify a spark config to work with Celeborn.
### Why are the changes needed?
After some tests, I found that Spark 3.1 and newer can work with Celeborn
with `spark.shuffle.service.enabled=true`.
ExternalShuffleBlockResolver won't check the shuffle manager's type since
Spark 3.1 and newer.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
I tested two scenarios about this PR.
1. Check whether Spark can release the executors in time.
2. Check data correctness by running TPC-DS.
All checks are good.
Closes #1955 from FMX/CELEBORN-1010.
Authored-by: mingji <[email protected]>
Signed-off-by: zky.zhoukeyong <[email protected]>
---
README.md | 1 +
docs/deploy.md | 1 +
docs/developers/glutensupport.md | 1 +
3 files changed, 3 insertions(+)
diff --git a/README.md b/README.md
index 5876f2b4f..1f86a9203 100644
--- a/README.md
+++ b/README.md
@@ -234,6 +234,7 @@ spark.serializer org.apache.spark.serializer.KryoSerializer
# celeborn master
spark.celeborn.master.endpoints clb-1:9097,clb-2:9097,clb-3:9097
+# This is not necessary if your Spark external shuffle service is Spark 3.1 or
newer
spark.shuffle.service.enabled false
# options: hash, sort
diff --git a/docs/deploy.md b/docs/deploy.md
index b618d8e5c..b91f74ced 100644
--- a/docs/deploy.md
+++ b/docs/deploy.md
@@ -167,6 +167,7 @@ spark.serializer org.apache.spark.serializer.KryoSerializer
# celeborn master
spark.celeborn.master.endpoints clb-1:9097,clb-2:9097,clb-3:9097
+# This is not necessary if your Spark external shuffle service is Spark 3.1 or
newer
spark.shuffle.service.enabled false
# options: hash, sort
diff --git a/docs/developers/glutensupport.md b/docs/developers/glutensupport.md
index d3b7b01df..3879e353f 100644
--- a/docs/developers/glutensupport.md
+++ b/docs/developers/glutensupport.md
@@ -47,6 +47,7 @@ spark.celeborn.master.endpoints clb-master:9097
spark.celeborn.client.push.replicate.enabled true
spark.celeborn.client.spark.shuffle.writer hash
+# This is not necessary if your Spark external shuffle service is Spark 3.1 or
newer
spark.shuffle.service.enabled false
spark.sql.adaptive.localShuffleReader.enabled false
