This is an automated email from the ASF dual-hosted git repository.
rexxiong pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 101c755e0 [CELEBORN-1635] Introduce Blaze support document
101c755e0 is described below
commit 101c755e0bfd9d4fdf94f27caa7da47604023c80
Author: SteNicholas <[email protected]>
AuthorDate: Wed Oct 9 11:48:05 2024 +0800
[CELEBORN-1635] Introduce Blaze support document
### What changes were proposed in this pull request?
Introduce Blaze support document.
### Why are the changes needed?
[Blaze](https://github.com/kwai/blaze) supports Celeborn as remote shuffle
service. It's recommened to Blaze support document for introduction of Blaze
usage.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No.
Closes #2787 from SteNicholas/CELEBORN-1635.
Authored-by: SteNicholas <[email protected]>
Signed-off-by: Shuang <[email protected]>
---
docs/developers/blazesupport.md | 47 ++++++++++++++++++++++++++++++++++++++++
docs/developers/glutensupport.md | 4 ----
mkdocs.yml | 4 +++-
3 files changed, 50 insertions(+), 5 deletions(-)
diff --git a/docs/developers/blazesupport.md b/docs/developers/blazesupport.md
new file mode 100644
index 000000000..a7533dd82
--- /dev/null
+++ b/docs/developers/blazesupport.md
@@ -0,0 +1,47 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Blaze Support
+
+## Build Blaze
+
+[Blaze](https://github.com/kwai/blaze) supports Celeborn as remote shuffle
service. Below introduction is used to enable this feature.
+
+First refer to [Build From
Source](https://github.com/kwai/blaze/blob/master/README.md#build-from-source)
or [Build With
Docker](https://github.com/kwai/blaze/blob/master/README.md#build-with-docker)
to build Blaze.
+
+## Blaze Configuration
+
+Currently, to use Blaze following configurations are required in
`spark-defaults.conf`.
+
+```
+spark.shuffle.manager
org.apache.spark.sql.execution.blaze.shuffle.celeborn.BlazeCelebornShuffleManager
+
+# celeborn master
+spark.celeborn.master.endpoints clb-master:9097
+
+spark.celeborn.client.spark.shuffle.writer hash
+spark.serializer org.apache.spark.serializer.KryoSerializer
+spark.sql.adaptive.localShuffleReader.enabled false
+```
+
+## Availability
+
+| Celeborn Version | Available in Blaze? |
+|:----------------:|:-------------------:|
+| < 0.5.0 | No |
+| \>= 0.5.0 | Yes |
diff --git a/docs/developers/glutensupport.md b/docs/developers/glutensupport.md
index 9a53927d1..32d057847 100644
--- a/docs/developers/glutensupport.md
+++ b/docs/developers/glutensupport.md
@@ -60,10 +60,6 @@ spark.shuffle.manager
org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleMa
# celeborn master
spark.celeborn.master.endpoints clb-master:9097
-# we recommend set `spark.celeborn.push.replicate.enabled` to true to enable
server-side data replication
-# If you have only one worker, this setting must be false
-spark.celeborn.client.push.replicate.enabled true
-
spark.celeborn.client.spark.shuffle.writer hash
# This is not necessary if your Spark external shuffle service is Spark 3.1 or
newer
spark.shuffle.service.enabled false
diff --git a/mkdocs.yml b/mkdocs.yml
index 8c197ab7f..92e03dfdd 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -97,5 +97,7 @@ nav:
- Worker Exclusion: developers/workerexclusion.md
- Integrating Celeborn: developers/integrate.md
- SBT Build: developers/sbt.md
- - Gluten Support: developers/glutensupport.md
+ - Native Engine Integration:
+ - Blaze: developers/blazesupport.md
+ - Gluten: developers/glutensupport.md
- Helm Charts: developers/helm-charts.md
