This is an automated email from the ASF dual-hosted git repository.
zhouky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 35fe63e4a [CELEBORN-870][DOC] Document on usage together with Gluten
35fe63e4a is described below
commit 35fe63e4a9d4dfa5c98920a5b1df7c7784ca23be
Author: xiyu.zk <[email protected]>
AuthorDate: Fri Aug 4 11:32:13 2023 +0800
[CELEBORN-870][DOC] Document on usage together with Gluten
### What changes were proposed in this pull request?
As title.
### Why are the changes needed?
As title.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test.
Closes #1784 from kerwin-zk/gluten_celeborn.
Lead-authored-by: xiyu.zk <[email protected]>
Co-authored-by: Kerwin Zhang <[email protected]>
Co-authored-by: Keyong Zhou <[email protected]>
Signed-off-by: zky.zhoukeyong <[email protected]>
---
docs/developers/glutensupport.md | 62 ++++++++++++++++++++++++++++++++++++++++
mkdocs.yml | 1 +
2 files changed, 63 insertions(+)
diff --git a/docs/developers/glutensupport.md b/docs/developers/glutensupport.md
new file mode 100644
index 000000000..370286a4f
--- /dev/null
+++ b/docs/developers/glutensupport.md
@@ -0,0 +1,62 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Gluten Support
+## Velox Backend
+
+[Gluten](https://github.com/oap-project/gluten) with velox backend supports
Celeborn as remote shuffle service. Below introduction is used to enable this
feature
+
+First refer to this
URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md)
to build Gluten with velox backend.
+
+When compiling the Gluten Java module, it's required to enable `rss` profile,
as follows:
+
+```
+mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+```
+
+Then add the Gluten and Spark Celeborn Client packages to your Spark
application's classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
+- Gluten: gluten-velox-bundle-spark3.x_2.12-xx-xx-SNAPSHOT.jar,
gluten-thirdparty-lib-xx.jar
+
+Currently to use Gluten following configurations are required in
`spark-defaults.conf`
+
+```
+spark.shuffle.manager
org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
+
+# celeborn master
+spark.celeborn.master.endpoints clb-master:9097
+
+# we recommend set spark.celeborn.push.replicate.enabled to true to enable
server-side data replication
+# If you have only one worker, this setting must be false
+spark.celeborn.push.replicate.enabled true
+
+spark.celeborn.shuffle.writer hash
+spark.shuffle.service.enabled false
+spark.sql.adaptive.localShuffleReader.enabled false
+
+# If you want to use dynamic resource allocation,
+# please refer to this URL
(https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to
apply the patch into your own Spark.
+spark.dynamicAllocation.enabled false
+```
+
+## Availability
+| Celeborn Version | Available in Gluten? |
+|:----------------:|:--------------------:|
+| < 0.2.0 | No |
+| \>= 0.2.0 | Yes |
\ No newline at end of file
diff --git a/mkdocs.yml b/mkdocs.yml
index f842a5732..b73e78110 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -95,3 +95,4 @@ nav:
- Fault Tolerant: developers/faulttolerant.md
- Worker Exclusion: developers/workerexclusion.md
- Integrating Celeborn: developers/integrate.md
+ - Gluten Support: developers/glutensupport.md
