This is an automated email from the ASF dual-hosted git repository.
kerwinzhang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 5a39fc743 [CELEBORN-1486] Introduce ClickHouse Backend in Gluten
Support document
5a39fc743 is described below
commit 5a39fc743b6715e5dea57b4225ea711fc7e6ebe6
Author: SteNicholas <[email protected]>
AuthorDate: Mon Jul 1 16:38:49 2024 +0800
[CELEBORN-1486] Introduce ClickHouse Backend in Gluten Support document
### What changes were proposed in this pull request?
Introduce `ClickHouse Backend` in `Gluten Support` document. Meanwhile, fix
the profile via `-Pceleborn` to compile gluten module.
### Why are the changes needed?
Gluten with ClickHouse backend supports Celeborn as remote shuffle service
at present. Gluten Support document should introduce ClickHouse Backend to
guide user usage of Gluten with ClickHouse backend.
Backport https://github.com/apache/incubator-gluten/pull/6282.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No.
Closes #2594 from SteNicholas/CELEBORN-1486.
Authored-by: SteNicholas <[email protected]>
Signed-off-by: xiyu.zk <[email protected]>
---
docs/developers/glutensupport.md | 24 +++++++++++++++++++++---
1 file changed, 21 insertions(+), 3 deletions(-)
diff --git a/docs/developers/glutensupport.md b/docs/developers/glutensupport.md
index 6092caec9..2cc80e5e7 100644
--- a/docs/developers/glutensupport.md
+++ b/docs/developers/glutensupport.md
@@ -21,12 +21,12 @@ license: |
[Gluten](https://github.com/apache/incubator-gluten) with velox backend
supports Celeborn as remote shuffle service. Below introduction is used to
enable this feature.
-First refer to [Get Started With
Velox](https://github.com/apache/incubator-gluten/blob/main/docs/get-started/Velox.md)
to build Gluten with velox backend.
+First refer to [Get Started With
Velox](https://gluten.apache.org/docs/velox/getting-started) to build Gluten
with velox backend.
-When compiling the Gluten Java module, it's required to enable `rss` profile,
as follows:
+When compiling the Gluten Java module, it's required to enable `celeborn`
profile, as follows:
```
-mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests
+mvn clean package -Pbackends-velox -Pspark-3.3 -Pceleborn -DskipTests
```
Then add the Gluten and Spark Celeborn Client packages to your Spark
application's classpath(usually add them into `$SPARK_HOME/jars`).
@@ -34,6 +34,24 @@ Then add the Gluten and Spark Celeborn Client packages to
your Spark application
- Celeborn: `celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar`
- Gluten: `gluten-velox-bundle-spark3.x_2.12-xx-xx-SNAPSHOT.jar`,
`gluten-thirdparty-lib-xx.jar`
+## ClickHouse Backend
+
+[Gluten](https://github.com/apache/incubator-gluten) with clickhouse backend
supports Celeborn as remote shuffle service. Below introduction is used to
enable this feature.
+
+First refer to [Get Started With
ClickHouse](https://gluten.apache.org/docs/clickhouse/getting-started) to build
Gluten with clickhouse backend.
+
+When compiling the Gluten Java module, it's required to enable `celeborn`
profile, as follows:
+
+```
+mvn clean package -Pbackends-clickhouse -Pspark-3.3 -Pceleborn -DskipTests
+```
+
+Then add the Spark Celeborn Client packages to your Spark application's
classpath(usually add them into `$SPARK_HOME/jars`).
+
+- Celeborn: `celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar`
+
+## Gluten Configuration
+
Currently, to use Gluten following configurations are required in
`spark-defaults.conf`.
```
