This is an automated email from the ASF dual-hosted git repository.
ulyssesyou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 3bc5387c0 [GLUTEN-5142][CELEBORN] Remove Incubating of Celeborn from
reference (#5143)
3bc5387c0 is described below
commit 3bc5387c0e50f3e012f6ffad55dabbb7c52229c9
Author: Nicholas Jiang <[email protected]>
AuthorDate: Wed Mar 27 13:44:40 2024 +0800
[GLUTEN-5142][CELEBORN] Remove Incubating of Celeborn from reference (#5143)
---
docs/get-started/ClickHouse.md | 8 ++++----
docs/get-started/Velox.md | 8 ++++----
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/docs/get-started/ClickHouse.md b/docs/get-started/ClickHouse.md
index ad7183b90..4167af2ee 100644
--- a/docs/get-started/ClickHouse.md
+++ b/docs/get-started/ClickHouse.md
@@ -629,14 +629,14 @@ public read-only account：gluten/hN2xX3uQ4m
### Celeborn support
-Gluten with clickhouse backend has not yet supportted
[Celeborn](https://github.com/apache/incubator-celeborn) natively as remote
shuffle service using columar shuffle. However, you can still use Celeborn with
row shuffle, which means a ColumarBatch will be converted to a row during
shuffle.
+Gluten with clickhouse backend has not yet supportted
[Celeborn](https://github.com/apache/celeborn) natively as remote shuffle
service using columar shuffle. However, you can still use Celeborn with row
shuffle, which means a ColumarBatch will be converted to a row during shuffle.
Below introduction is used to enable this feature:
-First refer to this URL(https://github.com/apache/incubator-celeborn) to setup
a celeborn cluster.
+First refer to this URL(https://github.com/apache/celeborn) to setup a
celeborn cluster.
Then add the Spark Celeborn Client packages to your Spark application's
classpath(usually add them into `$SPARK_HOME/jars`).
-- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
+- Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
Currently to use Celeborn following configurations are required in
`spark-defaults.conf`
@@ -666,7 +666,7 @@ spark.sql.adaptive.localShuffleReader.enabled false
spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
# If you want to use dynamic resource allocation,
-# please refer to this URL
(https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to
apply the patch into your own Spark.
+# please refer to this URL
(https://github.com/apache/celeborn/tree/main/assets/spark-patch) to apply the
patch into your own Spark.
spark.dynamicAllocation.enabled false
```
diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md
index 7c3d77abc..1fabfc0fe 100644
--- a/docs/get-started/Velox.md
+++ b/docs/get-started/Velox.md
@@ -203,11 +203,11 @@ Currently there are several ways to asscess S3 in Spark.
Please refer [Velox S3]
## Celeborn support
-Gluten with velox backend supports
[Celeborn](https://github.com/apache/incubator-celeborn) as remote shuffle
service. Currently, the supported Celeborn versions are `0.3.x` and `0.4.0`.
+Gluten with velox backend supports
[Celeborn](https://github.com/apache/celeborn) as remote shuffle service.
Currently, the supported Celeborn versions are `0.3.x` and `0.4.0`.
Below introduction is used to enable this feature
-First refer to this URL(https://github.com/apache/incubator-celeborn) to setup
a celeborn cluster.
+First refer to this URL(https://github.com/apache/celeborn) to setup a
celeborn cluster.
When compiling the Gluten Java module, it's required to enable `rss` profile,
as follows:
@@ -217,7 +217,7 @@ mvn clean package -Pbackends-velox -Pspark-3.3 -Prss
-DskipTests
Then add the Gluten and Spark Celeborn Client packages to your Spark
application's classpath(usually add them into `$SPARK_HOME/jars`).
-- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
+- Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar
- Gluten: gluten-velox-bundle-spark3.x_2.12-xx_xx_xx-SNAPSHOT.jar,
gluten-thirdparty-lib-xx-xx.jar
Currently to use Gluten following configurations are required in
`spark-defaults.conf`
@@ -248,7 +248,7 @@ spark.sql.adaptive.localShuffleReader.enabled false
spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
# If you want to use dynamic resource allocation,
-# please refer to this URL
(https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to
apply the patch into your own Spark.
+# please refer to this URL
(https://github.com/apache/celeborn/tree/main/assets/spark-patch) to apply the
patch into your own Spark.
spark.dynamicAllocation.enabled false
```
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
