This is an automated email from the ASF dual-hosted git repository.
kerwinzhang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 3986fd595 [CELEBORN] support celeborn 0.5.0 (#6264)
3986fd595 is described below
commit 3986fd59555ea03347951de37054c3f11c0606b9
Author: Kaifei Yi <[email protected]>
AuthorDate: Thu Jul 4 12:51:51 2024 +0800
[CELEBORN] support celeborn 0.5.0 (#6264)
Co-authored-by: yikaifei <[email protected]>
---
.github/workflows/velox_docker.yml | 6 ++++--
docs/get-started/ClickHouse.md | 2 +-
docs/get-started/Velox.md | 2 +-
tools/gluten-it/pom.xml | 6 ++++++
4 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/velox_docker.yml
b/.github/workflows/velox_docker.yml
index d07ceb93b..c4b24598c 100644
--- a/.github/workflows/velox_docker.yml
+++ b/.github/workflows/velox_docker.yml
@@ -532,7 +532,7 @@ jobs:
fail-fast: false
matrix:
spark: [ "spark-3.2" ]
- celeborn: [ "celeborn-0.4.1", "celeborn-0.3.2-incubating" ]
+ celeborn: [ "celeborn-0.5.0", "celeborn-0.4.1",
"celeborn-0.3.2-incubating" ]
runs-on: ubuntu-20.04
container: ubuntu:22.04
steps:
@@ -563,8 +563,10 @@ jobs:
- name: TPC-H SF1.0 && TPC-DS SF1.0 Parquet local spark3.2 with ${{
matrix.celeborn }}
run: |
EXTRA_PROFILE=""
- if [ "${{ matrix.celeborn }}" = "celeborn-0.4.0" ]; then
+ if [ "${{ matrix.celeborn }}" = "celeborn-0.4.1" ]; then
EXTRA_PROFILE="-Pceleborn-0.4"
+ elif [ "${{ matrix.celeborn }}" = "celeborn-0.5.0" ]; then
+ EXTRA_PROFILE="-Pceleborn-0.5"
fi
echo "EXTRA_PROFILE: ${EXTRA_PROFILE}"
cd /opt && mkdir -p celeborn && \
diff --git a/docs/get-started/ClickHouse.md b/docs/get-started/ClickHouse.md
index 38ce048fe..f0b7fc13b 100644
--- a/docs/get-started/ClickHouse.md
+++ b/docs/get-started/ClickHouse.md
@@ -629,7 +629,7 @@ public read-only account：gluten/hN2xX3uQ4m
### Celeborn support
-Gluten with clickhouse backend supports
[Celeborn](https://github.com/apache/celeborn) as remote shuffle service.
Currently, the supported Celeborn versions are `0.3.x` and `0.4.0`.
+Gluten with clickhouse backend supports
[Celeborn](https://github.com/apache/celeborn) as remote shuffle service.
Currently, the supported Celeborn versions are `0.3.x`, `0.4.x` and `0.5.0`.
Below introduction is used to enable this feature.
diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md
index 5f9ae2a46..ff3b8f4b9 100644
--- a/docs/get-started/Velox.md
+++ b/docs/get-started/Velox.md
@@ -222,7 +222,7 @@ Currently there are several ways to asscess S3 in Spark.
Please refer [Velox S3]
## Celeborn support
-Gluten with velox backend supports
[Celeborn](https://github.com/apache/celeborn) as remote shuffle service.
Currently, the supported Celeborn versions are `0.3.x` and `0.4.0`.
+Gluten with velox backend supports
[Celeborn](https://github.com/apache/celeborn) as remote shuffle service.
Currently, the supported Celeborn versions are `0.3.x`, `0.4.x` and `0.5.0`.
Below introduction is used to enable this feature.
diff --git a/tools/gluten-it/pom.xml b/tools/gluten-it/pom.xml
index 71db637a8..c092a0ebb 100644
--- a/tools/gluten-it/pom.xml
+++ b/tools/gluten-it/pom.xml
@@ -170,5 +170,11 @@
<celeborn.version>0.4.1</celeborn.version>
</properties>
</profile>
+ <profile>
+ <id>celeborn-0.5</id>
+ <properties>
+ <celeborn.version>0.5.0</celeborn.version>
+ </properties>
+ </profile>
</profiles>
</project>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
