This is an automated email from the ASF dual-hosted git repository. kerwinzhang pushed a commit to branch gluten_celeborn in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
commit d7d6a50598c4a136560a0b122e863e8f23c777a0 Author: xiyu.zk <[email protected]> AuthorDate: Wed Aug 2 11:43:18 2023 +0800 [CELEBORN-870][DOC] Document on usage together with Gluten --- docs/developers/glutensupport.md | 51 ++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 52 insertions(+) diff --git a/docs/developers/glutensupport.md b/docs/developers/glutensupport.md new file mode 100644 index 000000000..35888b817 --- /dev/null +++ b/docs/developers/glutensupport.md @@ -0,0 +1,51 @@ +--- +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +# Gluten Support +## Velox Backend + +[Gluten](https://github.com/oap-project/gluten) with velox backend supports Celeborn as remote shuffle service. Below introduction is used to enable this feature + +First refer to this URL(https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md) to build Gluten with velox backend. + +When compiling the Gluten java module, you need to add rss, as follows: + +``` +mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests +``` + +Then add the paths of Gluten and Spark Celeborn Client packages to class path. + +Currently to use Gluten following configurations are required in spark-defaults.conf + +``` +spark.shuffle.manager org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager + +# celeborn master +spark.celeborn.master.endpoints clb-master:9097 + +# we recommend set spark.celeborn.push.replicate.enabled to true to enable server-side data replication +# If you have only one worker, this setting must be false +spark.celeborn.push.replicate.enabled true + +spark.celeborn.shuffle.writer hash +spark.shuffle.service.enabled false +spark.sql.adaptive.localShuffleReader.enabled false + +# If you want to use dynamic resource allocation, +# please refer to this URL (https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark. +spark.dynamicAllocation.enabled false +``` \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 43640a777..833db3115 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -94,3 +94,4 @@ nav: - ShuffleClient: developers/shuffleclient.md - Fault Tolerant: developers/faulttolerant.md - Worker Exclusion: developers/workerexclusion.md + - Gluten Support: developers/glutensupport.md \ No newline at end of file
