(incubator-celeborn) branch main updated: [CELEBORN-1341] Improve Celeborn document

Wed, 20 Mar 2024 00:04:52 -0700 

This is an automated email from the ASF dual-hosted git repository.

ethanfeng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new a371f934c [CELEBORN-1341] Improve Celeborn document
a371f934c is described below

commit a371f934cff0507fc97c7596ed7f9b4be2f21e62
Author: SteNicholas <[email protected]>
AuthorDate: Wed Mar 20 15:02:05 2024 +0800

    [CELEBORN-1341] Improve Celeborn document
    
    ### What changes were proposed in this pull request?
    
    Improve Celeborn document to fix typos, table formats and wrong description 
of document. Meanwhile, `deploy.md` adds the document of MapReduce client 
deployment.
    
    ### Why are the changes needed?
    
    There are some typos and format fixes in Celeborn document at present. 
Meanwhile, the `deploy.md` does not contain the deployment of MapReduce client, 
which is inconsistent with `README.md` for Flink configuration.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    No.
    
    Closes #2407 from SteNicholas/CELEBORN-1341.
    
    Authored-by: SteNicholas <[email protected]>
    Signed-off-by: mingji <[email protected]>
---
 README.md                        | 47 +++++++++++++++++-------------------
 docs/README.md                   | 27 ++++++++++++---------
 docs/celeborn_ratis_shell.md     |  6 ++---
 docs/deploy.md                   | 52 ++++++++++++++++++++++++----------------
 docs/deploy_on_k8s.md            |  2 +-
 docs/developers/flink.md         | 19 ---------------
 docs/developers/glutensupport.md | 10 ++++----
 docs/developers/integrate.md     |  2 +-
 docs/developers/overview.md      | 12 +++++-----
 docs/developers/readdata.md      |  0
 docs/developers/sbt.md           | 10 ++++----
 docs/developers/shuffleclient.md |  2 +-
 docs/developers/spark.md         | 19 ---------------
 docs/developers/storage.md       |  2 --
 docs/quota_management.md         | 10 ++++----
 15 files changed, 97 insertions(+), 123 deletions(-)

diff --git a/README.md b/README.md
index 4c0cb76b5..8dac10312 100644
--- a/README.md
+++ b/README.md
@@ -44,12 +44,12 @@ Celeborn worker's slot count decreases when a partition is 
allocated and increme
 1. Celeborn supports Spark 2.4/3.0/3.1/3.2/3.3/3.4/3.5, Flink 
1.14/1.15/1.17/1.18/1.19 and Hadoop MapReduce 2/3.
 2. Celeborn tested under Scala 2.11/2.12/2.13 and Java 8/11/17 environment.
 
-Build Celeborn
+Build Celeborn via `make-distribution.sh`:
 ```shell
 ./build/make-distribution.sh 
-Pspark-2.4/-Pspark-3.0/-Pspark-3.1/-Pspark-3.2/-Pspark-3.3/-Pspark-3.4/-Pflink-1.14/-Pflink-1.15/-Pflink-1.17/-Pflink-1.18/-Pflink-1.19/-Pmr
 ```
 
-package apache-celeborn-${project.version}-bin.tgz will be generated.
+Package `apache-celeborn-${project.version}-bin.tgz` will be generated.
 
 > **_NOTE:_** The following table indicates the compatibility of Celeborn 
 > Spark and Flink clients with different versions of Spark and Flink for 
 > various Java and Scala versions.
 
@@ -68,7 +68,7 @@ package apache-celeborn-${project.version}-bin.tgz will be 
generated.
 | Flink 1.18 | &#x274C;          | &#10004;          | &#10004;           | 
&#x274C;           | &#x274C;          | &#x274C;           | &#x274C;          
 |
 | Flink 1.19 | &#x274C;          | &#10004;          | &#10004;           | 
&#x274C;           | &#x274C;          | &#x274C;           | &#x274C;          
 |
 
-To compile the client for Spark 2.4 with Scala 2.12, please use the following 
command
+To compile the client for Spark 2.4 with Scala 2.12, please use the following 
command:
 
 - Scala 2.12.8/2.12.9/2.12.10
 ```shell
@@ -108,8 +108,8 @@ Celeborn cluster composes of Master and Worker nodes, the 
Master supports both s
 
 ### Deploy Celeborn
 #### Deploy on host
-1. Unzip the tarball to `$CELEBORN_HOME`
-2. Modify environment variables in `$CELEBORN_HOME/conf/celeborn-env.sh`
+1. Unzip the tarball to `$CELEBORN_HOME`.
+2. Modify environment variables in `$CELEBORN_HOME/conf/celeborn-env.sh`.
 
 EXAMPLE:
 ```properties
@@ -118,7 +118,7 @@ CELEBORN_MASTER_MEMORY=4g
 CELEBORN_WORKER_MEMORY=2g
 CELEBORN_WORKER_OFFHEAP_MEMORY=4g
 ```
-3. Modify configurations in `$CELEBORN_HOME/conf/celeborn-defaults.conf`
+3. Modify configurations in `$CELEBORN_HOME/conf/celeborn-defaults.conf`.
 
 EXAMPLE: single master cluster
 ```properties
@@ -152,7 +152,7 @@ celeborn.worker.replicate.fastFail.duration 240s
 celeborn.storage.hdfs.kerberos.principal user@REALM
 celeborn.storage.hdfs.kerberos.keytab /path/to/user.keytab
 
-# If your hosts have disk raid or use lvm, set 
celeborn.worker.monitor.disk.enabled to false
+# If your hosts have disk raid or use lvm, set 
`celeborn.worker.monitor.disk.enabled` to false
 celeborn.worker.monitor.disk.enabled false
 ```   
 
@@ -199,26 +199,24 @@ celeborn.worker.flusher.hdfs.buffer.size 4m
 celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
 celeborn.worker.replicate.fastFail.duration 240s
 
-# If your hosts have disk raid or use lvm, set 
celeborn.worker.monitor.disk.enabled to false
+# If your hosts have disk raid or use lvm, set 
`celeborn.worker.monitor.disk.enabled` to false
 celeborn.worker.monitor.disk.enabled false
 ```
 
 Flink engine related configurations:
 ```properties
-# if you are using Celeborn for flink, these settings will be needed
+# If you are using Celeborn for flink, these settings will be needed.
 celeborn.worker.directMemoryRatioForReadBuffer 0.4
-celeborn.worker.directMemoryRatioToResume 0.6
-# these setting will affect performance. 
+celeborn.worker.directMemoryRatioToResume 0.5
+# These setting will affect performance. 
 # If there is enough off-heap memory, you can try to increase read buffers.
 # Read buffer max memory usage for a data partition is 
`taskmanager.memory.segment-size * readBuffersMax`
 celeborn.worker.partition.initial.readBuffersMin 512
 celeborn.worker.partition.initial.readBuffersMax 1024
 celeborn.worker.readBuffer.allocationWait 10ms
-# Currently, shuffle partitionSplit is not supported, so you should disable 
split in celeborn worker side or set 
`celeborn.client.shuffle.partitionSplit.threshold` to a high value in flink 
client side.
-celeborn.worker.shuffle.partitionSplit.enabled false
 ```
 
-4. Copy Celeborn and configurations to all nodes
+4. Copy Celeborn and configurations to all nodes.
 5. Start all services. If you install Celeborn distribution in the same path 
on every node and your
    cluster can perform SSH login then you can fill `$CELEBORN_HOME/conf/hosts` 
and
    use `$CELEBORN_HOME/sbin/start-all.sh` to start all
@@ -251,14 +249,14 @@ WorkerRef: null
 Please refer to our 
[website](https://celeborn.apache.org/docs/latest/deploy_on_k8s/)
 
 ### Deploy Spark client
-Copy $CELEBORN_HOME/spark/*.jar to $SPARK_HOME/jars/
+Copy `$CELEBORN_HOME/spark/*.jar` to `$SPARK_HOME/jars/`.
 
 #### Spark Configuration
-To use Celeborn,the following spark configurations should be added.
+To use Celeborn, the following spark configurations should be added.
 ```properties
 # Shuffle manager class name changed in 0.3.0:
-#    before 0.3.0: org.apache.spark.shuffle.celeborn.RssShuffleManager
-#    since 0.3.0: org.apache.spark.shuffle.celeborn.SparkShuffleManager
+#    before 0.3.0: `org.apache.spark.shuffle.celeborn.RssShuffleManager`
+#    since 0.3.0: `org.apache.spark.shuffle.celeborn.SparkShuffleManager`
 spark.shuffle.manager org.apache.spark.shuffle.celeborn.SparkShuffleManager
 # must use kryo serializer because java serializer do not support relocation
 spark.serializer org.apache.spark.serializer.KryoSerializer
@@ -273,13 +271,13 @@ spark.shuffle.service.enabled false
 # Sort shuffle writer uses less memory than hash shuffle writer, if your 
shuffle partition count is large, try to use sort hash writer.  
 spark.celeborn.client.spark.shuffle.writer hash
 
-# We recommend setting spark.celeborn.client.push.replicate.enabled to true to 
enable server-side data replication
+# We recommend setting `spark.celeborn.client.push.replicate.enabled` to true 
to enable server-side data replication
 # If you have only one worker, this setting must be false 
 # If your Celeborn is using HDFS, it's recommended to set this setting to false
 spark.celeborn.client.push.replicate.enabled true
 
 # Support for Spark AQE only tested under Spark 3
-# we recommend setting localShuffleReader to false to get better performance 
of Celeborn
+# we recommend setting localShuffleReader to false for getting better 
performance of Celeborn
 spark.sql.adaptive.localShuffleReader.enabled false
 
 # If Celeborn is using HDFS
@@ -297,7 +295,7 @@ spark.dynamicAllocation.shuffleTracking.enabled false
 ```
 
 ### Deploy Flink client
-Copy $CELEBORN_HOME/flink/*.jar to $FLINK_HOME/lib/
+Copy `$CELEBORN_HOME/flink/*.jar` to `$FLINK_HOME/lib/`.
 
 #### Flink Configuration
 To use Celeborn, the following flink configurations should be added.
@@ -323,9 +321,9 @@ taskmanager.memory.task.off-heap.size: 512m
 ```
 **Note**: The config option `execution.batch-shuffle-mode` should configure as 
`ALL_EXCHANGES_BLOCKING`.
 
-### Deploy mapreduce client 
-Add $CELEBORN_HOME/mr/*.jar to to `mapreduce.application.classpath` and 
`yarn.application.classpath`.
-And setting the following settings in YARN and MapReduce config.
+### Deploy MapReduce client 
+Copy `$CELEBORN_HOME/mr/*.jar` into `mapreduce.application.classpath` and 
`yarn.application.classpath`.
+Meanwhile, configure the following settings in YARN and MapReduce config.
 ```bash
 -Dyarn.app.mapreduce.am.job.recovery.enable=false
 -Dmapreduce.job.reduce.slowstart.completedmaps=1
@@ -335,7 +333,6 @@ And setting the following settings in YARN and MapReduce 
config.
 
-Dmapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.CelebornShuffleConsumer
 ```
 
-
 ### Best Practice
 If you want to set up a production-ready Celeborn cluster, your cluster should 
have at least 3 masters and at least 4 workers.
 Masters and works can be deployed on the same node but should not deploy 
multiple masters or workers on the same node.
diff --git a/docs/README.md b/docs/README.md
index d618a8176..613431984 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -20,11 +20,11 @@ license: |
 ---
 Quick Start
 ===
-This documentation gives a quick start guide for running Apache Spark/Flink 
with Apache Celeborn™(Incubating).
+This documentation gives a quick start guide for running Apache 
Spark/Flink/MapReduce with Apache Celeborn™(Incubating).
 
 ### Download Celeborn
 Download the latest Celeborn binary from the [Downloading 
Page](https://celeborn.apache.org/download/).
-Decompress the binary and set `$CELEBORN_HOME`
+Decompress the binary and set `$CELEBORN_HOME`.
 ```shell
 tar -C <DST_DIR> -zxvf apache-celeborn-<VERSION>-bin.tgz
 export CELEBORN_HOME=<Decompressed path>
@@ -37,7 +37,7 @@ cd $CELEBORN_HOME/conf
 cp log4j2.xml.template log4j2.xml
 ```
 #### Configure Storage
-Configure the directory to store shuffle data, for example 
`$CELEBORN_HOME/shuffle`
+Configure the directory to store shuffle data, for example 
`$CELEBORN_HOME/shuffle`.
 ```shell
 cd $CELEBORN_HOME/conf
 echo "celeborn.worker.storage.dirs=$CELEBORN_HOME/shuffle" > 
celeborn-defaults.conf
@@ -160,11 +160,15 @@ INFO [async-reply] Controller: CommitFiles for 
local-1690000152711-0 success wit
 ```
 
 ## Start MapReduce With Celeborn
-### Add Celeborn client jar to MapReduce's classpath
-1.Add $CELEBORN_HOME/mr/*.jar to `mapreduce.application.classpath` and 
`yarn.application.classpath`.
-2.Restart your yarn cluster.
-### Add Celeborn configurations to MapReduce's conf
-Modify `${HADOOP_CONF_DIR}/yarn-site.xml`
+### Copy Celeborn Client to MapReduce's classpath
+1. Copy `$CELEBORN_HOME/mr/*.jar` into `mapreduce.application.classpath` and 
`yarn.application.classpath`.
+```shell
+cp $CELEBORN_HOME/mr/<Celeborn Client Jar> <mapreduce.application.classpath>
+cp $CELEBORN_HOME/mr/<Celeborn Client Jar> <yarn.application.classpath>
+```
+2. Restart your yarn cluster.
+### Add Celeborn configuration to MapReduce's conf
+- Modify configurations in `${HADOOP_CONF_DIR}/yarn-site.xml`.
 ```xml
 <configuration>
     <property>
@@ -179,7 +183,7 @@ Modify `${HADOOP_CONF_DIR}/yarn-site.xml`
     </property>
 </configuration>
 ```
-Modify `${HADOOP_CONF_DIR}/mapred-site.xml`
+- Modify configurations in `${HADOOP_CONF_DIR}/mapred-site.xml`.
 ```xml
 <configuration>
     <property>
@@ -201,10 +205,11 @@ Modify `${HADOOP_CONF_DIR}/mapred-site.xml`
     </property>
 </configuration>
 ```
-Then you can run a word count to check whether your configs are correct.
+Then deploy the example word count to the running cluster for verifying 
whether above configurations are correct.
 ```shell
 cd $HADOOP_HOME
-hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar 
wordcount /sometext /someoutput
+
+./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar 
wordcount /someinput /someoutput
 ```
 During the MapReduce Job, you should see the following message in Celeborn 
Master's log:
 ```log
diff --git a/docs/celeborn_ratis_shell.md b/docs/celeborn_ratis_shell.md
index d1ffb3c03..6576a51b3 100644
--- a/docs/celeborn_ratis_shell.md
+++ b/docs/celeborn_ratis_shell.md
@@ -28,9 +28,9 @@ then it's convenient for Celeborn Admin to operate the master 
ratis service.
 
 ## Availability
 |  Version  | Available in src tarball? | Available in bin tarball? |
-|:---------:| :-----------------------: |:-------------------------:|
-|  < 0.3.0  | No                        |            No             |
-| \>= 0.3.0 | Yes                       |            Yes            |
+|:---------:|:-------------------------:|:-------------------------:|
+|  < 0.3.0  |            No             |            No             |
+| \>= 0.3.0 |            Yes            |            Yes            |
 
 ## Setting up the Celeborn ratis-shell
 
diff --git a/docs/deploy.md b/docs/deploy.md
index 445b76fa1..b6acd66d6 100644
--- a/docs/deploy.md
+++ b/docs/deploy.md
@@ -18,8 +18,8 @@ license: |
 
 # Deploy Celeborn
 
-1. Unzip the tarball to `$CELEBORN_HOME`
-2. Modify environment variables in `$CELEBORN_HOME/conf/celeborn-env.sh`
+1. Unzip the tarball to `$CELEBORN_HOME`.
+2. Modify environment variables in `$CELEBORN_HOME/conf/celeborn-env.sh`.
 
 EXAMPLE:
 ```properties
@@ -28,7 +28,7 @@ CELEBORN_MASTER_MEMORY=4g
 CELEBORN_WORKER_MEMORY=2g
 CELEBORN_WORKER_OFFHEAP_MEMORY=4g
 ```
-3. Modify configurations in `$CELEBORN_HOME/conf/celeborn-defaults.conf`
+3. Modify configurations in `$CELEBORN_HOME/conf/celeborn-defaults.conf`.
 
 EXAMPLE: single master cluster
 ```properties
@@ -44,7 +44,7 @@ celeborn.worker.flusher.buffer.size 256k
 
 # If Celeborn workers have local disks and HDFS. Following configs should be 
added.
 # If Celeborn workers have local disks, use following config.
-# Disk type is HDD by defaut.
+# Disk type is HDD by default.
 celeborn.worker.storage.dirs /mnt/disk1:disktype=SSD,/mnt/disk2:disktype=SSD
 
 # If Celeborn workers don't have local disks. You can use HDFS.
@@ -62,7 +62,7 @@ celeborn.worker.replicate.fastFail.duration 240s
 celeborn.storage.hdfs.kerberos.principal user@REALM
 celeborn.storage.hdfs.kerberos.keytab /path/to/user.keytab
 
-# If your hosts have disk raid or use lvm, set 
celeborn.worker.monitor.disk.enabled to false
+# If your hosts have disk raid or use lvm, set 
`celeborn.worker.monitor.disk.enabled` to false
 celeborn.worker.monitor.disk.enabled false
 ```   
 
@@ -109,25 +109,25 @@ celeborn.worker.flusher.hdfs.buffer.size 4m
 celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
 celeborn.worker.replicate.fastFail.duration 240s
 
-# If your hosts have disk raid or use lvm, set 
celeborn.worker.monitor.disk.enabled to false
+# If your hosts have disk raid or use lvm, set 
`celeborn.worker.monitor.disk.enabled` to false
 celeborn.worker.monitor.disk.enabled false
 ```
 
 Flink engine related configurations:
 ```properties
-# if you are using Celeborn for flink, these settings will be needed
+# If you are using Celeborn for flink, these settings will be needed.
 celeborn.worker.directMemoryRatioForReadBuffer 0.4
 celeborn.worker.directMemoryRatioToResume 0.5
-# these setting will affect performance. 
-# If there is enough off-heap memory you can try to increase read buffers.
+# These setting will affect performance. 
+# If there is enough off-heap memory, you can try to increase read buffers.
 # Read buffer max memory usage for a data partition is 
`taskmanager.memory.segment-size * readBuffersMax`
 celeborn.worker.partition.initial.readBuffersMin 512
 celeborn.worker.partition.initial.readBuffersMax 1024
 celeborn.worker.readBuffer.allocationWait 10ms
 ```
 
-4. Copy Celeborn and configurations to all nodes
-5. Start all services. If you install Celeborn distribution in same path on 
every node and your
+4. Copy Celeborn and configurations to all nodes.
+5. Start all services. If you install Celeborn distribution in the same path 
on every node and your
    cluster can perform SSH login then you can fill `$CELEBORN_HOME/conf/hosts` 
and
    use `$CELEBORN_HOME/sbin/start-all.sh` to start all
    services. If the installation paths are not identical, you will need to 
start service manually.  
@@ -135,7 +135,7 @@ celeborn.worker.readBuffer.allocationWait 10ms
    `$CELEBORN_HOME/sbin/start-master.sh`  
    Start Celeborn worker  
    `$CELEBORN_HOME/sbin/start-worker.sh`
-6. If Celeborn start success, the output of Master's log should be like this:
+6. If Celeborn starts success, the output of the Master's log should be like 
this:
 ```
 22/10/08 19:29:11,805 INFO [main] Dispatcher: Dispatcher numThreads: 64
 22/10/08 19:29:11,875 INFO [main] TransportClientFactory: mode NIO threads 64
@@ -156,14 +156,14 @@ WorkerRef: null
 ```
 
 ## Deploy Spark client
-Copy $CELEBORN_HOME/spark/*.jar to $SPARK_HOME/jars/
+Copy `$CELEBORN_HOME/spark/*.jar` to `$SPARK_HOME/jars/`.
 
 ### Spark Configuration
-To use Celeborn, following spark configurations should be added.
+To use Celeborn, the following spark configurations should be added.
 ```properties
 # Shuffle manager class name changed in 0.3.0:
-#    before 0.3.0: org.apache.spark.shuffle.celeborn.RssShuffleManager
-#    since 0.3.0: org.apache.spark.shuffle.celeborn.SparkShuffleManager
+#    before 0.3.0: `org.apache.spark.shuffle.celeborn.RssShuffleManager`
+#    since 0.3.0: `org.apache.spark.shuffle.celeborn.SparkShuffleManager`
 spark.shuffle.manager org.apache.spark.shuffle.celeborn.SparkShuffleManager
 # must use kryo serializer because java serializer do not support relocation
 spark.serializer org.apache.spark.serializer.KryoSerializer
@@ -178,13 +178,13 @@ spark.shuffle.service.enabled false
 # Sort shuffle writer uses less memory than hash shuffle writer, if your 
shuffle partition count is large, try to use sort hash writer.  
 spark.celeborn.client.spark.shuffle.writer hash
 
-# We recommend setting spark.celeborn.client.push.replicate.enabled to true to 
enable server-side data replication
+# We recommend setting `spark.celeborn.client.push.replicate.enabled` to true 
to enable server-side data replication
 # If you have only one worker, this setting must be false 
 # If your Celeborn is using HDFS, it's recommended to set this setting to false
 spark.celeborn.client.push.replicate.enabled true
 
 # Support for Spark AQE only tested under Spark 3
-# we recommend setting localShuffleReader to false to get better performance 
of Celeborn
+# we recommend setting localShuffleReader to false for getting better 
performance of Celeborn
 spark.sql.adaptive.localShuffleReader.enabled false
 
 # If Celeborn is using HDFS
@@ -202,10 +202,10 @@ spark.dynamicAllocation.shuffleTracking.enabled false
 ```
 
 ## Deploy Flink client
-Copy $CELEBORN_HOME/flink/*.jar to $FLINK_HOME/lib/
+Copy `$CELEBORN_HOME/flink/*.jar` to `$FLINK_HOME/lib/`.
 
 ### Flink Configuration
-To use Celeborn, following flink configurations should be added.
+To use Celeborn, the following flink configurations should be added.
 ```properties
 shuffle-service-factory.class: 
org.apache.celeborn.plugin.flink.RemoteShuffleServiceFactory
 execution.batch-shuffle-mode: ALL_EXCHANGES_BLOCKING
@@ -227,3 +227,15 @@ taskmanager.network.memory.buffers-per-channel: 0
 taskmanager.memory.task.off-heap.size: 512m
 ```
 **Note**: The config option `execution.batch-shuffle-mode` should configure as 
`ALL_EXCHANGES_BLOCKING`.
+
+## Deploy MapReduce client
+Copy `$CELEBORN_HOME/mr/*.jar` into `mapreduce.application.classpath` and 
`yarn.application.classpath`.
+Meanwhile, configure the following settings in YARN and MapReduce config.
+```bash
+-Dyarn.app.mapreduce.am.job.recovery.enable=false
+-Dmapreduce.job.reduce.slowstart.completedmaps=1
+-Dmapreduce.celeborn.master.endpoints=<master-1-1>:9097
+-Dyarn.app.mapreduce.am.command-opts=org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn
+-Dmapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.CelebornMapOutputCollector
+-Dmapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.CelebornShuffleConsumer
+```
diff --git a/docs/deploy_on_k8s.md b/docs/deploy_on_k8s.md
index 298979c5c..fa8ad91e6 100644
--- a/docs/deploy_on_k8s.md
+++ b/docs/deploy_on_k8s.md
@@ -131,7 +131,7 @@ 
celeborn-worker-0.celeborn-worker-svc.default.svc.cluster.local`
 After a restart, the StatefulSet Pod IP changes but the DNS name remains, this 
is important for rolling upgrade.
 
 When bind address is not set explicitly, Celeborn worker is going to find the 
first non-loopback address to bind. By default,
-it use IP address both for address binding and registering, that causes the 
Master and Client use the IP address to access the
+it uses IP address both for address binding and registering, that causes the 
Master and Client use the IP address to access the
 Worker, it's problematic after Worker restart as explained above, especially 
when Graceful Shutdown is enabled.
 
 You may want to set `celeborn.network.bind.preferIpAddress=false` to address 
such issue. Note that, depends on your Kubernetes
diff --git a/docs/developers/flink.md b/docs/developers/flink.md
deleted file mode 100644
index 28312b7c1..000000000
--- a/docs/developers/flink.md
+++ /dev/null
@@ -1,19 +0,0 @@
----
-license: |
-  Licensed to the Apache Software Foundation (ASF) under one or more
-  contributor license agreements.  See the NOTICE file distributed with
-  this work for additional information regarding copyright ownership.
-  The ASF licenses this file to You under the Apache License, Version 2.0
-  (the "License"); you may not use this file except in compliance with
-  the License.  You may obtain a copy of the License at
-
-      https://www.apache.org/licenses/LICENSE-2.0
-
-  Unless required by applicable law or agreed to in writing, software
-  distributed under the License is distributed on an "AS IS" BASIS,
-  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-  See the License for the specific language governing permissions and
-  limitations under the License.
----
-
-# Flink Plugin
\ No newline at end of file
diff --git a/docs/developers/glutensupport.md b/docs/developers/glutensupport.md
index f18bd9340..645c4af22 100644
--- a/docs/developers/glutensupport.md
+++ b/docs/developers/glutensupport.md
@@ -19,7 +19,7 @@ license: |
 # Gluten Support
 ## Velox Backend
 
-[Gluten](https://github.com/apache/incubator-gluten) with velox backend 
supports Celeborn as remote shuffle service. Below introduction is used to 
enable this feature
+[Gluten](https://github.com/apache/incubator-gluten) with velox backend 
supports Celeborn as remote shuffle service. Below introduction is used to 
enable this feature.
 
 First refer to [Get Started With 
Velox](https://github.com/apache/incubator-gluten/blob/main/docs/get-started/Velox.md)
 to build Gluten with velox backend.
 
@@ -31,10 +31,10 @@ mvn clean package -Pbackends-velox -Pspark-3.3 -Prss 
-DskipTests
 
 Then add the Gluten and Spark Celeborn Client packages to your Spark 
application's classpath(usually add them into `$SPARK_HOME/jars`).
 
-- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar
-- Gluten: gluten-velox-bundle-spark3.x_2.12-xx-xx-SNAPSHOT.jar, 
gluten-thirdparty-lib-xx.jar
+- Celeborn: `celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar`
+- Gluten: `gluten-velox-bundle-spark3.x_2.12-xx-xx-SNAPSHOT.jar`, 
`gluten-thirdparty-lib-xx.jar`
 
-Currently to use Gluten following configurations are required in 
`spark-defaults.conf`
+Currently, to use Gluten following configurations are required in 
`spark-defaults.conf`.
 
 ```
 spark.shuffle.manager 
org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
@@ -42,7 +42,7 @@ spark.shuffle.manager 
org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleMa
 # celeborn master
 spark.celeborn.master.endpoints clb-master:9097
 
-# we recommend set spark.celeborn.push.replicate.enabled to true to enable 
server-side data replication
+# we recommend set `spark.celeborn.push.replicate.enabled` to true to enable 
server-side data replication
 # If you have only one worker, this setting must be false 
 spark.celeborn.client.push.replicate.enabled true
 
diff --git a/docs/developers/integrate.md b/docs/developers/integrate.md
index d7f2882fb..22ec50df8 100644
--- a/docs/developers/integrate.md
+++ b/docs/developers/integrate.md
@@ -27,7 +27,7 @@ This article briefly describes an example of integrating 
Celeborn into a simple
 Celeborn `Client`.
 
 ## Background
-Say we have an distributed application who has two phases:
+Say we have a distributed application who has two phases:
 
 - Write phase that parallel tasks write data to some data service, each record 
is classified into some logical id,
   say partition id.
diff --git a/docs/developers/overview.md b/docs/developers/overview.md
index ca99ea94d..493453a37 100644
--- a/docs/developers/overview.md
+++ b/docs/developers/overview.md
@@ -89,8 +89,8 @@ Celeborn's primary components(i.e. Master, Worker, Client) 
are engine irrelevant
 and easy to implement plugins for various engines.
 
 Currently, Celeborn officially supports 
[Spark](https://spark.apache.org/)(both Spark 2.x and Spark 3.x),
-[Flink](https://flink.apache.org/)(1.14/1.15/1.17), and
-[Gluten](https://github.com/apache/incubator-gluten). Also developers are 
integrating Celeborn with other engines,
+[Flink](https://flink.apache.org/)(1.14/1.15/1.17/1.18/1.19), and
+[Gluten](https://github.com/apache/incubator-gluten). Also, developers are 
integrating Celeborn with other engines,
 for example [MR3](https://mr3docs.datamonad.com/docs/mr3/).
 
 Celeborn community is also working on integrating Celeborn with other engines.
@@ -100,11 +100,11 @@ In order not to impact running applications when 
upgrading Celeborn Cluster, Cel
 
 When graceful shutdown is turned on, upon shutdown, Celeborn will do the 
following things:
 
-1. Master will not allocate slots on the Worker
-2. Worker will inform Clients to split
-3. Client will send `CommitFiles` to the Worker
+1. Master will not allocate slots on the Worker.
+2. Worker will inform Clients to split.
+3. Client will send `CommitFiles` to the Worker.
 
-Then the Worker waits until all `PartitionLocation` flushes data to persistent 
storage, stores states in local leveldb,
+Then the Worker waits until all `PartitionLocation` flushes data to persistent 
storage, stores states in local leveldb/rocksdb,
 then stops itself. The process is typically within one minute.
 
 For more details, please refer to [Rolling 
upgrade](../../upgrading/#rolling-upgrade)
diff --git a/docs/developers/readdata.md b/docs/developers/readdata.md
deleted file mode 100644
index e69de29bb..000000000
diff --git a/docs/developers/sbt.md b/docs/developers/sbt.md
index f91fa0343..638e3cfa6 100644
--- a/docs/developers/sbt.md
+++ b/docs/developers/sbt.md
@@ -51,7 +51,7 @@ As an example, one can build a version of Celeborn as follows:
 ./build/sbt clean package
 ```
 
-To create a Celeborn distribution like those distributed by the [Celeborn 
Downloads](https://celeborn.apache.org/download/) page, and that is laid out so 
as to be runnable, use `./build/make-distribution.sh` in the project root 
directory.
+To create a Celeborn distribution like those distributed by the [Celeborn 
Downloads](https://celeborn.apache.org/download/) page, and that is laid out to 
be runnable, use `./build/make-distribution.sh` in the project root directory.
 
 ```
 ./build/make-distribution.sh --sbt-enabled --release
@@ -173,7 +173,7 @@ $ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
 
 When developing locally, it’s often convenient to run a single test or a few 
tests, rather than running the entire test suite.
 
-The fastest way to run individual tests is to use the sbt console. It’s 
fastest to keep a sbt console open, and use it to re-run tests as necessary. 
For example, to run all of the tests in a particular project, e.g., master:
+The fastest way to run individual tests is to use the sbt console. It’s 
fastest to keep a sbt console open, and use it to re-run tests as necessary. 
For example, to run all the tests in a particular project, e.g., master:
 
 ```
 $ ./build/sbt
@@ -190,7 +190,7 @@ The `testOnly` command accepts wildcards; e.g., you can 
also run the `SlotsAlloc
 ```
 > testOnly *SlotsAllocatorSuiteJ
 ```
-Or you could run all of the tests in the `master` package:
+Or you could run all the tests in the `master` package:
 
 ```
 > testOnly org.apache.celeborn.service.deploy.master.*
@@ -201,7 +201,7 @@ If you’d like to run just a single Java test in the 
`SlotsAllocatorSuiteJ`, e.
 > testOnly *SlotsAllocatorSuiteJ -- 
 > *SlotsAllocatorSuiteJ.testAllocateSlotsForSinglePartitionId
 ```
 
-If you’d like to run just a single Scala test in the 
`AppDiskUsageMetricSuite`, e.g., a test that incudes "app usage snapshot" in 
the name, you run the following command in the sbt console:
+If you’d like to run just a single Scala test in the 
`AppDiskUsageMetricSuite`, e.g., a test that includes "app usage snapshot" in 
the name, you run the following command in the sbt console:
 
 ```
 > testOnly *AppDiskUsageMetricSuite -- -z "app usage snapshot"
@@ -291,7 +291,7 @@ SBT supports publishing shade clients 
(Spark/Flink/MapReduce) to an internal Mav
 Before executing the publish command, ensure that the following environment 
variables are correctly set:
 
 | Environment Variable   | Description                                         
                                                                                
  |
-| ---------------------- | 
-------------------------------------------------------------------------------------------------------------------------------------
 |
+|------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
 | ASF_USERNAME           | Sonatype repository username                        
                                                                                
  |
 | ASF_PASSWORD           | Sonatype repository password                        
                                                                                
  |
 | SONATYPE_SNAPSHOTS_URL | Sonatype repository URL for snapshot version 
releases, default is 
"https://repository.apache.org/content/repositories/snapshots";      |
diff --git a/docs/developers/shuffleclient.md b/docs/developers/shuffleclient.md
index 0ad9e8257..b48179cad 100644
--- a/docs/developers/shuffleclient.md
+++ b/docs/developers/shuffleclient.md
@@ -65,7 +65,7 @@ responds to `LifecycleManager` with the allocated 
`PartitionLocation`s.
 In normal cases, the process of pushing data is as follows:
 
 - `ShuffleClient` compresses data, currently supports `zstd` and `lz4`
-- `ShuffleClient` addes Header for the data: `mapId`, `attemptId`, `batchId` 
and `size`. The `bastchId` is a unique
+- `ShuffleClient` adds Header for the data: `mapId`, `attemptId`, `batchId` 
and `size`. The `bastchId` is a unique
   id for the data batch inside the (`mapId`, `attemptId`), for the purpose of 
de-duplication
 - `ShuffleClient` sends `PushData` to the `Worker` on which the current 
`PartitionLocation` is allocated, and holds push
   state for this pushing
diff --git a/docs/developers/spark.md b/docs/developers/spark.md
deleted file mode 100644
index 6dd0bfc30..000000000
--- a/docs/developers/spark.md
+++ /dev/null
@@ -1,19 +0,0 @@
----
-license: |
-  Licensed to the Apache Software Foundation (ASF) under one or more
-  contributor license agreements.  See the NOTICE file distributed with
-  this work for additional information regarding copyright ownership.
-  The ASF licenses this file to You under the Apache License, Version 2.0
-  (the "License"); you may not use this file except in compliance with
-  the License.  You may obtain a copy of the License at
-
-      https://www.apache.org/licenses/LICENSE-2.0
-
-  Unless required by applicable law or agreed to in writing, software
-  distributed under the License is distributed on an "AS IS" BASIS,
-  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-  See the License for the specific language governing permissions and
-  limitations under the License.
----
-
-# Spark Plugin
\ No newline at end of file
diff --git a/docs/developers/storage.md b/docs/developers/storage.md
index e27b7452a..851e53d46 100644
--- a/docs/developers/storage.md
+++ b/docs/developers/storage.md
@@ -58,8 +58,6 @@ The layout of `MapPartition` is as follows:
 Each region has an in-memory index which points to start positions of each 
partition. Upon requesting data from
 some partition, `Worker` reads the partition data from every region.
 
-For more details about reading data, please refer to 
[ReadData](../../developers/readdata).
-
 ## Local Disk and Memory Buffer
 To the time this article is written, the most common case is local disk only. 
Users specify directories and
 capacity that Celeborn can use to store data. It is recommended to specify one 
directory per disk. If users
diff --git a/docs/quota_management.md b/docs/quota_management.md
index 078fa5786..b4a36c97a 100644
--- a/docs/quota_management.md
+++ b/docs/quota_management.md
@@ -52,7 +52,7 @@ Users can also implement their own identity provider by 
inheriting the `org.apac
 ## QuotaManager
 
 `QuotaManager` supports to check whether quota is available and manage quota 
configurations for `Master`.
-`QuotaManager` uses the [dynamic config 
service](developers/configuration.md#dynamic-configuration) to store quota 
settings.
+`QuotaManager` uses the [dynamic config 
service](developers/configuration#dynamic-configuration) to store quota 
settings.
 For example, there are some quota configurations as follows:
 
 The quota for user `tenant_01.Jerry` is
@@ -78,8 +78,8 @@ The quota for `system default` is
 
 ### FileSystem Store Backend
 
-This backend reads [quota](#Quota Indicators) settings from a user-specified 
dynamic config file.
-For more information on using the database store backend, refer to [filesystem 
config service](developers/configuration.md#filesystem-config-service).
+This backend reads [quota](#quota-indicators) settings from a user-specified 
dynamic config file.
+For more information on using the database store backend, refer to [filesystem 
config service](developers/configuration#filesystem-config-service).
 Here's an example quota setting YAML file of above quota examples:
 
 ```yaml
@@ -104,8 +104,8 @@ Here's an example quota setting YAML file of above quota 
examples:
 
 ### Database Store Backend
 
-This backend reads [quota](#Quota Indicators) settings from a user-specified 
database.
-For more information on using the database store backend, refer to [database 
config service](developers/configuration.md#database-config-service).
+This backend reads [quota](#quota-indicators) settings from a user-specified 
database.
+For more information on using the database store backend, refer to [database 
config service](developers/configuration#database-config-service).
 Here's an example quota setting sql of above quota examples:
 ```sql
 # SYSTEM level configuration

Reply via email to