This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 516bdc7e0 [CELEBORN-877][DOC] Document on SBT
516bdc7e0 is described below
commit 516bdc7e08f0d7cb2cde943b853394dda6592de9
Author: Fu Chen <[email protected]>
AuthorDate: Fri Aug 11 12:17:55 2023 +0800
[CELEBORN-877][DOC] Document on SBT
### What changes were proposed in this pull request?
As title
### Why are the changes needed?
As title
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manual test
Closes #1795 from cfmcgrady/sbt-docs.
Authored-by: Fu Chen <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
build/sbt-config/repositories | 17 ++
build/sbt-config/repositories-asia.template | 32 +++
build/sbt-config/repositories-cn.template | 17 ++
.../org/apache/celeborn/ConfigurationSuite.scala | 4 +-
docs/README.md | 4 +-
docs/celeborn_ratis_shell.md | 5 +-
docs/cluster_planning.md | 8 +-
docs/configuration/client.md | 4 +-
docs/configuration/columnar-shuffle.md | 4 +-
docs/configuration/ha.md | 4 +-
docs/configuration/index.md | 13 +-
docs/configuration/master.md | 4 +-
docs/configuration/metrics.md | 4 +-
docs/configuration/network.md | 4 +-
docs/configuration/quota.md | 4 +-
docs/configuration/worker.md | 4 +-
docs/deploy.md | 4 +-
docs/deploy_on_k8s.md | 4 +-
docs/developers/overview.md | 6 +-
docs/developers/sbt.md | 279 +++++++++++++++++++++
docs/developers/trafficcontrol.md | 6 +-
docs/developers/worker.md | 6 +-
docs/migration.md | 4 +-
docs/monitoring.md | 11 +-
docs/upgrading.md | 5 +-
mkdocs.yml | 1 +
26 files changed, 413 insertions(+), 45 deletions(-)
diff --git a/build/sbt-config/repositories b/build/sbt-config/repositories
index 55e1424cf..713581a32 100644
--- a/build/sbt-config/repositories
+++ b/build/sbt-config/repositories
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
[repositories]
local
mavenLocal: file://${user.home}/.m2/repository/
diff --git a/build/sbt-config/repositories-asia.template
b/build/sbt-config/repositories-asia.template
new file mode 100644
index 000000000..ae201139d
--- /dev/null
+++ b/build/sbt-config/repositories-asia.template
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# you can accelerate the download speed of bootstrap/plugin/dependencies jar
+# packages by executing the following command to configure the mirror:
+#
+# ```
+# cp build/sbt-config/repositories-asia.template
build/sbt-config/repositories-local
+# ```
+
+[repositories]
+ local
+ mavenLocal: file://${user.home}/.m2/repository/
+ # The system property value of `celeborn.sbt.default.artifact.repository` is
+ # fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
+ # assigned within the build/sbt-launch-lib.bash script.
+ private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
+ gcs: https://maven-central-asia.storage-download.googleapis.com/maven2/
diff --git a/build/sbt-config/repositories-cn.template
b/build/sbt-config/repositories-cn.template
index 63ee64251..eb75ff217 100644
--- a/build/sbt-config/repositories-cn.template
+++ b/build/sbt-config/repositories-cn.template
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
# As a Chinese developer facing network issues, you can accelerate the download
# speed of bootstrap/plugin/dependencies jar packages by executing the
# following command to configure the mirror:
diff --git a/common/src/test/scala/org/apache/celeborn/ConfigurationSuite.scala
b/common/src/test/scala/org/apache/celeborn/ConfigurationSuite.scala
index e5cd97366..1b7240fbc 100644
--- a/common/src/test/scala/org/apache/celeborn/ConfigurationSuite.scala
+++ b/common/src/test/scala/org/apache/celeborn/ConfigurationSuite.scala
@@ -125,9 +125,9 @@ class ConfigurationSuite extends AnyFunSuite {
output += " The ASF licenses this file to You under the Apache License,
Version 2.0"
output += " (the \"License\"); you may not use this file except in
compliance with"
output += " the License. You may obtain a copy of the License at"
- output += " "
+ output += ""
output += " https://www.apache.org/licenses/LICENSE-2.0";
- output += " "
+ output += ""
output += " Unless required by applicable law or agreed to in writing,
software"
output += " distributed under the License is distributed on an \"AS IS\"
BASIS,"
output += " WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
or implied."
diff --git a/docs/README.md b/docs/README.md
index 16aa57da5..c260ebca2 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -9,7 +9,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/celeborn_ratis_shell.md b/docs/celeborn_ratis_shell.md
index b7086bb90..d1ffb3c03 100644
--- a/docs/celeborn_ratis_shell.md
+++ b/docs/celeborn_ratis_shell.md
@@ -6,7 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -14,7 +16,6 @@ license: |
limitations under the License.
---
-
# Celeborn Ratis-shell
[Ratis-shell](https://github.com/apache/ratis/blob/master/ratis-docs/src/site/markdown/cli.md)
is the command line interface of Ratis.
diff --git a/docs/cluster_planning.md b/docs/cluster_planning.md
index c7806b7e3..d7f458e96 100644
--- a/docs/cluster_planning.md
+++ b/docs/cluster_planning.md
@@ -1,12 +1,14 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
+ contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/configuration/client.md b/docs/configuration/client.md
index bedc97ad5..1820bc9c3 100644
--- a/docs/configuration/client.md
+++ b/docs/configuration/client.md
@@ -6,9 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
https://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/configuration/columnar-shuffle.md
b/docs/configuration/columnar-shuffle.md
index c3581605d..65d60f0a3 100644
--- a/docs/configuration/columnar-shuffle.md
+++ b/docs/configuration/columnar-shuffle.md
@@ -6,9 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
https://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/configuration/ha.md b/docs/configuration/ha.md
index c7f41af79..494537ded 100644
--- a/docs/configuration/ha.md
+++ b/docs/configuration/ha.md
@@ -6,9 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
https://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index 2ee39c40a..a2166b9fa 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -3,12 +3,15 @@ hide:
- navigation
license: |
- Licensed under the Apache License, Version 2.0 (the "License");
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
https://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/configuration/master.md b/docs/configuration/master.md
index a9bdbb031..59baad7c6 100644
--- a/docs/configuration/master.md
+++ b/docs/configuration/master.md
@@ -6,9 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
https://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/configuration/metrics.md b/docs/configuration/metrics.md
index 817e03572..4847879a0 100644
--- a/docs/configuration/metrics.md
+++ b/docs/configuration/metrics.md
@@ -6,9 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
https://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/configuration/network.md b/docs/configuration/network.md
index e1001ee9b..2d28b81f7 100644
--- a/docs/configuration/network.md
+++ b/docs/configuration/network.md
@@ -6,9 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
https://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/configuration/quota.md b/docs/configuration/quota.md
index 5537de934..565dde953 100644
--- a/docs/configuration/quota.md
+++ b/docs/configuration/quota.md
@@ -6,9 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
https://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/configuration/worker.md b/docs/configuration/worker.md
index e724fa2aa..f846268cd 100644
--- a/docs/configuration/worker.md
+++ b/docs/configuration/worker.md
@@ -6,9 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
-
+
https://www.apache.org/licenses/LICENSE-2.0
-
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/deploy.md b/docs/deploy.md
index 4f93bb948..cba476336 100644
--- a/docs/deploy.md
+++ b/docs/deploy.md
@@ -6,7 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/deploy_on_k8s.md b/docs/deploy_on_k8s.md
index 2e1ad36a7..298979c5c 100644
--- a/docs/deploy_on_k8s.md
+++ b/docs/deploy_on_k8s.md
@@ -6,7 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/developers/overview.md b/docs/developers/overview.md
index 832c502c7..70cf2dc9f 100644
--- a/docs/developers/overview.md
+++ b/docs/developers/overview.md
@@ -6,7 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -105,4 +107,4 @@ When graceful shutdown is turned on, upon shutdown,
Celeborn will do the followi
Then the Worker waits until all `PartitionLocation` flushes data to persistent
storage, stores states in local leveldb,
then stops itself. The process is typically within one minute.
-For more details, please refer to [Rolling upgrade](/upgrade)
\ No newline at end of file
+For more details, please refer to [Rolling upgrade](/upgrade)
diff --git a/docs/developers/sbt.md b/docs/developers/sbt.md
new file mode 100644
index 000000000..4ca28d0c1
--- /dev/null
+++ b/docs/developers/sbt.md
@@ -0,0 +1,279 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Building via SBT
+
+Starting from version 0.4.0, the Celeborn project supports building and
packaging using SBT. This article provides a detailed guide on how to build the
Celeborn project using SBT.
+
+## System Requirements
+
+Celeborn Service (master/worker) supports Scala 2.11/2.12 and Java 8/11/17.
+
+The following table indicates the compatibility of Celeborn Spark and Flink
clients with different versions of Spark and Flink for various Java and Scala
versions:
+
+| | Java 8/Scala 2.11 | Java 8/Scala 2.12 | Java 11/Scala 2.12
| Java 17/Scala 2.12 |
+|----------------|-------------------|-------------------|--------------------|--------------------|
+| Spark 2.4 | ✔ | ❌ | ❌
| ❌ |
+| Spark 3.0 | ❌ | ✔ | ✔
| ❌ |
+| Spark 3.1 | ❌ | ✔ | ✔
| ❌ |
+| Spark 3.2 | ❌ | ✔ | ✔
| ❌ |
+| Spark 3.3 | ❌ | ✔ | ✔
| ✔ |
+| Spark 3.4 | ❌ | ✔ | ✔
| ✔ |
+| Flink 1.14 | ❌ | ✔ | ✔
| ❌ |
+| Flink 1.15 | ❌ | ✔ | ✔
| ❌ |
+| Flink 1.17 | ❌ | ✔ | ✔
| ❌ |
+
+## Useful SBT commands
+
+### Packaging the Project
+
+As an example, one can build a version of Celeborn as follows:
+
+```
+./build/sbt clean package
+```
+
+To create a Celeborn distribution like those distributed by the [Celeborn
Downloads](https://celeborn.apache.org/download/) page, and that is laid out so
as to be runnable, use `./build/make-distribution.sh` in the project root
directory.
+
+```
+./build/make-distribution.sh --sbt-enabled
+```
+
+### Maven-Style Profile Management
+
+We have adopted the Maven-style profile management for our Client module. For
example, you can enable the Spark 3.3 client module by adding `-Pspark-3.3`:
+
+```
+# ./build/sbt -Pspark-3.3 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-spark-3
+[info] celeborn-client-spark-3-shaded
+[info] celeborn-common
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-spark-common
+[info] celeborn-spark-group
+[info] celeborn-spark-it
+[info] celeborn-worker
+```
+
+To enable the Flink 1.15 client module, add `-Pflink-1.15`:
+
+```
+# ./build/sbt -Pflink-1.15 projects
+
+[info] set current project to celeborn (in build file:/root/celeborn/)
+[info] In file:/root/celeborn/
+[info] * celeborn
+[info] celeborn-client
+[info] celeborn-client-flink-1_15
+[info] celeborn-client-flink-1_15-shaded
+[info] celeborn-common
+[info] celeborn-flink-common
+[info] celeborn-flink-group
+[info] celeborn-flink-it
+[info] celeborn-master
+[info] celeborn-service
+[info] celeborn-worker
+```
+
+By using these profiles, you can easily switch between different client
modules for Spark and Flink. These profiles enable specific dependencies and
configurations relevant to the chosen version. This way, you can conveniently
manage and build the desired configurations of the Celeborn project.
+
+### Building Spark/Flink Assembly Client Jars
+
+For example, you can build the Spark 3.3 client assembly jar by running the
following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-spark-3-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pspark-3.3` profile:
+$ ./build/sbt -Pspark-3.3 celeborn-client-spark-3-shaded/assembly
+```
+
+Similarly, you can build the Flink 1.15 client assembly jar using the
following commands:
+
+```shell
+$ ./build/sbt
+> project celeborn-client-flink-1_15-shaded
+> assembly
+
+$ # Or, you can use sbt directly with the `-Pflink-1.15` profile:
+$ ./build/sbt -Pflink-1.15 celeborn-client-flink-1_15-shaded/assembly
+```
+
+By executing these commands, you will create assembly jar files for the
respective Spark and Flink client modules. The assembly jar bundles all the
dependencies, allowing the client module to be used independently with all
required dependencies included.
+
+### Building submodules individually
+
+For instance, you can build the Celeborn Master module using:
+
+```
+$ # sbt
+$ ./build/sbt
+> project celeborn-master
+> package
+
+$ # Or, you can build the celeborn-master module with sbt directly using:
+$ ./build/sbt celeborn-master/package
+```
+
+## Testing with SBT
+
+To run all tests for the Celeborn project, you can use the following command:
+
+```shell
+./build/sbt test
+```
+
+Running tests for specific versions of Spark/Flink client.
+
+For example, to run the test cases for the Spark 3.3 client, use the following
command:
+
+```shell
+$ ./build/sbt -Pspark-3.3 test
+
+$ # only run spark client related modules tests
+$ ./build/sbt -Pspark-3.3 celeborn-spark-group/test
+```
+
+Similarly, to run the test cases for the Flink 1.15 client, use the following
command:
+
+```shell
+$ ./build/sbt -Pflink-1.15 test
+
+$ # only run flink client related modules tests
+$ ./build/sbt -Pflink-1.15 celeborn-flink-group/test
+```
+
+### Running Individual Tests
+
+When developing locally, it’s often convenient to run a single test or a few
tests, rather than running the entire test suite.
+
+The fastest way to run individual tests is to use the sbt console. It’s
fastest to keep a sbt console open, and use it to re-run tests as necessary.
For example, to run all of the tests in a particular project, e.g., master:
+
+```
+$ ./build/sbt
+> project celeborn-master
+> test
+```
+You can run a single test suite using the `testOnly` command. For example, to
run the `SlotsAllocatorSuiteJ`:
+
+```
+> testOnly org.apache.celeborn.service.deploy.master.SlotsAllocatorSuiteJ
+```
+The `testOnly` command accepts wildcards; e.g., you can also run the
`SlotsAllocatorSuiteJ` with:
+
+```
+> testOnly *SlotsAllocatorSuiteJ
+```
+Or you could run all of the tests in the `master` package:
+
+```
+> testOnly org.apache.celeborn.service.deploy.master.*
+```
+If you’d like to run just a single Java test in the `SlotsAllocatorSuiteJ`,
e.g., a test that with the name `testAllocateSlotsForSinglePartitionId`, you
run the following command in the sbt console:
+
+```
+> testOnly *SlotsAllocatorSuiteJ --
*SlotsAllocatorSuiteJ.testAllocateSlotsForSinglePartitionId
+```
+
+If you’d like to run just a single Scala test in the
`AppDiskUsageMetricSuite`, e.g., a test that incudes "app usage snapshot" in
the name, you run the following command in the sbt console:
+
+```
+> testOnly *AppDiskUsageMetricSuite -- -z "app usage snapshot"
+```
+
+If you’d prefer, you can run all of these commands on the command line (but
this will be slower than running tests using an open console). To do this, you
need to surround `testOnly` and the following arguments in quotes:
+
+```
+$ ./build/sbt "celeborn-master/testOnly *AppDiskUsageMetricSuite -- -z \"app
usage snapshot\""
+```
+For more about how to run individual tests with sbt, see the [sbt
documentation](https://www.scala-sbt.org/1.x/docs/Testing.html) and [JUnit
Interface](https://github.com/sbt/junit-interface/#junit-interface).
+
+## Accelerating SBT
+
+This section provides instructions on setting up repository mirrors or proxies
for a smoother SBT experience. Depending on your location and network
conditions, you can choose the appropriate approach to accelerate SBT startup
and enhance dependency retrieval.
+
+### Accelerating SBT Startup
+
+The SBT startup process involves fetching the SBT bootstrap jar, which is
typically obtained from the Maven Central Repository
(https://repo1.maven.org/maven2/). If you encounter slow access to this
repository or if it's inaccessible in your network environment, you can
expedite the SBT startup by configuring a custom artifact repository using the
`DEFAULT_ARTIFACT_REPOSITORY` environment variable.
+
+
+```shell
+$ # The following command fetches sbt-launch-x.y.z.jar from
https://maven.aliyun.com/nexus/content/groups/public/
+$ # Ensure that the URL ends with a trailing slash "/"
+$ export
DEFAULT_ARTIFACT_REPOSITORY=https://maven.aliyun.com/nexus/content/groups/public/
+$ ./build/sbt
+```
+
+This will initiate SBT using the specified repository, allowing for faster
download and startup times.
+
+### Custom SBT Repositories
+
+The current repositories embedded within the Celeborn project are detailed
below:
+
+```
+[repositories]
+ local
+ mavenLocal: file://${user.home}/.m2/repository/
+ local-preloaded-ivy:
file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/},
[organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
+ local-preloaded:
file:///${sbt.preloaded-${sbt.global.base-${user.home}/.sbt}/preloaded/}
+ # The system property value of `celeborn.sbt.default.artifact.repository` is
+ # fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
+ # assigned within the build/sbt-launch-lib.bash script.
+ private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
+ gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/repos/central/data/
+ maven-central
+ typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/,
[organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext],
bootOnly
+ sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/,
[organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext],
bootOnly
+ sbt-plugin-releases:
https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/,
[organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-typesafe-sbt-plugin-releases:
https://dl.bintray.com/typesafe/sbt-plugins/,
[organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
+ bintray-spark-packages: https://dl.bintray.com/spark-packages/maven/
+ typesafe-releases: https://repo.typesafe.com/typesafe/releases/
+```
+
+For numerous developers across various regions, the default repository
download speeds are less than optimal. To address this concern, we have curated
a selection of verified public mirror templates tailored for specific regions
with a significant local developer presence. For instance, we provide the
`repositories-cn.template` template for developers situated within the expanse
of the Chinese mainland, and the `repositories-asia.template` template designed
for developers across the Asia [...]
+
+```
+cp build/sbt-config/repositories-cn.template
build/sbt-config/repositories-local
+```
+
+Furthermore, it is strongly encouraged that developers from various regions
contribute templates tailored to their respective areas.
+
+??? note
+ 1. `build/sbt-config/repositories-local` takes precedence over
`build/sbt-config/repositories` and is ignored by `.gitignore`.
+ 2. Should the environment variable `DEFAULT_ARTIFACT_REPOSITORY` be set,
it attains the highest priority among non-local repositories.
+ 3. Repository priority is determined by the file order; repositories
listed earlier possess higher precedence.
+
+Similarly, if your objective involves compiling and packaging within an
intranet environment, you can edit `build/sbt-config/repositories-local` as
demonstrated below:
+
+```
+[repositories]
+ local
+ mavenLocal: file://${user.home}/.m2/repository/
+ private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
+ private-central: https://example.com/repository/maven/
+```
+
+For more details on sbt repository configuration, please refer to the [SBT
documentation](https://www.scala-sbt.org/1.x/docs/Proxy-Repositories.html).
diff --git a/docs/developers/trafficcontrol.md
b/docs/developers/trafficcontrol.md
index d824fee5a..5f7a34e72 100644
--- a/docs/developers/trafficcontrol.md
+++ b/docs/developers/trafficcontrol.md
@@ -6,7 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -77,4 +79,4 @@ to `Slow Start` phase.
`Congestion Control` can be enabled and tuned by the following configurations:
-- `celeborn.worker.congestionControl.*`
\ No newline at end of file
+- `celeborn.worker.congestionControl.*`
diff --git a/docs/developers/worker.md b/docs/developers/worker.md
index 154e09205..284f30fc5 100644
--- a/docs/developers/worker.md
+++ b/docs/developers/worker.md
@@ -6,7 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -29,4 +31,4 @@ Celeborn `Worker` has four dedicated servers:
- `Controller` handles control messages, i.e. `ReserveSlots`, `CommitFiles`,
and `DestroyWorkerSlots`
- `Push Server` handles primary input data, i.e. `PushData` and
`PushMergedData`, and push related control messages
- `Replicate Server` handles replica input data, it has the same logic with
`Push Server`
-- `Fetch Server` handles fetch requests, i.e. `ChunkFetchRequest`, and fetch
related control messages
\ No newline at end of file
+- `Fetch Server` handles fetch requests, i.e. `ChunkFetchRequest`, and fetch
related control messages
diff --git a/docs/migration.md b/docs/migration.md
index 3b4f91faa..96632b804 100644
--- a/docs/migration.md
+++ b/docs/migration.md
@@ -9,7 +9,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
diff --git a/docs/monitoring.md b/docs/monitoring.md
index c0797a70b..cad97b4fa 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -1,12 +1,14 @@
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
+ contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -14,7 +16,6 @@ license: |
limitations under the License.
---
-
Monitoring
===
@@ -313,4 +314,4 @@ API path listed as below:
| /unavailablePeers | List the unavailable peers of the worker, this
always means the worker connect to the peer failed. |
| /isShutdown | Show if the worker is during the process of
shutdown. |
| /isRegistered | Show if the worker is registered to the master
success. |
-| /decommission | Trigger this worker to decommission from the
cluster |
\ No newline at end of file
+| /decommission | Trigger this worker to decommission from the
cluster |
diff --git a/docs/upgrading.md b/docs/upgrading.md
index 7504476c2..9c7e6cb28 100644
--- a/docs/upgrading.md
+++ b/docs/upgrading.md
@@ -6,7 +6,9 @@ license: |
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -14,7 +16,6 @@ license: |
limitations under the License.
---
-
Upgrading
===
diff --git a/mkdocs.yml b/mkdocs.yml
index b73e78110..eaf1e398b 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -95,4 +95,5 @@ nav:
- Fault Tolerant: developers/faulttolerant.md
- Worker Exclusion: developers/workerexclusion.md
- Integrating Celeborn: developers/integrate.md
+ - SBT Build: developers/sbt.md
- Gluten Support: developers/glutensupport.md
