This is an automated email from the ASF dual-hosted git repository.
angerszhuuuu pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new bd7c2ea35 [CELEBORN-746][BUILD] Rename project files from rss-xx to
celeborn-xx
bd7c2ea35 is described below
commit bd7c2ea35a88318f7356e85aa285707aaa6cf32c
Author: Angerszhuuuu <[email protected]>
AuthorDate: Thu Jun 29 16:30:02 2023 +0800
[CELEBORN-746][BUILD] Rename project files from rss-xx to celeborn-xx
### What changes were proposed in this pull request?
Rename project files from rss-xx to celeborn-xx
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
Closes #1660 from AngersZhuuuu/CELEBORN-746.
Lead-authored-by: Angerszhuuuu <[email protected]>
Co-authored-by: Cheng Pan <[email protected]>
Signed-off-by: Angerszhuuuu <[email protected]>
---
.gitignore | 6 ------
METRICS.md | 4 ++--
README.md | 6 +++---
assets/diagram/celeborn.drawio | 1 +
..._balance.drawio => celeborn_load_balance.drawio} | 0
assets/diagram/emr_shuffle.drawio | 1 -
assets/diagram/rss.drawio | 1 -
assets/diagram/rss_fault_tolerance.drawio | 1 -
assets/diagram/shuffle-procedure.drawio | 2 +-
assets/diagram/spark_shuffle.drawio | 1 -
.../{rss-dashboard.json => celeborn-dashboard.json} | 0
.../adaptive_query_execution_coalesce_partition.jpg | Bin 197120 -> 0 bytes
.../adaptive_query_execution_local_shuffle_read.jpg | Bin 194121 -> 0 bytes
assets/img/celeborn.jpg | Bin 0 -> 101739 bytes
...s_load_balance.jpg => celeborn_load_balance.jpg} | Bin
assets/img/emr_shuffle.jpg | Bin 124233 -> 0 bytes
assets/img/rss.jpg | Bin 68263 -> 0 bytes
assets/img/rss_fault_tolerance.jpg | Bin 1360776 -> 0 bytes
assets/img/shuffle-procedure.jpg | Bin 50939 -> 59048 bytes
assets/img/spark_shuffle.jpg | Bin 61434 -> 0 bytes
20 files changed, 7 insertions(+), 16 deletions(-)
diff --git a/.gitignore b/.gitignore
index eeba8b376..6b756767d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -58,13 +58,7 @@ reports/
scalastyle-on-compile.generated.xml
scalastyle-output.xml
scalastyle.txt
-rss-*.tgz
-rss-*-bin*
-emr-*.tgz
-emr-*-bin*
-celeborn-*.tgz
apache-celeborn-*.tgz
-celeborn-*-bin*
apache-celeborn-*-bin*
src_managed/
target/
diff --git a/METRICS.md b/METRICS.md
index 1ddcb85ce..041f6b51b 100644
--- a/METRICS.md
+++ b/METRICS.md
@@ -27,7 +27,7 @@ scrape_configs:
3.You need to install Grafana server(https://grafana.com/grafana/download)
4.Import Celeborn dashboard into grafana.
-You can find Celeborn dashboard at assets/grafana/rss-dashboard.json.
+You can find Celeborn dashboard at assets/grafana/celeborn-dashboard.json.
### Optional
@@ -100,7 +100,7 @@ Celeborn worker metric :
`org/apache/celeborn/service/deploy/worker/WorkerSource
## Grafana Dashboard
-We provide a grafana dashboard for Celeborn
[Grafana-Dashboard](assets/grafana/rss-dashboard.json). The dashboard was
generated by grafana of version 8.5.0.
+We provide a grafana dashboard for Celeborn
[Grafana-Dashboard](assets/grafana/celeborn-dashboard.json). The dashboard was
generated by grafana of version 8.5.0.
Here are some snapshots:
diff --git a/README.md b/README.md
index f6b2ee6b5..ecb2bbd2b 100644
--- a/README.md
+++ b/README.md
@@ -7,7 +7,7 @@ management service for intermediate data including shuffle
data, spilled data, r
## Internals
### Architecture
-
+
Celeborn has three primary components: Master, Worker, and Client.
Master manages all resources and syncs shared states based on Raft.
Worker processes read-write requests and merges data for each reducer.
@@ -33,7 +33,7 @@ LifecycleManager maintains metadata of each shuffle and runs
within the Spark dr
11. Reducers read shuffle data.
### Load Balance
-
+
We introduce slots to achieve load balance. We will equally distribute
partitions on every Celeborn worker by tracking slot usage.
The Slot is a logical concept in Celeborn Worker that represents how many
partitions can be allocated to each Celeborn Worker.
@@ -153,7 +153,7 @@ celeborn.master.ha.node.2.ratis.port 9872
celeborn.master.ha.node.3.host clb-3
celeborn.master.ha.node.3.port 9097
celeborn.master.ha.node.3.ratis.port 9872
-celeborn.master.ha.ratis.raft.server.storage.dir /mnt/disk1/rss_ratis/
+celeborn.master.ha.ratis.raft.server.storage.dir /mnt/disk1/celeborn_ratis/
celeborn.metrics.enabled true
# If you want to use HDFS as shuffle storage, make sure that flush buffer size
is at least 4MB or larger.
diff --git a/assets/diagram/celeborn.drawio b/assets/diagram/celeborn.drawio
new file mode 100644
index 000000000..b91e50e5d
--- /dev/null
+++ b/assets/diagram/celeborn.drawio
@@ -0,0 +1 @@
+<mxfile host="drawio-plugin" modified="2021-12-21T08:26:44.168Z" agent="5.0
(Macintosh; Intel Mac OS X 11_6_0) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/89.0.4389.114 Safari/537.36" etag="LH2JKBTcqSlmpDd78EZn"
version="14.7.4" type="embed"><diagram id="xOzwDYJeCoK_-xXMeRwq" name="第 1
页"><mxGraphModel dx="926" dy="813" grid="1" gridSize="10" guides="1"
tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1"
pageWidth="1169" pageHeight="827" math="0" shadow="0"><root><mxCel [...]
\ No newline at end of file
diff --git a/assets/diagram/rss_load_balance.drawio
b/assets/diagram/celeborn_load_balance.drawio
similarity index 100%
rename from assets/diagram/rss_load_balance.drawio
rename to assets/diagram/celeborn_load_balance.drawio
diff --git a/assets/diagram/emr_shuffle.drawio
b/assets/diagram/emr_shuffle.drawio
deleted file mode 100644
index 1d4fc05c3..000000000
--- a/assets/diagram/emr_shuffle.drawio
+++ /dev/null
@@ -1 +0,0 @@
-<mxfile host="Electron" modified="2021-05-26T10:28:12.634Z" agent="5.0
(Macintosh; Intel Mac OS X 11_3_1) AppleWebKit/537.36 (KHTML, like Gecko)
draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36"
etag="d1AASNl-txOqgHCv0_24" version="14.6.13" type="device"><diagram
id="JZ4pPKlbDS7XLyerruGA" name="第 1
页">7V1Nd6s4Ev01WXYfJCE+lul0ZnoxmdPTWbx5qz48I9vMOMaN8Yszv37AINtIcoJfkEoIr2IEJuZWUVVXVSXdkYeX/d+LZLN8ylO2usNeur8jv95hjILYr/7UI2/tiEeCZmRRZGk7dhp4zv7H+IXt6C5L2bZzYZnnqzLbdAdn+Xr
[...]
\ No newline at end of file
diff --git a/assets/diagram/rss.drawio b/assets/diagram/rss.drawio
deleted file mode 100644
index f3f467af4..000000000
--- a/assets/diagram/rss.drawio
+++ /dev/null
@@ -1 +0,0 @@
-<mxfile host="drawio-plugin" modified="2021-12-21T08:26:44.168Z" agent="5.0
(Macintosh; Intel Mac OS X 11_6_0) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/89.0.4389.114 Safari/537.36" etag="LH2JKBTcqSlmpDd78EZn"
version="14.7.4" type="embed"><diagram id="xOzwDYJeCoK_-xXMeRwq" name="第 1
页"><mxGraphModel dx="926" dy="813" grid="1" gridSize="10" guides="1"
tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1"
pageWidth="1169" pageHeight="827" math="0" shadow="0"><root><mxCel [...]
\ No newline at end of file
diff --git a/assets/diagram/rss_fault_tolerance.drawio
b/assets/diagram/rss_fault_tolerance.drawio
deleted file mode 100644
index 7940dffec..000000000
--- a/assets/diagram/rss_fault_tolerance.drawio
+++ /dev/null
@@ -1 +0,0 @@
-<mxfile host="drawio-plugin" modified="2021-12-21T08:28:06.780Z" agent="5.0
(Macintosh; Intel Mac OS X 11_6_0) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/89.0.4389.114 Safari/537.36" etag="Ch5jsyxn_7B1-Vp8TIPE"
version="14.7.4" type="embed"><diagram id="7pMcRzmeSCS7Ap43zZdk" name="第 1
页"><mxGraphModel dx="926" dy="813" grid="1" gridSize="10" guides="1"
tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1"
pageWidth="1169" pageHeight="827" math="0" shadow="0"><root><mxCel [...]
\ No newline at end of file
diff --git a/assets/diagram/shuffle-procedure.drawio
b/assets/diagram/shuffle-procedure.drawio
index 8ab8e95bc..9f97d2b18 100644
--- a/assets/diagram/shuffle-procedure.drawio
+++ b/assets/diagram/shuffle-procedure.drawio
@@ -1 +1 @@
-<mxfile host="Electron" modified="2021-12-21T09:35:31.789Z" agent="5.0
(Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
draw.io/16.0.0 Chrome/91.0.4472.164 Electron/13.6.3 Safari/537.36"
etag="JKgcKgqyZ9ssTqmfpi8u" version="16.0.0" type="device"><diagram
id="xOzwDYJeCoK_-xXMeRwq" name="第 1
页">7Vxbd5s4EP41frQPQlwfUyftNk16cuLd03bfFBCGBlsuFkncX7/CSDZCGHzBmJxtHhIjJAzzzXwzoxkygOPZ26cELcJ74uN4oGv+2wBeD3QdAM1if7KRFR+xbDcfmSaRz8e2A5PoN+aDGh9NIx8vpYmUkJhGC3nQI/M59qg0hpKE
[...]
\ No newline at end of file
+<mxfile host="app.diagrams.net" modified="2023-06-29T08:14:37.564Z" agent="5.0
(Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/114.0.0.0 Safari/537.36" etag="fyVJYGiounE1p992S9DX" version="20.0.1"
type="device"><diagram id="xOzwDYJeCoK_-xXMeRwq" name="第 1
页">7Vxdd5s4EP01foyPhPh8TJ202zTpyUl2T9t9k0EYGoxcLJK4v36FETYgDHbAQM42D4kZJMBzZ+6MNEMmaLZ8/RThlXdHHRJMFOC8TtDVRFGgYpn8TyLZCIkO1FSyiHxHyPaCR/83EUIgpLHvkHVhIKM0YP6qKLRpGBKbFWQ4iuhLcZhLg+JdV3hBJMGjjQNZ+s13mCe
[...]
\ No newline at end of file
diff --git a/assets/diagram/spark_shuffle.drawio
b/assets/diagram/spark_shuffle.drawio
deleted file mode 100644
index 384013924..000000000
--- a/assets/diagram/spark_shuffle.drawio
+++ /dev/null
@@ -1 +0,0 @@
-<mxfile host="Electron" modified="2021-05-26T10:00:23.748Z" agent="5.0
(Macintosh; Intel Mac OS X 11_3_1) AppleWebKit/537.36 (KHTML, like Gecko)
draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36"
etag="jj1yCrwMuHbPFE98KNZ3" version="14.6.13" type="device"><diagram
id="ExNYNv40toBjj5x9zH2e" name="第 1
页">7V3fk5s2EP5r/JgMICTw451zbWYymen02mnaNw5kmwlGDsaxr399hRFnI+lu7AZpbZF7OesHAn27rPZbrewJmq32v1bJevmZZbSYBF62n6APkyDw/Zjwf03Nc1sTBbitWFR5JjodKx7zf6mo9ETtNs/optexZqyo83W/MmVlSdO
[...]
\ No newline at end of file
diff --git a/assets/grafana/rss-dashboard.json
b/assets/grafana/celeborn-dashboard.json
similarity index 100%
rename from assets/grafana/rss-dashboard.json
rename to assets/grafana/celeborn-dashboard.json
diff --git a/assets/img/adaptive_query_execution_coalesce_partition.jpg
b/assets/img/adaptive_query_execution_coalesce_partition.jpg
deleted file mode 100644
index 3dc47d202..000000000
Binary files a/assets/img/adaptive_query_execution_coalesce_partition.jpg and
/dev/null differ
diff --git a/assets/img/adaptive_query_execution_local_shuffle_read.jpg
b/assets/img/adaptive_query_execution_local_shuffle_read.jpg
deleted file mode 100644
index 651530e8c..000000000
Binary files a/assets/img/adaptive_query_execution_local_shuffle_read.jpg and
/dev/null differ
diff --git a/assets/img/celeborn.jpg b/assets/img/celeborn.jpg
new file mode 100644
index 000000000..149ecf70a
Binary files /dev/null and b/assets/img/celeborn.jpg differ
diff --git a/assets/img/rss_load_balance.jpg
b/assets/img/celeborn_load_balance.jpg
similarity index 100%
rename from assets/img/rss_load_balance.jpg
rename to assets/img/celeborn_load_balance.jpg
diff --git a/assets/img/emr_shuffle.jpg b/assets/img/emr_shuffle.jpg
deleted file mode 100644
index 213fc2881..000000000
Binary files a/assets/img/emr_shuffle.jpg and /dev/null differ
diff --git a/assets/img/rss.jpg b/assets/img/rss.jpg
deleted file mode 100644
index 36589747e..000000000
Binary files a/assets/img/rss.jpg and /dev/null differ
diff --git a/assets/img/rss_fault_tolerance.jpg
b/assets/img/rss_fault_tolerance.jpg
deleted file mode 100644
index 6fffa255b..000000000
Binary files a/assets/img/rss_fault_tolerance.jpg and /dev/null differ
diff --git a/assets/img/shuffle-procedure.jpg b/assets/img/shuffle-procedure.jpg
index e950b10f3..6f19289e8 100644
Binary files a/assets/img/shuffle-procedure.jpg and
b/assets/img/shuffle-procedure.jpg differ
diff --git a/assets/img/spark_shuffle.jpg b/assets/img/spark_shuffle.jpg
deleted file mode 100644
index 9d1d21691..000000000
Binary files a/assets/img/spark_shuffle.jpg and /dev/null differ
