This is an automated email from the ASF dual-hosted git repository.
roryqi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-uniffle-website.git
The following commit(s) were added to refs/heads/master by this push:
new 0eb11e7 Polish the blog (#47)
0eb11e7 is described below
commit 0eb11e7311fe75626b277b3f37181a8cc7c6775a
Author: roryqi <[email protected]>
AuthorDate: Sun Jul 23 11:40:24 2023 +0800
Polish the blog (#47)
---
...ter for the shuffle in the cloud native era.md} | 12 +++++++++---
blog/2023-07-21/img/select.png | Bin 165995 -> 164164 bytes
2 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/blog/2023-07-21/current_state.md b/blog/2023-07-21/Uniffle: New
chapter for the shuffle in the cloud native era.md
similarity index 97%
rename from blog/2023-07-21/current_state.md
rename to blog/2023-07-21/Uniffle: New chapter for the shuffle in the cloud
native era.md
index 5d3c61b..e99f972 100644
--- a/blog/2023-07-21/current_state.md
+++ b/blog/2023-07-21/Uniffle: New chapter for the shuffle in the cloud native
era.md
@@ -20,12 +20,18 @@
## Background
Shuffle is the process in distributed computing frameworks used to
redistribute data between upstream and downstream tasks. It is a crucial
component within computing frameworks and directly impacts their performance
and stability.
However, with the exploration of cloud-native architectures, traditional
Shuffle solutions have revealed various issues.
-In a cloud-native architecture, techniques such as storage-compute separation
and mixed deployment are also applied simultaneously.
-The computational nodes have relatively small disk capacities, poor IO
performance, and an imbalance between CPU and IO resources.
+
+In a cloud-native architecture, techniques such as storage-compute separation
and mixed deployment are also applied simultaneously.The computational nodes
have relatively small disk capacities, poor IO performance, and an imbalance
between CPU and IO resources.
Additionally, computational nodes may be preempted by high-priority jobs due
to mixed deployments.
+
In traditional Shuffle implementations, Shuffle nodes are tightly coupled with
computational nodes. However, due to the different resource requirements for
disk, memory, CPU, and node stability between computational and Shuffle nodes,
it is challenging to independently scale them based on their resource needs.
-By separating the computational nodes from Shuffle nodes, the computational
node's state becomes more lightweight after offloading the Shuffle state to
Shuffle nodes, reducing the need for job recomputation when computational nodes
are preempted. Decoupling computational and Shuffle nodes also reduces the
demand for disk specifications on computational nodes, enabling an increase in
the number of accessible computational nodes.
+By separating the computational nodes from Shuffle nodes, the computational
node's state becomes more lightweight after offloading the Shuffle state to
Shuffle nodes, reducing the need for job recomputation when computational nodes
are preempted.
+
+Decoupling computational and Shuffle nodes also reduces the demand for disk
specifications on computational nodes, enabling an increase in the number of
accessible computational nodes.
+
In cloud-native architectures, large Shuffle jobs can exert significant
pressure on local disk drives, leading to issues such as insufficient disk
capacity on computational nodes and higher disk random IO, thus affecting the
performance and stability of large Shuffle jobs.
+
+
The industry has explored various new Shuffle technologies, including Google's
BigQuery, Baidu DCE Shuffle, Facebook's Cosco Shuffle, Uber Zeus Shuffle,
Alibaba's Celeborn Shuffle, and many others.
Each system has made its own trade-offs based on different scenarios. Uniffle
aims to create a fast, accurate, stable, and cost-efficient cloud-native Remote
Shuffle Service, considering performance, correctness, stability, and cost as
its core aspects.
diff --git a/blog/2023-07-21/img/select.png b/blog/2023-07-21/img/select.png
index 93a4350..5224d18 100644
Binary files a/blog/2023-07-21/img/select.png and
b/blog/2023-07-21/img/select.png differ
