This is an automated email from the ASF dual-hosted git repository.
kirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/master by this push:
new 8f8e61c update 2.0 architecture (#506)
8f8e61c is described below
commit 8f8e61cda8afe6c1dcc6470411d5b61b63ca63bb
Author: OS <[email protected]>
AuthorDate: Mon Nov 8 09:47:03 2021 +0800
update 2.0 architecture (#506)
---
docs/en-us/2.0.0/user_doc/architecture/design.md | 20 ++++++++++----------
docs/zh-cn/2.0.0/user_doc/architecture/design.md | 7 ++-----
2 files changed, 12 insertions(+), 15 deletions(-)
diff --git a/docs/en-us/2.0.0/user_doc/architecture/design.md
b/docs/en-us/2.0.0/user_doc/architecture/design.md
index 7c69078..79ec65b 100644
--- a/docs/en-us/2.0.0/user_doc/architecture/design.md
+++ b/docs/en-us/2.0.0/user_doc/architecture/design.md
@@ -45,7 +45,7 @@ Before explaining the architecture of the scheduling system,
let's first underst
#### 2.2 Start process activity diagram
<p align="center">
- <img src="/img/process-start-flow-1.3.0.png" alt="Start process activity
diagram" width="70%" />
+ <img src="/img/master-process-2.0-en.png" alt="Start process activity
diagram" width="70%" />
<p align="center">
<em>Start process activity diagram</em>
</p>
@@ -104,6 +104,7 @@ Before explaining the architecture of the scheduling
system, let's first underst
###### Centralized thinking
The centralized design concept is relatively simple. The nodes in the
distributed cluster are divided into roles according to roles, which are
roughly divided into two roles:
+
<p align="center">
<img
src="https://analysys.github.io/easyscheduler_docs_cn/images/master_slave.png"
alt="master-slave character" width="50%" />
</p>
@@ -130,21 +131,20 @@ Problems in centralized thought design:
- In fact, truly decentralized distributed systems are rare. Instead, dynamic
centralized distributed systems are constantly pouring out. Under this
architecture, the managers in the cluster are dynamically selected, rather than
preset, and when the cluster fails, the nodes of the cluster will automatically
hold "meetings" to elect new "managers" To preside over the work. The most
typical case is Etcd implemented by ZooKeeper and Go language.
--The decentralization of DolphinScheduler is that the Master/Worker is
registered in Zookeeper to realize the non-centralization of the Master cluster
and the Worker cluster. The sharding mechanism is used to fairly distribute the
workflow for execution on the master, and tasks are sent to the workers for
execution through different sending strategies. Specific task
+- The decentralization of DolphinScheduler is that the Master/Worker is
registered in Zookeeper to realize the non-centralization of the Master cluster
and the Worker cluster. The sharding mechanism is used to fairly distribute the
workflow for execution on the master, and tasks are sent to the workers for
execution through different sending strategies. Specific task
##### Second, the master execution process
1. DolphinScheduler uses the sharding algorithm to modulate the command and
assigns it according to the sort id of the master. The master converts the
received command into a workflow instance, and uses the thread pool to process
the workflow instance
+2. DolphinScheduler's process of workflow:
-2. Dolphinscheduler's process of workflow:
-
- -Start the workflow through UI or API calls, and persist a command to the
database
- -The Master scans the Command table through the sharding algorithm,
generates a workflow instance ProcessInstance, and deletes the Command data at
the same time
- -The Master uses the thread pool to run WorkflowExecuteThread to execute the
process of the workflow instance, including building DAG, creating task
instance TaskInstance, and sending TaskInstance to worker through netty
- -After the worker receives the task, it modifies the task status and returns
the execution information to the Master
- -The Master receives the task information, persists it to the database, and
stores the state change event in the EventExecuteService event queue
- -EventExecuteService calls WorkflowExecuteThread according to the event
queue to submit subsequent tasks and modify workflow status
+ - Start the workflow through UI or API calls, and persist a command to the
database
+ - The Master scans the Command table through the sharding algorithm,
generates a workflow instance ProcessInstance, and deletes the Command data at
the same time
+ - The Master uses the thread pool to run WorkflowExecuteThread to execute
the process of the workflow instance, including building DAG, creating task
instance TaskInstance, and sending TaskInstance to worker through netty
+ - After the worker receives the task, it modifies the task status and
returns the execution information to the Master
+ - The Master receives the task information, persists it to the database, and
stores the state change event in the EventExecuteService event queue
+ - EventExecuteService calls WorkflowExecuteThread according to the event
queue to submit subsequent tasks and modify workflow status
##### Three、Insufficient thread loop waiting problem
diff --git a/docs/zh-cn/2.0.0/user_doc/architecture/design.md
b/docs/zh-cn/2.0.0/user_doc/architecture/design.md
index 81b0c30..9ac86a4 100644
--- a/docs/zh-cn/2.0.0/user_doc/architecture/design.md
+++ b/docs/zh-cn/2.0.0/user_doc/architecture/design.md
@@ -45,7 +45,7 @@
#### 2.2 启动流程活动图
<p align="center">
- <img src="/img/process-start-flow-1.3.0.png" alt="启动流程活动图" width="70%" />
+ <img src="/img/master-process-2.0-zh_cn.png" alt="Start process activity
diagram" width="70%" />
<p align="center">
<em>启动流程活动图</em>
</p>
@@ -141,7 +141,7 @@
1.
DolphinScheduler使用分片算法将command取模,根据master的排序id分配,master将拿到的command转换成工作流实例,使用线程池处理工作流实例
-2. dolphinscheduler对工作流的处理流程:
+2. DolphinScheduler对工作流的处理流程:
- 通过UI或者API调用,启动工作流,持久化一条command到数据库中
- Master通过分片算法,扫描Command表,生成工作流实例ProcessInstance,同时删除Command数据
@@ -150,9 +150,6 @@
- Master收到任务信息,持久化到数据库,并且将状态变化事件存入EventExecuteService事件队列
- EventExecuteService根据事件队列调用WorkflowExecuteThread进行后续任务的提交和工作流状态的修改
- <p align="center">
- <img src="/img/master-process-2.0-zh_cn.png" alt="master执行流程" width="50%"
/>
- </p>
##### 三、容错设计
容错分为服务宕机容错和任务重试,服务宕机容错又分为Master容错和Worker容错两种情况