This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0ff5047 Automated deployment: 8f8e61cda8afe6c1dcc6470411d5b61b63ca63bb
0ff5047 is described below
commit 0ff50471c88ab051ab6b7780d9aa1df97acd7294
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Nov 8 01:49:12 2021 +0000
Automated deployment: 8f8e61cda8afe6c1dcc6470411d5b61b63ca63bb
---
en-us/docs/2.0.0/user_doc/architecture/design.html | 34 ++++++++++++++--------
en-us/docs/2.0.0/user_doc/architecture/design.json | 2 +-
.../docs/latest/user_doc/architecture/design.html | 34 ++++++++++++++--------
.../docs/latest/user_doc/architecture/design.json | 2 +-
zh-cn/docs/2.0.0/user_doc/architecture/design.html | 7 ++---
zh-cn/docs/2.0.0/user_doc/architecture/design.json | 2 +-
.../docs/latest/user_doc/architecture/design.html | 7 ++---
.../docs/latest/user_doc/architecture/design.json | 2 +-
8 files changed, 52 insertions(+), 38 deletions(-)
diff --git a/en-us/docs/2.0.0/user_doc/architecture/design.html
b/en-us/docs/2.0.0/user_doc/architecture/design.html
index 1b8afcb..d013768 100644
--- a/en-us/docs/2.0.0/user_doc/architecture/design.html
+++ b/en-us/docs/2.0.0/user_doc/architecture/design.html
@@ -41,7 +41,7 @@
</p>
<h4>2.2 Start process activity diagram</h4>
<p align="center">
- <img src="/img/process-start-flow-1.3.0.png" alt="Start process activity
diagram" width="70%" />
+ <img src="/img/master-process-2.0-en.png" alt="Start process activity
diagram" width="70%" />
<p align="center">
<em>Start process activity diagram</em>
</p>
@@ -122,26 +122,36 @@ When the WorkerServer service starts, it registers a
temporary node with Zookeep
<img
src="https://analysys.github.io/easyscheduler_docs_cn/images/decentralization.png";
alt="Decentralization" width="50%" />
</p>
<ul>
-<li>In the decentralized design, there is usually no concept of Master/Slave,
all roles are the same, the status is equal, the global Internet is a typical
decentralized distributed system, any node equipment connected to the network
is down, All will only affect a small range of functions.</li>
-<li>The core design of decentralized design is that there is no
"manager" different from other nodes in the entire distributed
system, so there is no single point of failure. However, because there is no
"manager" node, each node needs to communicate with other nodes to
obtain the necessary machine information, and the unreliability of distributed
system communication greatly increases the difficulty of implementing the above
functions.</li>
-<li>In fact, truly decentralized distributed systems are rare. Instead,
dynamic centralized distributed systems are constantly pouring out. Under this
architecture, the managers in the cluster are dynamically selected, rather than
preset, and when the cluster fails, the nodes of the cluster will automatically
hold "meetings" to elect new "managers" To preside over the
work. The most typical case is Etcd implemented by ZooKeeper and Go
language.</li>
+<li>
+<p>In the decentralized design, there is usually no concept of Master/Slave,
all roles are the same, the status is equal, the global Internet is a typical
decentralized distributed system, any node equipment connected to the network
is down, All will only affect a small range of functions.</p>
+</li>
+<li>
+<p>The core design of decentralized design is that there is no
"manager" different from other nodes in the entire distributed
system, so there is no single point of failure. However, because there is no
"manager" node, each node needs to communicate with other nodes to
obtain the necessary machine information, and the unreliability of distributed
system communication greatly increases the difficulty of implementing the above
functions.</p>
+</li>
+<li>
+<p>In fact, truly decentralized distributed systems are rare. Instead, dynamic
centralized distributed systems are constantly pouring out. Under this
architecture, the managers in the cluster are dynamically selected, rather than
preset, and when the cluster fails, the nodes of the cluster will automatically
hold "meetings" to elect new "managers" To preside over the
work. The most typical case is Etcd implemented by ZooKeeper and Go
language.</p>
+</li>
+<li>
+<p>The decentralization of DolphinScheduler is that the Master/Worker is
registered in Zookeeper to realize the non-centralization of the Master cluster
and the Worker cluster. The sharding mechanism is used to fairly distribute the
workflow for execution on the master, and tasks are sent to the workers for
execution through different sending strategies. Specific task</p>
+</li>
</ul>
-<p>-The decentralization of DolphinScheduler is that the Master/Worker is
registered in Zookeeper to realize the non-centralization of the Master cluster
and the Worker cluster. The sharding mechanism is used to fairly distribute the
workflow for execution on the master, and tasks are sent to the workers for
execution through different sending strategies. Specific task</p>
<h5>Second, the master execution process</h5>
<ol>
<li>
<p>DolphinScheduler uses the sharding algorithm to modulate the command and
assigns it according to the sort id of the master. The master converts the
received command into a workflow instance, and uses the thread pool to process
the workflow instance</p>
</li>
<li>
-<p>Dolphinscheduler's process of workflow:</p>
+<p>DolphinScheduler's process of workflow:</p>
</li>
</ol>
-<p>-Start the workflow through UI or API calls, and persist a command to the
database
--The Master scans the Command table through the sharding algorithm, generates
a workflow instance ProcessInstance, and deletes the Command data at the same
time
--The Master uses the thread pool to run WorkflowExecuteThread to execute the
process of the workflow instance, including building DAG, creating task
instance TaskInstance, and sending TaskInstance to worker through netty
--After the worker receives the task, it modifies the task status and returns
the execution information to the Master
--The Master receives the task information, persists it to the database, and
stores the state change event in the EventExecuteService event queue
--EventExecuteService calls WorkflowExecuteThread according to the event queue
to submit subsequent tasks and modify workflow status</p>
+<ul>
+<li>Start the workflow through UI or API calls, and persist a command to the
database</li>
+<li>The Master scans the Command table through the sharding algorithm,
generates a workflow instance ProcessInstance, and deletes the Command data at
the same time</li>
+<li>The Master uses the thread pool to run WorkflowExecuteThread to execute
the process of the workflow instance, including building DAG, creating task
instance TaskInstance, and sending TaskInstance to worker through netty</li>
+<li>After the worker receives the task, it modifies the task status and
returns the execution information to the Master</li>
+<li>The Master receives the task information, persists it to the database, and
stores the state change event in the EventExecuteService event queue</li>
+<li>EventExecuteService calls WorkflowExecuteThread according to the event
queue to submit subsequent tasks and modify workflow status</li>
+</ul>
<h5>Three、Insufficient thread loop waiting problem</h5>
<ul>
<li>If there is no sub-process in a DAG, if the number of data in the Command
is greater than the threshold set by the thread pool, the process directly
waits or fails.</li>
diff --git a/en-us/docs/2.0.0/user_doc/architecture/design.json
b/en-us/docs/2.0.0/user_doc/architecture/design.json
index cfe2054..c128604 100644
--- a/en-us/docs/2.0.0/user_doc/architecture/design.json
+++ b/en-us/docs/2.0.0/user_doc/architecture/design.json
@@ -1,6 +1,6 @@
{
"filename": "design.md",
- "__html": "<h2>System Architecture Design</h2>\n<p>Before explaining the
architecture of the scheduling system, let's first understand the commonly used
terms of the scheduling
system</p>\n<h3>1.Glossary</h3>\n<p><strong>DAG：</strong> The full name is
Directed Acyclic Graph, referred to as DAG. Task tasks in the workflow are
assembled in the form of a directed acyclic graph, and topological traversal is
performed from nodes with zero degrees of entry until there are no subsequent
nodes [...]
+ "__html": "<h2>System Architecture Design</h2>\n<p>Before explaining the
architecture of the scheduling system, let's first understand the commonly used
terms of the scheduling
system</p>\n<h3>1.Glossary</h3>\n<p><strong>DAG：</strong> The full name is
Directed Acyclic Graph, referred to as DAG. Task tasks in the workflow are
assembled in the form of a directed acyclic graph, and topological traversal is
performed from nodes with zero degrees of entry until there are no subsequent
nodes [...]
"link": "/dist/en-us/docs/2.0.0/user_doc/architecture/design.html",
"meta": {}
}
\ No newline at end of file
diff --git a/en-us/docs/latest/user_doc/architecture/design.html
b/en-us/docs/latest/user_doc/architecture/design.html
index 1b8afcb..d013768 100644
--- a/en-us/docs/latest/user_doc/architecture/design.html
+++ b/en-us/docs/latest/user_doc/architecture/design.html
@@ -41,7 +41,7 @@
</p>
<h4>2.2 Start process activity diagram</h4>
<p align="center">
- <img src="/img/process-start-flow-1.3.0.png" alt="Start process activity
diagram" width="70%" />
+ <img src="/img/master-process-2.0-en.png" alt="Start process activity
diagram" width="70%" />
<p align="center">
<em>Start process activity diagram</em>
</p>
@@ -122,26 +122,36 @@ When the WorkerServer service starts, it registers a
temporary node with Zookeep
<img
src="https://analysys.github.io/easyscheduler_docs_cn/images/decentralization.png";
alt="Decentralization" width="50%" />
</p>
<ul>
-<li>In the decentralized design, there is usually no concept of Master/Slave,
all roles are the same, the status is equal, the global Internet is a typical
decentralized distributed system, any node equipment connected to the network
is down, All will only affect a small range of functions.</li>
-<li>The core design of decentralized design is that there is no
"manager" different from other nodes in the entire distributed
system, so there is no single point of failure. However, because there is no
"manager" node, each node needs to communicate with other nodes to
obtain the necessary machine information, and the unreliability of distributed
system communication greatly increases the difficulty of implementing the above
functions.</li>
-<li>In fact, truly decentralized distributed systems are rare. Instead,
dynamic centralized distributed systems are constantly pouring out. Under this
architecture, the managers in the cluster are dynamically selected, rather than
preset, and when the cluster fails, the nodes of the cluster will automatically
hold "meetings" to elect new "managers" To preside over the
work. The most typical case is Etcd implemented by ZooKeeper and Go
language.</li>
+<li>
+<p>In the decentralized design, there is usually no concept of Master/Slave,
all roles are the same, the status is equal, the global Internet is a typical
decentralized distributed system, any node equipment connected to the network
is down, All will only affect a small range of functions.</p>
+</li>
+<li>
+<p>The core design of decentralized design is that there is no
"manager" different from other nodes in the entire distributed
system, so there is no single point of failure. However, because there is no
"manager" node, each node needs to communicate with other nodes to
obtain the necessary machine information, and the unreliability of distributed
system communication greatly increases the difficulty of implementing the above
functions.</p>
+</li>
+<li>
+<p>In fact, truly decentralized distributed systems are rare. Instead, dynamic
centralized distributed systems are constantly pouring out. Under this
architecture, the managers in the cluster are dynamically selected, rather than
preset, and when the cluster fails, the nodes of the cluster will automatically
hold "meetings" to elect new "managers" To preside over the
work. The most typical case is Etcd implemented by ZooKeeper and Go
language.</p>
+</li>
+<li>
+<p>The decentralization of DolphinScheduler is that the Master/Worker is
registered in Zookeeper to realize the non-centralization of the Master cluster
and the Worker cluster. The sharding mechanism is used to fairly distribute the
workflow for execution on the master, and tasks are sent to the workers for
execution through different sending strategies. Specific task</p>
+</li>
</ul>
-<p>-The decentralization of DolphinScheduler is that the Master/Worker is
registered in Zookeeper to realize the non-centralization of the Master cluster
and the Worker cluster. The sharding mechanism is used to fairly distribute the
workflow for execution on the master, and tasks are sent to the workers for
execution through different sending strategies. Specific task</p>
<h5>Second, the master execution process</h5>
<ol>
<li>
<p>DolphinScheduler uses the sharding algorithm to modulate the command and
assigns it according to the sort id of the master. The master converts the
received command into a workflow instance, and uses the thread pool to process
the workflow instance</p>
</li>
<li>
-<p>Dolphinscheduler's process of workflow:</p>
+<p>DolphinScheduler's process of workflow:</p>
</li>
</ol>
-<p>-Start the workflow through UI or API calls, and persist a command to the
database
--The Master scans the Command table through the sharding algorithm, generates
a workflow instance ProcessInstance, and deletes the Command data at the same
time
--The Master uses the thread pool to run WorkflowExecuteThread to execute the
process of the workflow instance, including building DAG, creating task
instance TaskInstance, and sending TaskInstance to worker through netty
--After the worker receives the task, it modifies the task status and returns
the execution information to the Master
--The Master receives the task information, persists it to the database, and
stores the state change event in the EventExecuteService event queue
--EventExecuteService calls WorkflowExecuteThread according to the event queue
to submit subsequent tasks and modify workflow status</p>
+<ul>
+<li>Start the workflow through UI or API calls, and persist a command to the
database</li>
+<li>The Master scans the Command table through the sharding algorithm,
generates a workflow instance ProcessInstance, and deletes the Command data at
the same time</li>
+<li>The Master uses the thread pool to run WorkflowExecuteThread to execute
the process of the workflow instance, including building DAG, creating task
instance TaskInstance, and sending TaskInstance to worker through netty</li>
+<li>After the worker receives the task, it modifies the task status and
returns the execution information to the Master</li>
+<li>The Master receives the task information, persists it to the database, and
stores the state change event in the EventExecuteService event queue</li>
+<li>EventExecuteService calls WorkflowExecuteThread according to the event
queue to submit subsequent tasks and modify workflow status</li>
+</ul>
<h5>Three、Insufficient thread loop waiting problem</h5>
<ul>
<li>If there is no sub-process in a DAG, if the number of data in the Command
is greater than the threshold set by the thread pool, the process directly
waits or fails.</li>
diff --git a/en-us/docs/latest/user_doc/architecture/design.json
b/en-us/docs/latest/user_doc/architecture/design.json
index cfe2054..c128604 100644
--- a/en-us/docs/latest/user_doc/architecture/design.json
+++ b/en-us/docs/latest/user_doc/architecture/design.json
@@ -1,6 +1,6 @@
{
"filename": "design.md",
- "__html": "<h2>System Architecture Design</h2>\n<p>Before explaining the
architecture of the scheduling system, let's first understand the commonly used
terms of the scheduling
system</p>\n<h3>1.Glossary</h3>\n<p><strong>DAG：</strong> The full name is
Directed Acyclic Graph, referred to as DAG. Task tasks in the workflow are
assembled in the form of a directed acyclic graph, and topological traversal is
performed from nodes with zero degrees of entry until there are no subsequent
nodes [...]
+ "__html": "<h2>System Architecture Design</h2>\n<p>Before explaining the
architecture of the scheduling system, let's first understand the commonly used
terms of the scheduling
system</p>\n<h3>1.Glossary</h3>\n<p><strong>DAG：</strong> The full name is
Directed Acyclic Graph, referred to as DAG. Task tasks in the workflow are
assembled in the form of a directed acyclic graph, and topological traversal is
performed from nodes with zero degrees of entry until there are no subsequent
nodes [...]
"link": "/dist/en-us/docs/2.0.0/user_doc/architecture/design.html",
"meta": {}
}
\ No newline at end of file
diff --git a/zh-cn/docs/2.0.0/user_doc/architecture/design.html
b/zh-cn/docs/2.0.0/user_doc/architecture/design.html
index 8fda821..6e060b7 100644
--- a/zh-cn/docs/2.0.0/user_doc/architecture/design.html
+++ b/zh-cn/docs/2.0.0/user_doc/architecture/design.html
@@ -41,7 +41,7 @@
</p>
<h4>2.2 启动流程活动图</h4>
<p align="center">
- <img src="/img/process-start-flow-1.3.0.png" alt="启动流程活动图" width="70%" />
+ <img src="/img/master-process-2.0-zh_cn.png" alt="Start process activity
diagram" width="70%" />
<p align="center">
<em>启动流程活动图</em>
</p>
@@ -147,7 +147,7 @@ WorkerServer服务启动时向Zookeeper注册临时节点，并维持心跳。</
<p>DolphinScheduler使用分片算法将command取模，根据master的排序id分配，master将拿到的command转换成工作流实例，使用线程池处理工作流实例</p>
</li>
<li>
-<p>dolphinscheduler对工作流的处理流程:</p>
+<p>DolphinScheduler对工作流的处理流程:</p>
</li>
</ol>
<ul>
@@ -158,9 +158,6 @@ WorkerServer服务启动时向Zookeeper注册临时节点，并维持心跳。</
<li>Master收到任务信息，持久化到数据库，并且将状态变化事件存入EventExecuteService事件队列</li>
<li>EventExecuteService根据事件队列调用WorkflowExecuteThread进行后续任务的提交和工作流状态的修改</li>
</ul>
- <p align="center">
- <img src="/img/master-process-2.0-zh_cn.png" alt="master执行流程" width="50%"
/>
- </p>
<h5>三、容错设计</h5>
<p>容错分为服务宕机容错和任务重试，服务宕机容错又分为Master容错和Worker容错两种情况</p>
<h6>1. 宕机容错</h6>
diff --git a/zh-cn/docs/2.0.0/user_doc/architecture/design.json
b/zh-cn/docs/2.0.0/user_doc/architecture/design.json
index 3adcc8b..d24fd18 100644
--- a/zh-cn/docs/2.0.0/user_doc/architecture/design.json
+++ b/zh-cn/docs/2.0.0/user_doc/architecture/design.json
@@ -1,6 +1,6 @@
{
"filename": "design.md",
- "__html":
"<h2>系统架构设计</h2>\n<p>在对调度系统架构说明之前，我们先来认识一下调度系统常用的名词</p>\n<h3>1.名词解释</h3>\n<p><strong>DAG：</strong>
全称Directed Acyclic
Graph，简称DAG。工作流中的Task任务以有向无环图的形式组装起来，从入度为零的节点进行拓扑遍历，直到无后继节点为止。举例如下图：</p>\n<p
align=\"center\">\n <img src=\"/img/dag_examples_cn.jpg\" alt=\"dag示例\"
width=\"60%\" />\n <p align=\"center\">\n <em>dag示例</em>\n
</p>\n</p>\n<p><strong>流程定义</strong>：通过拖拽任务节点并建立任务节点的关联所形成的可视化<strong>DAG</strong></p>\n<p><strong>流程实例</strong>：流程实例是流程定义的实例化，可以通过手动启动或定时调度生成,
[...]
+ "__html":
"<h2>系统架构设计</h2>\n<p>在对调度系统架构说明之前，我们先来认识一下调度系统常用的名词</p>\n<h3>1.名词解释</h3>\n<p><strong>DAG：</strong>
全称Directed Acyclic
Graph，简称DAG。工作流中的Task任务以有向无环图的形式组装起来，从入度为零的节点进行拓扑遍历，直到无后继节点为止。举例如下图：</p>\n<p
align=\"center\">\n <img src=\"/img/dag_examples_cn.jpg\" alt=\"dag示例\"
width=\"60%\" />\n <p align=\"center\">\n <em>dag示例</em>\n
</p>\n</p>\n<p><strong>流程定义</strong>：通过拖拽任务节点并建立任务节点的关联所形成的可视化<strong>DAG</strong></p>\n<p><strong>流程实例</strong>：流程实例是流程定义的实例化，可以通过手动启动或定时调度生成,
[...]
"link": "/dist/zh-cn/docs/2.0.0/user_doc/architecture/design.html",
"meta": {}
}
\ No newline at end of file
diff --git a/zh-cn/docs/latest/user_doc/architecture/design.html
b/zh-cn/docs/latest/user_doc/architecture/design.html
index 8fda821..6e060b7 100644
--- a/zh-cn/docs/latest/user_doc/architecture/design.html
+++ b/zh-cn/docs/latest/user_doc/architecture/design.html
@@ -41,7 +41,7 @@
</p>
<h4>2.2 启动流程活动图</h4>
<p align="center">
- <img src="/img/process-start-flow-1.3.0.png" alt="启动流程活动图" width="70%" />
+ <img src="/img/master-process-2.0-zh_cn.png" alt="Start process activity
diagram" width="70%" />
<p align="center">
<em>启动流程活动图</em>
</p>
@@ -147,7 +147,7 @@ WorkerServer服务启动时向Zookeeper注册临时节点，并维持心跳。</
<p>DolphinScheduler使用分片算法将command取模，根据master的排序id分配，master将拿到的command转换成工作流实例，使用线程池处理工作流实例</p>
</li>
<li>
-<p>dolphinscheduler对工作流的处理流程:</p>
+<p>DolphinScheduler对工作流的处理流程:</p>
</li>
</ol>
<ul>
@@ -158,9 +158,6 @@ WorkerServer服务启动时向Zookeeper注册临时节点，并维持心跳。</
<li>Master收到任务信息，持久化到数据库，并且将状态变化事件存入EventExecuteService事件队列</li>
<li>EventExecuteService根据事件队列调用WorkflowExecuteThread进行后续任务的提交和工作流状态的修改</li>
</ul>
- <p align="center">
- <img src="/img/master-process-2.0-zh_cn.png" alt="master执行流程" width="50%"
/>
- </p>
<h5>三、容错设计</h5>
<p>容错分为服务宕机容错和任务重试，服务宕机容错又分为Master容错和Worker容错两种情况</p>
<h6>1. 宕机容错</h6>
diff --git a/zh-cn/docs/latest/user_doc/architecture/design.json
b/zh-cn/docs/latest/user_doc/architecture/design.json
index 3adcc8b..d24fd18 100644
--- a/zh-cn/docs/latest/user_doc/architecture/design.json
+++ b/zh-cn/docs/latest/user_doc/architecture/design.json
@@ -1,6 +1,6 @@
{
"filename": "design.md",
- "__html":
"<h2>系统架构设计</h2>\n<p>在对调度系统架构说明之前，我们先来认识一下调度系统常用的名词</p>\n<h3>1.名词解释</h3>\n<p><strong>DAG：</strong>
全称Directed Acyclic
Graph，简称DAG。工作流中的Task任务以有向无环图的形式组装起来，从入度为零的节点进行拓扑遍历，直到无后继节点为止。举例如下图：</p>\n<p
align=\"center\">\n <img src=\"/img/dag_examples_cn.jpg\" alt=\"dag示例\"
width=\"60%\" />\n <p align=\"center\">\n <em>dag示例</em>\n
</p>\n</p>\n<p><strong>流程定义</strong>：通过拖拽任务节点并建立任务节点的关联所形成的可视化<strong>DAG</strong></p>\n<p><strong>流程实例</strong>：流程实例是流程定义的实例化，可以通过手动启动或定时调度生成,
[...]
+ "__html":
"<h2>系统架构设计</h2>\n<p>在对调度系统架构说明之前，我们先来认识一下调度系统常用的名词</p>\n<h3>1.名词解释</h3>\n<p><strong>DAG：</strong>
全称Directed Acyclic
Graph，简称DAG。工作流中的Task任务以有向无环图的形式组装起来，从入度为零的节点进行拓扑遍历，直到无后继节点为止。举例如下图：</p>\n<p
align=\"center\">\n <img src=\"/img/dag_examples_cn.jpg\" alt=\"dag示例\"
width=\"60%\" />\n <p align=\"center\">\n <em>dag示例</em>\n
</p>\n</p>\n<p><strong>流程定义</strong>：通过拖拽任务节点并建立任务节点的关联所形成的可视化<strong>DAG</strong></p>\n<p><strong>流程实例</strong>：流程实例是流程定义的实例化，可以通过手动启动或定时调度生成,
[...]
"link": "/dist/zh-cn/docs/2.0.0/user_doc/architecture/design.html",
"meta": {}
}
\ No newline at end of file
![https://analysys.github.io/easyscheduler_docs_cn/images/decentralization.png"](https://analysys.github.io/easyscheduler_docs_cn/images/decentralization.png"){kind=link}