This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e516c11 Automated deployment: ab35ab4d9a76baada1972057505c47a08d5d3eca
e516c11 is described below
commit e516c1138bdeb9e5100a7f0bf78a6a5fa7ff8799
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Sun Jul 18 13:44:37 2021 +0000
Automated deployment: ab35ab4d9a76baada1972057505c47a08d5d3eca
---
build/{blog.fea92b2.js => blog.e4df554.js} | 2 +-
en-us/blog/index.html | 2 +-
img/architecture-1.2.0.jpg | Bin 0 -> 64444 bytes
zh-cn/blog/DS_architecture_evolution.html | 101 +++++++++++++++++++++++++++++
zh-cn/blog/DS_architecture_evolution.json | 6 ++
zh-cn/blog/index.html | 4 +-
6 files changed, 111 insertions(+), 4 deletions(-)
diff --git a/build/blog.fea92b2.js b/build/blog.e4df554.js
similarity index 56%
rename from build/blog.fea92b2.js
rename to build/blog.e4df554.js
index 60bd25e..bce5c66 100644
--- a/build/blog.fea92b2.js
+++ b/build/blog.e4df554.js
@@ -1 +1 @@
-webpackJsonp([1],{1:function(e,t){e.exports=React},2:function(e,t){e.exports=ReactDOM},401:function(e,t,n){e.exports=n(402)},402:function(e,t,n){"use
strict";function r(e){return e&&e.__esModule?e:{default:e}}function
o(e,t){if(!(e instanceof t))throw new TypeError("Cannot call a class as a
function")}function a(e,t){if(!e)throw new ReferenceError("this hasn't been
initialised - super() hasn't been called");return!t||"object"!=typeof
t&&"function"!=typeof t?e:t}function l(e,t){if("functi [...]
\ No newline at end of file
+webpackJsonp([1],{1:function(e,t){e.exports=React},2:function(e,t){e.exports=ReactDOM},401:function(e,t,n){e.exports=n(402)},402:function(e,t,n){"use
strict";function r(e){return e&&e.__esModule?e:{default:e}}function
o(e,t){if(!(e instanceof t))throw new TypeError("Cannot call a class as a
function")}function a(e,t){if(!e)throw new ReferenceError("this hasn't been
initialised - super() hasn't been called");return!t||"object"!=typeof
t&&"function"!=typeof t?e:t}function l(e,t){if("functi [...]
\ No newline at end of file
diff --git a/en-us/blog/index.html b/en-us/blog/index.html
index 0f02599..9b65e58 100644
--- a/en-us/blog/index.html
+++ b/en-us/blog/index.html
@@ -16,7 +16,7 @@
<script
src="//cdn.jsdelivr.net/npm/[email protected]/dist/react-dom.min.js"></script>
<script>window.rootPath = '';</script>
<script src="/build/vendor.e1cbde6.js"></script>
- <script src="/build/blog.fea92b2.js"></script>
+ <script src="/build/blog.e4df554.js"></script>
<script>
var _hmt = _hmt || [];
(function() {
diff --git a/img/architecture-1.2.0.jpg b/img/architecture-1.2.0.jpg
new file mode 100644
index 0000000..ce60c2b
Binary files /dev/null and b/img/architecture-1.2.0.jpg differ
diff --git a/zh-cn/blog/DS_architecture_evolution.html
b/zh-cn/blog/DS_architecture_evolution.html
new file mode 100644
index 0000000..97ae393
--- /dev/null
+++ b/zh-cn/blog/DS_architecture_evolution.html
@@ -0,0 +1,101 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+ <meta charset="UTF-8">
+ <meta name="viewport" content="width=device-width, initial-scale=1.0,
maximum-scale=1.0, user-scalable=no">
+ <meta name="keywords" content="DS_architecture_evolution">
+ <meta name="description" content="DS_architecture_evolution">
+ <title>DS_architecture_evolution</title>
+ <link rel="shortcut icon" href="/img/favicon.ico">
+ <link rel="stylesheet" href="/build/vendor.c5ba65d.css">
+ <link rel="stylesheet" href="/build/blog.md.fd8b187.css">
+</head>
+<body>
+ <div id="root"><div class="blog-detail-page" data-reactroot=""><header
class="header-container header-container-dark"><div class="header-body"><a
href="/zh-cn/index.html"><img class="logo" src="/img/hlogo_white.svg"/></a><div
class="search search-dark"><span class="icon-search"></span></div><span
class="language-switch language-switch-dark">En</span><div
class="header-menu"><img class="header-menu-toggle"
src="/img/system/menu_white.png"/><div><ul class="ant-menu whiteClass
ant-menu-li [...]
+<p>[toc]</p>
+<h2>演讲者介绍</h2>
+<p>阮文俊，eBay开发工程师，DolphinScheduler贡献者。</p>
+<p>视频分享参见https://www.bilibili.com/video/BV11M4y1T7VT?from=search&seid=4448322552403036491</p>
+<h2>Apache DolphinScheduler介绍</h2>
+<p>Apache
DolphinScheduler是一个分布式去中心化，易扩展的可视化DAG工作流任务调度平台。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。DolphinScheduler以有向无环图的方式将任务连接起来，可实时监控任务的运行状态，同时支持取消、暂停、恢复、从指定任务节点重跑等操作。</p>
+<p>DolphinScheduler具有以下几个优良功能特性：</p>
+<ul>
+<li><strong>简单易用</strong>，提供可视化的工作流编排能力，可通过拖拉拽任务定制DAG，支持一键部署</li>
+<li><strong>高可靠性</strong>，整个调度系统高度可用，各个组件具备自我容错能力</li>
+<li><strong>高可扩展性</strong>，采用去中心化的多master+多worker的设计框架，集群可用横向扩展，并支持节点的动态上下线</li>
+<li><strong>丰富的使用场景</strong>，DolphinScheduler为面向复杂的使用场景，所有扩展点都采用插件化的实现方式</li>
+</ul>
+<p>关于DolphinSheduler更多功能介绍和开发文档请查阅官网详细信息https://dolphinscheduler.apache.org/zh-cn/。</p>
+<h2>架构演进过程</h2>
+<p><strong>1.2.x架构</strong></p>
+<p>DolphinScheduler最初进入Apache孵化器的版本是1.2，在这一版本中采用的架构由以下几个重要部分组成：</p>
+<ul>
+<li>去中心化的master节点，负责工作流调度、DAG任务切分、任务提交监控和监听其它节点健康状态等任务</li>
+<li>去中心化的worker节点，负责执行任务和维护任务的生命周期等</li>
+<li>数据库，存储工作流元数据，运行实例数据</li>
+<li>Zookeeper，主要负责注册中心、分布式锁、任务队列等工作任务</li>
+</ul>
+<p>1.2版本基本实现了高可靠的工作流调度系统，但是也存在多个问题：</p>
+<ul>
+<li><strong>重量级的worker</strong>，worker节点需要负责多种任务</li>
+<li>异步派发任务会导致<strong>任务执行延迟</strong></li>
+<li>由于masker和worker都需要依赖数据库，导致数据库压力大</li>
+</ul>
+<p><a href="https://imgtu.com/i/W15rAH";><img
src="https://z3.ax1x.com/2021/07/17/W15rAH.jpg"; alt="W15rAH.jpg"></a></p>
+<p><strong>1.3.x架构</strong></p>
+<p>针对1.2版本存在的问题，1.3架构进行了如下改进：</p>
+<ul>
+<li>去任务队列，保证master节点同步派发任务，降低任务执行延迟</li>
+<li>轻量级worker，worker节点只负责执行任务，单一化worker职责</li>
+<li>减小数据库压力，worker不再连接数据库</li>
+<li>采用多任务负载均衡策略，master根据worker节点资源使用情况分配任务，提高worker资源利用率</li>
+</ul>
+<p>这些改进有效改进了1.2版本的缺陷，但仍存在一些问题，例如：</p>
+<ul>
+<li>master调度工作流时需要依赖<strong>分布式锁</strong>，导致工作流吞吐量难以提升</li>
+<li>因为需要创建大量线程池，多数线程处于轮询数据库，导致<strong>master资源利用率低</strong></li>
+<li>master轮询数据库，仍然导致<strong>数据库压力大</strong></li>
+<li>各组件存在<strong>耦合</strong>情况</li>
+</ul>
+<p><img src="https://dolphinscheduler.apache.org/img/architecture-1.3.0.jpg";
alt="系统架构图"></p>
+<p><strong>2.0架构</strong></p>
+<p>针对1.3版本的缺陷，2.0架构进一步做出改进：</p>
+<ul>
+<li><strong>去分布式锁</strong>，对master进行分区编号，实现错位查询数据库，避免多个节点同时访问同一个工作流造成的冲突问题</li>
+<li>重构master线程模型，对所有工作流使用<strong>统一的线程池</strong></li>
+<li>重构数据库中<strong>DAG元数据</strong>模型</li>
+<li><strong>彻底的插件化</strong>，所有扩展点都采用插件化实现</li>
+<li><strong>数据血缘关系分析</strong></li>
+</ul>
+<h2>Apache DolphinScheduler发展方向</h2>
+<p>开发者阮文俊针对dolphinsheduler的未来发展方向，也分享了一些看法：</p>
+<ul>
+<li>系统更稳、速度更快（高吞吐、低延迟、智能化运维、高可用）</li>
+<li>支持更多的任务集成（深度学习任务、CI/CD等其它系统集成、存储过程和数据质量任务、容器调度任务、复杂调度场景等）</li>
+<li>轻量化dolphinscheduler内核，提供基础调度服务</li>
+</ul>
+<h2>如何参与开源贡献</h2>
+<p>最后，开发者阮文俊针对入门新手如何参与开源贡献的问题，提出了宝贵的指导意见：</p>
+<ul>
+<li>从小事做起，积累开发经验</li>
+<li>关注社区动态，积极参与讨论，更好融入社区</li>
+<li>坚持开源精神，乐于帮助他人</li>
+<li>持之以恒</li>
+</ul>
+</section><footer class="footer-container"><div
class="footer-body"><div><h3>联系我们</h3><h4>有问题需要反馈？请通过以下方式联系我们。</h4></div><div
class="contact-container"><ul><li><img class="img-base"
src="/img/emailgray.png"/><img class="img-change" src="/img/emailblue.png"/><a
href="/zh-cn/community/development/subscribe.html"><p>邮件列表</p></a></li><li><img
class="img-base" src="/img/twittergray.png"/><img class="img-change"
src="/img/twitterblue.png"/><a
href="https://twitter.com/dolphinschedule";><p>Twitt [...]
+ <script
src="//cdn.jsdelivr.net/npm/[email protected]/dist/react-with-addons.min.js"></script>
+ <script
src="//cdn.jsdelivr.net/npm/[email protected]/dist/react-dom.min.js"></script>
+ <script>window.rootPath = '';</script>
+ <script src="/build/vendor.e1cbde6.js"></script>
+ <script src="/build/blog.md.57874be.js"></script>
+ <script>
+ var _hmt = _hmt || [];
+ (function() {
+ var hm = document.createElement("script");
+ hm.src = "https://hm.baidu.com/hm.js?4e7b4b400dd31fa015018a435c64d06f";;
+ var s = document.getElementsByTagName("script")[0];
+ s.parentNode.insertBefore(hm, s);
+ })();
+ </script>
+</body>
+</html>
\ No newline at end of file
diff --git a/zh-cn/blog/DS_architecture_evolution.json
b/zh-cn/blog/DS_architecture_evolution.json
new file mode 100644
index 0000000..af2447e
--- /dev/null
+++ b/zh-cn/blog/DS_architecture_evolution.json
@@ -0,0 +1,6 @@
+{
+ "filename": "DS_architecture_evolution.md",
+ "__html": "<h1>Apache DolphinScheduler
架构演进及开源经验分享</h1>\n<p>[toc]</p>\n<h2>演讲者介绍</h2>\n<p>阮文俊，eBay开发工程师，DolphinScheduler贡献者。</p>\n<p>视频分享参见https://www.bilibili.com/video/BV11M4y1T7VT?from=search&seid=4448322552403036491</p>\n<h2>Apache
DolphinScheduler介绍</h2>\n<p>Apache
DolphinScheduler是一个分布式去中心化，易扩展的可视化DAG工作流任务调度平台。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中开箱即用。DolphinScheduler以有向无环图的方式将任务连接起来，可实时监控任务的运行状态，同时支持取消、暂停、恢复、从指定任务节点重跑等操作。</p>\n<p>DolphinScheduler具有以下几个优良功能特性：</p>\n<ul>\n<li><s
[...]
+ "link": "/dist/zh-cn/blog/DS_architecture_evolution.html",
+ "meta": {}
+}
\ No newline at end of file
diff --git a/zh-cn/blog/index.html b/zh-cn/blog/index.html
index 27cd7af..3b2fb4e 100644
--- a/zh-cn/blog/index.html
+++ b/zh-cn/blog/index.html
@@ -11,12 +11,12 @@
<link rel="stylesheet" href="/build/blog.acc2955.css">
</head>
<body>
- <div id="root"><div class="blog-list-page" data-reactroot=""><header
class="header-container header-container-dark"><div class="header-body"><a
href="/zh-cn/index.html"><img class="logo" src="/img/hlogo_white.svg"/></a><div
class="search search-dark"><span class="icon-search"></span></div><span
class="language-switch language-switch-dark">En</span><div
class="header-menu"><img class="header-menu-toggle"
src="/img/system/menu_white.png"/><div><ul class="ant-menu whiteClass
ant-menu-ligh [...]
+ <div id="root"><div class="blog-list-page" data-reactroot=""><header
class="header-container header-container-dark"><div class="header-body"><a
href="/zh-cn/index.html"><img class="logo" src="/img/hlogo_white.svg"/></a><div
class="search search-dark"><span class="icon-search"></span></div><span
class="language-switch language-switch-dark">En</span><div
class="header-menu"><img class="header-menu-toggle"
src="/img/system/menu_white.png"/><div><ul class="ant-menu whiteClass
ant-menu-ligh [...]
<script
src="//cdn.jsdelivr.net/npm/[email protected]/dist/react-with-addons.min.js"></script>
<script
src="//cdn.jsdelivr.net/npm/[email protected]/dist/react-dom.min.js"></script>
<script>window.rootPath = '';</script>
<script src="/build/vendor.e1cbde6.js"></script>
- <script src="/build/blog.fea92b2.js"></script>
+ <script src="/build/blog.e4df554.js"></script>
<script>
var _hmt = _hmt || [];
(function() {
![https://z3.ax1x.com/2021/07/17/W15rAH.jpg"](https://z3.ax1x.com/2021/07/17/W15rAH.jpg"){kind=link}
![https://dolphinscheduler.apache.org/img/architecture-1.3.0.jpg"](https://dolphinscheduler.apache.org/img/architecture-1.3.0.jpg"){kind=link}