This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/master by this push:
new 1315037 proof writing documents under architecture directory (#718)
1315037 is described below
commit 1315037f5f5c8443d67c9ad96c4f19ad1e933155
Author: Tq <[email protected]>
AuthorDate: Tue Mar 8 10:40:27 2022 +0800
proof writing documents under architecture directory (#718)
---
.../About_DolphinScheduler.md | 16 +-
docs/en-us/dev/user_doc/architecture/cache.md | 16 +-
.../dev/user_doc/architecture/configuration.md | 131 ++++++++--------
docs/en-us/dev/user_doc/architecture/design.md | 164 ++++++++++-----------
.../dev/user_doc/architecture/load-balance.md | 28 ++--
docs/en-us/dev/user_doc/architecture/metadata.md | 135 ++++++++---------
.../dev/user_doc/architecture/task-structure.md | 40 ++---
7 files changed, 268 insertions(+), 262 deletions(-)
diff --git
a/docs/en-us/dev/user_doc/About_DolphinScheduler/About_DolphinScheduler.md
b/docs/en-us/dev/user_doc/About_DolphinScheduler/About_DolphinScheduler.md
index aafcca1..a0d314e 100644
--- a/docs/en-us/dev/user_doc/About_DolphinScheduler/About_DolphinScheduler.md
+++ b/docs/en-us/dev/user_doc/About_DolphinScheduler/About_DolphinScheduler.md
@@ -1,19 +1,19 @@
# About DolphinScheduler
-Apache DolphinScheduler is a cloud-native visual Big Data workflow scheduler
system, committed to “solving complex big-data task dependencies and triggering
relationships in data OPS orchestration so that various types of big data tasks
can be used out of the box”.
+Apache DolphinScheduler is a distributed, easy to extend visual DAG workflow
task scheduling open-source system. Solves the intricate dependencies of data
R&D ETL and the inability to monitor the health status of tasks.
DolphinScheduler assembles tasks in the DAG streaming way, which can monitor
the execution status of tasks in time, and supports operations like retry,
recovery failure from specified nodes, pause, resume and kill tasks, etc.
-## High Reliability
+## Simple to Use
-- Decentralized multi-master and multi-worker, HA is supported by itself,
overload processing
+- DolphinScheduler has DAG monitoring user interfaces, users can customize DAG
by dragging and dropping. All process definitions are visualized, supports rich
third-party systems APIs and one-click deployment.
-## User-Friendly
+## High Reliability
-- All process definition operations are visualized, Visualization process
defines key information at a glance, One-click deployment
+- Decentralized multi-masters and multi-workers, support HA, select queues to
avoid overload.
## Rich Scenarios
-- Support multi-tenant. Support many task types e.g., spark,flink,hive, mr,
shell, python, sub_process
+- Support features like multi-tenants, suspend and resume operations to cope
with big data scenarios. Support many task types like Spark, Flink, Hive, MR,
shell, python, sub_process.
-## High Expansibility
+## High Scalability
-- Support custom task types, Distributed scheduling, and the overall
scheduling capability will increase linearly with the scale of the cluster
+- Supports customized task types, distributed scheduling, and the overall
scheduling capability increases linearly with the scale of the cluster.
\ No newline at end of file
diff --git a/docs/en-us/dev/user_doc/architecture/cache.md
b/docs/en-us/dev/user_doc/architecture/cache.md
index a07190d..a0251e3 100644
--- a/docs/en-us/dev/user_doc/architecture/cache.md
+++ b/docs/en-us/dev/user_doc/architecture/cache.md
@@ -2,9 +2,9 @@
## Purpose
-Due to the master-server scheduling process, there will be a large number of
database read operations, such as `tenant`, `user`, `processDefinition`, etc.
On the one hand, it will put a lot of pressure on the DB, and on the other
hand, it will slow down the entire core scheduling process.
+Due to the large database read operations during the master-server scheduling
process. Such as read tables like `tenant`, `user`, `processDefinition`, etc.
Operations stress read pressure to the DB, and slow down the entire core
scheduling process.
-Considering that this part of the business data is a scenario where more reads
and less writes are performed, a cache module is introduced to reduce the DB
read pressure and speed up the core scheduling process;
+By considering this part of the business data is a high-read and low-write
scenario, a cache module is introduced to reduce the DB read pressure and speed
up the core scheduling process.
## Cache Settings
@@ -23,20 +23,20 @@ spring:
spec: maximumSize=100,expireAfterWrite=300s,recordStats
```
-The cache-module use [spring-cache](https://spring.io/guides/gs/caching/), so
you can set cache config in the spring application.yaml directly. Default
disable cache, and you can enable it by `type: caffeine`.
+The cache module uses [spring-cache](https://spring.io/guides/gs/caching/), so
you can set cache config like whether to enable cache (`none` to disable by
default), cache types in the spring `application.yaml` directly.
-With the config of [caffeine](https://github.com/ben-manes/caffeine), you can
set the cache size, expire time, etc.
+Currently, implements the config of
[caffeine](https://github.com/ben-manes/caffeine), you can assign cache configs
like cache size, expire time, etc.
## Cache Read
-The cache adopts the annotation `@Cacheable` of spring-cache and is configured
in the mapper layer. For example: `TenantMapper`.
+The cache module adopts the `@Cacheable` annotation from spring-cache and you
can annotate the annotation in the related mapper layer. Refer to the
`TenantMapper`.
## Cache Evict
-The business data update comes from the api-server, and the cache end is in
the master-server. So it is necessary to monitor the data update of the
api-server (aspect intercept `@CacheEvict`), and the master-server will be
notified when the cache eviction is required.
+The business data updates come from the api-server, and the cache side is in
the master-server. Then it is necessary to monitor the data updates from the
api-server (use aspect point cut interceptor `@CacheEvict`), and notify the
master-server of `cacheEvictCommand` when processing a cache eviction.
-It should be noted that the final strategy for cache update comes from the
user's expiration strategy configuration in caffeine, so please configure it in
conjunction with the business;
+Note: the final strategy for cache update comes from the expiration strategy
configuration in caffeine, therefore configure it under the business scenarios;
-The sequence diagram is shown in the following figure:
+The sequence diagram shows below:
<img src="/img/cache-evict.png" alt="cache-evict" style="zoom: 67%;" />
\ No newline at end of file
diff --git a/docs/en-us/dev/user_doc/architecture/configuration.md
b/docs/en-us/dev/user_doc/architecture/configuration.md
index 37063ea..7f3afb2 100644
--- a/docs/en-us/dev/user_doc/architecture/configuration.md
+++ b/docs/en-us/dev/user_doc/architecture/configuration.md
@@ -1,4 +1,5 @@
<!-- markdown-link-check-disable -->
+
# Configuration
## Preface
@@ -7,26 +8,28 @@ This document explains the DolphinScheduler application
configurations according
## Directory Structure
-Currently, all the configuration files are under [conf ] directory. Please
check the following simplified DolphinScheduler installation directories to
have a direct view about the position [conf] directory in and configuration
files inside. This document only describes DolphinScheduler configurations and
other modules are not going into.
+Currently, all the configuration files are under [conf ] directory.
+Check the following simplified DolphinScheduler installation directories to
have a direct view about the position of [conf] directory and configuration
files it has.
+This document only describes DolphinScheduler configurations and other topics
are not going into.
[Note: the DolphinScheduler (hereinafter called the ‘DS’) .]
```
├─bin DS application commands directory
-│ ├─dolphinscheduler-daemon.sh startup/shutdown DS application
-│ ├─start-all.sh A startup all DS services with
configurations
+│ ├─dolphinscheduler-daemon.sh startup or shutdown DS application
+│ ├─start-all.sh startup all DS services with
configurations
│ ├─stop-all.sh shutdown all DS services with
configurations
├─conf configurations directory
│ ├─application-api.properties API-service config properties
│ ├─datasource.properties datasource config properties
│ ├─zookeeper.properties ZooKeeper config properties
-│ ├─master.properties master config properties
-│ ├─worker.properties worker config properties
+│ ├─master.properties master-service config properties
+│ ├─worker.properties worker-service config properties
│ ├─quartz.properties quartz config properties
-│ ├─common.properties common-service[storage] config
properties
+│ ├─common.properties common-service [storage] config
properties
│ ├─alert.properties alert-service config properties
│ ├─config environment variables config directory
-│ ├─install_config.conf DS environment variables
configuration script[install/start DS]
+│ ├─install_config.conf DS environment variables
configuration script [install or start DS]
│ ├─env load environment variables configs
script directory
│ ├─dolphinscheduler_env.sh load environment variables configs
[eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
│ ├─org mybatis mapper files directory
@@ -35,13 +38,13 @@ Currently, all the configuration files are under [conf ]
directory. Please check
│ ├─logback-master.xml master-service log config
│ ├─logback-worker.xml worker-service log config
│ ├─logback-alert.xml alert-service log config
-├─sql DS metadata to create/upgrade .sql
directory
+├─sql .sql files to create or upgrade DS
metadata
│ ├─create create SQL scripts directory
│ ├─upgrade upgrade SQL scripts directory
-│ ├─dolphinscheduler_postgre.sql postgre database init script
-│ ├─dolphinscheduler_mysql.sql mysql database init script
+│ ├─dolphinscheduler_postgre.sql PostgreSQL database init script
+│ ├─dolphinscheduler_mysql.sql MySQL database init script
│ ├─soft_version current DS version-id file
-├─script DS services deployment, database
create/upgrade scripts directory
+├─script DS services deployment, database create or
upgrade scripts directory
│ ├─create-dolphinscheduler.sh DS database init script
│ ├─upgrade-dolphinscheduler.sh DS database upgrade script
│ ├─monitor-server.sh DS monitor-server start script
@@ -56,13 +59,13 @@ Currently, all the configuration files are under [conf ]
directory. Please check
serial number| service classification| config file|
|--|--|--|
-1|startup/shutdown DS application|dolphinscheduler-daemon.sh
-2|datasource config properties| datasource.properties
+1|startup or shutdown DS application|dolphinscheduler-daemon.sh
+2|datasource config properties|datasource.properties
3|ZooKeeper config properties|zookeeper.properties
4|common-service[storage] config properties|common.properties
5|API-service config properties|application-api.properties
-6|master config properties|master.properties
-7|worker config properties|worker.properties
+6|master-service config properties|master.properties
+7|worker-service config properties|worker.properties
8|alert-service config properties|alert.properties
9|quartz config properties|quartz.properties
10|DS environment variables configuration script[install/start
DS]|install_config.conf
@@ -70,11 +73,11 @@ serial number| service classification| config file|
12|services log config files|API-service log config : logback-api.xml <br />
master-service log config : logback-master.xml <br /> worker-service log
config : logback-worker.xml <br /> alert-service log config : logback-alert.xml
-### dolphinscheduler-daemon.sh [startup/shutdown DS application]
+### dolphinscheduler-daemon.sh [startup or shutdown DS application]
dolphinscheduler-daemon.sh is responsible for DS startup and shutdown.
-Essentially, start-all.sh/stop-all.sh startup/shutdown the cluster via
dolphinscheduler-daemon.sh.
-Currently, DS just makes a basic config, please config further JVＭ options
based on your practical situation of resources.
+Essentially, start-all.sh or stop-all.sh startup and shutdown the cluster via
dolphinscheduler-daemon.sh.
+Currently, DS just makes a basic config, remember to config further JVM
options based on your practical situation of resources.
Default simplified parameters are:
```bash
@@ -102,10 +105,10 @@ spring.datasource.driver-class-name||datasource driver
spring.datasource.url||datasource connection url
spring.datasource.username||datasource username
spring.datasource.password||datasource password
-spring.datasource.initialSize|5| initail connection pool size number
+spring.datasource.initialSize|5| initial connection pool size number
spring.datasource.minIdle|5| minimum connection pool size number
spring.datasource.maxActive|5| maximum connection pool size number
-spring.datasource.maxWait|60000| max wait mili-seconds
+spring.datasource.maxWait|60000| max wait milliseconds
spring.datasource.timeBetweenEvictionRunsMillis|60000| idle connection check
interval
spring.datasource.timeBetweenConnectErrorMillis|60000| retry interval
spring.datasource.minEvictableIdleTimeMillis|300000| connections over
minEvictableIdleTimeMillis will be collect when idle check
@@ -116,7 +119,7 @@ spring.datasource.testOnBorrow|true| validity check when
the program requests a
spring.datasource.testOnReturn|false| validity check when the program recalls
a connection
spring.datasource.defaultAutoCommit|true| whether auto commit
spring.datasource.keepAlive|true| runs validationQuery SQL to avoid the
connection closed by pool when the connection idles over
minEvictableIdleTimeMillis
-spring.datasource.poolPreparedStatements|true| Open PSCache
+spring.datasource.poolPreparedStatements|true| open PSCache
spring.datasource.maxPoolPreparedStatementPerConnectionSize|20| specify the
size of PSCache on each connection
@@ -135,7 +138,7 @@ zookeeper.retry.maxtime|10| maximum retry times
### common.properties [hadoop、s3、yarn config properties]
-Currently, common.properties mainly configures hadoop/s3a related
configurations.
+Currently, common.properties mainly configures Hadoop,s3a related
configurations.
|Parameters | Default value| Description|
|--|--|--|
data.basedir.path|/tmp/dolphinscheduler| local directory used to store temp
files
@@ -148,12 +151,12 @@ login.user.keytab.path|/opt/hdfs.headless.keytab|kerberos
user keytab
kerberos.expire.time|2|kerberos expire time,integer,the unit is hour
resource.view.suffixs| txt,log,sh,conf,cfg,py,java,sql,hql,xml,properties|
file types supported by resource center
hdfs.root.user|hdfs| configure users with corresponding permissions if storage
type is HDFS
-fs.defaultFS|hdfs://mycluster:8020|If resource.storage.type=S3, then the
request url would be similar to 's3a://dolphinscheduler'. Otherwise if
resource.storage.type=HDFS and hadoop supports HA, please copy core-site.xml
and hdfs-site.xml into 'conf' directory
+fs.defaultFS|hdfs://mycluster:8020|If resource.storage.type=S3, then the
request url would be similar to 's3a://dolphinscheduler'. Otherwise if
resource.storage.type=HDFS and hadoop supports HA, copy core-site.xml and
hdfs-site.xml into 'conf' directory
fs.s3a.endpoint||s3 endpoint url
fs.s3a.access.key||s3 access key
fs.s3a.secret.key||s3 secret key
yarn.resourcemanager.ha.rm.ids||specify the yarn resourcemanager url. if
resourcemanager supports HA, input HA IP addresses (separated by comma), or
input null for standalone
-yarn.application.status.address|http://ds1:8088/ws/v1/cluster/apps/%s|keep
default if resourcemanager supports HA or not use resourcemanager. Or replace
ds1 with corresponding hostname if resourcemanager in standalone mode
+yarn.application.status.address|http://ds1:8088/ws/v1/cluster/apps/%s|keep
default if ResourceManager supports HA or not use ResourceManager, or replace
ds1 with corresponding hostname if ResourceManager in standalone mode
dolphinscheduler.env.path|env/dolphinscheduler_env.sh|load environment
variables configs [eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
development.state|false| specify whether in development state
@@ -179,10 +182,10 @@ security.authentication.type|PASSWORD| authentication type
|Parameters | Default value| Description|
|--|--|--|
master.listen.port|5678|master listen port
-master.exec.threads|100|master execute thread number to limit process
instances in parallel
-master.exec.task.num|20|master execute task number in parallel per process
instance
-master.dispatch.task.num|3|master dispatch task number per batch
-master.host.selector|LowerWeight|master host selector to select a suitable
worker, default value: LowerWeight. Optional values include Random, RoundRobin,
LowerWeight
+master.exec.threads|100|master-service execute thread number, used to limit
the number of process instances in parallel
+master.exec.task.num|20|defines the number of parallel tasks for each process
instance of the master-service
+master.dispatch.task.num|3|defines the number of dispatch tasks for each batch
of the master-service
+master.host.selector|LowerWeight|master host selector, to select a suitable
worker to run the task, optional value: random, round-robin, lower weight
master.heartbeat.interval|10|master heartbeat interval, the unit is second
master.task.commit.retryTimes|5|master commit task retry times
master.task.commit.interval|1000|master commit task interval, the unit is
millisecond
@@ -194,12 +197,12 @@ master.reserved.memory|0.3|master reserved memory, only
lower than system availa
|Parameters | Default value| Description|
|--|--|--|
-worker.listen.port|1234|worker listen port
-worker.exec.threads|100|worker execute thread number to limit task instances
in parallel
-worker.heartbeat.interval|10|worker heartbeat interval, the unit is second
+worker.listen.port|1234|worker-service listen port
+worker.exec.threads|100|worker-service execute thread number, used to limit
the number of task instances in parallel
+worker.heartbeat.interval|10|worker-service heartbeat interval, the unit is
second
worker.max.cpuload.avg|-1|worker max CPU load avg, only higher than the system
CPU load average, worker server can be dispatched tasks. default value -1: the
number of CPU cores * 2
worker.reserved.memory|0.3|worker reserved memory, only lower than system
available memory, worker server can be dispatched tasks. default value 0.3, the
unit is G
-worker.groups|default|worker groups separated by comma, like
'worker.groups=default,test' <br> worker will join corresponding group
according to this config when startup
+worker.groups|default|worker groups separated by comma, e.g.,
'worker.groups=default,test' <br> worker will join corresponding group
according to this config when startup
### alert.properties [alert-service log config]
@@ -232,40 +235,40 @@ plugin.dir|/Users/xx/your/path/to/plugin/dir|plugin
directory
### quartz.properties [quartz config properties]
-This part describes quartz configs and please configure them based on your
practical situation and resources.
+This part describes quartz configs and configure them based on your practical
situation and resources.
|Parameters | Default value| Description|
|--|--|--|
-org.quartz.jobStore.driverDelegateClass |
org.quartz.impl.jdbcjobstore.StdJDBCDelegate
-org.quartz.jobStore.driverDelegateClass |
org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
-org.quartz.scheduler.instanceName | DolphinScheduler
-org.quartz.scheduler.instanceId | AUTO
-org.quartz.scheduler.makeSchedulerThreadDaemon | true
-org.quartz.jobStore.useProperties | false
-org.quartz.threadPool.class | org.quartz.simpl.SimpleThreadPool
-org.quartz.threadPool.makeThreadsDaemons | true
-org.quartz.threadPool.threadCount | 25
-org.quartz.threadPool.threadPriority | 5
-org.quartz.jobStore.class | org.quartz.impl.jdbcjobstore.JobStoreTX
-org.quartz.jobStore.tablePrefix | QRTZ_
-org.quartz.jobStore.isClustered | true
-org.quartz.jobStore.misfireThreshold | 60000
-org.quartz.jobStore.clusterCheckinInterval | 5000
-org.quartz.jobStore.acquireTriggersWithinLock|true
-org.quartz.jobStore.dataSource | myDs
-org.quartz.dataSource.myDs.connectionProvider.class |
org.apache.dolphinscheduler.service.quartz.DruidConnectionProvider
-
-
-### install_config.conf [DS environment variables configuration
script[install/start DS]]
+org.quartz.jobStore.driverDelegateClass |
org.quartz.impl.jdbcjobstore.StdJDBCDelegate |
+org.quartz.jobStore.driverDelegateClass |
org.quartz.impl.jdbcjobstore.PostgreSQLDelegate |
+org.quartz.scheduler.instanceName | DolphinScheduler |
+org.quartz.scheduler.instanceId | AUTO |
+org.quartz.scheduler.makeSchedulerThreadDaemon | true |
+org.quartz.jobStore.useProperties | false |
+org.quartz.threadPool.class | org.quartz.simpl.SimpleThreadPool |
+org.quartz.threadPool.makeThreadsDaemons | true |
+org.quartz.threadPool.threadCount | 25 |
+org.quartz.threadPool.threadPriority | 5 |
+org.quartz.jobStore.class | org.quartz.impl.jdbcjobstore.JobStoreTX |
+org.quartz.jobStore.tablePrefix | QRTZ_ |
+org.quartz.jobStore.isClustered | true |
+org.quartz.jobStore.misfireThreshold | 60000 |
+org.quartz.jobStore.clusterCheckinInterval | 5000 |
+org.quartz.jobStore.acquireTriggersWithinLock|true |
+org.quartz.jobStore.dataSource | myDs |
+org.quartz.dataSource.myDs.connectionProvider.class |
org.apache.dolphinscheduler.service.quartz.DruidConnectionProvider |
+
+
+### install_config.conf [DS environment variables configuration script[install
or start DS]]
install_config.conf is a bit complicated and is mainly used in the following
two places.
-* DS Cluster Auto Installation
+* DS Cluster Auto Installation.
> System will load configs in the install_config.conf and auto-configure files
> below, based on the file content when executing 'install.sh'.
-> Files such as
dolphinscheduler-daemon.sh、datasource.properties、zookeeper.properties、common.properties、application-api.properties、master.properties、worker.properties、alert.properties、quartz.properties
and etc.
+> Files such as dolphinscheduler-daemon.sh, datasource.properties,
zookeeper.properties, common.properties, application-api.properties,
master.properties, worker.properties, alert.properties, quartz.properties, etc.
+* Startup and Shutdown DS Cluster.
-* Startup/Shutdown DS Cluster
-> The system will load masters, workers, alertServer, apiServers and other
parameters inside the file to startup/shutdown DS cluster.
+> The system will load masters, workers, alert-server, API-servers and other
parameters inside the file to startup or shutdown DS cluster.
#### File Content as Follows:
@@ -297,7 +300,7 @@
zkQuorum="192.168.xx.xx:2181,192.168.xx.xx:2181,192.168.xx.xx:2181"
installPath="/data1_1T/dolphinscheduler"
# Deployment user
-# Note: Deployment user needs 'sudo' privilege and has rights to operate HDFS
+# Note: Deployment user needs 'sudo' privilege and has rights to operate HDFS.
# Root directory must be created by the same user if using HDFS, otherwise
permission related issues will be raised.
deployUser="dolphinscheduler"
@@ -318,16 +321,16 @@ mailUser="xxxxxxxxxx"
# Mail password
mailPassword="xxxxxxxxxx"
-# Mail supports TLS set true if not set false
+# Whether mail supports TLS
starttlsEnable="true"
-# Mail supports SSL set true if not set false. Note: starttlsEnable and
sslEnable cannot both set true
+# Whether mail supports SSL. Note: starttlsEnable and sslEnable cannot both
set true.
sslEnable="false"
# Mail server host, same as mailServerHost
sslTrust="smtp.exmail.qq.com"
-# Specify which resource upload function to use for resources storage such as
sql files. And supported options are HDFS, S3 and NONE. HDFS for upload to HDFS
and NONE for not using this function.
+# Specify which resource upload function to use for resources storage, such as
sql files. And supported options are HDFS, S3 and NONE. HDFS for upload to HDFS
and NONE for not using this function.
resourceStorageType="NONE"
# if S3, write S3 address. HA, for example: s3a://dolphinscheduler，
@@ -355,7 +358,7 @@ hdfsRootUser="hdfs"
# Followings are Kerberos configs
-# Spicify Kerberos enable or not
+# Specify Kerberos enable or not
kerberosStartUp="false"
# Kdc krb5 config file path
@@ -395,7 +398,7 @@ apiServers="ds1"
### dolphinscheduler_env.sh [load environment variables configs]
When using shell to commit tasks, DS will load environment variables inside
dolphinscheduler_env.sh into the host.
-Types of tasks involved are: Shell task、Python task、Spark task、Flink
task、Datax task and etc.
+Types of tasks involved are: Shell, Python, Spark, Flink, DataX, etc.
```bash
export HADOOP_HOME=/opt/soft/hadoop
export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop
diff --git a/docs/en-us/dev/user_doc/architecture/design.md
b/docs/en-us/dev/user_doc/architecture/design.md
index 396e29c..5e0bdf8 100644
--- a/docs/en-us/dev/user_doc/architecture/design.md
+++ b/docs/en-us/dev/user_doc/architecture/design.md
@@ -1,9 +1,10 @@
# System Architecture Design
-Before explaining the architecture of the scheduling system, let's first
understand the commonly used terms of the scheduling system
+Before explain the architecture of the scheduling system, let's first get to
know the terms commonly used in scheduling system.
## Glossary
-**DAG：** The full name is Directed Acyclic Graph, referred to as DAG. Task
tasks in the workflow are assembled in the form of a directed acyclic graph,
and topological traversal is performed from nodes with zero degrees of entry
until there are no subsequent nodes. Examples are as follows:
+
+**DAG：** The full name is Directed Acyclic Graph, the abbreviation is DAG.
Tasks in the workflow are assembled in the form of a directed acyclic graph,
and topological traversal performs from zero degree entry nodes until there are
no subsequent nodes. Examples are as follows:
<p align="center">
<img src="/img/dag_examples_cn.jpg" alt="dag example" width="60%" />
@@ -12,27 +13,27 @@ Before explaining the architecture of the scheduling
system, let's first underst
</p>
</p>
-**Process definition**: Visualization formed by dragging task nodes and
establishing task node associations**DAG**
+**Process definition**: A visualized **DAG** formed by association of task
nodes which is created by dragging and dropping.
-**Process instance**: The process instance is the instantiation of the process
definition, which can be generated by manual start or scheduled scheduling.
Each time the process definition runs, a process instance is generated
+**Process instance**: The process instance is the instantiation of a process
definition, which can be generated by manual start or scheduled scheduling. A
process instance generates by everytime process definition runs.
-**Task instance**: The task instance is the instantiation of the task node in
the process definition, which identifies the specific task execution status
+**Task instance**: The task instance is the instantiation of the task node in
the process definition, which identifies specific task execution status.
-**Task type**: Currently supports SHELL, SQL, SUB_PROCESS (sub-process),
PROCEDURE, MR, SPARK, PYTHON, DEPENDENT (depends), and plans to support dynamic
plug-in expansion, note: **SUB_PROCESS** It is also a separate process
definition that can be started and executed separately
+**Task type**: Currently supports shell, SQL, SUB_PROCESS (sub-process),
PROCEDURE, MR, SPARK, PYTHON, DEPENDENT (dependent), and plans to support
dynamic plug-in extension. Note: **SUB_PROCESS** is also an individual process
definition that can be start and execute separately.
-**Scheduling method**: The system supports scheduled scheduling and manual
scheduling based on cron expressions. Command type support: start workflow,
start execution from current node, resume fault-tolerant workflow, resume pause
process, start execution from failed node, complement, timing, rerun, pause,
stop, resume waiting thread. Among them **Resume fault-tolerant workflow** and
**Resume waiting thread** The two command types are used by the internal
control of scheduling, and canno [...]
+**Scheduling method**: The system supports cron expressions based scheduling
and manual scheduling. Command type support: start workflow, start execution
from current node, resume fault-tolerant workflow, resume pause process, start
execution from failed node, complement, timing, rerun, pause, stop, resume
waiting thread. Among them **Resume fault-tolerant workflow** and **Resume
waiting thread** these command types are controlled by the internal scheduling,
and cannot be called from the [...]
-**Scheduled**: System adopts **quartz** distributed scheduler, and supports
the visual generation of cron expressions
+**Scheduled**: System adopts **quartz** distributed scheduler, and supports
the visual generation of cron expressions.
-**Rely**: The system not only supports **DAG** simple dependencies between the
predecessor and successor nodes, but also provides **task dependent** nodes,
supporting **between processes**
+**Rely**: The system not only supports **DAG** simple dependencies between the
predecessor and successor nodes, but also provides **task dependent** nodes,
supporting **dependencies between customized tasks of processes**.
-**Priority**: Support the priority of process instances and task instances, if
the priority of process instances and task instances is not set, the default is
first-in-first-out
+**Priority**: Support the priority of process instances and task instances. By
default, the priority is first-in-first-out.
-**Email alert**: Support **SQL task** Query result email sending, process
instance running result email alert and fault tolerance alert notification
+**Email alert**: Support send **SQL task** query result email, process
instance execution result email alert and fault tolerance alert notification.
-**Failure strategy**: For tasks running in parallel, if a task fails, two
failure strategy processing methods are provided. **Continue** refers to
regardless of the status of the task running in parallel until the end of the
process failure. **End** means that once a failed task is found, Kill will also
run the parallel task at the same time, and the process fails and ends
+**Failure strategy**: For parallel tasks, if a task fails, there are two
failure strategy remedy. **Continue** refers to regardless of the status of the
task running in parallel until the end of the process failure. **End** means
that once a task failed, kill the parallel task, and the process has a failure
result and end.
-**Complement**: Supplement historical data，Supports **interval parallel and
serial** two complement methods
+**Complement**: Complement historical data, Supports **interval parallel and
serial** two complement methods.
## System Structure
@@ -58,40 +59,40 @@ Before explaining the architecture of the scheduling
system, let's first underst
* **MasterServer**
- MasterServer adopts a distributed and centerless design concept.
MasterServer is mainly responsible for DAG task segmentation, task submission
monitoring, and monitoring the health status of other MasterServer and
WorkerServer at the same time.
+ MasterServer adopts a distributed and decentralized design concept.
MasterServer is mainly responsible for DAG task segmentation, task submission
monitoring, and monitoring the health status of other MasterServer and
WorkerServer at the same time.
When the MasterServer service starts, register a temporary node with
ZooKeeper, and perform fault tolerance by monitoring changes in the temporary
node of ZooKeeper.
MasterServer provides monitoring services based on netty.
#### The Service Mainly Includes:
- - **Distributed Quartz** distributed scheduling component, which is mainly
responsible for the start and stop operations of scheduled tasks. When Quartz
starts the task, there will be a thread pool inside the Master that is
specifically responsible for the follow-up operation of the processing task
+ - **Distributed Quartz** distributed scheduling component, which is mainly
responsible for the start and stop operations of schedule tasks. When Quartz
starts the task, there will be a thread pool inside the Master responsible for
the follow-up operation of the processing task.
- - **MasterSchedulerThread** is a scanning thread that regularly scans the
**command** table in the database and performs different business operations
according to different **command types**
+ - **MasterSchedulerThread** is a scanning thread that regularly scans the
**command** table in the database and runs different business operations
according to different **command types**.
- - **MasterExecThread** is mainly responsible for DAG task segmentation,
task submission monitoring, and logical processing of various command types
+ - **MasterExecThread** is mainly responsible for DAG task segmentation,
task submission monitoring, and logical processing to different command types.
- - **MasterTaskExecThread** is mainly responsible for the persistence of
tasks
+ - **MasterTaskExecThread** is mainly responsible for the persistence to
tasks.
* **WorkerServer**
WorkerServer also adopts a distributed and decentralized design concept.
WorkerServer is mainly responsible for task execution and providing log
services.
When the WorkerServer service starts, register a temporary node with
ZooKeeper and maintain a heartbeat.
- Server provides monitoring services based on netty. Worker
+ Server provides monitoring services based on netty.
#### The Service Mainly Includes:
- - **Fetch TaskThread** is mainly responsible for continuously getting
tasks from **Task Queue**, and calling **TaskScheduleThread** corresponding
executor according to different task types.
+ - **Fetch TaskThread** is mainly responsible for continuously getting
tasks from the **Task Queue**, and calling **TaskScheduleThread** corresponding
executor according to different task types.
* **ZooKeeper**
- ZooKeeper service, MasterServer and WorkerServer nodes in the system all
use ZooKeeper for cluster management and fault tolerance. In addition, the
system is based on ZooKeeper for event monitoring and distributed locks.
+ ZooKeeper service, MasterServer and WorkerServer nodes in the system all
use ZooKeeper for cluster management and fault tolerance. In addition, the
system implements event monitoring and distributed locks based on ZooKeeper.
- We have also implemented queues based on Redis, but we hope that
DolphinScheduler depends on as few components as possible, so we finally
removed the Redis implementation.
+ We have also implemented queues based on Redis, but we hope
DolphinScheduler depends on as few components as possible, so we finally
removed the Redis implementation.
* **Task Queue**
- Provide task queue operation, the current queue is also implemented based
on ZooKeeper. Because there is less information stored in the queue, there is
no need to worry about too much data in the queue. In fact, we have tested the
millions of data storage queues, which has no impact on system stability and
performance.
+ Provide task queue operation, the current queue is also implement base on
ZooKeeper. Due to little information stored in the queue, there is no need to
worry about excessive data in the queue. In fact, we have tested the millions
of data storage in queues, which has no impact on system stability and
performance.
* **Alert**
@@ -99,12 +100,12 @@ Before explaining the architecture of the scheduling
system, let's first underst
* **API**
- The API interface layer is mainly responsible for processing requests from
the front-end UI layer. The service uniformly provides RESTful APIs to provide
request services to the outside world. Interfaces include workflow creation,
definition, query, modification, release, logoff, manual start, stop, pause,
resume, start execution from the node and so on.
+ The API interface layer is mainly responsible for processing requests from
the front-end UI layer. The service uniformly provides RESTful APIs to provide
request services to external.
+ Interfaces include workflow creation, definition, query, modification,
release, logoff, manual start, stop, pause, resume, start execution from
specific node, etc.
* **UI**
- The front-end page of the system provides various visual operation
interfaces of the system,See more
- at [Introduction to Functions](../guide/homepage.md) section。
+ The front-end page of the system provides various visual operation
interfaces of the system, see more at [Introduction to
Functions](../guide/homepage.md) section.
### Architecture Design Ideas
@@ -112,17 +113,17 @@ Before explaining the architecture of the scheduling
system, let's first underst
##### Centralized Thinking
-The centralized design concept is relatively simple. The nodes in the
distributed cluster are divided into roles according to roles, which are
roughly divided into two roles:
+The centralized design concept is relatively simple. The nodes in the
distributed cluster are roughly divided into two roles according to
responsibilities:
<p align="center">
<img
src="https://analysys.github.io/easyscheduler_docs_cn/images/master_slave.png";
alt="master-slave character" width="50%" />
</p>
-- The role of the master is mainly responsible for task distribution and
monitoring the health status of the slave, and can dynamically balance the task
to the slave, so that the slave node will not be in a "busy dead" or "idle
dead" state.
-- The role of Worker is mainly responsible for task execution and maintenance
and Master's heartbeat, so that Master can assign tasks to Slave.
+- The role of the master is mainly responsible for task distribution and
monitoring the health status of the slave, and can dynamically balance the task
to the slave, so that the slave node won't be in a "busy dead" or "idle dead"
state.
+- The role of Worker is mainly responsible for task execution and heartbeat
maintenance to the Master, so that Master can assign tasks to Slave.
Problems in centralized thought design:
-- Once there is a problem with the Master, the dragons are headless and the
entire cluster will collapse. In order to solve this problem, most of the
Master/Slave architecture models adopt the design scheme of active and standby
Master, which can be hot standby or cold standby, or automatic switching or
manual switching, and more and more new systems are beginning to have The
ability to automatically elect and switch Master to improve the availability of
the system.
+- Once there is a problem with the Master, the team grow aimless without
commander and the entire cluster collapse. In order to solve this problem, most
of the Master and Slave architecture models adopt the design scheme of active
and standby Master, which can be hot standby or cold standby, or automatic
switching or manual switching. More and more new systems are beginning to have
ability to automatically elect and switch Master to improve the availability of
the system.
- Another problem is that if the Scheduler is on the Master, although it can
support different tasks in a DAG running on different machines, it will cause
the Master to be overloaded. If the Scheduler is on the slave, all tasks in a
DAG can only submit jobs on a certain machine. When there are more parallel
tasks, the pressure on the slave may be greater.
##### Decentralized
@@ -131,23 +132,20 @@ Problems in centralized thought design:
<img
src="https://analysys.github.io/easyscheduler_docs_cn/images/decentralization.png";
alt="Decentralization" width="50%" />
</p>
-- In the decentralized design, there is usually no concept of Master/Slave,
all roles are the same, the status is equal, the global Internet is a typical
decentralized distributed system, any node equipment connected to the network
is down, All will only affect a small range of functions.
-- The core design of decentralized design is that there is no "manager"
different from other nodes in the entire distributed system, so there is no
single point of failure. However, because there is no "manager" node, each node
needs to communicate with other nodes to obtain the necessary machine
information, and the unreliability of distributed system communication greatly
increases the difficulty of implementing the above functions.
+- In the decentralized design, there is usually no concept of Master or Slave.
All roles are the same, the status is equal, the global Internet is a typical
decentralized distributed system. Any node connected to the network goes down,
will only affect a small range of functions.
+- The core design of decentralized design is that there is no distinct
"manager" different from other nodes in the entire distributed system, so there
is no single point failure. However, because there is no "manager" node, each
node needs to communicate with other nodes to obtain the necessary machine
information, and the unreliability of distributed system communication greatly
increases the difficulty to implement the above functions.
- In fact, truly decentralized distributed systems are rare. Instead, dynamic
centralized distributed systems are constantly pouring out. Under this
architecture, the managers in the cluster are dynamically selected, rather than
preset, and when the cluster fails, the nodes of the cluster will automatically
hold "meetings" to elect new "managers" To preside over the work. The most
typical case is Etcd implemented by ZooKeeper and Go language.
-
-
-
-- The decentralization of DolphinScheduler is that the Master/Worker is
registered in ZooKeeper, and the Master cluster and Worker cluster are
centerless, and the ZooKeeper distributed lock is used to elect one of the
Master or Worker as the "manager" to perform the task.
+- The decentralization of DolphinScheduler is that the Master and Worker
register in ZooKeeper, for implement the centerless feature to Master cluster
and Worker cluster. Use the ZooKeeper distributed lock to elect one of the
Master or Worker as the "manager" to perform the task.
#### Distributed Lock Practice
-DolphinScheduler uses ZooKeeper distributed lock to realize that only one
Master executes Scheduler at the same time, or only one Worker executes the
submission of tasks.
+DolphinScheduler uses ZooKeeper distributed lock to implement only one Master
executes Scheduler at the same time, or only one Worker executes the submission
of tasks.
1. The core process algorithm for acquiring distributed locks is as follows:
<p align="center">
<img
src="https://analysys.github.io/easyscheduler_docs_cn/images/distributed_lock.png";
alt="Obtain distributed lock process" width="50%" />
</p>
-2. Flow chart of implementation of Scheduler thread distributed lock in
DolphinScheduler:
+2. Flow diagram of implementation of Scheduler thread distributed lock in
DolphinScheduler:
<p align="center">
<img src="/img/distributed_lock_procss.png" alt="Obtain distributed lock
process" width="50%" />
</p>
@@ -155,36 +153,37 @@ DolphinScheduler uses ZooKeeper distributed lock to
realize that only one Master
#### Insufficient Thread Loop Waiting Problem
-- If there is no sub-process in a DAG, if the number of data in the Command
is greater than the threshold set by the thread pool, the process directly
waits or fails.
-- If many sub-processes are nested in a large DAG, the following figure will
produce a "dead" state:
+- If there is no sub-process in a DAG, when the number of data in the Command
is greater than the threshold set by the thread pool, the process directly
waits or fails.
+- If a large DAG nests many sub-processes, there will produce a "dead" state
as the following figure:
<p align="center">
<img
src="https://analysys.github.io/easyscheduler_docs_cn/images/lack_thread.png";
alt="Insufficient threads waiting loop problem" width="50%" />
</p>
-In the above figure, MainFlowThread waits for the end of SubFlowThread1,
SubFlowThread1 waits for the end of SubFlowThread2, SubFlowThread2 waits for
the end of SubFlowThread3, and SubFlowThread3 waits for a new thread in the
thread pool, then the entire DAG process cannot end, so that the threads cannot
be released. In this way, the state of the child-parent process loop waiting is
formed. At this time, unless a new Master is started to add threads to break
such a "stalemate", the sched [...]
+In the above figure, MainFlowThread waits for the end of SubFlowThread1,
SubFlowThread1 waits for the end of SubFlowThread2, SubFlowThread2 waits for
the end of SubFlowThread3, and SubFlowThread3 waits for a new thread in the
thread pool, then the entire DAG process cannot finish, and the threads cannot
be released. In this situation, the state of the child-parent process loop
waiting is formed. At this moment, unless a new Master is started and add
threads to break such a "stalemate", t [...]
It seems a bit unsatisfactory to start a new Master to break the deadlock, so
we proposed the following three solutions to reduce this risk:
-1. Calculate the sum of all Master threads, and then calculate the number of
threads required for each DAG, that is, pre-calculate before the DAG process is
executed. Because it is a multi-master thread pool, the total number of threads
is unlikely to be obtained in real time.
-2. Judge the single-master thread pool. If the thread pool is full, let the
thread fail directly.
+1. Calculate the sum of all Master threads, and then calculate the number of
threads required for each DAG, that is, pre-calculate before the DAG process
executes. Because it is a multi-master thread pool, it is unlikely to obtain
the total number of threads in real time.
+2. Judge whether the single-master thread pool is full, let the thread fail
directly when fulfilled.
3. Add a Command type with insufficient resources. If the thread pool is
insufficient, suspend the main process. In this way, there are new threads in
the thread pool, which can make the process suspended by insufficient resources
wake up to execute again.
-note: The Master Scheduler thread is executed by FIFO when acquiring the
Command.
+Note: The Master Scheduler thread executes by FIFO when acquiring the Command.
-So we chose the third way to solve the problem of insufficient threads.
+So we choose the third way to solve the problem of insufficient threads.
#### Fault-Tolerant Design
-Fault tolerance is divided into service downtime fault tolerance and task
retry, and service downtime fault tolerance is divided into master fault
tolerance and worker fault tolerance.
+
+Fault tolerance divides into service downtime fault tolerance and task retry,
and service downtime fault tolerance divides into master fault tolerance and
worker fault tolerance.
##### Downtime Fault Tolerance
-The service fault-tolerance design relies on ZooKeeper's Watcher mechanism,
and the implementation principle is shown in the figure:
+The service fault-tolerance design relies on ZooKeeper's Watcher mechanism,
and the implementation principle shows in the figure:
<p align="center">
<img
src="https://analysys.github.io/easyscheduler_docs_cn/images/fault-tolerant.png";
alt="DolphinScheduler fault-tolerant design" width="40%" />
</p>
-Among them, the Master monitors the directories of other Masters and Workers.
If the remove event is heard, fault tolerance of the process instance or task
instance will be performed according to the specific business logic.
+Among them, the Master monitors the directories of other Masters and Workers.
If the remove event is triggered, perform fault tolerance of the process
instance or task instance according to the specific business logic.
@@ -194,11 +193,11 @@ Among them, the Master monitors the directories of other
Masters and Workers. If
<img src="/img/failover-master.jpg" alt="failover-master" width="50%" />
</p>
-Fault tolerance range: From the perspective of host, the fault tolerance range
of Master includes: own host + node host that does not exist in the registry,
and the entire process of fault tolerance will be locked;
+Fault tolerance range: From the perspective of host, the fault tolerance range
of Master includes: own host and node host that does not exist in the registry,
and the entire process of fault tolerance will be locked;
-Fault-tolerant content: Master's fault-tolerant content includes:
fault-tolerant process instances and task instances. Before fault-tolerant, it
compares the start time of the instance with the server start-up time, and
skips fault-tolerance if after the server start time;
+Fault-tolerant content: Master's fault-tolerant content includes:
fault-tolerant process instances and task instances. Before fault-tolerant,
compares the start time of the instance with the server start-up time, and
skips fault-tolerance if after the server start time;
-Fault-tolerant post-processing: After the fault tolerance of ZooKeeper Master
is completed, it is re-scheduled by the Scheduler thread in DolphinScheduler,
traverses the DAG to find the "running" and "submit successful" tasks, monitors
the status of its task instances for the "running" tasks, and "commits
successful" tasks It is necessary to determine whether the task queue already
exists. If it exists, the status of the task instance is also monitored. If it
does not exist, resubmit the [...]
+Fault-tolerant post-processing: After the fault tolerance of ZooKeeper Master
completed, then re-schedule by the Scheduler thread in DolphinScheduler,
traverses the DAG to find the "running" and "submit successful" tasks. Monitor
the status of its task instances for the "running" tasks, and for the "commits
successful" tasks, it is necessary to find out whether the task queue already
exists. If exists, monitor the status of the task instance. Otherwise, resubmit
the task instance.
- Worker fault tolerance：
@@ -208,60 +207,60 @@ Fault-tolerant post-processing: After the fault tolerance
of ZooKeeper Master is
Fault tolerance range: From the perspective of process instance, each Master
is only responsible for fault tolerance of its own process instance; it will
lock only when `handleDeadServer`;
-Fault-tolerant content: When sending the remove event of the Worker node, the
Master only fault-tolerant task instances. Before fault-tolerant, it compares
the start time of the instance with the server start-up time, and skips
fault-tolerance if after the server start time;
+Fault-tolerant content: When sending the remove event of the Worker node, the
Master only fault-tolerant task instances. Before fault-tolerant, compares the
start time of the instance with the server start-up time, and skips
fault-tolerance if after the server start time;
Fault-tolerant post-processing: Once the Master Scheduler thread finds that
the task instance is in the "fault-tolerant" state, it takes over the task and
resubmits it.
- Note: Due to "network jitter", the node may lose its heartbeat with ZooKeeper
in a short period of time, and the node's remove event may occur. For this
situation, we use the simplest way, that is, once the node and ZooKeeper
timeout connection occurs, then directly stop the Master or Worker service.
+ Note: Due to "network jitter", the node may lose heartbeat with ZooKeeper in
a short period of time, and the node's remove event may occur. For this
situation, we use the simplest way, that is, once the node and ZooKeeper
timeout connection occurs, then directly stop the Master or Worker service.
##### Task Failed and Try Again
-Here we must first distinguish the concepts of task failure retry, process
failure recovery, and process failure rerun:
+Here we must first distinguish the concepts of task failure retry, process
failure recovery, and process failure re-run:
-- Task failure retry is at the task level and is automatically performed by
the scheduling system. For example, if a Shell task is set to retry for 3
times, it will try to run it again up to 3 times after the Shell task fails.
-- Process failure recovery is at the process level and is performed manually.
Recovery can only be performed **from the failed node** or **from the current
node**
-- Process failure rerun is also at the process level and is performed
manually, rerun is performed from the start node
+- Task failure retry is at the task level and is automatically performed by
the schedule system. For example, if a Shell task sets to retry for 3 times, it
will try to run it again up to 3 times after the Shell task fails.
+- Process failure recovery is at the process level and is performed manually.
Recovery can only perform **from the failed node** or **from the current node**.
+- Process failure re-run is also at the process level and is performed
manually, re-run perform from the beginning node.
-Next to the topic, we divide the task nodes in the workflow into two types.
+Next to the main point, we divide the task nodes in the workflow into two
types.
-- One is a business node, which corresponds to an actual script or processing
statement, such as Shell node, MR node, Spark node, and dependent node.
+- One is a business node, which corresponds to an actual script or process
command, such as shell node, MR node, Spark node, and dependent node.
-- There is also a logical node, which does not do actual script or statement
processing, but only logical processing of the entire process flow, such as
sub-process sections.
+- Another is a logical node, which does not operate actual script or process
command, but only logical processing to the entire process flow, such as
sub-process sections.
-Each **business node** can be configured with the number of failed retries.
When the task node fails, it will automatically retry until it succeeds or
exceeds the configured number of retries. **Logical node** Failure retry is not
supported. But the tasks in the logical node support retry.
+Each **business node** can configure the number of failed retries. When the
task node fails, it will automatically retry until it succeeds or exceeds the
retry times. **Logical node** failure retry is not supported, but the tasks in
the logical node support.
-If there is a task failure in the workflow that reaches the maximum number of
retries, the workflow will fail to stop, and the failed workflow can be
manually rerun or process recovery operation
+If there is a task failure in the workflow that reaches the maximum retry
times, the workflow will fail and stop, and the failed workflow can be manually
re-run or process recovery operations.
#### Task Priority Design
-In the early scheduling design, if there is no priority design and the fair
scheduling design is used, the task submitted first may be completed at the
same time as the task submitted later, and the process or task priority cannot
be set, so We have redesigned this, and our current design is as follows:
+In the early schedule design, if there is no priority design and use the fair
scheduling, the task submitted first may complete at the same time with the
task submitted later, thus invalid the priority of process or task. So we have
re-designed this, and our current design is as follows:
-- According to **priority of different process instances** priority over
**priority of the same process instance** priority over **priority of tasks
within the same process**priority over **tasks within the same
process**submission order from high to Low task processing.
- - The specific implementation is to parse the priority according to the
JSON of the task instance, and then save the **process instance
priority_process instance id_task priority_task id** information in the
ZooKeeper task queue, when obtained from the task queue, pass String comparison
can get the tasks that need to be executed first
+- According to **the priority of different process instances** prior over
**priority of the same process instance** prior over **priority of tasks within
the same process** prior over **tasks within the same process**, process task
submission order from highest to Lowest.
+ - The specific implementation is to parse the priority according to the
JSON of the task instance, and then save the **process instance
priority_process instance id_task priority_task id** information to the
ZooKeeper task queue. When obtain from the task queue, we can get the highest
priority task by comparing string.
- - The priority of the process definition is to consider that some
processes need to be processed before other processes. This can be configured
when the process is started or scheduled to start. There are 5 levels in total,
which are HIGHEST, HIGH, MEDIUM, LOW, and LOWEST. As shown below
+ - The priority of the process definition is to consider that some
processes need to process before other processes. Configure the priority when
the process starts or schedules. There are 5 levels in total, which are
HIGHEST, HIGH, MEDIUM, LOW, and LOWEST. As shown below
<p align="center">
<img
src="https://user-images.githubusercontent.com/10797147/146744784-eb351b14-c94a-4ed6-8ba4-5132c2a3d116.png";
alt="Process priority configuration" width="40%" />
</p>
- - The priority of the task is also divided into 5 levels, followed by
HIGHEST, HIGH, MEDIUM, LOW, LOWEST. As shown below
+ - The priority of the task is also divides into 5 levels, ordered by
HIGHEST, HIGH, MEDIUM, LOW, LOWEST. As shown below:
<p align="center">
<img
src="https://user-images.githubusercontent.com/10797147/146744830-5eac611f-5933-4f53-a0c6-31613c283708.png";
alt="Task priority configuration" width="35%" />
</p>
#### Logback and Netty Implement Log Access
-- Since Web (UI) and Worker are not necessarily on the same machine, viewing
the log cannot be like querying a local file. There are two options:
- - Put logs on the ES search engine
- - Obtain remote log information through netty communication
+- Since Web (UI) and Worker are not always on the same machine, to view the
log cannot be like querying a local file. There are two options:
+ - Put logs on the ES search engine.
+ - Obtain remote log information through netty communication.
-- In consideration of the lightness of DolphinScheduler as much as possible,
so I chose gRPC to achieve remote access to log information.
+- In consideration of the lightness of DolphinScheduler as much as possible,
so choose gRPC to achieve remote access to log information.
<p align="center">
<img src="https://analysys.github.io/easyscheduler_docs_cn/images/grpc.png";
alt="grpc remote access" width="50%" />
</p>
-- We use the FileAppender and Filter functions of the custom Logback to
realize that each task instance generates a log file.
+- We use the customized FileAppender and Filter functions from Logback to
implement each task instance generates one log file.
- FileAppender is mainly implemented as follows：
```java
@@ -290,7 +289,7 @@ In the early scheduling design, if there is no priority
design and the fair sche
}
```
-Generate logs in the form of /process definition id/process instance id/task
instance id.log
+Generate logs in the form of /process definition id /process instance id /task
instance id.log
- Filter to match the thread name starting with TaskLogInfo:
@@ -314,22 +313,23 @@ public class TaskLogFilter extends Filter<ILoggingEvent> {
## Module Introduction
-- dolphinscheduler-alert alarm module, providing AlertServer service.
+- dolphinscheduler-alert: alarm module, providing AlertServer service.
-- dolphinscheduler-api web application module, providing ApiServer service.
+- dolphinscheduler-api: web application module, providing ApiServer service.
-- dolphinscheduler-common General constant enumeration, utility class, data
structure or base class
+- dolphinscheduler-common: contains general constant enumeration, utility
class, data structure and base class.
-- dolphinscheduler-dao provides operations such as database access.
+- dolphinscheduler-dao: provides operations such as database access.
-- dolphinscheduler-remote client and server based on netty
+- dolphinscheduler-remote: client and server based on netty.
-- dolphinscheduler-server MasterServer and WorkerServer services
+- dolphinscheduler-server: MasterServer and WorkerServer services.
-- dolphinscheduler-service service module, including Quartz, ZooKeeper, log
client access service, easy to call server module and api module
+- dolphinscheduler-service: service module, including Quartz, ZooKeeper, log
client access service, convenient for calling from server module and API module.
-- dolphinscheduler-ui front-end module
+- dolphinscheduler-ui: front-end module.
## Sum Up
-From the perspective of scheduling, this article preliminarily introduces the
architecture principles and implementation ideas of the big data distributed
workflow scheduling system-DolphinScheduler. To be continued
+
+From the perspective of scheduling, this article preliminarily introduces the
architecture principles and implementation ideas of the big data distributed
workflow scheduling system: DolphinScheduler. To be continued.
diff --git a/docs/en-us/dev/user_doc/architecture/load-balance.md
b/docs/en-us/dev/user_doc/architecture/load-balance.md
index e21abba..5cf2d4e 100644
--- a/docs/en-us/dev/user_doc/architecture/load-balance.md
+++ b/docs/en-us/dev/user_doc/architecture/load-balance.md
@@ -6,17 +6,17 @@ Load balancing refers to the reasonable allocation of server
pressure through ro
DolphinScheduler-Master allocates tasks to workers, and by default provides
three algorithms:
-Weighted random (random)
+- Weighted random (random)
-Smoothing polling (roundrobin)
+- Smoothing polling (round-robin)
-Linear load (lowerweight)
+- Linear load (lower weight)
The default configuration is the linear load.
-As the routing is done on the client side, the master service, you can change
master.host.selector in master.properties to configure the algorithm what you
want.
+As the routing sets on the client side, the master service, you can change
master.host.selector in master.properties to configure the algorithm.
-e.g. master.host.selector = random (case-insensitive)
+e.g. master.host.selector=random (case-insensitive)
## Worker Load Balancing Configuration
@@ -24,34 +24,34 @@ The configuration file is worker.properties
### Weight
-All of the above load algorithms are weighted based on weights, which affect
the outcome of the triage. You can set different weights for different machines
by modifying the worker.weight value.
+All the load algorithms above are weighted based on weights, which affect the
routing outcome. You can set different weights for different machines by
modifying the `worker.weight` value.
### Preheating
-With JIT optimisation in mind, we will let the worker run at low power for a
period of time after startup so that it can gradually reach its optimal state,
a process we call preheating. If you are interested, you can read some articles
about JIT.
+Consider JIT optimization, worker runs at low power for a period of time after
startup, so that it can gradually reach its optimal state, a process we call
preheating. If you are interested, you can read some articles about JIT.
-So the worker will gradually reach its maximum weight over time after it
starts (by default ten minutes, we don't provide a configuration item, you can
change it and submit a PR if needed).
+So the worker gradually reaches its maximum weight with time after starts up (
by default ten minutes, there is no configuration about the pre-heating
duration, it's recommend to submit a PR if have needs to change the duration).
-## Load Balancing Algorithm Breakdown
+## Load Balancing Algorithm in Details
### Random (Weighted)
-This algorithm is relatively simple, one of the matched workers is selected at
random (the weighting affects his weighting).
+This algorithm is relatively simple, select a worker by random (the weight
affects its weighting).
### Smoothed Polling (Weighted)
-An obvious drawback of the weighted polling algorithm. Namely, under certain
specific weights, weighted polling scheduling generates an uneven sequence of
instances, and this unsmoothed load may cause some instances to experience
transient high loads, leading to a risk of system downtime. To address this
scheduling flaw, we provide a smooth weighted polling algorithm.
+An obvious drawback of the weighted polling algorithm, which is under special
weights circumstance, weighted polling scheduling generates an imbalanced
sequence of instances, and this unsmooth load may cause some instances to
experience transient high loads, leading to a risk of system crash. To address
this scheduling flaw, we provide a smooth weighted polling algorithm.
-Each worker is given two weights, weight (which remains constant after warm-up
is complete) and current_weight (which changes dynamically), for each route.
The current_weight + weight is iterated over all the workers, and the weight of
all the workers is added up and counted as total_weight, then the worker with
the largest current_weight is selected as the worker for this task.
current_weight-total_weight.
+Each worker has two weights parameters, weight (which remains constant after
warm-up is complete) and current_weight (which changes dynamically). For every
route, calculate the current_weight + weight and is iterated over all the
workers, the weight of all the workers sum up and count as total_weight, then
the worker with the largest current_weight is selected as the worker for this
task. By meantime, set worker's current_weight-total_weight.
### Linear Weighting (Default Algorithm)
-The algorithm reports its own load information to the registry at regular
intervals. We base our judgement on two main pieces of information
+This algorithm reports its own load information to the registry at regular
intervals. Make decision on two main pieces of information:
- load average (default is the number of CPU cores * 2)
- available physical memory (default is 0.3, in G)
-If either of the two is lower than the configured item, then this worker will
not participate in the load. (no traffic will be allocated)
+If either of these is lower than the configured item, then this worker will
not participate in the load. (no traffic will be allocated)
You can customise the configuration by changing the following properties in
worker.properties
diff --git a/docs/en-us/dev/user_doc/architecture/metadata.md
b/docs/en-us/dev/user_doc/architecture/metadata.md
index 616e50f..54ebc56 100644
--- a/docs/en-us/dev/user_doc/architecture/metadata.md
+++ b/docs/en-us/dev/user_doc/architecture/metadata.md
@@ -1,18 +1,16 @@
# MetaData
-<a name="V5KOl"></a>
-
## DolphinScheduler DB Table Overview
| Table Name | Comment |
| :---: | :---: |
-| t_ds_access_token | token for access ds backend |
+| t_ds_access_token | token for access DolphinScheduler backend |
| t_ds_alert | alert detail |
| t_ds_alertgroup | alert group |
| t_ds_command | command detail |
| t_ds_datasource | data source |
| t_ds_error_command | error command detail |
-| t_ds_process_definition | process difinition |
+| t_ds_process_definition | process definition |
| t_ds_process_instance | process instance |
| t_ds_project | project |
| t_ds_queue | queue |
@@ -20,62 +18,52 @@
| t_ds_relation_process_instance | sub process |
| t_ds_relation_project_user | project related to user |
| t_ds_relation_resources_user | resource related to user |
-| t_ds_relation_udfs_user | UDF related to user |
+| t_ds_relation_udfs_user | UDF functions related to user |
| t_ds_relation_user_alertgroup | alert group related to user |
| t_ds_resources | resoruce center file |
-| t_ds_schedules | process difinition schedule |
+| t_ds_schedules | process definition schedule |
| t_ds_session | user login session |
| t_ds_task_instance | task instance |
| t_ds_tenant | tenant |
| t_ds_udfs | UDF resource |
| t_ds_user | user detail |
-| t_ds_version | ds version |
+| t_ds_version | DolphinScheduler version |
---
-<a name="XCLy1"></a>
-
## E-R Diagram
-<a name="5hWWZ"></a>
-
### User Queue DataSource
-- Multiple users can belong to one tenant
-- The queue field in the t_ds_user table stores the queue_name information in
the t_ds_queue table, but t_ds_tenant stores queue information using queue_id.
During the execution of the process definition, the user queue has the highest
priority. If the user queue is empty, the tenant queue is used.
-- The user_id field in the t_ds_datasource table indicates the user who
created the data source. The user_id in t_ds_relation_datasource_user indicates
the user who has permission to the data source.
-<a name="7euSN"></a>
+- One tenant can own Multiple users.
+- The queue field in the t_ds_user table stores the queue_name information in
the t_ds_queue table, t_ds_tenant stores queue information using queue_id
column. During the execution of the process definition, the user queue has the
highest priority. If the user queue is null, use the tenant queue.
+- The user_id field in the t_ds_datasource table shows the user who create the
data source. The user_id in t_ds_relation_datasource_user shows the user who
has permission to the data source.
### Project Resource Alert
-- User can have multiple projects, User project authorization completes the
relationship binding using project_id and user_id in t_ds_relation_project_user
table
-- The user_id in the t_ds_projcet table represents the user who created the
project, and the user_id in the t_ds_relation_project_user table represents
users who have permission to the project
-- The user_id in the t_ds_resources table represents the user who created the
resource, and the user_id in t_ds_relation_resources_user represents the user
who has permissions to the resource
-- The user_id in the t_ds_udfs table represents the user who created the UDF,
and the user_id in the t_ds_relation_udfs_user table represents a user who has
permission to the UDF
-<a name="JEw4v"></a>
+- User can have multiple projects, user project authorization completes the
relationship binding using project_id and user_id in t_ds_relation_project_user
table.
+- The user_id in the t_ds_projcet table represents the user who create the
project, and the user_id in the t_ds_relation_project_user table represents
users who have permission to the project.
+- The user_id in the t_ds_resources table represents the user who create the
resource, and the user_id in t_ds_relation_resources_user represents the user
who has permissions to the resource.
+- The user_id in the t_ds_udfs table represents the user who create the UDF,
and the user_id in the t_ds_relation_udfs_user table represents a user who has
permission to the UDF.
### Command Process Task
<br
/>
-- A project has multiple process definitions, a process definition can
generate multiple process instances, and a process instance can generate
multiple task instances
-- The t_ds_schedulers table stores the timing schedule information for process
difinition
-- The data stored in the t_ds_relation_process_instance table is used to deal
with that the process definition contains sub-processes,
parent_process_instance_id field represents the id of the main process instance
containing the child process, process_instance_id field represents the id of
the sub-process instance, parent_task_instance_id field represents the task
instance id of the sub-process node
+- A project has multiple process definitions, a process definition can
generate multiple process instances, and a process instance can generate
multiple task instances.
+- The t_ds_schedulers table stores the specified time schedule information for
process definition.
+- The data stored in the t_ds_relation_process_instance table is used to deal
with the sub-processes of a process definition, parent_process_instance_id
field represents the id of the main process instance who contains child
processes, process_instance_id field represents the id of the sub-process
instance, parent_task_instance_id field represents the task instance id of the
sub-process node.
- The process instance table and the task instance table correspond to the
t_ds_process_instance table and the t_ds_task_instance table, respectively.
---
-<a name="yd79T"></a>
-
## Core Table Schema
-<a name="6bVhH"></a>
-
### t_ds_process_definition
| Field | Type | Comment |
@@ -86,10 +74,10 @@
| release_state | tinyint | process definition release
state：0:offline,1:online |
| project_id | int | project id |
| user_id | int | process definition creator id |
-| process_definition_json | longtext | process definition json content |
-| description | text | process difinition desc |
+| process_definition_json | longtext | process definition JSON content |
+| description | text | process definition description |
| global_params | text | global parameters |
-| flag | tinyint | process is available: 0 not available, 1 available |
+| flag | tinyint | whether process available: 0 not available, 1 available |
| locations | text | Node location information |
| connects | text | Node connection information |
| receivers | text | receivers |
@@ -98,8 +86,8 @@
| timeout | int | timeout |
| tenant_id | int | tenant id |
| update_time | datetime | update time |
-
-<a name="t5uxM"></a>
+| modify_by | varchar | define user modify the process |
+| resource_ids | varchar | resource id set |
### t_ds_process_instance
@@ -108,38 +96,36 @@
| id | int | primary key |
| name | varchar | process instance name |
| process_definition_id | int | process definition id |
-| state | tinyint | process instance Status: 0 commit succeeded, 1 running, 2
prepare to pause, 3 pause, 4 prepare to stop, 5 stop, 6 fail, 7 succeed, 8 need
fault tolerance, 9 kill, 10 wait for thread, 11 wait for dependency to complete
|
-| recovery | tinyint | process instance failover flag：0:normal,1:failover
instance |
+| state | tinyint | process instance Status: 0 successful commit, 1 running, 2
prepare to pause, 3 pause, 4 prepare to stop, 5 stop, 6 fail, 7 succeed, 8 need
fault tolerance, 9 kill, 10 wait for thread, 11 wait for dependency to complete
|
+| recovery | tinyint | process instance failover flag：0: normal,1: failover
instance needs restart |
| start_time | datetime | process instance start time |
| end_time | datetime | process instance end time |
| run_times | int | process instance run times |
| host | varchar | process instance host |
-| command_type | tinyint | command type：0 start ,1 Start from the current
node,2 Resume a fault-tolerant process,3 Resume Pause Process, 4 Execute from
the failed node,5 Complement, 6 dispatch, 7 re-run, 8 pause, 9 stop ,10 Resume
waiting thread |
-| command_param | text | json command parameters |
-| task_depend_type | tinyint | task depend type. 0: only current node,1:before
the node,2:later nodes |
+| command_type | tinyint | command type：0 start ,1 start from the current
node,2 resume a fault-tolerant process,3 resume from pause process, 4 execute
from the failed node,5 complement, 6 dispatch, 7 re-run, 8 pause, 9 stop, 10
resume waiting thread |
+| command_param | text | JSON command parameters |
+| task_depend_type | tinyint | node dependency type: 0 current node, 1
forward, 2 backward |
| max_try_times | tinyint | max try times |
-| failure_strategy | tinyint | failure strategy. 0:end the process when node
failed,1:continue running the other nodes when node failed |
-| warning_type | tinyint | warning type. 0:no warning,1:warning if process
success,2:warning if process failed,3:warning if success |
+| failure_strategy | tinyint | failure strategy, 0: end the process when node
failed,1: continue run the other nodes when failed |
+| warning_type | tinyint | warning type 0: no warning, 1: warning if process
success, 2: warning if process failed, 3: warning whatever results |
| warning_group_id | int | warning group id |
| schedule_time | datetime | schedule time |
| command_start_time | datetime | command start time |
| global_params | text | global parameters |
-| process_instance_json | longtext | process instance json |
-| flag | tinyint | process instance is available: 0 not available, 1 available
|
+| process_instance_json | longtext | process instance JSON |
+| flag | tinyint | whether process instance is available: 0 not available, 1
available |
| update_time | timestamp | update time |
| is_sub_process | int | whether the process is sub process: 1 sub-process, 0
not sub-process |
| executor_id | int | executor id |
-| locations | text | Node location information |
-| connects | text | Node connection information |
-| history_cmd | text | history commands of process instance operation |
-| dependence_schedule_times | text | depend schedule fire time |
-| process_instance_priority | int | process instance priority. 0 Highest,1
High,2 Medium,3 Low,4 Lowest |
-| worker_group_id | int | worker group id |
-| timeout | int | time out |
+| locations | text | node location information |
+| connects | text | node connection information |
+| history_cmd | text | history commands, record all the commands to a instance
|
+| dependence_schedule_times | text | depend schedule estimate time |
+| process_instance_priority | int | process instance priority. 0 highest,1
high,2 medium,3 low,4 lowest |
+| worker_group | varchar | worker group who assign the task |
+| timeout | int | timeout |
| tenant_id | int | tenant id |
-<a name="tHZsY"></a>
-
### t_ds_task_instance
| Field | Type | Comment |
@@ -149,7 +135,7 @@
| task_type | varchar | task type |
| process_definition_id | int | process definition id |
| process_instance_id | int | process instance id |
-| task_json | longtext | task content json |
+| task_json | longtext | task content JSON |
| state | tinyint | Status: 0 commit succeeded, 1 running, 2 prepare to pause,
3 pause, 4 prepare to stop, 5 stop, 6 fail, 7 succeed, 8 need fault tolerance,
9 kill, 10 wait for thread, 11 wait for dependency to complete |
| submit_time | datetime | task submit time |
| start_time | datetime | task start time |
@@ -160,31 +146,48 @@
| alert_flag | tinyint | whether alert |
| retry_times | int | task retry times |
| pid | int | pid of task |
-| app_link | varchar | yarn app id |
-| flag | tinyint | taskinstance is available: 0 not available, 1 available |
-| retry_interval | int | retry interval when task failed |
+| app_link | varchar | Yarn app id |
+| flag | tinyint | task instance is available : 0 not available, 1 available |
+| retry_interval | int | retry interval when task failed |
| max_retry_times | int | max retry times |
-| task_instance_priority | int | task instance priority:0 Highest,1 High,2
Medium,3 Low,4 Lowest |
-| worker_group_id | int | worker group id |
+| task_instance_priority | int | task instance priority:0 highest,1 high,2
medium,3 low,4 lowest |
+| worker_group | varchar | worker group who assign the task |
-<a name="gLGtm"></a>
+#### t_ds_schedules
+
+| Field | Type | Comment |
+| --- | --- | --- |
+| id | int | primary key |
+| process_definition_id | int | process definition id |
+| start_time | datetime | schedule start time |
+| end_time | datetime | schedule end time |
+| crontab | varchar | crontab expression |
+| failure_strategy | tinyint | failure strategy: 0 end,1 continue |
+| user_id | int | user id |
+| release_state | tinyint | release status: 0 not yet released,1 released |
+| warning_type | tinyint | warning type: 0: no warning, 1: warning if process
success, 2: warning if process failed, 3: warning whatever results |
+| warning_group_id | int | warning group id |
+| process_instance_priority | int | process instance priority:0 highest,1
high,2 medium,3 low,4 lowest |
+| worker_group | varchar | worker group who assign the task |
+| create_time | datetime | create time |
+| update_time | datetime | update time |
### t_ds_command
| Field | Type | Comment |
| --- | --- | --- |
| id | int | primary key |
-| command_type | tinyint | Command type: 0 start workflow, 1 start execution
from current node, 2 resume fault-tolerant workflow, 3 resume pause process, 4
start execution from failed node, 5 complement, 6 schedule, 7 rerun, 8 pause, 9
stop, 10 resume waiting thread |
+| command_type | tinyint | command type: 0 start workflow, 1 start execution
from current node, 2 resume fault-tolerant workflow, 3 resume pause process, 4
start execution from failed node, 5 complement, 6 schedule, 7 re-run, 8 pause,
9 stop, 10 resume waiting thread |
| process_definition_id | int | process definition id |
-| command_param | text | json command parameters |
-| task_depend_type | tinyint | Node dependency type: 0 current node, 1
forward, 2 backward |
-| failure_strategy | tinyint | Failed policy: 0 end, 1 continue |
-| warning_type | tinyint | Alarm type: 0 is not sent, 1 process is sent
successfully, 2 process is sent failed, 3 process is sent successfully and all
failures are sent |
-| warning_group_id | int | warning group |
+| command_param | text | JSON command parameters |
+| task_depend_type | tinyint | node dependency type: 0 current node, 1
forward, 2 backward |
+| failure_strategy | tinyint | failed policy: 0 end, 1 continue |
+| warning_type | tinyint | alarm type: 0 no alarm, 1 alarm if process success,
2: alarm if process failed, 3: warning whatever results |
+| warning_group_id | int | warning group id |
| schedule_time | datetime | schedule time |
| start_time | datetime | start time |
| executor_id | int | executor id |
-| dependence | varchar | dependence |
+| dependence | varchar | dependence column |
| update_time | datetime | update time |
-| process_instance_priority | int | process instance priority: 0 Highest,1
High,2 Medium,3 Low,4 Lowest |
-| worker_group_id | int | worker group id |
\ No newline at end of file
+| process_instance_priority | int | process instance priority: 0 highest,1
high,2 medium,3 low,4 lowest |
+| worker_group_id | int | worker group who assign the task |
\ No newline at end of file
diff --git a/docs/en-us/dev/user_doc/architecture/task-structure.md
b/docs/en-us/dev/user_doc/architecture/task-structure.md
index 0b1c28a..37864fe 100644
--- a/docs/en-us/dev/user_doc/architecture/task-structure.md
+++ b/docs/en-us/dev/user_doc/architecture/task-structure.md
@@ -2,16 +2,16 @@
## Overall Tasks Storage Structure
-All tasks created in DolphinScheduler are saved in the t_ds_process_definition
table.
+All tasks in DolphinScheduler are saved in the `t_ds_process_definition` table.
-The following shows the 't_ds_process_definition' table structure:
+The following shows the `t_ds_process_definition` table structure:
No. | field | type | description
-------- | ---------| -------- | ---------
1|id|int(11)|primary key
2|name|varchar(255)|process definition name
3|version|int(11)|process definition version
-4|release_state|tinyint(4)|release status of process definition: 0 not online,
1 online
+4|release_state|tinyint(4)|release status of process definition: 0 not
released, 1 released
5|project_id|int(11)|project id
6|user_id|int(11)|user id of the process definition
7|process_definition_json|longtext|process definition JSON
@@ -26,10 +26,10 @@ No. | field | type | description
16|timeout|int(11) |timeout
17|tenant_id|int(11) |tenant id
18|update_time|datetime|update time
-19|modify_by|varchar(36)|specifics of the user that made the modification
+19|modify_by|varchar(36)|specify the user that made the modification
20|resource_ids|varchar(255)|resource ids
-The 'process_definition_json' field is the core field, which defines the task
information in the DAG diagram, and it is stored in JSON format.
+The `process_definition_json` field is the core field, which defines the task
information in the DAG diagram, and it is stored in JSON format.
The following table describes the common data structure.
No. | field | type | description
@@ -64,9 +64,9 @@ Data example:
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
1|id | |String| task Id|
-2|type ||String |task type |SHELL
+2|type | |String |task type |SHELL
3| name| |String|task name |
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| |rawScript |String| Shell script |
6| | localParams| Array|customized local parameters||
7| | resourceList| Array|resource files||
@@ -141,7 +141,7 @@ No.|parameter name||type|description |note
1|id | |String|task id|
2|type ||String |task type |SQL
3| name| |String|task name|
-4| params| |Object|customized parameters|Json format
+4| params| |Object|customized parameters|JSON format
5| |type |String |database type
6| |datasource |Int |datasource id
7| |sql |String |query SQL statement
@@ -150,10 +150,10 @@ No.|parameter name||type|description |note
10| |title |String | mail title
11| |receivers |String |receivers
12| |receiversCc |String |CC receivers
-13| |showType | String|display type of mail|optionals: TABLE or ATTACHMENT
+13| |showType | String|display type of mail|options: TABLE or ATTACHMENT
14| |connParams | String|connect parameters
15| |preStatements | Array|preposition SQL statements
-16| | postStatements| Array|postposition SQL statements||
+16| | postStatements| Array|post-position SQL statements||
17| | localParams| Array|customized parameters||
18|description | |String|description | |
19|runFlag | |String |execution flag| |
@@ -243,7 +243,7 @@ No.|parameter name||type|description |notes
1|id | |String| task Id|
2|type ||String |task type |SPARK
3| name| |String|task name |
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| |mainClass |String | main class
6| |mainArgs | String| execution arguments
7| |others | String| other arguments
@@ -341,7 +341,7 @@ No.|parameter name||type|description |notes
1|id | |String| task Id|
2|type ||String |task type |MR
3| name| |String|task name |
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| |mainClass |String | main class
6| |mainArgs | String|execution arguments
7| |others | String|other arguments
@@ -424,7 +424,7 @@ No.|parameter name||type|description |notes
1|id | |String| task Id|
2|type ||String |task type|PYTHON
3| name| |String|task name|
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| |rawScript |String| Python script|
6| | localParams| Array|customized local parameters||
7| | resourceList| Array|resource files||
@@ -498,7 +498,7 @@ No.|parameter name||type|description |notes
1|id | |String|task Id|
2|type ||String |task type|FLINK
3| name| |String|task name|
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| |mainClass |String |main class
6| |mainArgs | String|execution arguments
7| |others | String|other arguments
@@ -593,7 +593,7 @@ No.|parameter name||type|description |notes
1|id | |String|task Id|
2|type ||String |task type|HTTP
3| name| |String|task name|
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| |url |String |request url
6| |httpMethod | String|http method|GET,POST,HEAD,PUT,DELETE
7| | httpParams| Array|http parameters
@@ -677,7 +677,7 @@ No.|parameter name||type|description |notes
1|id | |String| task Id|
2|type ||String |task type|DATAX
3| name| |String|task name|
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| |customConfig |Int |specify whether use customized config| 0 none
customized, 1 customized
6| |dsType |String | datasource type
7| |dataSource |Int | datasource ID
@@ -688,7 +688,7 @@ No.|parameter name||type|description |notes
12| |jobSpeedByte |Int |job speed limiting(bytes)
13| |jobSpeedRecord | Int|job speed limiting(records)
14| |preStatements | Array|preposition SQL
-15| | postStatements| Array|postposition SQL
+15| | postStatements| Array|post-position SQL
16| | json| String|customized configs|valid if customConfig=1
17| | localParams| Array|customized parameters|valid if customConfig=1
18|description | |String|description| |
@@ -764,7 +764,7 @@ No.|parameter name||type|description |notes
1|id | |String|task ID|
2|type ||String |task type|SQOOP
3| name| |String|task name|
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| | concurrency| Int|concurrency rate
6| | modelType|String |flow direction|import,export
7| |sourceType|String |datasource type|
@@ -903,7 +903,7 @@ No.|parameter name||type|description |notes
1|id | |String| task ID|
2|type ||String |task type|SHELL
3| name| |String|task name|
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| |processDefinitionId |Int| process definition ID
6|description | |String|description | |
7|runFlag | |String |execution flag| |
@@ -962,7 +962,7 @@ No.|parameter name||type|description |notes
1|id | |String| task ID|
2|type ||String |task type|DEPENDENT
3| name| |String|task name|
-4| params| |Object|customized parameters |Json format
+4| params| |Object|customized parameters |JSON format
5| |rawScript |String|Shell script|
6| | localParams| Array|customized local parameters||
7| | resourceList| Array|resource files||
![https://analysys.github.io/easyscheduler_docs_cn/images/master_slave.png"](https://analysys.github.io/easyscheduler_docs_cn/images/master_slave.png"){kind=link}
![https://analysys.github.io/easyscheduler_docs_cn/images/decentralization.png"](https://analysys.github.io/easyscheduler_docs_cn/images/decentralization.png"){kind=link}
![https://analysys.github.io/easyscheduler_docs_cn/images/distributed_lock.png"](https://analysys.github.io/easyscheduler_docs_cn/images/distributed_lock.png"){kind=link}
![https://analysys.github.io/easyscheduler_docs_cn/images/lack_thread.png"](https://analysys.github.io/easyscheduler_docs_cn/images/lack_thread.png"){kind=link}
![https://analysys.github.io/easyscheduler_docs_cn/images/fault-tolerant.png"](https://analysys.github.io/easyscheduler_docs_cn/images/fault-tolerant.png"){kind=link}
![https://user-images.githubusercontent.com/10797147/146744784-eb351b14-c94a-4ed6-8ba4-5132c2a3d116.png"](https://user-images.githubusercontent.com/10797147/146744784-eb351b14-c94a-4ed6-8ba4-5132c2a3d116.png"){kind=link}
![https://user-images.githubusercontent.com/10797147/146744830-5eac611f-5933-4f53-a0c6-31613c283708.png"](https://user-images.githubusercontent.com/10797147/146744830-5eac611f-5933-4f53-a0c6-31613c283708.png"){kind=link}
![https://analysys.github.io/easyscheduler_docs_cn/images/grpc.png"](https://analysys.github.io/easyscheduler_docs_cn/images/grpc.png"){kind=link}