This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch 3.0.0-alpha-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
The following commit(s) were added to refs/heads/3.0.0-alpha-prepare by this
push:
new a82f5026b7 [cherry-pick] some commit until apr 11 (#9446)
a82f5026b7 is described below
commit a82f5026b78bfe21f2ee370503d6a88d48c9ff80
Author: Jiajie Zhong <[email protected]>
AuthorDate: Mon Apr 11 21:47:46 2022 +0800
[cherry-pick] some commit until apr 11 (#9446)
* [python] Add missing document
which including `configuration`, `run example`
`how to connect remote server`
close: #9286, #9284, #8917
* [python] Recover python release properties
This patch recovers the properties `python.sign.skip=false`
when the combined profile `release,python` is used.
also close: #9433
* [doc] Add some dev missing doc
Including general-setting, task-definition, audit-log
and they related img
Co-authored-by: Tq <[email protected]>
Co-authored-by: Tq <[email protected]>
---
docs/configs/docsdev.js | 26 ++++++
docs/docs/en/guide/howto/general-setting.md | 22 +++++
docs/docs/en/guide/monitor.md | 15 +++-
docs/docs/en/guide/project/task-definition.md | 13 +++
docs/docs/zh/guide/howto/general-setting.md | 19 ++++
docs/docs/zh/guide/monitor.md | 14 ++-
docs/docs/zh/guide/project/task-definition.md | 9 ++
docs/img/new_ui/dev/monitor/audit-log.jpg | Bin 0 -> 227221 bytes
docs/img/new_ui/dev/project/task-definition.jpg | Bin 0 -> 801293 bytes
dolphinscheduler-dist/pom.xml | 3 -
dolphinscheduler-python/pom.xml | 6 ++
.../pydolphinscheduler/docs/source/config.rst | 96 +++++++++++++++++++++
.../docs/source/{ => howto}/index.rst | 27 ++----
.../docs/source/howto/remote-submit.rst | 51 +++++++++++
.../pydolphinscheduler/docs/source/index.rst | 2 +
.../pydolphinscheduler/docs/source/start.rst | 32 +++++++
dolphinscheduler-standalone-server/pom.xml | 5 --
17 files changed, 304 insertions(+), 36 deletions(-)
diff --git a/docs/configs/docsdev.js b/docs/configs/docsdev.js
index ad39afefb6..b552ff4d50 100644
--- a/docs/configs/docsdev.js
+++ b/docs/configs/docsdev.js
@@ -97,6 +97,10 @@ export default {
title: 'Task Instance',
link:
'/en-us/docs/dev/user_doc/guide/project/task-instance.html',
},
+ {
+ title: 'Task Definition',
+ link:
'/zh-cn/docs/dev/user_doc/guide/project/task-definition.html',
+ },
]
},
{
@@ -251,6 +255,15 @@ export default {
title: 'Security',
link: '/en-us/docs/dev/user_doc/guide/security.html',
},
+ {
+ title: 'How-To',
+ children: [
+ {
+ title: 'General Setting',
+ link:
'/en-us/docs/dev/user_doc/guide/howto/general-setting.html',
+ }
+ ],
+ },
{
title: 'Open API',
link: '/en-us/docs/dev/user_doc/guide/open-api.html',
@@ -398,6 +411,10 @@ export default {
title: '任务实例',
link:
'/zh-cn/docs/dev/user_doc/guide/project/task-instance.html',
},
+ {
+ title: '任务定义',
+ link:
'/zh-cn/docs/dev/user_doc/guide/project/task-definition.html',
+ },
]
},
{
@@ -552,6 +569,15 @@ export default {
title: '安全中心',
link: '/zh-cn/docs/dev/user_doc/guide/security.html',
},
+ {
+ title: '常见操作',
+ children: [
+ {
+ title: '通用设置',
+ link:
'/zh-cn/docs/dev/user_doc/guide/howto/general-setting.html',
+ }
+ ],
+ },
{
title: 'API调用',
link: '/zh-cn/docs/dev/user_doc/guide/open-api.html',
diff --git a/docs/docs/en/guide/howto/general-setting.md
b/docs/docs/en/guide/howto/general-setting.md
new file mode 100644
index 0000000000..fc2588bba4
--- /dev/null
+++ b/docs/docs/en/guide/howto/general-setting.md
@@ -0,0 +1,22 @@
+# General Setting
+
+## Language
+
+DolphinScheduler supports two types of built-in language which include
`English` and `Chinese`. You could click the button
+on the top control bar named `English` and `Chinese` and change it to another
one when you want to switch the language.
+The entire DolphinScheduler page language will shift when you switch the
language selection.
+
+## Theme
+
+DolphinScheduler supports two types of built-in theme which include `Dark` and
`Light`. When you want to change the theme
+of DolphinScheduler, all you have to do is click the button named `Dark`(or
`Light`) on the top control bar and on the left
+of to [language](#language) control button.
+
+## Time Zone
+
+DolphinScheduler support time zone setting. The build-in time zone is based on
the server you run DolphinScheduler. You could
+click the button on the right of the [language](#language) button and then
click `Choose timeZone` to choose the time zone
+you want to switch. All time related components will adjust their time zone
according to the time zone setting you select.
+
+DolphinScheduler uses UTC time for the internal communication, the time zone
you choose only changes the display time format
+base on the UTC time. When you choose time zone `TIME ZONE`, we just convert
the UTC time to the zone you prefer.
diff --git a/docs/docs/en/guide/monitor.md b/docs/docs/en/guide/monitor.md
index 420a190728..5ccb95d0da 100644
--- a/docs/docs/en/guide/monitor.md
+++ b/docs/docs/en/guide/monitor.md
@@ -4,19 +4,19 @@
- Service management is mainly to monitor and display the health status and
basic information of each service in the system.
-## Monitor Master Server
+### Master Server
- Mainly related to master information.
-## Monitor Worker Server
+### Worker Server
- Mainly related to worker information.
-## Monitor DB
+### Database
- Mainly the health status of the DB.
@@ -24,9 +24,18 @@
## Statistics Management
+### Statistics
+
- Number of commands wait to be executed: statistics of the `t_ds_command`
table data.
- The number of failed commands: statistics of the `t_ds_error_command` table
data.
- Number of tasks wait to run: count the data of `task_queue` in the ZooKeeper.
- Number of tasks wait to be killed: count the data of `task_kill` in the
ZooKeeper.
+
+### Audit Log
+
+The audit log provides information about who accesses the system and the
operations made to the system and record related
+time, which strengthen the security of the system and maintenance.
+
+
diff --git a/docs/docs/en/guide/project/task-definition.md
b/docs/docs/en/guide/project/task-definition.md
new file mode 100644
index 0000000000..53646eb2d9
--- /dev/null
+++ b/docs/docs/en/guide/project/task-definition.md
@@ -0,0 +1,13 @@
+# Task Definition
+
+Task definition allows to modify or operate tasks at the task level rather
than modifying them in the workflow definition.
+We already have workflow level task editor in [workflow
definition](workflow-definition.md) which you can click the specific
+workflow and then edit its task definition. It is depressing when you want to
edit the task definition but do not remember
+which workflow it belongs to. So we decide to add `Task Definition` view under
`Task` menu.
+
+
+
+In this view, you can create, query, update, delete task definition by click
the related button in `operation` column. The
+most exciting thing is you could query task by task name in the wildcard, and
it is useful when you only remember the task
+name but forget which workflow it belongs to. It is also supported query by
the task name alone with `Task Type` or
+`Workflow Name`
diff --git a/docs/docs/zh/guide/howto/general-setting.md
b/docs/docs/zh/guide/howto/general-setting.md
new file mode 100644
index 0000000000..1b7744b3f9
--- /dev/null
+++ b/docs/docs/zh/guide/howto/general-setting.md
@@ -0,0 +1,19 @@
+# 通用配置
+
+## 语言
+
+DolphinScheduler 支持两种内置语言，包括 `English` 和 `Chinese` 。您可以点击顶部控制栏名为 `English` 或
`Chinese` 的按钮切换语言。
+当您将语言从一种切换为另一种时，您所有 DolphinScheduler 的页面语言页面将发生变化。
+
+## 主题
+
+DolphinScheduler 支持两种类型的内置主题，包括 `Dark` 和 `Light`。当您想改变主题时，只需单击顶部控制栏在 [语言](#语言)
左侧名为 `Dark`(or `Light`)
+的按钮即可。
+
+## 时区
+
+DolphinScheduler 支持时区设置。默认时区基于您运行 DolphinScheduler 服务器的时区。如果你想要切换时区，可以点击
[语言](#语言) 按钮右侧的时区按钮，
+然后点击 `请选择时区` 进行时区选择。当切换完成后，所有与时间相关的组件都将更改。
+
+DolphinScheduler 内部通讯，以及任务主流程中使用 UTC 时间，在界面上显示的时间均是在 UTC
时间基础上进行时区格式化的，意味着当你切换了时区后，
+只是将格式化的时间从一个时区切换到另一个时区。
diff --git a/docs/docs/zh/guide/monitor.md b/docs/docs/zh/guide/monitor.md
index 34c15fe8cc..f5e8709f8e 100644
--- a/docs/docs/zh/guide/monitor.md
+++ b/docs/docs/zh/guide/monitor.md
@@ -4,19 +4,19 @@
- 服务管理主要是对系统中的各个服务的健康状况和基本信息的监控和显示
-### Master 监控
+### Master
- 主要是 master 的相关信息。
-### Worker 监控
+### Worker
- 主要是 worker 的相关信息。
-### DB 监控
+### Database
- 主要是 DB 的健康状况
@@ -24,9 +24,17 @@
## 统计管理
+### Statistics
+
- 待执行命令数：统计 t_ds_command 表的数据
- 执行失败的命令数：统计 t_ds_error_command 表的数据
- 待运行任务数：统计 Zookeeper 中 task_queue 的数据
- 待杀死任务数：统计 Zookeeper 中 task_kill 的数据
+
+### 审计日志
+
+审计日志的记录提供了有关谁访问了系统，以及他或她在给定时间段内执行了哪些操作的信息，他对于维护安全都很有用。
+
+
diff --git a/docs/docs/zh/guide/project/task-definition.md
b/docs/docs/zh/guide/project/task-definition.md
new file mode 100644
index 0000000000..8079484378
--- /dev/null
+++ b/docs/docs/zh/guide/project/task-definition.md
@@ -0,0 +1,9 @@
+# 任务定义
+
+任务定义允许您在基于任务级别而不是在工作流中操作修改任务。再此之前，我们已经有了工作流级别的任务编辑器，你可以在[工作流定义](workflow-definition.md)
+单击特定的工作流，然后编辑任务的定义。当您想编辑特定的任务定义但不记得它属于哪个工作流时，这是令人沮丧的。所以我们决定在 `任务` 菜单下添加 `任务定义`
视图。
+
+
+
+在该视图中，您可以通过单击 `操作` 列中的相关按钮来进行创建、查询、更新、删除任务定义。最令人兴奋的是您可以通过通配符进行全部任务查询，当您只
+记得任务名称但忘记它属于哪个工作流时是非常有用的。也支持通过任务名称结合使用 `任务类型` 或 `工作流程名称` 进行查询。
diff --git a/docs/img/new_ui/dev/monitor/audit-log.jpg
b/docs/img/new_ui/dev/monitor/audit-log.jpg
new file mode 100644
index 0000000000..bc396ad303
Binary files /dev/null and b/docs/img/new_ui/dev/monitor/audit-log.jpg differ
diff --git a/docs/img/new_ui/dev/project/task-definition.jpg
b/docs/img/new_ui/dev/project/task-definition.jpg
new file mode 100644
index 0000000000..cd2db4e8ce
Binary files /dev/null and b/docs/img/new_ui/dev/project/task-definition.jpg
differ
diff --git a/dolphinscheduler-dist/pom.xml b/dolphinscheduler-dist/pom.xml
index 34455743a0..facd272043 100644
--- a/dolphinscheduler-dist/pom.xml
+++ b/dolphinscheduler-dist/pom.xml
@@ -70,9 +70,6 @@
<profiles>
<profile>
<id>release</id>
- <properties>
- <python.sign.skip>false</python.sign.skip>
- </properties>
<build>
<plugins>
<plugin>
diff --git a/dolphinscheduler-python/pom.xml b/dolphinscheduler-python/pom.xml
index 4a66459cae..9d46e14fa3 100644
--- a/dolphinscheduler-python/pom.xml
+++ b/dolphinscheduler-python/pom.xml
@@ -29,6 +29,12 @@
<packaging>jar</packaging>
<profiles>
+ <profile>
+ <id>release</id>
+ <properties>
+ <python.sign.skip>false</python.sign.skip>
+ </properties>
+ </profile>
<profile>
<id>python</id>
<build>
diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst
b/dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst
new file mode 100644
index 0000000000..bfba2ecfde
--- /dev/null
+++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/config.rst
@@ -0,0 +1,96 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Configuration
+=============
+
+Export Configuration File
+-------------------------
+
+pydolphinscheduler allows you to change the built-in configurations via CLI or
editor you like. pydolphinscheduler
+integrated built-in configurations in its package, but you could also export
it locally by CLI
+
+.. code-block:: bash
+
+ $ pydolphinscheduler config --init
+
+And it will create a new YAML file in the path
`~/pydolphinscheduler/config.yaml` by default. If you want to export
+it to another path, you should set `PYDOLPHINSCHEDULER_HOME` before you run
command :code:`pydolphinscheduler config --init`.
+
+.. code-block:: bash
+
+ $ export PYDOLPHINSCHEDULER_HOME=<CUSTOM_PATH>
+ $ pydolphinscheduler config --init
+
+After that, your configuration file will export into
`<CUSTOM_PATH>/config.yaml` instead of the default path.
+
+Change Configuration
+--------------------
+
+In section `export configuration file`_ you export the configuration file
locally, and as a local file, you could
+edit it with any editor you like. After you save your change in your editor,
the latest configuration will work
+when you run your workflow code.
+
+You could also query or change the configuration via CLI :code:`config --get
<config>` or :code:`config --get <config> <val>`.
+Both `--get` and `--set` could be call one or more times in single command,
and you could only set the leaf
+node of the configuration but could get the parent configuration, there are
simple examples below:
+
+.. code-block:: bash
+
+ # Get single configuration in the leaf node
+ $ pydolphinscheduler config --get java_gateway.address
+ The configuration query as below:
+
+ java_gateway.address = 127.0.0.1
+
+ # Get multiple configuration in the leaf node
+ $ pydolphinscheduler config --get java_gateway.address --get
java_gateway.port
+ The configuration query as below:
+
+ java_gateway.address = 127.0.0.1
+ java_gateway.port = 25333
+
+ # Get parent configuration which contain multiple leaf nodes
+ $ pydolphinscheduler config --get java_gateway
+ The configuration query as below:
+
+ java_gateway = ordereddict([('address', '127.0.0.1'), ('port', 25333),
('auto_convert', True)])
+
+ # Set single configuration
+ $ pydolphinscheduler config --set java_gateway.address 192.168.1.1
+ Set configuration done.
+
+ # Set multiple configuration
+ $ pydolphinscheduler config --set java_gateway.address 192.168.1.1 --set
java_gateway.port 25334
+ Set configuration done.
+
+ # Set configuration not in leaf node will fail
+ $ pydolphinscheduler config --set java_gateway 192.168.1.1,25334,True
+ Raise error.
+
+For more information about our CLI, you could see document :doc:`cli`.
+
+All Configurations
+------------------
+
+Here are all our configurations for pydolphinscheduler.
+
+.. literalinclude:: ../../src/pydolphinscheduler/core/default_config.yaml
+ :language: yaml
+ :lines: 18-
+
+
diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst
b/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst
similarity index 60%
copy from dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst
copy to dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst
index ad4c93d1f9..e83b1631cf 100644
--- a/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst
+++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/index.rst
@@ -15,29 +15,12 @@
specific language governing permissions and limitations
under the License.
-PyDolphinScheduler
-==================
-
-**PyDolphinScheduler** is Python API for `Apache DolphinScheduler
<https://dolphinscheduler.apache.org>`_,
-which allow you definition your workflow by Python code, aka workflow-as-codes.
-
-I could go and find how to :ref:`install <start:getting started>` the project.
Or if you want to see simply example
-then go and see :doc:`tutorial` for more detail.
+How To
+======
+In this section
.. toctree::
:maxdepth: 2
-
- start
- tutorial
- concept
- tasks/index
- cli
- api
-
-Indices and tables
-==================
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
+
+ remote-submit
diff --git
a/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst
b/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst
new file mode 100644
index 0000000000..43d03cce38
--- /dev/null
+++
b/dolphinscheduler-python/pydolphinscheduler/docs/source/howto/remote-submit.rst
@@ -0,0 +1,51 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Submit Your Code from Different machine
+=======================================
+
+Generally, we use pydolphinscheduler as a client to DolphinScheduler, and
consider we may change our workflow
+code frequently, the best practice is running :ref:`python gateway service
<start:start python gateway service>`
+in your server machine and submit the workflow code from your development
machine, like a laptop or PC. This behavior
+is supported by pydolphinscheduler out of box with one or two single command
lines.
+
+Export Configuration File
+-------------------------
+
+.. code-block:: bash
+
+ $ pydolphinscheduler config --init
+
+your could find more detail in :ref:`configuration exporting <config:export
configuration file>`
+
+Run API Server in Other Host
+----------------------------
+
+.. code-block:: bash
+
+ $ pydolphinscheduler config --set java_gateway.address
<your-api-server-ip-or-hostname>
+
+your could find more detail in :ref:`configuration setting <config:change
configuration>`
+
+Run API Server in Other Port
+----------------------------
+
+.. code-block:: bash
+
+ $ pydolphinscheduler config --set java_gateway.port
<your-python-gateway-service-port>
+
+your could find more detail in :ref:`configuration setting <config:change
configuration>`
diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst
b/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst
index ad4c93d1f9..24ad107ad6 100644
--- a/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst
+++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/index.rst
@@ -32,7 +32,9 @@ then go and see :doc:`tutorial` for more detail.
tutorial
concept
tasks/index
+ howto/index
cli
+ config
api
Indices and tables
diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst
b/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst
index 489f4e8eb3..e411d7bf72 100644
--- a/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst
+++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst
@@ -107,6 +107,37 @@ the server is health if keyword `ApiApplicationServer` in
the console.
yaml config path `python-gateway.enabled : true` in api-server's
configuration path in `api-server/conf/application.yaml`.
The default value is true and Python gateway service start when api server
is been started.
+Run an Example
+--------------
+
+Before run an example for pydolphinscheduler, you should get the example code
from it source code. You could run
+single bash command to get it
+
+.. code-block:: bash
+
+ $ wget
https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py
+
+or you could copy-paste the content from `tutorial source code`_. And then you
could run the example in your
+terminal
+
+.. code-block:: bash
+
+ $ python tutorial.py
+
+If you want to submit your workflow to a remote API server, which means that
your workflow script is different
+from the API server, you should first change pydolphinscheduler configuration
and then submit the workflow script
+
+.. code-block:: bash
+
+ $ pydolphinscheduler config --init
+ $ pydolphinscheduler config --set java_gateway.address
<your-api-server-ip-or-hostname>
+ $ python tutorial.py
+
+.. note::
+
+ You could see more information in :doc:`config` about all the
configurations pydolphinscheduler supported.
+
+
What's More
-----------
@@ -117,3 +148,4 @@ maybe you could go and play with all :doc:`tasks/index`
*PyDolphinScheduler* sup
.. _`instructions for all platforms here`:
https://wiki.python.org/moin/BeginnersGuide/Download
.. _`Apache DolphinScheduler`: https://dolphinscheduler.apache.org
.. _`install Apache DolphinScheduler`:
https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/guide/installation/standalone.html
+.. _`tutorial source code`:
https://raw.githubusercontent.com/apache/dolphinscheduler/dev/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/examples/tutorial.py
diff --git a/dolphinscheduler-standalone-server/pom.xml
b/dolphinscheduler-standalone-server/pom.xml
index 1b32b0b074..b0827b7a4f 100644
--- a/dolphinscheduler-standalone-server/pom.xml
+++ b/dolphinscheduler-standalone-server/pom.xml
@@ -50,11 +50,6 @@
<artifactId>dolphinscheduler-log-server</artifactId>
</dependency>
- <dependency>
- <groupId>org.apache.dolphinscheduler</groupId>
- <artifactId>dolphinscheduler-python</artifactId>
- </dependency>
-
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-test</artifactId>
