This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch main
in repository
https://gitbox.apache.org/repos/asf/dolphinscheduler-sdk-python.git
The following commit(s) were added to refs/heads/main by this push:
new c55c0f7 [migrate] Add tag 3.0.1 source code (#5)
c55c0f7 is described below
commit c55c0f7c6fab26fa4df58f2b6c5237bb0c1db621
Author: Jay Chung <[email protected]>
AuthorDate: Wed Nov 9 11:48:23 2022 +0800
[migrate] Add tag 3.0.1 source code (#5)
Add version 3.0.1 code from
https://github.com/apache/dolphinscheduler/tree/3.0.1/dolphinscheduler-python/pydolphinscheduler
---
DEVELOP.md | 25 ++-
README.md | 7 +-
RELEASE.md | 2 +-
docs/Makefile | 5 +
docs/source/_templates/versioning.html | 27 +++
docs/source/_templates/versions.html | 46 +++++
docs/source/api.rst | 6 +-
docs/source/conf.py | 38 +++-
docs/source/config.rst | 2 +-
setup.py | 14 +-
src/pydolphinscheduler/cli/commands.py | 8 +-
src/pydolphinscheduler/configuration.py | 193 +++++++++++++++++++++
src/pydolphinscheduler/core/__init__.py | 2 +-
src/pydolphinscheduler/core/process_definition.py | 13 +-
src/pydolphinscheduler/core/resource.py | 2 +-
src/pydolphinscheduler/core/task.py | 4 +-
src/pydolphinscheduler/default_config.yaml | 58 +++++++
.../examples/task_dependent_example.py | 2 +-
src/pydolphinscheduler/java_gateway.py | 2 +-
.../{core => models}/__init__.py | 24 ++-
src/pydolphinscheduler/models/base.py | 74 ++++++++
.../{core/resource.py => models/base_side.py} | 33 ++--
.../{core/resource.py => models/project.py} | 29 ++--
.../{core/resource.py => models/queue.py} | 31 ++--
.../{core/resource.py => models/tenant.py} | 32 ++--
src/pydolphinscheduler/models/user.py | 78 +++++++++
.../{core/__init__.py => models/worker_group.py} | 22 +--
src/pydolphinscheduler/tasks/condition.py | 2 +-
src/pydolphinscheduler/tasks/dependent.py | 6 +-
src/pydolphinscheduler/tasks/sql.py | 2 +-
src/pydolphinscheduler/tasks/switch.py | 27 ++-
tests/cli/test_config.py | 2 +-
tests/cli/test_version.py | 24 ++-
tests/core/test_configuration.py | 4 +-
tests/core/test_process_definition.py | 4 +-
tests/tasks/test_sql.py | 1 +
tests/testing/path.py | 2 +-
tox.ini | 18 +-
38 files changed, 713 insertions(+), 158 deletions(-)
diff --git a/DEVELOP.md b/DEVELOP.md
index bdd0416..9972094 100644
--- a/DEVELOP.md
+++ b/DEVELOP.md
@@ -34,7 +34,7 @@ Now, we should install all dependence to make sure we could
run test or check co
```shell
cd dolphinscheduler/dolphinscheduler-python/pydolphinscheduler
-python -m pip install .[dev]
+python -m pip install -e '.[dev]'
```
Next, we have to open pydolphinscheduler project in you editor. We recommend
you use [pycharm][pycharm]
@@ -146,10 +146,24 @@ python -m flake8
#### Testing
-## Build Docs
+## Build Document
We use [sphinx][sphinx] to build docs. Dolphinscheduler Python API CI would
automatically build docs when you submit pull request in
-GitHub. You may locally ensure docs could be built suceessfully in case the
failure blocks CI.
+GitHub. You may locally ensure docs could be built successfully in case the
failure blocks CI, you can build by tox or manual.
+
+### Build Document Automatically with tox
+
+We integrated document build process into tox, you can build the latest
document and all document(including history documents) via
+single command
+
+```shell
+# Build the latest document in dev branch
+tox -e doc-build
+# Build all documents, which including the latest and all history documents
+tox -e doc-build-multi
+```
+
+### Build Document Manually
To build docs locally, install sphinx and related python modules first via:
@@ -157,13 +171,16 @@ To build docs locally, install sphinx and related python
modules first via:
python -m pip install '.[doc]'
```
-Then
+Then go to document directory and execute the build command
```shell
cd pydolphinscheduler/docs/
make clean && make html
```
+> NOTE: We support build multiple versions of documents with
[sphinx-multiversion](https://holzhaus.github.io/sphinx-multiversion/master/index.html),
+> you can build with command `git fetch --tags && make clean && make
multiversion`
+
## Testing
pydolphinscheduler using [pytest][pytest] to test our codebase. GitHub Action
will run our test when you create
diff --git a/README.md b/README.md
index 316d604..71119bd 100644
--- a/README.md
+++ b/README.md
@@ -45,12 +45,15 @@ pydolphinscheduler version
# 0.1.0
```
+> NOTE: package apache-dolphinscheduler not work on above Python version
3.10(including itself) in Window operating system
+> due to dependence [py4j](https://pypi.org/project/py4j/) not work on those
environments.
+
Here we show you how to install and run a simple example of pydolphinscheduler
### Start Server And Run Example
Before you run an example, you have to start backend server. You could follow
-[development
setup](https://dolphinscheduler.apache.org/en-us/development/development-environment-setup.html)
+[development
setup](../../docs/docs/en/contribute/development-environment-setup.md)
section "DolphinScheduler Standalone Quick Start" to set up developer
environment. You have to start backend
and frontend server in this step, which mean that you could view
DolphinScheduler UI in your browser with URL
http://localhost:12345/dolphinscheduler
@@ -84,4 +87,4 @@ If you are interested in how to release
**PyDolphinScheduler**, you could go and
## What's more
-For more detail information, please go to see **PyDolphinScheduler**
[document](https://dolphinscheduler.apache.org/python/index.html)
+For more detail information, please go to see **PyDolphinScheduler**
latest(unreleased)
[document](https://dolphinscheduler.apache.org/python/dev/index.html)
diff --git a/RELEASE.md b/RELEASE.md
index 6c2b46e..c90107a 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -25,7 +25,7 @@ and it should be released together with
[apache-dolphinscheduler](https://github
## To ASF Distribution Directory
You could release to [ASF Distribution
Directory](https://downloads.apache.org/dolphinscheduler/) according to
-[release
guide](https://dolphinscheduler.apache.org/en-us/community/release-prepare.html)
in DolphinScheduler
+[release guide](../../docs/docs/en/contribute/release/release-prepare.md) in
DolphinScheduler
website.
## To PyPi
diff --git a/docs/Makefile b/docs/Makefile
index 985198a..ff2c4eb 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -24,6 +24,7 @@
# Add opts `turn warnings into errors` strict sphinx-build behavior
SPHINXOPTS ?= -W
SPHINXBUILD ?= sphinx-build
+SPHINXMULTIVERSION ?= sphinx-multiversion
SOURCEDIR = source
BUILDDIR = build
@@ -37,3 +38,7 @@ help:
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+# Create multiple version of docs
+multiversion:
+ @$(SPHINXMULTIVERSION) "$(SOURCEDIR)" "$(BUILDDIR)/html"
diff --git a/docs/source/_templates/versioning.html
b/docs/source/_templates/versioning.html
new file mode 100644
index 0000000..47136c4
--- /dev/null
+++ b/docs/source/_templates/versioning.html
@@ -0,0 +1,27 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% if versions %}
+<h3>{{ _('Versions') }}</h3>
+<ul>
+ {%- for item in versions %}
+ <li><a href="{{ item.url }}">{{ item.name }}</a></li>
+ {%- endfor %}
+</ul>
+{% endif %}
diff --git a/docs/source/_templates/versions.html
b/docs/source/_templates/versions.html
new file mode 100644
index 0000000..51b7271
--- /dev/null
+++ b/docs/source/_templates/versions.html
@@ -0,0 +1,46 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{%- if current_version %}
+<div class="rst-versions" data-toggle="rst-versions" role="note"
aria-label="versions">
+ <span class="rst-current-version" data-toggle="rst-current-version">
+ <span class="fa fa-book"> Other Versions</span>
+ v: {{ current_version.name }}
+ <span class="fa fa-caret-down"></span>
+ </span>
+ <div class="rst-other-versions">
+ {%- if versions.tags %}
+ <dl>
+ <dt>Tags</dt>
+ {%- for item in versions.tags %}
+ <dd><a href="{{ item.url }}">{{ item.name }}</a></dd>
+ {%- endfor %}
+ </dl>
+ {%- endif %}
+ {%- if versions.branches %}
+ <dl>
+ <dt>Branches</dt>
+ {%- for item in versions.branches %}
+ <dd><a href="{{ item.url }}">{{ item.name }}</a></dd>
+ {%- endfor %}
+ </dl>
+ {%- endif %}
+ </div>
+</div>
+{%- endif %}
diff --git a/docs/source/api.rst b/docs/source/api.rst
index 8e55ea5..b170b6f 100644
--- a/docs/source/api.rst
+++ b/docs/source/api.rst
@@ -24,10 +24,10 @@ Core
.. automodule:: pydolphinscheduler.core
:inherited-members:
-Sides
------
+Models
+------
-.. automodule:: pydolphinscheduler.side
+.. automodule:: pydolphinscheduler.models
:inherited-members:
Tasks
diff --git a/docs/source/conf.py b/docs/source/conf.py
index b162e0c..23fc117 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -26,17 +26,31 @@
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+
+import os
+import sys
+from pathlib import Path
+
+# For sphinx-multiversion, we need to build API docs of the corresponding
package version, related issue:
+# https://github.com/Holzhaus/sphinx-multiversion/issues/42
+pkg_src_dir = (
+ Path(os.environ.get("SPHINX_MULTIVERSION_SOURCEDIR", default="."))
+ .joinpath("../../src")
+ .resolve()
+)
+sys.path.insert(0, str(pkg_src_dir))
+# Debug to uncomment this to see the source path
+# print("=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=")
+# print(pkg_src_dir)
+# [print(p) for p in sys.path]
+# print("=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=")
# -- Project information -----------------------------------------------------
project = "pydolphinscheduler"
copyright = "2022, apache"
-author = "apache"
+author = "apache dolphinscheduler contributors"
# The full version, including alpha/beta/rc tags
release = "0.0.1"
@@ -60,11 +74,25 @@ extensions = [
# Add inline tabbed content
"sphinx_inline_tabs",
"sphinx_copybutton",
+ "sphinx_multiversion",
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
+# sphinx_multiversion configuration
+html_sidebars = {
+ "**": [
+ "versioning.html",
+ ],
+}
+# Match all exists tag for pydolphinscheduler expect version 2.0.4(not release
apache dolphinscheduler)
+smv_tag_whitelist = r"^(?!2.0.4)\d+\.\d+\.\d+$"
+smv_branch_whitelist = "dev"
+smv_remote_whitelist = r"^(origin|upstream)$"
+smv_released_pattern = "^refs/tags/.*$"
+smv_outputdir_format = "versions/{ref.name}"
+
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
diff --git a/docs/source/config.rst b/docs/source/config.rst
index feb147a..29a143d 100644
--- a/docs/source/config.rst
+++ b/docs/source/config.rst
@@ -200,7 +200,7 @@ All Configurations in File
Here are all our configurations for pydolphinscheduler.
-.. literalinclude:: ../../src/pydolphinscheduler/core/default_config.yaml
+.. literalinclude:: ../../src/pydolphinscheduler/default_config.yaml
:language: yaml
:lines: 18-
diff --git a/setup.py b/setup.py
index b1e3742..5d1903c 100644
--- a/setup.py
+++ b/setup.py
@@ -32,7 +32,7 @@ if sys.version_info[0] < 3:
logger = logging.getLogger(__name__)
-version = "3.0.0"
+version = "3.0.1"
# Start package required
prod = [
@@ -53,6 +53,10 @@ doc = [
"sphinx-click>=3.0",
"sphinx-inline-tabs",
"sphinx-copybutton>=0.4.0",
+ # Unreleased package have a feature we want(use correct version package
for API ref), so we install from
+ # GitHub directly, see also:
+ #
https://github.com/Holzhaus/sphinx-multiversion/issues/42#issuecomment-1210539786
+ "sphinx-multiversion @
git+https://github.com/Holzhaus/sphinx-multiversion#egg=sphinx-multiversion";,
]
test = [
@@ -135,7 +139,7 @@ setup(
],
project_urls={
"Homepage": "https://dolphinscheduler.apache.org";,
- "Documentation":
"https://dolphinscheduler.apache.org/python/index.html";,
+ "Documentation":
"https://dolphinscheduler.apache.org/python/dev/index.html";,
"Source":
"https://github.com/apache/dolphinscheduler/tree/dev/dolphinscheduler-python/";
"pydolphinscheduler",
"Issue Tracker": "https://github.com/apache/dolphinscheduler/issues?";
@@ -147,12 +151,12 @@ setup(
package_dir={"": "src"},
include_package_data=True,
package_data={
- "pydolphinscheduler": ["core/default_config.yaml"],
+ "pydolphinscheduler": ["default_config.yaml"],
},
platforms=["any"],
classifiers=[
# complete classifier list:
http://pypi.python.org/pypi?%3Aaction=list_classifiers
- "Development Status :: 3 - Alpha",
+ "Development Status :: 4 - Beta",
"Environment :: Console",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
@@ -165,6 +169,8 @@ setup(
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
"Programming Language :: Python :: Implementation :: CPython",
"Programming Language :: Python :: Implementation :: PyPy",
"Topic :: Software Development :: User Interfaces",
diff --git a/src/pydolphinscheduler/cli/commands.py
b/src/pydolphinscheduler/cli/commands.py
index e2ca86b..d78e503 100644
--- a/src/pydolphinscheduler/cli/commands.py
+++ b/src/pydolphinscheduler/cli/commands.py
@@ -20,8 +20,8 @@
import click
from click import echo
-from pydolphinscheduler import __version__
-from pydolphinscheduler.core.configuration import (
+import pydolphinscheduler
+from pydolphinscheduler.configuration import (
get_single_config,
init_config_file,
set_single_config,
@@ -48,9 +48,9 @@ def version(part: str) -> None:
"""Show current version of pydolphinscheduler."""
if part:
idx = version_option_val.index(part)
- echo(f"{__version__.split('.')[idx]}")
+ echo(f"{pydolphinscheduler.__version__.split('.')[idx]}")
else:
- echo(f"{__version__}")
+ echo(f"{pydolphinscheduler.__version__}")
@cli.command()
diff --git a/src/pydolphinscheduler/configuration.py
b/src/pydolphinscheduler/configuration.py
new file mode 100644
index 0000000..860f986
--- /dev/null
+++ b/src/pydolphinscheduler/configuration.py
@@ -0,0 +1,193 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Configuration module for pydolphinscheduler."""
+import os
+from pathlib import Path
+from typing import Any
+
+from pydolphinscheduler.exceptions import PyDSConfException
+from pydolphinscheduler.utils import file
+from pydolphinscheduler.utils.yaml_parser import YamlParser
+
+BUILD_IN_CONFIG_PATH =
Path(__file__).resolve().parent.joinpath("default_config.yaml")
+
+
+def config_path() -> Path:
+ """Get the path of pydolphinscheduler configuration file."""
+ pyds_home = os.environ.get("PYDS_HOME", "~/pydolphinscheduler")
+ config_file_path = Path(pyds_home).joinpath("config.yaml").expanduser()
+ return config_file_path
+
+
+def get_configs() -> YamlParser:
+ """Get all configuration settings from configuration file.
+
+ Will use custom configuration file first if it exists, otherwise default
configuration file in
+ default path.
+ """
+ path = str(config_path()) if config_path().exists() else
BUILD_IN_CONFIG_PATH
+ with open(path, mode="r") as f:
+ return YamlParser(f.read())
+
+
+def init_config_file() -> None:
+ """Initialize configuration file by default configs."""
+ if config_path().exists():
+ raise PyDSConfException(
+ "Initialize configuration false to avoid overwrite configure by
accident, file already exists "
+ "in %s, if you wan to overwrite the exists configure please remove
the exists file manually.",
+ str(config_path()),
+ )
+ file.write(content=str(get_configs()), to_path=str(config_path()))
+
+
+def get_single_config(key: str) -> Any:
+ """Get single config to configuration file.
+
+ Support get from nested keys by delimiter ``.``.
+
+ For example, yaml config as below:
+
+ .. code-block:: yaml
+
+ one:
+ two1:
+ three: value1
+ two2: value2
+
+ you could get ``value1`` and ``value2`` by nested path
+
+ .. code-block:: python
+
+ value1 = get_single_config("one.two1.three")
+ value2 = get_single_config("one.two2")
+
+ :param key: The config key want to get it value.
+ """
+ config = get_configs()
+ if key not in config:
+ raise PyDSConfException(
+ "Configuration path %s do not exists. Can not get configuration.",
key
+ )
+ return config[key]
+
+
+def set_single_config(key: str, value: Any) -> None:
+ """Change single config to configuration file.
+
+ For example, yaml config as below:
+
+ .. code-block:: yaml
+
+ one:
+ two1:
+ three: value1
+ two2: value2
+
+ you could change ``value1`` to ``value3``, also change ``value2`` to
``value4`` by nested path assigned
+
+ .. code-block:: python
+
+ set_single_config["one.two1.three"] = "value3"
+ set_single_config["one.two2"] = "value4"
+
+ :param key: The config key want change.
+ :param value: The new value want to set.
+ """
+ config = get_configs()
+ if key not in config:
+ raise PyDSConfException(
+ "Configuration path %s do not exists. Can not set configuration.",
key
+ )
+ config[key] = value
+ file.write(content=str(config), to_path=str(config_path()), overwrite=True)
+
+
+def get_int(val: Any) -> int:
+ """Covert value to int."""
+ return int(val)
+
+
+def get_bool(val: Any) -> bool:
+ """Covert value to boolean."""
+ if isinstance(val, str):
+ return val.lower() in {"true", "t"}
+ elif isinstance(val, int):
+ return val == 1
+ else:
+ return bool(val)
+
+
+# Start Common Configuration Settings
+
+# Add configs as module variables to avoid read configuration multiple times
when
+# Get common configuration setting
+# Set or get multiple configs in single time
+configs: YamlParser = get_configs()
+
+# Java Gateway Settings
+JAVA_GATEWAY_ADDRESS = os.environ.get(
+ "PYDS_JAVA_GATEWAY_ADDRESS", configs.get("java_gateway.address")
+)
+JAVA_GATEWAY_PORT = get_int(
+ os.environ.get("PYDS_JAVA_GATEWAY_PORT", configs.get("java_gateway.port"))
+)
+JAVA_GATEWAY_AUTO_CONVERT = get_bool(
+ os.environ.get(
+ "PYDS_JAVA_GATEWAY_AUTO_CONVERT",
configs.get("java_gateway.auto_convert")
+ )
+)
+
+# User Settings
+USER_NAME = os.environ.get("PYDS_USER_NAME", configs.get("default.user.name"))
+USER_PASSWORD = os.environ.get(
+ "PYDS_USER_PASSWORD", configs.get("default.user.password")
+)
+USER_EMAIL = os.environ.get("PYDS_USER_EMAIL",
configs.get("default.user.email"))
+USER_PHONE = str(os.environ.get("PYDS_USER_PHONE",
configs.get("default.user.phone")))
+USER_STATE = get_int(
+ os.environ.get("PYDS_USER_STATE", configs.get("default.user.state"))
+)
+
+# Workflow Settings
+WORKFLOW_PROJECT = os.environ.get(
+ "PYDS_WORKFLOW_PROJECT", configs.get("default.workflow.project")
+)
+WORKFLOW_TENANT = os.environ.get(
+ "PYDS_WORKFLOW_TENANT", configs.get("default.workflow.tenant")
+)
+WORKFLOW_USER = os.environ.get(
+ "PYDS_WORKFLOW_USER", configs.get("default.workflow.user")
+)
+WORKFLOW_QUEUE = os.environ.get(
+ "PYDS_WORKFLOW_QUEUE", configs.get("default.workflow.queue")
+)
+WORKFLOW_RELEASE_STATE = os.environ.get(
+ "PYDS_WORKFLOW_RELEASE_STATE",
configs.get("default.workflow.release_state")
+)
+WORKFLOW_WORKER_GROUP = os.environ.get(
+ "PYDS_WORKFLOW_WORKER_GROUP", configs.get("default.workflow.worker_group")
+)
+WORKFLOW_TIME_ZONE = os.environ.get(
+ "PYDS_WORKFLOW_TIME_ZONE", configs.get("default.workflow.time_zone")
+)
+WORKFLOW_WARNING_TYPE = os.environ.get(
+ "PYDS_WORKFLOW_WARNING_TYPE", configs.get("default.workflow.warning_type")
+)
+
+# End Common Configuration Setting
diff --git a/src/pydolphinscheduler/core/__init__.py
b/src/pydolphinscheduler/core/__init__.py
index 7497d1f..b997c3e 100644
--- a/src/pydolphinscheduler/core/__init__.py
+++ b/src/pydolphinscheduler/core/__init__.py
@@ -23,8 +23,8 @@ from pydolphinscheduler.core.process_definition import
ProcessDefinition
from pydolphinscheduler.core.task import Task
__all__ = [
+ "Database",
"Engine",
"ProcessDefinition",
"Task",
- "Database",
]
diff --git a/src/pydolphinscheduler/core/process_definition.py
b/src/pydolphinscheduler/core/process_definition.py
index 63e0808..69dcbc1 100644
--- a/src/pydolphinscheduler/core/process_definition.py
+++ b/src/pydolphinscheduler/core/process_definition.py
@@ -21,12 +21,11 @@ import json
from datetime import datetime
from typing import Any, Dict, List, Optional, Set
+from pydolphinscheduler import configuration
from pydolphinscheduler.constants import TaskType
-from pydolphinscheduler.core import configuration
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.exceptions import PyDSParamException,
PyDSTaskNoFoundException
from pydolphinscheduler.java_gateway import launch_gateway
-from pydolphinscheduler.side import Project, Tenant, User
+from pydolphinscheduler.models import Base, Project, Tenant, User
from pydolphinscheduler.utils.date import MAX_DATETIME, conv_from_str,
conv_to_schedule
@@ -170,7 +169,7 @@ class ProcessDefinition(Base):
def user(self) -> User:
"""Get user object.
- For now we just get from python side but not from java gateway side,
so it may not correct.
+ For now we just get from python models but not from java gateway
models, so it may not correct.
"""
return User(name=self._user, tenant=self._tenant)
@@ -358,10 +357,10 @@ class ProcessDefinition(Base):
self.start()
def _ensure_side_model_exists(self):
- """Ensure process definition side model exists.
+ """Ensure process definition models model exists.
- For now, side object including
:class:`pydolphinscheduler.side.project.Project`,
- :class:`pydolphinscheduler.side.tenant.Tenant`,
:class:`pydolphinscheduler.side.user.User`.
+ For now, models object including
:class:`pydolphinscheduler.models.project.Project`,
+ :class:`pydolphinscheduler.models.tenant.Tenant`,
:class:`pydolphinscheduler.models.user.User`.
If these model not exists, would create default value in
:class:`pydolphinscheduler.constants.ProcessDefinitionDefault`.
"""
diff --git a/src/pydolphinscheduler/core/resource.py
b/src/pydolphinscheduler/core/resource.py
index bd4ffd4..a3aab81 100644
--- a/src/pydolphinscheduler/core/resource.py
+++ b/src/pydolphinscheduler/core/resource.py
@@ -19,7 +19,7 @@
from typing import Optional
-from pydolphinscheduler.core.base import Base
+from pydolphinscheduler.models import Base
class Resource(Base):
diff --git a/src/pydolphinscheduler/core/task.py
b/src/pydolphinscheduler/core/task.py
index 90c0e89..4d4e67e 100644
--- a/src/pydolphinscheduler/core/task.py
+++ b/src/pydolphinscheduler/core/task.py
@@ -20,6 +20,7 @@
from logging import getLogger
from typing import Dict, List, Optional, Sequence, Set, Tuple, Union
+from pydolphinscheduler import configuration
from pydolphinscheduler.constants import (
Delimiter,
ResourceKey,
@@ -27,13 +28,12 @@ from pydolphinscheduler.constants import (
TaskPriority,
TaskTimeoutFlag,
)
-from pydolphinscheduler.core import configuration
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.core.process_definition import (
ProcessDefinition,
ProcessDefinitionContext,
)
from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models import Base
logger = getLogger(__name__)
diff --git a/src/pydolphinscheduler/default_config.yaml
b/src/pydolphinscheduler/default_config.yaml
new file mode 100644
index 0000000..98d7b99
--- /dev/null
+++ b/src/pydolphinscheduler/default_config.yaml
@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Setting about Java gateway server
+java_gateway:
+ # The address of Python gateway server start. Set its value to `0.0.0.0` if
your Python API run in different
+ # between Python gateway server. It could be be specific to other address
like `127.0.0.1` or `localhost`
+ address: 127.0.0.1
+
+ # The port of Python gateway server start. Define which port you could
connect to Python gateway server from
+ # Python API models.
+ port: 25333
+
+ # Whether automatically convert Python objects to Java Objects. Default
value is ``True``. There is some
+ # performance lost when set to ``True`` but for now pydolphinscheduler do
not handle the convert issue between
+ # java and Python, mark it as TODO item in the future.
+ auto_convert: true
+
+# Setting about dolphinscheduler default value, will use the value set below
if property do not set, which
+# including ``user``, ``workflow``
+default:
+ # Default value for dolphinscheduler's user object
+ user:
+ name: userPythonGateway
+ password: userPythonGateway
+ email: [email protected]
+ tenant: tenant_pydolphin
+ phone: 11111111111
+ state: 1
+ # Default value for dolphinscheduler's workflow object
+ workflow:
+ project: project-pydolphin
+ tenant: tenant_pydolphin
+ user: userPythonGateway
+ queue: queuePythonGateway
+ worker_group: default
+ # Release state of workflow, default value is ``online`` which mean
setting workflow online when it submits
+ # to Java gateway, if you want to set workflow offline set its value to
``offline``
+ release_state: online
+ time_zone: Asia/Shanghai
+ # Warning type of the workflow, default value is ``NONE`` mean do not warn
user in any cases of workflow state,
+ # change to ``FAILURE`` if you want to warn users when workflow failed.
All available enum value are
+ # ``NONE``, ``SUCCESS``, ``FAILURE``, ``ALL``
+ warning_type: NONE
diff --git a/src/pydolphinscheduler/examples/task_dependent_example.py
b/src/pydolphinscheduler/examples/task_dependent_example.py
index 88d6ea2..db53bcc 100644
--- a/src/pydolphinscheduler/examples/task_dependent_example.py
+++ b/src/pydolphinscheduler/examples/task_dependent_example.py
@@ -35,7 +35,7 @@ task_dependent:
task_dependent(this task dependent on task_dependent_external.task_1 and
task_dependent_external.task_2).
"""
-from pydolphinscheduler.core import configuration
+from pydolphinscheduler import configuration
from pydolphinscheduler.core.process_definition import ProcessDefinition
from pydolphinscheduler.tasks.dependent import And, Dependent, DependentItem,
Or
from pydolphinscheduler.tasks.shell import Shell
diff --git a/src/pydolphinscheduler/java_gateway.py
b/src/pydolphinscheduler/java_gateway.py
index 8560638..7b85902 100644
--- a/src/pydolphinscheduler/java_gateway.py
+++ b/src/pydolphinscheduler/java_gateway.py
@@ -22,8 +22,8 @@ from typing import Any, Optional
from py4j.java_collections import JavaMap
from py4j.java_gateway import GatewayParameters, JavaGateway
+from pydolphinscheduler import configuration
from pydolphinscheduler.constants import JavaGatewayDefault
-from pydolphinscheduler.core import configuration
from pydolphinscheduler.exceptions import PyDSJavaGatewayException
diff --git a/src/pydolphinscheduler/core/__init__.py
b/src/pydolphinscheduler/models/__init__.py
similarity index 58%
copy from src/pydolphinscheduler/core/__init__.py
copy to src/pydolphinscheduler/models/__init__.py
index 7497d1f..b289954 100644
--- a/src/pydolphinscheduler/core/__init__.py
+++ b/src/pydolphinscheduler/models/__init__.py
@@ -15,16 +15,22 @@
# specific language governing permissions and limitations
# under the License.
-"""Init pydolphinscheduler.core package."""
+"""Init Models package, keeping object related to DolphinScheduler covert from
Java Gateway Service."""
-from pydolphinscheduler.core.database import Database
-from pydolphinscheduler.core.engine import Engine
-from pydolphinscheduler.core.process_definition import ProcessDefinition
-from pydolphinscheduler.core.task import Task
+from pydolphinscheduler.models.base import Base
+from pydolphinscheduler.models.base_side import BaseSide
+from pydolphinscheduler.models.project import Project
+from pydolphinscheduler.models.queue import Queue
+from pydolphinscheduler.models.tenant import Tenant
+from pydolphinscheduler.models.user import User
+from pydolphinscheduler.models.worker_group import WorkerGroup
__all__ = [
- "Engine",
- "ProcessDefinition",
- "Task",
- "Database",
+ "Base",
+ "BaseSide",
+ "Project",
+ "Tenant",
+ "User",
+ "Queue",
+ "WorkerGroup",
]
diff --git a/src/pydolphinscheduler/models/base.py
b/src/pydolphinscheduler/models/base.py
new file mode 100644
index 0000000..2647714
--- /dev/null
+++ b/src/pydolphinscheduler/models/base.py
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""DolphinScheduler Base object."""
+
+from typing import Dict, Optional
+
+# from pydolphinscheduler.models.user import User
+from pydolphinscheduler.utils.string import attr2camel
+
+
+class Base:
+ """DolphinScheduler Base object."""
+
+ # Object key attribute, to test whether object equals and so on.
+ _KEY_ATTR: set = {"name", "description"}
+
+ # Object defines attribute, use when needs to communicate with Java
gateway server.
+ _DEFINE_ATTR: set = set()
+
+ # Object default attribute, will add those attribute to `_DEFINE_ATTR`
when init assign missing.
+ _DEFAULT_ATTR: Dict = {}
+
+ def __init__(self, name: str, description: Optional[str] = None):
+ self.name = name
+ self.description = description
+
+ def __repr__(self) -> str:
+ return f'<{type(self).__name__}: name="{self.name}">'
+
+ def __eq__(self, other):
+ return type(self) == type(other) and all(
+ getattr(self, a, None) == getattr(other, a, None) for a in
self._KEY_ATTR
+ )
+
+ def get_define_custom(
+ self, camel_attr: bool = True, custom_attr: set = None
+ ) -> Dict:
+ """Get object definition attribute by given attr set."""
+ content = {}
+ for attr in custom_attr:
+ val = getattr(self, attr, None)
+ if camel_attr:
+ content[attr2camel(attr)] = val
+ else:
+ content[attr] = val
+ return content
+
+ def get_define(self, camel_attr: bool = True) -> Dict:
+ """Get object definition attribute communicate to Java gateway server.
+
+ use attribute `self._DEFINE_ATTR` to determine which attributes should
including when
+ object tries to communicate with Java gateway server.
+ """
+ content = self.get_define_custom(camel_attr, self._DEFINE_ATTR)
+ update_default = {
+ k: self._DEFAULT_ATTR.get(k) for k in self._DEFAULT_ATTR if k not
in content
+ }
+ content.update(update_default)
+ return content
diff --git a/src/pydolphinscheduler/core/resource.py
b/src/pydolphinscheduler/models/base_side.py
similarity index 59%
copy from src/pydolphinscheduler/core/resource.py
copy to src/pydolphinscheduler/models/base_side.py
index bd4ffd4..67ac88d 100644
--- a/src/pydolphinscheduler/core/resource.py
+++ b/src/pydolphinscheduler/models/base_side.py
@@ -15,29 +15,26 @@
# specific language governing permissions and limitations
# under the License.
-"""Module resource."""
+"""Module for models object."""
from typing import Optional
-from pydolphinscheduler.core.base import Base
+from pydolphinscheduler import configuration
+from pydolphinscheduler.models import Base
-class Resource(Base):
- """resource object, will define the resources that you want to create or
update.
+class BaseSide(Base):
+ """Base class for models object, it declare base behavior for them."""
- :param name: The fullname of resource.Includes path and suffix.
- :param content: The description of resource.
- :param description: The description of resource.
- """
-
- _DEFINE_ATTR = {"name", "content", "description"}
+ def __init__(self, name: str, description: Optional[str] = None):
+ super().__init__(name, description)
- def __init__(
- self,
- name: str,
- content: str,
- description: Optional[str] = None,
+ @classmethod
+ def create_if_not_exists(
+ cls,
+ # TODO comment for avoiding cycle import
+ # user: Optional[User] = ProcessDefinitionDefault.USER
+ user=configuration.WORKFLOW_USER,
):
- super().__init__(name, description)
- self.content = content
- self._resource_code = None
+ """Create Base if not exists."""
+ raise NotImplementedError
diff --git a/src/pydolphinscheduler/core/resource.py
b/src/pydolphinscheduler/models/project.py
similarity index 59%
copy from src/pydolphinscheduler/core/resource.py
copy to src/pydolphinscheduler/models/project.py
index bd4ffd4..ad72211 100644
--- a/src/pydolphinscheduler/core/resource.py
+++ b/src/pydolphinscheduler/models/project.py
@@ -15,29 +15,28 @@
# specific language governing permissions and limitations
# under the License.
-"""Module resource."""
+"""DolphinScheduler Project object."""
from typing import Optional
-from pydolphinscheduler.core.base import Base
+from pydolphinscheduler import configuration
+from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models import BaseSide
-class Resource(Base):
- """resource object, will define the resources that you want to create or
update.
-
- :param name: The fullname of resource.Includes path and suffix.
- :param content: The description of resource.
- :param description: The description of resource.
- """
-
- _DEFINE_ATTR = {"name", "content", "description"}
+class Project(BaseSide):
+ """DolphinScheduler Project object."""
def __init__(
self,
- name: str,
- content: str,
+ name: str = configuration.WORKFLOW_PROJECT,
description: Optional[str] = None,
):
super().__init__(name, description)
- self.content = content
- self._resource_code = None
+
+ def create_if_not_exists(self, user=configuration.USER_NAME) -> None:
+ """Create Project if not exists."""
+ gateway = launch_gateway()
+ gateway.entry_point.createOrGrantProject(user, self.name,
self.description)
+ # TODO recover result checker
+ # gateway_result_checker(result, None)
diff --git a/src/pydolphinscheduler/core/resource.py
b/src/pydolphinscheduler/models/queue.py
similarity index 55%
copy from src/pydolphinscheduler/core/resource.py
copy to src/pydolphinscheduler/models/queue.py
index bd4ffd4..3f8f81d 100644
--- a/src/pydolphinscheduler/core/resource.py
+++ b/src/pydolphinscheduler/models/queue.py
@@ -15,29 +15,28 @@
# specific language governing permissions and limitations
# under the License.
-"""Module resource."""
+"""DolphinScheduler User object."""
from typing import Optional
-from pydolphinscheduler.core.base import Base
+from pydolphinscheduler import configuration
+from pydolphinscheduler.java_gateway import gateway_result_checker,
launch_gateway
+from pydolphinscheduler.models import BaseSide
-class Resource(Base):
- """resource object, will define the resources that you want to create or
update.
-
- :param name: The fullname of resource.Includes path and suffix.
- :param content: The description of resource.
- :param description: The description of resource.
- """
-
- _DEFINE_ATTR = {"name", "content", "description"}
+class Queue(BaseSide):
+ """DolphinScheduler Queue object."""
def __init__(
self,
- name: str,
- content: str,
- description: Optional[str] = None,
+ name: str = configuration.WORKFLOW_QUEUE,
+ description: Optional[str] = "",
):
super().__init__(name, description)
- self.content = content
- self._resource_code = None
+
+ def create_if_not_exists(self, user=configuration.USER_NAME) -> None:
+ """Create Queue if not exists."""
+ gateway = launch_gateway()
+ # Here we set Queue.name and Queue.queueName same as self.name
+ result = gateway.entry_point.createProject(user, self.name, self.name)
+ gateway_result_checker(result, None)
diff --git a/src/pydolphinscheduler/core/resource.py
b/src/pydolphinscheduler/models/tenant.py
similarity index 57%
copy from src/pydolphinscheduler/core/resource.py
copy to src/pydolphinscheduler/models/tenant.py
index bd4ffd4..148a8f6 100644
--- a/src/pydolphinscheduler/core/resource.py
+++ b/src/pydolphinscheduler/models/tenant.py
@@ -15,29 +15,31 @@
# specific language governing permissions and limitations
# under the License.
-"""Module resource."""
+"""DolphinScheduler Tenant object."""
from typing import Optional
-from pydolphinscheduler.core.base import Base
+from pydolphinscheduler import configuration
+from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models import BaseSide
-class Resource(Base):
- """resource object, will define the resources that you want to create or
update.
-
- :param name: The fullname of resource.Includes path and suffix.
- :param content: The description of resource.
- :param description: The description of resource.
- """
-
- _DEFINE_ATTR = {"name", "content", "description"}
+class Tenant(BaseSide):
+ """DolphinScheduler Tenant object."""
def __init__(
self,
- name: str,
- content: str,
+ name: str = configuration.WORKFLOW_TENANT,
+ queue: str = configuration.WORKFLOW_QUEUE,
description: Optional[str] = None,
):
super().__init__(name, description)
- self.content = content
- self._resource_code = None
+ self.queue = queue
+
+ def create_if_not_exists(
+ self, queue_name: str, user=configuration.USER_NAME
+ ) -> None:
+ """Create Tenant if not exists."""
+ gateway = launch_gateway()
+ gateway.entry_point.createTenant(self.name, self.description,
queue_name)
+ # gateway_result_checker(result, None)
diff --git a/src/pydolphinscheduler/models/user.py
b/src/pydolphinscheduler/models/user.py
new file mode 100644
index 0000000..de2d8b1
--- /dev/null
+++ b/src/pydolphinscheduler/models/user.py
@@ -0,0 +1,78 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""DolphinScheduler User object."""
+
+from typing import Optional
+
+from pydolphinscheduler import configuration
+from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models import BaseSide, Tenant
+
+
+class User(BaseSide):
+ """DolphinScheduler User object."""
+
+ _KEY_ATTR = {
+ "name",
+ "password",
+ "email",
+ "phone",
+ "tenant",
+ "queue",
+ "status",
+ }
+
+ def __init__(
+ self,
+ name: str,
+ password: Optional[str] = configuration.USER_PASSWORD,
+ email: Optional[str] = configuration.USER_EMAIL,
+ phone: Optional[str] = configuration.USER_PHONE,
+ tenant: Optional[str] = configuration.WORKFLOW_TENANT,
+ queue: Optional[str] = configuration.WORKFLOW_QUEUE,
+ status: Optional[int] = configuration.USER_STATE,
+ ):
+ super().__init__(name)
+ self.password = password
+ self.email = email
+ self.phone = phone
+ self.tenant = tenant
+ self.queue = queue
+ self.status = status
+
+ def create_tenant_if_not_exists(self) -> None:
+ """Create tenant object."""
+ tenant = Tenant(name=self.tenant, queue=self.queue)
+ tenant.create_if_not_exists(self.queue)
+
+ def create_if_not_exists(self, **kwargs):
+ """Create User if not exists."""
+ # Should make sure queue already exists.
+ self.create_tenant_if_not_exists()
+ gateway = launch_gateway()
+ gateway.entry_point.createUser(
+ self.name,
+ self.password,
+ self.email,
+ self.phone,
+ self.tenant,
+ self.queue,
+ self.status,
+ )
+ # TODO recover result checker
+ # gateway_result_checker(result, None)
diff --git a/src/pydolphinscheduler/core/__init__.py
b/src/pydolphinscheduler/models/worker_group.py
similarity index 68%
copy from src/pydolphinscheduler/core/__init__.py
copy to src/pydolphinscheduler/models/worker_group.py
index 7497d1f..bc55eaf 100644
--- a/src/pydolphinscheduler/core/__init__.py
+++ b/src/pydolphinscheduler/models/worker_group.py
@@ -15,16 +15,16 @@
# specific language governing permissions and limitations
# under the License.
-"""Init pydolphinscheduler.core package."""
+"""DolphinScheduler Worker Group object."""
-from pydolphinscheduler.core.database import Database
-from pydolphinscheduler.core.engine import Engine
-from pydolphinscheduler.core.process_definition import ProcessDefinition
-from pydolphinscheduler.core.task import Task
+from typing import Optional
-__all__ = [
- "Engine",
- "ProcessDefinition",
- "Task",
- "Database",
-]
+from pydolphinscheduler.models import BaseSide
+
+
+class WorkerGroup(BaseSide):
+ """DolphinScheduler Worker Group object."""
+
+ def __init__(self, name: str, address: str, description: Optional[str] =
None):
+ super().__init__(name, description)
+ self.address = address
diff --git a/src/pydolphinscheduler/tasks/condition.py
b/src/pydolphinscheduler/tasks/condition.py
index 50aac25..cb139f1 100644
--- a/src/pydolphinscheduler/tasks/condition.py
+++ b/src/pydolphinscheduler/tasks/condition.py
@@ -20,9 +20,9 @@
from typing import Dict, List
from pydolphinscheduler.constants import TaskType
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.core.task import Task
from pydolphinscheduler.exceptions import PyDSParamException
+from pydolphinscheduler.models.base import Base
class Status(Base):
diff --git a/src/pydolphinscheduler/tasks/dependent.py
b/src/pydolphinscheduler/tasks/dependent.py
index 7cff24c..cc6d25b 100644
--- a/src/pydolphinscheduler/tasks/dependent.py
+++ b/src/pydolphinscheduler/tasks/dependent.py
@@ -20,10 +20,10 @@
from typing import Dict, Optional, Tuple
from pydolphinscheduler.constants import TaskType
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.core.task import Task
from pydolphinscheduler.exceptions import PyDSJavaGatewayException,
PyDSParamException
from pydolphinscheduler.java_gateway import launch_gateway
+from pydolphinscheduler.models.base import Base
DEPENDENT_ALL_TASK_IN_WORKFLOW = "0"
@@ -31,8 +31,8 @@ DEPENDENT_ALL_TASK_IN_WORKFLOW = "0"
class DependentDate(str):
"""Constant of Dependent date value.
- These values set according to Java server side, if you want to add and
change it,
- please change Java server side first.
+ These values set according to Java server models, if you want to add and
change it,
+ please change Java server models first.
"""
# TODO Maybe we should add parent level to DependentDate for easy to use,
such as
diff --git a/src/pydolphinscheduler/tasks/sql.py
b/src/pydolphinscheduler/tasks/sql.py
index a125982..716a024 100644
--- a/src/pydolphinscheduler/tasks/sql.py
+++ b/src/pydolphinscheduler/tasks/sql.py
@@ -98,7 +98,7 @@ class Sql(Task):
return self.param_sql_type
pattern_select_str = (
"^(?!(.* |)insert |(.* |)delete |(.* |)drop "
- "|(.* |)update |(.* |)alter |(.* |)create ).*"
+ "|(.* |)update |(.* |)truncate |(.* |)alter |(.* |)create ).*"
)
pattern_select = re.compile(pattern_select_str, re.IGNORECASE)
if pattern_select.match(self.sql) is None:
diff --git a/src/pydolphinscheduler/tasks/switch.py
b/src/pydolphinscheduler/tasks/switch.py
index 0c9a2b8..35eece8 100644
--- a/src/pydolphinscheduler/tasks/switch.py
+++ b/src/pydolphinscheduler/tasks/switch.py
@@ -20,9 +20,9 @@
from typing import Dict, Optional
from pydolphinscheduler.constants import TaskType
-from pydolphinscheduler.core.base import Base
from pydolphinscheduler.core.task import Task
from pydolphinscheduler.exceptions import PyDSParamException
+from pydolphinscheduler.models.base import Base
class SwitchBranch(Base):
@@ -99,23 +99,22 @@ class SwitchCondition(Base):
result = []
num_branch_default = 0
for condition in self.args:
- if isinstance(condition, SwitchBranch):
- if num_branch_default < 1:
- if isinstance(condition, Default):
- self._DEFINE_ATTR.add("next_node")
- setattr(self, "next_node", condition.next_node)
- num_branch_default += 1
- elif isinstance(condition, Branch):
- result.append(condition.get_define())
- else:
- raise PyDSParamException(
- "Task Switch's parameter only support exactly one
default branch."
- )
- else:
+ if not isinstance(condition, SwitchBranch):
raise PyDSParamException(
"Task Switch's parameter only support SwitchBranch but got
%s.",
type(condition),
)
+ # Default number branch checker
+ if num_branch_default >= 1 and isinstance(condition, Default):
+ raise PyDSParamException(
+ "Task Switch's parameter only support exactly one default
branch."
+ )
+ if isinstance(condition, Default):
+ self._DEFINE_ATTR.add("next_node")
+ setattr(self, "next_node", condition.next_node)
+ num_branch_default += 1
+ elif isinstance(condition, Branch):
+ result.append(condition.get_define())
# Handle switch default branch, default value is `""` if not provide.
if num_branch_default == 0:
self._DEFINE_ATTR.add("next_node")
diff --git a/tests/cli/test_config.py b/tests/cli/test_config.py
index d913277..516ad75 100644
--- a/tests/cli/test_config.py
+++ b/tests/cli/test_config.py
@@ -23,7 +23,7 @@ from pathlib import Path
import pytest
from pydolphinscheduler.cli.commands import cli
-from pydolphinscheduler.core.configuration import BUILD_IN_CONFIG_PATH,
config_path
+from pydolphinscheduler.configuration import BUILD_IN_CONFIG_PATH, config_path
from tests.testing.cli import CliTestWrapper
from tests.testing.constants import DEV_MODE, ENV_PYDS_HOME
from tests.testing.file import get_file_content
diff --git a/tests/cli/test_version.py b/tests/cli/test_version.py
index f0dcb0e..b61d26d 100644
--- a/tests/cli/test_version.py
+++ b/tests/cli/test_version.py
@@ -17,9 +17,11 @@
"""Test command line interface subcommand `version`."""
+from unittest.mock import patch
+
import pytest
-from pydolphinscheduler import __version__
+import pydolphinscheduler
from pydolphinscheduler.cli.commands import cli
from tests.testing.cli import CliTestWrapper
@@ -27,21 +29,27 @@ from tests.testing.cli import CliTestWrapper
def test_version():
"""Test whether subcommand `version` correct."""
cli_test = CliTestWrapper(cli, ["version"])
- cli_test.assert_success(output=f"{__version__}")
+ cli_test.assert_success(output=f"{pydolphinscheduler.__version__}")
@pytest.mark.parametrize(
- "part, idx",
+ "version, part, idx",
[
- ("major", 0),
- ("minor", 1),
- ("micro", 2),
+ ("1.2.3", "major", 0),
+ ("0.1.3", "minor", 1),
+ ("3.1.0", "micro", 2),
+ ("1.2.3-beta-1", "micro", 2),
+ ("1.2.3-alpha", "micro", 2),
+ ("1.2.3a2", "micro", 2),
+ ("1.2.3b1", "micro", 2),
],
)
-def test_version_part(part: str, idx: int):
+@patch("pydolphinscheduler.__version__")
+def test_version_part(mock_version, version: str, part: str, idx: int):
"""Test subcommand `version` option `--part`."""
+ mock_version.return_value = version
cli_test = CliTestWrapper(cli, ["version", "--part", part])
- cli_test.assert_success(output=f"{__version__.split('.')[idx]}")
+
cli_test.assert_success(output=f"{pydolphinscheduler.__version__.split('.')[idx]}")
@pytest.mark.parametrize(
diff --git a/tests/core/test_configuration.py b/tests/core/test_configuration.py
index c7e217a..b9dc8cb 100644
--- a/tests/core/test_configuration.py
+++ b/tests/core/test_configuration.py
@@ -24,8 +24,8 @@ from typing import Any
import pytest
-from pydolphinscheduler.core import configuration
-from pydolphinscheduler.core.configuration import (
+from pydolphinscheduler import configuration
+from pydolphinscheduler.configuration import (
BUILD_IN_CONFIG_PATH,
config_path,
get_single_config,
diff --git a/tests/core/test_process_definition.py
b/tests/core/test_process_definition.py
index 5cb6dab..30445bf 100644
--- a/tests/core/test_process_definition.py
+++ b/tests/core/test_process_definition.py
@@ -24,11 +24,11 @@ from unittest.mock import patch
import pytest
from freezegun import freeze_time
-from pydolphinscheduler.core import configuration
+from pydolphinscheduler import configuration
from pydolphinscheduler.core.process_definition import ProcessDefinition
from pydolphinscheduler.core.resource import Resource
from pydolphinscheduler.exceptions import PyDSParamException
-from pydolphinscheduler.side import Project, Tenant, User
+from pydolphinscheduler.models import Project, Tenant, User
from pydolphinscheduler.tasks.switch import Branch, Default, Switch,
SwitchCondition
from pydolphinscheduler.utils.date import conv_to_schedule
from tests.testing.task import Task
diff --git a/tests/tasks/test_sql.py b/tests/tasks/test_sql.py
index ee0acc4..50ccd94 100644
--- a/tests/tasks/test_sql.py
+++ b/tests/tasks/test_sql.py
@@ -54,6 +54,7 @@ from pydolphinscheduler.tasks.sql import Sql, SqlType
("delete from table_name where id < 10", None, SqlType.NOT_SELECT),
("alter table table_name add column col1 int", None,
SqlType.NOT_SELECT),
("create table table_name2 (col1 int)", None, SqlType.NOT_SELECT),
+ ("truncate table table_name", None, SqlType.NOT_SELECT),
("create table table_name2 (col1 int)", SqlType.SELECT,
SqlType.SELECT),
("select 1", SqlType.NOT_SELECT, SqlType.NOT_SELECT),
("create table table_name2 (col1 int)", SqlType.NOT_SELECT,
SqlType.NOT_SELECT),
diff --git a/tests/testing/path.py b/tests/testing/path.py
index d1e520b..68d93c4 100644
--- a/tests/testing/path.py
+++ b/tests/testing/path.py
@@ -26,7 +26,7 @@ path_code_tasks = project_root.joinpath("src",
"pydolphinscheduler", "tasks")
path_example = project_root.joinpath("src", "pydolphinscheduler", "examples")
path_doc_tasks = project_root.joinpath("docs", "source", "tasks")
path_default_config_yaml = project_root.joinpath(
- "src", "pydolphinscheduler", "core", "default_config.yaml"
+ "src", "pydolphinscheduler", "default_config.yaml"
)
diff --git a/tox.ini b/tox.ini
index 8e9280f..d90e8a3 100644
--- a/tox.ini
+++ b/tox.ini
@@ -16,10 +16,12 @@
# under the License.
[tox]
-envlist = local-ci, auto-lint, lint, doc-build-test, code-test,
integrate-test, py{36,37,38,39}
+envlist = local-ci, auto-lint, lint, doc-build, doc-build-multi, code-test,
integrate-test, local-integrate-test, py{36,37,38,39,310,311}
[testenv]
-whitelist_externals = make
+allowlist_externals =
+ make
+ git
[testenv:auto-lint]
extras = style
@@ -42,12 +44,20 @@ extras = test
commands =
python -m pytest --cov=pydolphinscheduler
--cov-config={toxinidir}/.coveragerc tests/
-[testenv:doc-build-test]
+[testenv:doc-build]
extras = doc
commands =
make -C {toxinidir}/docs clean
make -C {toxinidir}/docs html
+[testenv:doc-build-multi]
+extras = doc
+commands =
+ # Get all tags for `multiversion` subcommand
+ git fetch --tags
+ make -C {toxinidir}/docs clean
+ make -C {toxinidir}/docs multiversion
+
[testenv:integrate-test]
extras = test
commands =
@@ -58,4 +68,4 @@ extras = dev
commands =
{[testenv:lint]commands}
{[testenv:code-test]commands}
- {[testenv:doc-build-test]commands}
+ {[testenv:doc-build]commands}
