Hi community, I think we can use the pydolphinscheduler to dynamically generate workflows from the configuration file.
Dynamically generate workflows from YAML configuration files, aka workflows-as-code. Now we can use python-dolphinscheduler to build workflow easily. Workflows-as-code would be easier to use if we supported configuration file definition workflows. It will make the workflow clearer, shareable, and reviewable. Then we can upload our project as a git repository. This can help us do richer operations, such as CICD, etc We can create a workload by defining the following fields, all the field definitions are derived from [PydolphinScheduler](https://dolphinscheduler.apache.org/python/concept.html#). A YAML file defines a workflow : ```yaml # Define the process Process: #process(workflow) name name: prepare_datas # Parameters for process creation param: project: /data/project # Whether to run the workflow after the creation is complete run: True # Define the tasks under the process Tasks: - # task type: Shell, python, SubProcess, Spark, etc TaskType: Shell # Upstream Task List dependencies: [xxxx] # Parameters for task creation params: name: task1 other parameters: ... - TaskType: Python params: name: task2 - ... - ... ``` I already add a GitHub Issue for my proposal, which you could see in https://github.com/apache/dolphinscheduler/issues/10995 Looking forward any feedback for this thread.
