Hi Jieguang, Generating workflows from `yaml` template is a better way for users who need to create tons of workflows at a time. Furthermore, if users have little knowledge in python, `yaml` helps them get hands dirty more easily if they want to try `workflow as code`. BTW, if users want to migrate from other big data orchestration tools such as Airflow, Azkaban, Oozie, etc. to DS, converting workflows from previous systems into `yaml` files helps ease the pain.
Above all, I'm +1 to this and looking forward to a detailed design. Thanks. *Best Regards,* *Chufeng (Eric) Gao* Jieguang Zhou <[email protected]> 于2022年7月29日周五 15:20写道： > Hi community, > > I think we can use the pydolphinscheduler to dynamically generate > workflows from the configuration file. > > Dynamically generate workflows from YAML configuration files, aka > workflows-as-code. > > Now we can use python-dolphinscheduler to build workflow easily. > Workflows-as-code would be easier to use if we supported configuration file > definition workflows. It will make the workflow clearer, shareable, and > reviewable. > > Then we can upload our project as a git repository. This can help us do > richer operations, such as CICD, etc > > We can create a workload by defining the following fields, all the field > definitions are derived from [PydolphinScheduler]( > https://dolphinscheduler.apache.org/python/concept.html#). > > A YAML file defines a workflow : > > ```yaml > # Define the process > Process: > #process(workflow) name > name: prepare_datas > > # Parameters for process creation > param: > project: /data/project > > # Whether to run the workflow after the creation is complete > run: True > > # Define the tasks under the process > Tasks: > - > # task type: Shell, python, SubProcess, Spark, etc > TaskType: Shell > > # Upstream Task List > dependencies: [xxxx] > > # Parameters for task creation > params: > name: task1 > other parameters: ... > > - > TaskType: Python > params: > name: task2 > - ... > - ... > ``` > > I already add a GitHub Issue for my proposal, which you could see in > https://github.com/apache/dolphinscheduler/issues/10995 > > Looking forward any feedback for this thread.
