good job, you did very well, and I found that these PRs have been merged. Looking forward to your new suggestions.
Best Regards --------------- Apache DolphinScheduler PMC Chair & Apache SeaTunnel PPMC David Linkedin: https://www.linkedin.com/in/davidzollo Twitter: @WorkflowEasy <https://twitter.com/WorkflowEasy> --------------- On Fri, Sep 16, 2022 at 11:55 AM rj chen <[email protected]> wrote: > Hi community, > At present, the command executed by the task in the python Api is to pass > the specific content through the parameter. For example, shell task: > Shell(name="task_parent", command="echo hello pydolphinscheduler"). > File is a good choice when the command executed by the task is very long. > > Teacher Zhongjiajie guided me, we called it New mechanism file plugins to > Python API. Resource plugins can be used in workflows and tasks. When both > have resource plugins, the resource plugin in the task is used first. > Plugins I will implement include but are not limited to local file system, > GitHub, GitLab, Amazon S3, Alibaba Cloud OSS. > > This is a tutorial for the local resource plugin, you can find it in PR[1], > it is the local resource plugin. > with ProcessDefinition( > name="tutorial_resource_plugin", > schedule="0 0 0 * * ? *", > start_time="2022-01-01", > tenant="tenant_exists", > resource_plugin=Local("/tmp"), > ) as process_definition: > file = "resource.sh" > path = Path("/tmp").joinpath(file) > with open(str(path), "w") as f: > f.write("echo tutorial resource plugin") > task_parent = Shell( > name="local-resource-example", > command=file, > ) > print(task_parent.task_params) > os.remove(path) > process_definition.run() > > Below is my draft design for the shell task. > You can see the GitLab resource plugin in PR[2] to understand my design. > > I abstracted a Resource class and created an abstract function read_file > for it. All resource plugins need to inherit from it and implement the > abstract function read_file. > > Add property in shell > ext: set = {".sh", ".zsh"} > ext is used to detect that shell tasks can only accept files ending in .sh > and .zsh > ext_attr: str = "_raw_script" > At the same time, it is added in the init function > self._raw_script = command > ext_attr is used to save the file path, > > Add a new parameter resource_plugin to the init function of the task to > accept resource plug-ins, add a new method to obtain specific resource > plug-ins, and also need to add attributes ext and ext_attr. > > Additional details can be found in pr[1] and pr[2]. > I'm very sorry, it's my first time to contribute and I didn't follow the > DSIP process, sorry. > > I already add a GitHub Issue for my proposal, which you could see in > https://github.com/apache/dolphinscheduler/issues/10911 > <https://github.com/apache/dolphinscheduler/issues/10911>. > > Looking forward any feedback for this thread. > > [1]: https://github.com/apache/dolphinscheduler/pull/11360 < > https://github.com/apache/dolphinscheduler/pull/11360> > [2]: https://github.com/apache/dolphinscheduler/pull/11831 < > https://github.com/apache/dolphinscheduler/pull/11831> >
