well done Ruijie On Tue, Sep 27, 2022 at 11:07 PM david zollo <[email protected]> wrote: > > good job, you did very well, and I found that these PRs have been merged. > > Looking forward to your new suggestions. > > > Best Regards > > --------------- > Apache DolphinScheduler PMC Chair & Apache SeaTunnel PPMC > David > Linkedin: https://www.linkedin.com/in/davidzollo > Twitter: @WorkflowEasy <https://twitter.com/WorkflowEasy> > --------------- > > > On Fri, Sep 16, 2022 at 11:55 AM rj chen <[email protected]> wrote: > > > Hi community, > > At present, the command executed by the task in the python Api is to pass > > the specific content through the parameter. For example, shell task: > > Shell(name="task_parent", command="echo hello pydolphinscheduler"). > > File is a good choice when the command executed by the task is very long. > > > > Teacher Zhongjiajie guided me, we called it New mechanism file plugins to > > Python API. Resource plugins can be used in workflows and tasks. When both > > have resource plugins, the resource plugin in the task is used first. > > Plugins I will implement include but are not limited to local file system, > > GitHub, GitLab, Amazon S3, Alibaba Cloud OSS. > > > > This is a tutorial for the local resource plugin, you can find it in PR[1], > > it is the local resource plugin. > > with ProcessDefinition( > > name="tutorial_resource_plugin", > > schedule="0 0 0 * * ? *", > > start_time="2022-01-01", > > tenant="tenant_exists", > > resource_plugin=Local("/tmp"), > > ) as process_definition: > > file = "resource.sh" > > path = Path("/tmp").joinpath(file) > > with open(str(path), "w") as f: > > f.write("echo tutorial resource plugin") > > task_parent = Shell( > > name="local-resource-example", > > command=file, > > ) > > print(task_parent.task_params) > > os.remove(path) > > process_definition.run() > > > > Below is my draft design for the shell task. > > You can see the GitLab resource plugin in PR[2] to understand my design. > > > > I abstracted a Resource class and created an abstract function read_file > > for it. All resource plugins need to inherit from it and implement the > > abstract function read_file. > > > > Add property in shell > > ext: set = {".sh", ".zsh"} > > ext is used to detect that shell tasks can only accept files ending in .sh > > and .zsh > > ext_attr: str = "_raw_script" > > At the same time, it is added in the init function > > self._raw_script = command > > ext_attr is used to save the file path, > > > > Add a new parameter resource_plugin to the init function of the task to > > accept resource plug-ins, add a new method to obtain specific resource > > plug-ins, and also need to add attributes ext and ext_attr. > > > > Additional details can be found in pr[1] and pr[2]. > > I'm very sorry, it's my first time to contribute and I didn't follow the > > DSIP process, sorry. > > > > I already add a GitHub Issue for my proposal, which you could see in > > https://github.com/apache/dolphinscheduler/issues/10911 > > <https://github.com/apache/dolphinscheduler/issues/10911>. > > > > Looking forward any feedback for this thread. > > > > [1]: https://github.com/apache/dolphinscheduler/pull/11360 < > > https://github.com/apache/dolphinscheduler/pull/11360> > > [2]: https://github.com/apache/dolphinscheduler/pull/11831 < > > https://github.com/apache/dolphinscheduler/pull/11831> > >
-- Best Wish — Jiajie
