Hi guys:
Currently we support two types of task submission: native submission and submission to yarn (resource scheduling platform), the final result of the task needs to be obtained from the resource scheduling platform (excluding the native submission), for better scaling, I suggest we separate the two. 1:Tasks can be separate as a component in which users can extend other task plugins that 2:Resource scheduling platform as another component in which the user can extend, such as k8s, yarn, etc. 3:Task plugins need to know which resource scheduling platforms they can submit to, and the final result of the task needs to be obtained through the resource scheduling platform. This needs to be left to the kernel to take care of the interaction, which in my opinion is not a big deal. 4:Associated with this is the data source plugin as well as the resource storage plugin. I am currently working on these two components. Again, the interaction between the task plugins and these components needs to be left to the kernel to manage. @geosmart, @blackberrier have discussed this in [1], you can refer to it, I would like to hear your opinion. [1]https://github.com/apache/dolphinscheduler/issues/5648 Best Wishes！ CalvinKirs, Apache DolphinScheduler PMC
