For the moment, I agree with you.
But from a plugin design perspective, the changes are split out for better scaling as well as maintenance. Hadoop 1.x to 2.x did a similar job. Best Wishes！ CalvinKirs, Apache DolphinScheduler PMC On 06/22/2021 18:52，Shiwen Cheng<[email protected]> wrote： At present, this demand does not seem to be strong, because there are not many types of tasks that can be run in yarn or k8s (mesos is also retired). For this reason, the solution of separation and plugin are performed, and the cost of revenue is relatively low. If you want to submit flink or spark tasks on k8s, we only need to do some simple integrations. For docker/k8s, we can plan to support cloud native better, like dolphinscheduler native on k8s (Maybe it caters to the needs of users better and is more valuable for commercialization) Shiwen Cheng / 程世文 DolphinScheduler Committer Mobile: (+86)15201523580 E-mail: [email protected] CalvinKirs <[email protected]> 于2021年6月17日周四 下午3:50写道： Hi guys: Currently we support two types of task submission: native submission and submission to yarn (resource scheduling platform), the final result of the task needs to be obtained from the resource scheduling platform (excluding the native submission), for better scaling, I suggest we separate the two. 1:Tasks can be separate as a component in which users can extend other task plugins that 2:Resource scheduling platform as another component in which the user can extend, such as k8s, yarn, etc. 3:Task plugins need to know which resource scheduling platforms they can submit to, and the final result of the task needs to be obtained through the resource scheduling platform. This needs to be left to the kernel to take care of the interaction, which in my opinion is not a big deal. 4:Associated with this is the data source plugin as well as the resource storage plugin. I am currently working on these two components. Again, the interaction between the task plugins and these components needs to be left to the kernel to manage. @geosmart, @blackberrier have discussed this in [1], you can refer to it, I would like to hear your opinion. [1]https://github.com/apache/dolphinscheduler/issues/5648 Best Wishes！ CalvinKirs, Apache DolphinScheduler PMC
