In other words, we have not reached this stage or moment like the Hadoop 1.x to 2.x I think that do not make plugin for plugin, and avoid over design and development
Shiwen Cheng / 程世文 DolphinScheduler Committer Mobile: (+86)15201523580 E-mail: [email protected] Calvin Kirs <[email protected]> 于2021年6月22日周二 下午7:19写道： > For the moment, I agree with you. > > > But from a plugin design perspective, the changes are split out for better > scaling as well as maintenance. > > > Hadoop 1.x to 2.x did a similar job. > > > Best Wishes！ > CalvinKirs, Apache DolphinScheduler PMC > > > On 06/22/2021 18:52，Shiwen Cheng<[email protected]> wrote： > At present, this demand does not seem to be strong, because there are not > many types of tasks that can be run in yarn or k8s (mesos is also retired). > For this reason, the solution of separation and plugin are performed, and > the cost of revenue is relatively low. If you want to submit flink or spark > tasks on k8s, we only need to do some simple integrations. > > For docker/k8s, we can plan to support cloud native better, like > dolphinscheduler native on k8s (Maybe it caters to the needs of users > better and is more valuable for commercialization) > > > Shiwen Cheng / 程世文 > DolphinScheduler Committer > Mobile: (+86)15201523580 > E-mail: [email protected] > > > CalvinKirs <[email protected]> 于2021年6月17日周四 下午3:50写道： > > Hi guys: > > > Currently we support two types of task submission: native submission and > submission to yarn (resource scheduling platform), the final result of the > task needs to be obtained from the resource scheduling platform (excluding > the native submission), for better scaling, I suggest we separate the two. > > > 1:Tasks can be separate as a component in which users can extend other > task plugins that > > > 2:Resource scheduling platform as another component in which the user can > extend, such as k8s, yarn, etc. > > > 3:Task plugins need to know which resource scheduling platforms they can > submit to, and the final result of the task needs to be obtained through > the resource scheduling platform. This needs to be left to the kernel to > take care of the interaction, which in my opinion is not a big deal. > > > 4:Associated with this is the data source plugin as well as the resource > storage plugin. I am currently working on these two components. Again, the > interaction between the task plugins and these components needs to be left > to the kernel to manage. > > > @geosmart, @blackberrier have discussed this in [1], you can refer to it, > I would like to hear your opinion. > > > [1]https://github.com/apache/dolphinscheduler/issues/5648 > > > Best Wishes！ > CalvinKirs, Apache DolphinScheduler PMC > > >
