Hello Everyone: DS has good horizontal scalability with its non central design architecture, which attracts many developers. With more and more users, the demand for scheduling is becoming more and more complex. At the same time, the functional design of DS is required to be more scalable，for example: the plug-in function of alarm mode. So we can discuss what parts of plug-ins DS can do at present. We can reconstruct DolphinScheduler according to the results of this discussion. At present, there are several parts of demand:
- alert model: refer to: https://github.com/apache/incubator-dolphinscheduler/issues/3049 - task plugin: refer to: https://github.com/apache/incubator-dolphinscheduler/issues/2869 - register center: refer to: https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E https://github.com/apache/incubator-dolphinscheduler/issues/3961 - log model: The current log is recorded by writing local files of the server. Can we make this plug-in type, which can facilitate users to extend the log reading and writing types, such as writing to the database or other third-party systems. - global task queue At present, tasks are stored in the memory queue of the master, which results in the priority of a task can only work within the scope of a master. In order to make the priority of a task effective globally, we need a global queue to make the global priority work. (in version 1.2, we used zookeeper as the global queue, which was removed because of the delay of ZK operation) Implementation details can be discussed within each topic. Here, we only discuss the requirements. Very appreciate you can put forward more opinions. ================================================================================================================================================================== DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。 目前已经有需求的几个部分： - 告警插件(running) 相关讨论： https://github.com/apache/incubator-dolphinscheduler/issues/3049 - 任务插件 https://github.com/apache/incubator-dolphinscheduler/issues/2869 - 注册中心 相关讨论： https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E https://github.com/apache/incubator-dolphinscheduler/issues/3961 - 日志插件 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。 - 全局队列插件 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。 -- DolphinScheduler(Incubator) PPMC BaoLiang 鲍亮 [email protected]
