gaojun2048 opened a new issue #3049: URL: https://github.com/apache/incubator-dolphinscheduler/issues/3049
# Summary design ## Plugin definition and plugin examples The core of the plug-in design is that the upper layer logic and the specific implementation are decoupled by SPI. When the system uses the plug-in, it should create a plug-in instance based on the plug-in supported by the current system. For example, the mail alert plug-in will provide the required parameter list. When used in the system, an instance of the plug-in needs to be created based on the parameter list. Another example is the Job of MR task, the Job plug-in will define the required parameters, and then in the DAG we create the Job and generate an instance of the Job plug-in. ## Configuration parameters & configuration files In addition, there are some problems in configuring the alarm through the alert.properties method. For example, in some enterprises, multiple departments work on the DS, and different departments need to configure different WeChat alarms and e-mail alarm stmp services. This scenario cannot be met by alert.properties. Therefore, we need to support the function of creating multiple alarm mode instances under one alarm mode plug-in. In order to instantiate this capability and reduce the configuration files to simplify installation and deployment, I provide all the parameters required by the plug-in to the front end as a parameter template for the plug-in, fill in when creating an instance of the plug-in on the front-end page, you can achieve more An email alert function, and no longer need to modify the alert.properties file through the background. ## Relationship between alarm group and DS user In the current DS version, the alarm group is a collection of users. The alarm service finds the user list through the alarm group when sending an alarm, and then obtains the email address of each user and adds it to the recipient of the email alarm. However, we found that we can't manage WeChat alarms and nail alarms through alarm groups. WeChat and nail alarms can only be configured and used through the backend alert.properties configuration file. All we intend to redefine the alarm group, the alarm group is defined as a collection of alarm plug-in instances, rather than just a collection of users. ## Relationship between process definition and alarm There are two fields in the current process schedule, e-mail recipient and cc sender, which binds the process definition to e-mail alerts to a certain extent. The actual situation is that when the process is defined, the alarm may not be used, or an alarm method other than email may be used. So we need to redefine the relationship between the process definition and the alarm and alarm group. ## User path Based on the above design considerations, we have redefined the user path of the alarm function to make it more scalable and easier to understand and use. The modified user path is as follows: If the user needs to use the alarm function, the user can choose to create an alarm instance based on the existing alarm plug-in in the alarm group creation and management page. For example, the user creates an alert group AlertGroupA, and then on the management page of AlertGroupA, you can choose to create a specific alert method through the "Add Alert Method" operation. When "Add Alert Method" is displayed, the existing alert plug-in (plug-in The information can be put into plugin_define in the plug-in definition table in db). The user selects the corresponding plug-in, and opens the creation page, which displays all the required parameters of the alarm plug-in. After the user completes, an alarm method instance is created. Multiple instances of multiple alarm methods can be created in an alarm group. The process definition is associated with an alarm group ID. The process definition is no longer associated with a specific alarm method, but is instead associated with an alarm group. In this way, even if you need to modify the alarm method in the future, you only need to modify the alarm method instance in the alarm group management, and no longer need to modify the process definition. # detailed design ## Database table design ### Plugin definition table plugin_define id, plugin_name, plugin_params ### Plugin instance table plugin_instance id plugin_define_id, plugin_instance_params ## Interface design ### incubator-dolphinscheduler-maven-plugin A custom maven plug-in, its role is to automatically check the spi and plug-in related dependencies, and can support adding dolphinscheduler-plugin logo in the pom file to allow DS to automatically generate related META-INF/services files at compile time. For detailed information, please refer to: https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin ## SPI ### DolphinSchedulerPlugin DS plug-in top-level interface, all DS plug-ins must implement this interface ### AlertChannelFactory Alarm plug-in factory interface. All alarm plug-ins need to implement this interface. This interface is used to define the name of the alarm plug-in and the required parameters. The create method is used to create a specific alarm plug-in instance. ### AlertChannel The interface of the alarm plug-in. The alarm plug-in needs to implement this interface. There is only one method process in the interface. The upper alarm system will use this method and obtain the return information of the alarm through the AlertResult returned by the method. ### AlertData Alarm content information, including id, title, content, log. ### AlertInfo For alarm-related information, when an upper-layer system calls an alarm plug-in instance, the instance of this type is passed into the specific alarm plug-in through the process method. It contains the alarm content AlertData and the parameter information filled in by the front end of the alarm plug-in instance called. ### AlertResult The alarm plugin sends alarm return information. ### org.apache.dolphinscheduler.spi.params Under this package is the plug-in parameter definition. Our front-end uses the front-end library alpacajs http://www.alpacajs.org/docs/api/forms.html, which can dynamically generate the front-end based on the parameter list json returned by the plug-in definition Ui. Under this package, we currently only encapsulate RadioParam, TextParam, and PasswordParam, which are used to define text type parameters, radio parameters, and password type parameters, respectively. AbsPluginParams This class is the base class for all parameters, and these classes inherit from RadioParam. Each DS alarm plugin will return a list of AbsPluginParams in the implementation of AlertChannelFactory. ### org.apache.dolphinscheduler.alert.plugin Under this package is the relevant code for the DS Alert module to load the Alert plug-in. In the AlertChannelManager class, we also need to save the loaded plugin to the database's plugin_define table. # todo list 1. Improvement of the alarm group function, in the alarm group, an alarm instance can be created based on the existing alarm plug-in. 2. Delete recipients and CCs in the process definition and select the associated alarm group instead. 3. Submit the dolphinscheduler-maven-plugin to the community. 4. Pull a separate branch from dev for development. 5. Transformation of WeChat alarm function. 6. Renovation of the nail warning function. 7. Improvement of mail alarm function. 8. Upgrade related tool development from the old version. # 概要设计 ## 插件定义与插件实例 插件化设计的核心是上层逻辑和具体实现以SPI的方式解耦，系统在使用插件时，应该基于当前系统支持的插件创建插件实例。比如邮件告警插件，该插件会提供需要的参数列表，在系统中使用时需要基于参数列表来创建该插件的实例。再比如MR任务的Job，该Job插件会定义好需要的参数，然后在DAG中我们创建该Job并生成一个该Job插件的实例。 ## 配置参数&配置文件 并且之前通过alert.properties的方式配置告警会有一些问题。比如在有些企业，多个部门都在DS上工作，不同的部门需要配置不同的微信告警和邮件告警stmp服务，这种场景是alert.properties这种方式无法满足的。所以我们需要支持在一个告警方式插件下创建多个告警方式实例的功能。为了实例这个能力并且能够减少配置文件简化安装部署，我把插件需要的所有参数都提供给前端，做为该插件的参数模板，在前端页面上创建该插件的实例时填写，即可实现创建多个邮件告警的功能，同时不再需要通过后台修改alert.properties文件。 ## 告警组和DS用户的关系 在目前的DS版本中，告警组是用户的集合，告警服务在发送告警中通过告警组找到用户列表，然后获取每个用户的邮箱地址并添加到邮件告警的接收人中。但是，我们发现我们无法将微信告警和钉钉告警通过告警组来管理，微信和钉钉告警也只能通过后端的alert.properties配置文件来配置使用。所有我们打算重新定义告警组，将告警组定义为告警插件实例的集合，而不再只是用户的集合。 ## 流程定义与告警的关系 当前流程定表中有邮件接收人和抄送人这两个字段，这在一定程度上将流程定义与邮件告警绑定了。实际情况是，流程定义时可能不使用告警，也可能使用除了邮件之外的告警方式。所以我们需要重新定义流程定义与告警和告警组三者之间的关系。 ## 用户路径 基于以上设计考虑，我们重新定义了告警功能的用户路径，使其使用起来有更好的扩展性，更容易被理解和使用。 修改后的用户路径如下： 如果用户需要使用告警功能，用户可以在告警组创建和管理页面中选择基于已有的告警插件来创建告警实例。比如用户创建了告警组AlertGroupA，然后在AlertGroupA的管理页面可以通过“添加告警方式”操作来选择创建具体的告警方式，“添加告警方式”时通过下拉菜单展示目前系统中已有的告警插件（插件信息可以放到db里的插件定义表中plugin_define），用户选择对应的插件，会打开创建页面，该页面会展示该告警插件所有需要的参数，用户填写完成后即创建了一个告警方式的实例。 一个告警组中可以创建多种告警方式的多个实例。 流程定义关联告警组ID，流程定义不再关联具体的告警方式，而是改为关联一个告警组。这样即使以后需要修改告警方式，也只需要在告警组管理中修改告警方式实例即可，不再需要修改流程定义。 # 详细设计 ## 数据库表设计 ### 插件定义表 plugin_define id, plugin_name, plugin_params ### 插件实例表 plugin_instance id plugin_define_id, plugin_instance_params ## 接口设计 ### incubator-dolphinscheduler-maven-plugin 一个自定义maven插件，它的作用是可以自动检查spi和插件相关的依赖，并且可以支持在pom文件中添加dolphinscheduler-plugin标识从而让DS在编译时自动生成相关的META-INF/services文件。详细的信息可以参考 ： https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin ## SPI ### DolphinSchedulerPlugin DS插件顶层接口，所有DS的插件都必须实现该接口 ### AlertChannelFactory 告警插件工厂接口，所有告警插件需要实现该接口，该接口用来定义告警插件的名称，需要的参数，create方法用来创建具体的告警插件实例. ### AlertChannel 告警插件的接口，告警插件需要实现该接口，该接口中只有一个方法process ，上层告警系统会该用该方法并通过该方法返回的AlertResult来获取告警的返回信息。 ### AlertData 告警内容信息，包括id，标题，内容，日志。 ### AlertInfo 告警相关信息，上层系统调用告警插件实例时，将该类的实例通过process方法传入具体的告警插件。内部包含告警内容AlertData和调用的告警插件实例的前端填写的参数信息。 ### AlertResult 告警插件发送告警返回信息。 ### org.apache.dolphinscheduler.spi.params 该包下是插件化的参数定义，我们前端使用alpacajs这个前端库http://www.alpacajs.org/docs/api/forms.html，该库可以基于插件定义返回的参数列表json来动态生成前端的ui。 该package下我们目前只封装了RadioParam，TextParam，PasswordParam，分别用来定义text类型的参数，radio参数和 password类型的参数。 AbsPluginParams 该类是所有参数的基类，RadioParam这些类都继承了该类。每个DS的告警插件都会在AlertChannelFactory的实现中返回一个AbsPluginParams的list。 ### org.apache.dolphinscheduler.alert.plugin 该包下是DS Alert模块加载Alert插件的相关代码，已经大部分。AlertChannelManager类中我们还需要将加载到的插件保存到数据库的plugin_define表中。 # todo list 1. 告警组功能改造，在告警组中可以基于已有的告警插件创建告警实例。 2. 流程定义中删除收件人，抄送人，改为选择关联告警组。 3. 将dolphinscheduler-maven-plugin提交到社区。 4. 从dev拉单独分支，用于开发。 5. 微信告警功能改造。 6. 钉钉告警功能的改造。 7. 邮件告警功能的完善。 8. 从老版本升级相关的工具开发。 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
