@JUN GAO ,I think changqun.xu is right, using different alert type according to task priority is a good pratice
On 2020/6/24, 17:58, "changqun.xu" <[email protected]> wrote: Hi JUN GAO It’s a very good design, and I’d like want a feature to use different alert type according to different task priority, for example ‘Middle task’ use email to send alarms, ‘High task’ use dingding … Glad to have more communication, thanks. > 在 2020年6月24日，16:33，JUN GAO <[email protected]> 写道： > > Hi All, This is a part of the [Proposal] DolphinScheduler Plugin Design > https://github.com/apache/incubator-dolphinscheduler/issues/2869 > > Welcome all partners who are interested in this issue to participate in the > discussion and development contribution > > > Summary designPlugin definition & plugin instances > > The core of the plug-in design is that the upper layer logic and the > specific implementation are decoupled by SPI. When the system uses the > plug-in, it should create a plug-in instance based on the plug-in supported > by the current system. For example, the mail alert plug-in will provide the > required parameter list. When used in the system, an instance of the > plug-in needs to be created based on the parameter list. Another example is > the Job of MR task, the Job plug-in will define the required parameters, > and then in the DAG we create the Job and generate an instance of the Job > plug-in. > Configuration parameters & configuration files > > In addition, there are some problems in configuring the alert through the > alert.properties method. For example, in some enterprises, multiple > departments work on the DS, and different departments need to configure > different WeChat alert and e-mail alert stmp services. This scenario cannot > be met by alert.properties. Therefore, we need to support the function of > creating multiple alert mode instances under one alert mode plug-in. In > order to instantiate this capability and reduce the configuration files to > simplify installation and deployment, I provide all the parameters required > by the plug-in to the front end as a parameter template for the plug-in, > fill in when creating an instance of the plug-in on the front-end page, you > can achieve more An email alert function, and no longer need to modify the > alert.properties file through the background. > alert group & DS user > > In the current DS version, the alert group is a collection of users. The > alert service finds the user list through the alert group when sending an > alert, and then obtains the email address of each user and adds it to the > recipient of the email alert. However, we found that we can't manage WeChat > alert and DingTalk alert through alert groups. WeChat and nail alert can > only be configured and used through the backend alert.properties > configuration file. All we intend to redefine the alert group, the alert > group is defined as a collection of alert plug-in instances, rather than > just a collection of users. > workflow define & alert group > > There are two fields in the current process schedule, e-mail recipient and > cc sender, which binds the process definition to e-mail alerts to a certain > extent. The actual situation is that when the process is defined, the alert > may not be used, or an alert method other than email may be used. So we > need to redefine the relationship between the process definition and the > alert and alert group. > Front-end UI plug-in > > http://www.alpacajs.org/docs/api/forms.html The front-end architecture > supports the generation of front-end UI components based on json. In SPI, I > encapsulated the parameters of the plug-in, and all the plug-in parameters > will be automatically converted into json supported by alpacajs. > User path > > Based on the above design considerations, we have redefined the user path > of the alert function to make it more scalable and easier to understand and > use. > > The modified user path is as follows: > > If the user needs to use the alert function, the user can choose to create > an alert instance based on the existing alert plug-in in the alert group > creation and management page. For example, the user creates an alert group > AlertGroupA, and then on the management page of AlertGroupA, you can choose > to create a specific alert method through the "Add Alert Method" operation. > When "Add Alert Method" is displayed, the existing alert plug-in (plug-in > The information can be put into plugin_define in the plug-in definition > table in db). The user selects the corresponding plug-in, and opens the > creation page, which displays all the required parameters of the alert > plug-in. After the user completes, an alert method instance is created. > > Multiple instances of multiple alert methods can be created in an alert > group. > > The process definition is associated with an alert group ID. The process > definition is no longer associated with a specific alert method, but is > instead associated with an alert group. In this way, even if you need to > modify the alert method in the future, you only need to modify the alert > method instance in the alert group management, and no longer need to modify > the process definition. > detailed designDatabase table designPlugin definition table plugin_define > > id, > plugin_name, > plugin_params > Plugin instance table plugin_instance > > id > plugin_define_id, > plugin_instance_params > Interface designincubator-dolphinscheduler-maven-plugin > > https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin > > A custom maven plug-in, its role is to automatically check the spi and > plug-in related dependencies, and can support adding > dolphinscheduler-plugin logo in the pom file to allow DS to automatically > generate related META-INF/services files at compile time. For detailed > information, please refer to: > https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin > SPIDolphinSchedulerPlugin > > DS plug-in top-level interface, all DS plug-ins must implement this > interface > AlertChannelFactory > > Alert plug-in factory interface. All alert plug-ins need to implement this > interface. This interface is used to define the name of the alert plug-in > and the required parameters. The create method is used to create a specific > alert plug-in instance. > AlertChannel > > The interface of the alert plug-in. The alert plug-in needs to implement > this interface. There is only one method process in the interface. The > upper alert system will use this method and obtain the return information > of the alert through the AlertResult returned by the method. > AlertData > > alert content information, including id, title, content, log. > AlertInfo > > For alert-related information, when an upper-layer system calls an alert > plug-in instance, the instance of this type is passed into the specific > alert plug-in through the process method. It contains the alert content > AlertData and the parameter information filled in by the front end of the > alert plug-in instance called. > AlertResult > > The alert plugin sends alert return information. > org.apache.dolphinscheduler.spi.params > > Under this package is the plug-in parameter definition. Our front-end uses > the front-end library alpacajs http://www.alpacajs.org/docs/api/forms.html, > which can dynamically generate the front-end based on the parameter list > json returned by the plug-in definition Ui. > > Under this package, we currently only encapsulate RadioParam, TextParam, > and PasswordParam, which are used to define text type parameters, radio > parameters, and password type parameters, respectively. > > AbsPluginParams This class is the base class for all parameters, and these > classes inherit from RadioParam. Each DS alert plugin will return a list of > AbsPluginParams in the implementation of AlertChannelFactory. > org.apache.dolphinscheduler.alert.plugin > > Under this package is the relevant code for the DS Alert module to load the > Alert plug-in. In the AlertChannelManager class, we also need to save the > loaded plugin to the database's plugin_define table. > some code of this issue > > gaojun2048#1 > <https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1> > todo list > > 1. > > Improvement of the alert group function, in the alert group, an alert > instance can be created based on the existing alert plug-in. > 2. > > Delete recipients and CCs in the process definition and select the > associated alert group instead. > 3. > > Submit the dolphinscheduler-maven-plugin to the community. > 4. > > Pull a separate branch from dev for development. > 5. > > Transformation of WeChat alert function. > 6. > > Renovation of the nail warning function. > 7. > > Improvement of mail alert function. > 8. > > Upgrade related tool development from the old version. > > 概要设计插件定义与插件实例 > > 插件化设计的核心是上层逻辑和具体实现以SPI的方式解耦，系统在使用插件时，应该基于当前系统支持的插件创建插件实例。比如邮件告警插件，该插件会提供需要的参数列表，在系统中使用时需要基于参数列表来创建该插件的实例。再比如MR任务的Job，该Job插件会定义好需要的参数，然后在DAG中我们创建该Job并生成一个该Job插件的实例。 > 配置参数&配置文件 > > 并且之前通过alert.properties的方式配置告警会有一些问题。比如在有些企业，多个部门都在DS上工作，不同的部门需要配置不同的微信告警和邮件告警stmp服务，这种场景是alert.properties这种方式无法满足的。所以我们需要支持在一个告警方式插件下创建多个告警方式实例的功能。为了实例这个能力并且能够减少配置文件简化安装部署，我把插件需要的所有参数都提供给前端，做为该插件的参数模板，在前端页面上创建该插件的实例时填写，即可实现创建多个邮件告警的功能，同时不再需要通过后台修改alert.properties文件。 > 告警组和DS用户的关系 > > 在目前的DS版本中，告警组是用户的集合，告警服务在发送告警中通过告警组找到用户列表，然后获取每个用户的邮箱地址并添加到邮件告警的接收人中。但是，我们发现我们无法将微信告警和钉钉告警通过告警组来管理，微信和钉钉告警也只能通过后端的alert.properties配置文件来配置使用。所有我们打算重新定义告警组，将告警组定义为告警插件实例的集合，而不再只是用户的集合。 > 流程定义与告警的关系 > > 当前流程定表中有邮件接收人和抄送人这两个字段，这在一定程度上将流程定义与邮件告警绑定了。实际情况是，流程定义时可能不使用告警，也可能使用除了邮件之外的告警方式。所以我们需要重新定义流程定义与告警和告警组三者之间的关系。 > 前端UI插件化 > > http://www.alpacajs.org/docs/api/forms.html > 该前端架构支持基于json生成前端ui组件。SPI里我对插件的参数做了封装，所有的插件参数最终会自动转换成alpacajs支持的json。 > 用户路径 > > 基于以上设计考虑，我们重新定义了告警功能的用户路径，使其使用起来有更好的扩展性，更容易被理解和使用。 > > 修改后的用户路径如下： > 如果用户需要使用告警功能，用户可以在告警组创建和管理页面中选择基于已有的告警插件来创建告警实例。比如用户创建了告警组AlertGroupA，然后在AlertGroupA的管理页面可以通过“添加告警方式”操作来选择创建具体的告警方式，“添加告警方式”时通过下拉菜单展示目前系统中已有的告警插件（插件信息可以放到db里的插件定义表中plugin_define），用户选择对应的插件，会打开创建页面，该页面会展示该告警插件所有需要的参数，用户填写完成后即创建了一个告警方式的实例。 > > 一个告警组中可以创建多种告警方式的多个实例。 > > 流程定义关联告警组ID，流程定义不再关联具体的告警方式，而是改为关联一个告警组。这样即使以后需要修改告警方式，也只需要在告警组管理中修改告警方式实例即可，不再需要修改流程定义。 > 详细设计数据库表设计插件定义表 plugin_define > > id, > plugin_name, > plugin_params > 插件实例表 plugin_instance > > id > plugin_define_id, > plugin_instance_params > 接口设计incubator-dolphinscheduler-maven-plugin > > https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin > > 一个自定义maven插件，它的作用是可以自动检查spi和插件相关的依赖，并且可以支持在pom文件中添加dolphinscheduler-plugin标识从而让DS在编译时自动生成相关的META-INF/services文件。详细的信息可以参考 > ： https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin > SPIDolphinSchedulerPlugin > > DS插件顶层接口，所有DS的插件都必须实现该接口 > AlertChannelFactory > > 告警插件工厂接口，所有告警插件需要实现该接口，该接口用来定义告警插件的名称，需要的参数，create方法用来创建具体的告警插件实例. > AlertChannel > > 告警插件的接口，告警插件需要实现该接口，该接口中只有一个方法process > ，上层告警系统会该用该方法并通过该方法返回的AlertResult来获取告警的返回信息。 > AlertData > > 告警内容信息，包括id，标题，内容，日志。 > AlertInfo > > 告警相关信息，上层系统调用告警插件实例时，将该类的实例通过process方法传入具体的告警插件。内部包含告警内容AlertData和调用的告警插件实例的前端填写的参数信息。 > AlertResult > > 告警插件发送告警返回信息。 > org.apache.dolphinscheduler.spi.params > > 该包下是插件化的参数定义，我们前端使用alpacajs这个前端库 > http://www.alpacajs.org/docs/api/forms.html，该库可以基于插件定义返回的参数列表json来动态生成前端的ui。 > > 该package下我们目前只封装了RadioParam，TextParam，PasswordParam，分别用来定义text类型的参数，radio参数和 > password类型的参数。 > > AbsPluginParams > 该类是所有参数的基类，RadioParam这些类都继承了该类。每个DS的告警插件都会在AlertChannelFactory的实现中返回一个AbsPluginParams的list。 > org.apache.dolphinscheduler.alert.plugin > > 该包下是DS > Alert模块加载Alert插件的相关代码，已经大部分。AlertChannelManager类中我们还需要将加载到的插件保存到数据库的plugin_define表中。 > 关于该Issue的一些代码 > > gaojun2048#1 > <https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1> > todo list > > 1. > > 告警组功能改造，在告警组中可以基于已有的告警插件创建告警实例。 > 2. > > 流程定义中删除收件人，抄送人，改为选择关联告警组。 > 3. > > 将dolphinscheduler-maven-plugin提交到社区。 > 4. > > 从dev拉单独分支，用于开发。 > 5. > > 微信告警功能改造。 > 6. > > 钉钉告警功能的改造。 > 7. > > 邮件告警功能的完善。 > 8. > > 从老版本升级相关的工具开发。 > > > 欢迎对这个issue感兴趣的小伙伴参与讨论和开发贡献。 > > > -- > > DolphinScheduler(Incubator) PPMC > Jun Gao 高俊 > [email protected]
