Hi @YiChao Yang . That`s great !
Yichao Yang <[email protected]> 于2020年7月15日周三 下午1:03写道： > Hi, > I am interested in 2, I will create an issue about this subtask. > Thx a lot. > > > Best, > Yichao Yang > > > ------------------ Original ------------------ > From: JUN GAO <[email protected]> > Date: Wed,Jul 15,2020 0:58 PM > To: dev <[email protected]>, 向偲彪 <[email protected]> > Subject: Re: [Proposal] Alert Plugin Design > > > > Hi all. > > I created a branch > > https://github.com/apache/incubator-dolphinscheduler/tree/alert_plugin_design > for > this feature. > > The issue about this feature is > https://github.com/apache/incubator-dolphinscheduler/issues/3049 > > > *The todo list that I currently think of is as follows:* > > - [ 1] https://github.com/apache/incubator-dolphinscheduler/issues/3189 > Add > DolphinScheduler SPI module, Plugin database schema, PluginLoader. Alert > SPI and Alert Plugin basic code development. Improvement of mail Alert > Plugin. @gaojun2048 > > - [ 2] Improvement of the alert group function, in the alert group, an > alert instance can be created based on the existing alert plug-in. > > - [ 3] Delete recipients and CCs in the process definition and select the > associated alert group instead. > > - [ 4] Implement WeChat Alert Plugin. > > - [5 ] Implement Dingtalk Alert Plugin > > - [ 6] Upgrade tool development from the old version. > > Does anyone interested in this feature want to participate in this feature? > You can reply to the email to pick up the tasks you are interested in. > > > > JUN GAO <[email protected]> 于2020年7月9日周四 下午7:32写道： > > > @qiao zhanwei > > > > Can you help me create a branch for this proposal ? > > > > JUN GAO <[email protected]> 于2020年7月9日周四 下午7:24写道： > > > >> @向偲彪 <[email protected]> I finished update the web ui > to form-create. > >> > >> JUN GAO <[email protected]> 于2020年6月30日周二 下午1:44写道： > >> > >>> @向偲彪 <[email protected]> I read the document of > >>> http://www.form-create.com/ , It`s a good web ui tool for DS > Plugin-in > >>> . I will update the code in package > >>> org.apache.dolphinscheduler.spi.params > >>> > >>> I noticed that the license of form-create is MIT, so don’t > forget to add > >>> the reference of the license. > >>> > >>> JUN GAO <[email protected]> 于2020年6月29日周一 上午11:54写道： > >>> > >>>> Ok, I will read the document about > http://form-create.com/ . And then > >>>> I will update the code of the PluginParams. > >>>> > >>>> 向偲彪 <[email protected]> 于2020年6月28日周日 上午9:30写道： > >>>> > >>>>> Regarding plug-inization, the front-end can use > form-create, which > >>>>> supports json format data, but the UI only supports > iview and element-ui, > >>>>> and the existing ans-ui needs to be replaced. The > form-create that > >>>>> personally feels is better compatible with the page > style, and the document > >>>>> details Wait, form-create link address: > http://form-create.com/ > >>>>> <http://form-create.com/>>>>>>; > >>>>> > >>>>> > >>>>> > >>>>> DolphinScheduler(Incubator) committer > >>>>> Caibiao Xiang 向偲彪 > >>>>> [email protected] > >>>>> > >>>>> > >>>>> > ------------------&nbsp;原始邮件&nbsp;------------------ > >>>>> 发件人:&nbsp;"JUN GAO"<[email protected]&gt;; > >>>>> 发送时间:&nbsp;2020年6月24日(星期三) 下午4:33 > >>>>> 收件人:&nbsp;"dev"<[email protected] > &gt;; > >>>>> > >>>>> 主题:&nbsp;[Proposal] Alert Plugin Design > >>>>> > >>>>> > >>>>> > >>>>> Hi All, This is a part of the [Proposal] > DolphinScheduler Plugin Design > >>>>> > https://github.com/apache/incubator-dolphinscheduler/issues/2869 > >>>>> > <https://github.com/apache/incubator-dolphinscheduler/issues/2869>>>>>> > ; > >>>>> Welcome all partners who are interested in this issue > to participate > >>>>> in the > >>>>> discussion and development contribution > >>>>> > >>>>> > >>>>> Summary designPlugin definition &amp; plugin > instances > >>>>> > >>>>> The core of the plug-in design is that the upper > layer logic and the > >>>>> specific implementation are decoupled by SPI. When > the system uses the > >>>>> plug-in, it should create a plug-in instance based on > the plug-in > >>>>> supported > >>>>> by the current system. For example, the mail alert > plug-in will > >>>>> provide the > >>>>> required parameter list. When used in the system, an > instance of the > >>>>> plug-in needs to be created based on the parameter > list. Another > >>>>> example is > >>>>> the Job of MR task, the Job plug-in will define the > required > >>>>> parameters, > >>>>> and then in the DAG we create the Job and generate an > instance of the > >>>>> Job > >>>>> plug-in. > >>>>> Configuration parameters &amp; configuration files > >>>>> > >>>>> In addition, there are some problems in configuring > the alert through > >>>>> the > >>>>> alert.properties method. For example, in some > enterprises, multiple > >>>>> departments work on the DS, and different departments > need to configure > >>>>> different WeChat alert and e-mail alert stmp > services. This scenario > >>>>> cannot > >>>>> be met by alert.properties. Therefore, we need to > support the function > >>>>> of > >>>>> creating multiple alert mode instances under one > alert mode plug-in. In > >>>>> order to instantiate this capability and reduce the > configuration > >>>>> files to > >>>>> simplify installation and deployment, I provide all > the parameters > >>>>> required > >>>>> by the plug-in to the front end as a parameter > template for the > >>>>> plug-in, > >>>>> fill in when creating an instance of the plug-in on > the front-end > >>>>> page, you > >>>>> can achieve more An email alert function, and no > longer need to modify > >>>>> the > >>>>> alert.properties file through the background. > >>>>> alert group &amp; DS user > >>>>> > >>>>> In the current DS version, the alert group is a > collection of users. > >>>>> The > >>>>> alert service finds the user list through the alert > group when sending > >>>>> an > >>>>> alert, and then obtains the email address of each > user and adds it to > >>>>> the > >>>>> recipient of the email alert. However, we found that > we can't manage > >>>>> WeChat > >>>>> alert and DingTalk alert through alert groups. WeChat > and nail alert > >>>>> can > >>>>> only be configured and used through the backend > alert.properties > >>>>> configuration file. All we intend to redefine the > alert group, the > >>>>> alert > >>>>> group is defined as a collection of alert plug-in > instances, rather > >>>>> than > >>>>> just a collection of users. > >>>>> workflow define &amp; alert group > >>>>> > >>>>> There are two fields in the current process schedule, > e-mail recipient > >>>>> and > >>>>> cc sender, which binds the process definition to > e-mail alerts to a > >>>>> certain > >>>>> extent. The actual situation is that when the process > is defined, the > >>>>> alert > >>>>> may not be used, or an alert method other than email > may be used. So we > >>>>> need to redefine the relationship between the process > definition and > >>>>> the > >>>>> alert and alert group. > >>>>> Front-end UI plug-in > >>>>> > >>>>> http://www.alpacajs.org/docs/api/forms.html The > front-end architecture > >>>>> supports the generation of front-end UI components > based on json. In > >>>>> SPI, I > >>>>> encapsulated the parameters of the plug-in, and all > the plug-in > >>>>> parameters > >>>>> will be automatically converted into json supported > by alpacajs. > >>>>> User path > >>>>> > >>>>> Based on the above design considerations, we have > redefined the user > >>>>> path > >>>>> of the alert function to make it more scalable and > easier to > >>>>> understand and > >>>>> use. > >>>>> > >>>>> The modified user path is as follows: > >>>>> > >>>>> If the user needs to use the alert function, the user > can choose to > >>>>> create > >>>>> an alert instance based on the existing alert plug-in > in the alert > >>>>> group > >>>>> creation and management page. For example, the user > creates an alert > >>>>> group > >>>>> AlertGroupA, and then on the management page of > AlertGroupA, you can > >>>>> choose > >>>>> to create a specific alert method through the "Add > Alert Method" > >>>>> operation. > >>>>> When "Add Alert Method" is displayed, the existing > alert plug-in > >>>>> (plug-in > >>>>> The information can be put into plugin_define in the > plug-in definition > >>>>> table in db). The user selects the corresponding > plug-in, and opens the > >>>>> creation page, which displays all the required > parameters of the alert > >>>>> plug-in. After the user completes, an alert method > instance is created. > >>>>> > >>>>> Multiple instances of multiple alert methods can be > created in an alert > >>>>> group. > >>>>> > >>>>> The process definition is associated with an alert > group ID. The > >>>>> process > >>>>> definition is no longer associated with a specific > alert method, but is > >>>>> instead associated with an alert group. In this way, > even if you need > >>>>> to > >>>>> modify the alert method in the future, you only need > to modify the > >>>>> alert > >>>>> method instance in the alert group management, and no > longer need to > >>>>> modify > >>>>> the process definition. > >>>>> detailed designDatabase table designPlugin definition > table > >>>>> plugin_define > >>>>> > >>>>> id, > >>>>> plugin_name, > >>>>> plugin_params > >>>>> Plugin instance table plugin_instance > >>>>> > >>>>> id > >>>>> plugin_define_id, > >>>>> plugin_instance_params > >>>>> Interface > designincubator-dolphinscheduler-maven-plugin > >>>>> > >>>>> > https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin > >>>>> > <https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin>>>>>> > ; > >>>>> A custom maven plug-in, its role is to automatically > check the spi and > >>>>> plug-in related dependencies, and can support adding > >>>>> dolphinscheduler-plugin logo in the pom file to allow > DS to > >>>>> automatically > >>>>> generate related META-INF/services files at compile > time. For detailed > >>>>> information, please refer to: > >>>>> > https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin > >>>>> > <https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin>>>>>>; > SPIDolphinSchedulerPlugin > >>>>> > >>>>> DS plug-in top-level interface, all DS plug-ins must > implement this > >>>>> interface > >>>>> AlertChannelFactory > >>>>> > >>>>> Alert plug-in factory interface. All alert plug-ins > need to implement > >>>>> this > >>>>> interface. This interface is used to define the name > of the alert > >>>>> plug-in > >>>>> and the required parameters. The create method is > used to create a > >>>>> specific > >>>>> alert plug-in instance. > >>>>> AlertChannel > >>>>> > >>>>> The interface of the alert plug-in. The alert plug-in > needs to > >>>>> implement > >>>>> this interface. There is only one method process in > the interface. The > >>>>> upper alert system will use this method and obtain > the return > >>>>> information > >>>>> of the alert through the AlertResult returned by the > method. > >>>>> AlertData > >>>>> > >>>>> alert content information, including id, title, > content, log. > >>>>> AlertInfo > >>>>> > >>>>> For alert-related information, when an upper-layer > system calls an > >>>>> alert > >>>>> plug-in instance, the instance of this type is passed > into the specific > >>>>> alert plug-in through the process method. It contains > the alert content > >>>>> AlertData and the parameter information filled in by > the front end of > >>>>> the > >>>>> alert plug-in instance called. > >>>>> AlertResult > >>>>> > >>>>> The alert plugin sends alert return information. > >>>>> org.apache.dolphinscheduler.spi.params > >>>>> > >>>>> Under this package is the plug-in parameter > definition. Our front-end > >>>>> uses > >>>>> the front-end library alpacajs > >>>>> http://www.alpacajs.org/docs/api/forms.html, > >>>>> which can dynamically generate the front-end based on > the parameter > >>>>> list > >>>>> json returned by the plug-in definition Ui. > >>>>> > >>>>> Under this package, we currently only encapsulate > RadioParam, > >>>>> TextParam, > >>>>> and PasswordParam, which are used to define text type > parameters, radio > >>>>> parameters, and password type parameters, > respectively. > >>>>> > >>>>> AbsPluginParams This class is the base class for all > parameters, and > >>>>> these > >>>>> classes inherit from RadioParam. Each DS alert plugin > will return a > >>>>> list of > >>>>> AbsPluginParams in the implementation of > AlertChannelFactory. > >>>>> org.apache.dolphinscheduler.alert.plugin > >>>>> > >>>>> Under this package is the relevant code for the DS > Alert module to > >>>>> load the > >>>>> Alert plug-in. In the AlertChannelManager class, we > also need to save > >>>>> the > >>>>> loaded plugin to the database's plugin_define table. > >>>>> some code of this issue > >>>>> > >>>>> gaojun2048#1 > >>>>> < > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1&gt; > >>>>> todo list > >>>>> > >>>>> &nbsp;&nbsp; 1. > >>>>> > >>>>> &nbsp;&nbsp; Improvement of the alert group > function, in the alert > >>>>> group, an alert > >>>>> &nbsp;&nbsp; instance can be created based on > the existing alert > >>>>> plug-in. > >>>>> &nbsp;&nbsp; 2. > >>>>> > >>>>> &nbsp;&nbsp; Delete recipients and CCs in the > process definition and > >>>>> select the > >>>>> &nbsp;&nbsp; associated alert group instead. > >>>>> &nbsp;&nbsp; 3. > >>>>> > >>>>> &nbsp;&nbsp; Submit the > dolphinscheduler-maven-plugin to the community. > >>>>> &nbsp;&nbsp; 4. > >>>>> > >>>>> &nbsp;&nbsp; Pull a separate branch from dev > for development. > >>>>> &nbsp;&nbsp; 5. > >>>>> > >>>>> &nbsp;&nbsp; Transformation of WeChat alert > function. > >>>>> &nbsp;&nbsp; 6. > >>>>> > >>>>> &nbsp;&nbsp; Renovation of the nail warning > function. > >>>>> &nbsp;&nbsp; 7. > >>>>> > >>>>> &nbsp;&nbsp; Improvement of mail alert > function. > >>>>> &nbsp;&nbsp; 8. > >>>>> > >>>>> &nbsp;&nbsp; Upgrade related tool development > from the old version. > >>>>> > >>>>> 概要设计插件定义与插件实例 > >>>>> > >>>>> > >>>>> > 插件化设计的核心是上层逻辑和具体实现以SPI的方式解耦，系统在使用插件时，应该基于当前系统支持的插件创建插件实例。比如邮件告警插件，该插件会提供需要的参数列表，在系统中使用时需要基于参数列表来创建该插件的实例。再比如MR任务的Job，该Job插件会定义好需要的参数，然后在DAG中我们创建该Job并生成一个该Job插件的实例。 > >>>>> 配置参数&amp;配置文件 > >>>>> > >>>>> > >>>>> > 并且之前通过alert.properties的方式配置告警会有一些问题。比如在有些企业，多个部门都在DS上工作，不同的部门需要配置不同的微信告警和邮件告警stmp服务，这种场景是alert.properties这种方式无法满足的。所以我们需要支持在一个告警方式插件下创建多个告警方式实例的功能。为了实例这个能力并且能够减少配置文件简化安装部署，我把插件需要的所有参数都提供给前端，做为该插件的参数模板，在前端页面上创建该插件的实例时填写，即可实现创建多个邮件告警的功能，同时不再需要通过后台修改alert.properties文件。 > >>>>> 告警组和DS用户的关系 > >>>>> > >>>>> > >>>>> > 在目前的DS版本中，告警组是用户的集合，告警服务在发送告警中通过告警组找到用户列表，然后获取每个用户的邮箱地址并添加到邮件告警的接收人中。但是，我们发现我们无法将微信告警和钉钉告警通过告警组来管理，微信和钉钉告警也只能通过后端的alert.properties配置文件来配置使用。所有我们打算重新定义告警组，将告警组定义为告警插件实例的集合，而不再只是用户的集合。 > >>>>> 流程定义与告警的关系 > >>>>> > >>>>> > >>>>> > 当前流程定表中有邮件接收人和抄送人这两个字段，这在一定程度上将流程定义与邮件告警绑定了。实际情况是，流程定义时可能不使用告警，也可能使用除了邮件之外的告警方式。所以我们需要重新定义流程定义与告警和告警组三者之间的关系。 > >>>>> 前端UI插件化 > >>>>> > >>>>> http://www.alpacajs.org/docs/api/forms.html > >>>>> > >>>>> > &nbsp;该前端架构支持基于json生成前端ui组件。SPI里我对插件的参数做了封装，所有的插件参数最终会自动转换成alpacajs支持的json。 > >>>>> 用户路径 > >>>>> > >>>>> 基于以上设计考虑，我们重新定义了告警功能的用户路径，使其使用起来有更好的扩展性，更容易被理解和使用。 > >>>>> > >>>>> 修改后的用户路径如下： > >>>>> > >>>>> > 如果用户需要使用告警功能，用户可以在告警组创建和管理页面中选择基于已有的告警插件来创建告警实例。比如用户创建了告警组AlertGroupA，然后在AlertGroupA的管理页面可以通过“添加告警方式”操作来选择创建具体的告警方式，“添加告警方式”时通过下拉菜单展示目前系统中已有的告警插件（插件信息可以放到db里的插件定义表中plugin_define），用户选择对应的插件，会打开创建页面，该页面会展示该告警插件所有需要的参数，用户填写完成后即创建了一个告警方式的实例。 > >>>>> > >>>>> 一个告警组中可以创建多种告警方式的多个实例。 > >>>>> > >>>>> > >>>>> > 流程定义关联告警组ID，流程定义不再关联具体的告警方式，而是改为关联一个告警组。这样即使以后需要修改告警方式，也只需要在告警组管理中修改告警方式实例即可，不再需要修改流程定义。 > >>>>> 详细设计数据库表设计插件定义表 plugin_define > >>>>> > >>>>> id, > >>>>> plugin_name, > >>>>> plugin_params > >>>>> 插件实例表 plugin_instance > >>>>> > >>>>> id > >>>>> plugin_define_id, > >>>>> plugin_instance_params > >>>>> 接口设计incubator-dolphinscheduler-maven-plugin > >>>>> > >>>>> > https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin > >>>>> > <https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin>>>>>> > ; > >>>>> > >>>>> > 一个自定义maven插件，它的作用是可以自动检查spi和插件相关的依赖，并且可以支持在pom文件中添加dolphinscheduler-plugin标识从而让DS在编译时自动生成相关的META-INF/services文件。详细的信息可以参考 > >>>>> ： > >>>>> > https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin > >>>>> > <https://github.com/gaojun2048/incubator-dolphinscheduler-maven-plugin>>>>>>; > SPIDolphinSchedulerPlugin > >>>>> > >>>>> DS插件顶层接口，所有DS的插件都必须实现该接口 > >>>>> AlertChannelFactory > >>>>> > >>>>> > 告警插件工厂接口，所有告警插件需要实现该接口，该接口用来定义告警插件的名称，需要的参数，create方法用来创建具体的告警插件实例. > >>>>> AlertChannel > >>>>> > >>>>> 告警插件的接口，告警插件需要实现该接口，该接口中只有一个方法process > >>>>> ，上层告警系统会该用该方法并通过该方法返回的AlertResult来获取告警的返回信息。 > >>>>> AlertData > >>>>> > >>>>> 告警内容信息，包括id，标题，内容，日志。 > >>>>> AlertInfo > >>>>> > >>>>> > >>>>> > 告警相关信息，上层系统调用告警插件实例时，将该类的实例通过process方法传入具体的告警插件。内部包含告警内容AlertData和调用的告警插件实例的前端填写的参数信息。 > >>>>> AlertResult > >>>>> > >>>>> 告警插件发送告警返回信息。 > >>>>> org.apache.dolphinscheduler.spi.params > >>>>> > >>>>> 该包下是插件化的参数定义，我们前端使用alpacajs这个前端库 > >>>>> http://www.alpacajs.org/docs/api/forms.html > >>>>> ，该库可以基于插件定义返回的参数列表json来动态生成前端的ui。 > >>>>> > >>>>> > >>>>> > 该package下我们目前只封装了RadioParam，TextParam，PasswordParam，分别用来定义text类型的参数，radio参数和 > >>>>> password类型的参数。 > >>>>> > >>>>> AbsPluginParams > >>>>> > >>>>> > 该类是所有参数的基类，RadioParam这些类都继承了该类。每个DS的告警插件都会在AlertChannelFactory的实现中返回一个AbsPluginParams的list。 > >>>>> org.apache.dolphinscheduler.alert.plugin > >>>>> > >>>>> 该包下是DS > >>>>> > >>>>> > Alert模块加载Alert插件的相关代码，已经大部分。AlertChannelManager类中我们还需要将加载到的插件保存到数据库的plugin_define表中。 > >>>>> 关于该Issue的一些代码 > >>>>> > >>>>> gaojun2048#1 > >>>>> < > https://github.com/gaojun2048/incubator-dolphinscheduler/pull/1&gt; > >>>>> todo list > >>>>> > >>>>> &nbsp;&nbsp; 1. > >>>>> > >>>>> &nbsp;&nbsp; 告警组功能改造，在告警组中可以基于已有的告警插件创建告警实例。 > >>>>> &nbsp;&nbsp; 2. > >>>>> > >>>>> &nbsp;&nbsp; 流程定义中删除收件人，抄送人，改为选择关联告警组。 > >>>>> &nbsp;&nbsp; 3. > >>>>> > >>>>> &nbsp;&nbsp; > 将dolphinscheduler-maven-plugin提交到社区。 > >>>>> &nbsp;&nbsp; 4. > >>>>> > >>>>> &nbsp;&nbsp; 从dev拉单独分支，用于开发。 > >>>>> &nbsp;&nbsp; 5. > >>>>> > >>>>> &nbsp;&nbsp; 微信告警功能改造。 > >>>>> &nbsp;&nbsp; 6. > >>>>> > >>>>> &nbsp;&nbsp; 钉钉告警功能的改造。 > >>>>> &nbsp;&nbsp; 7. > >>>>> > >>>>> &nbsp;&nbsp; 邮件告警功能的完善。 > >>>>> &nbsp;&nbsp; 8. > >>>>> > >>>>> &nbsp;&nbsp; 从老版本升级相关的工具开发。 > >>>>> > >>>>> > >>>>> 欢迎对这个issue感兴趣的小伙伴参与讨论和开发贡献。 > >>>>> > >>>>> > >>>>> -- > >>>>> > >>>>> DolphinScheduler(Incubator)&nbsp; PPMC > >>>>> Jun Gao 高俊 > >>>>> [email protected] > >>>> > >>>> > >>>> > >>>> -- > >>>> > >>>> DolphinScheduler(Incubator) PPMC > >>>> Jun Gao 高俊 > >>>> [email protected] > >>>> > >>>> > >>> > >>> -- > >>> > >>> DolphinScheduler(Incubator) PPMC > >>> Jun Gao 高俊 > >>> [email protected] > >>> > >>> > >> > >> -- > >> > >> DolphinScheduler(Incubator) PPMC > >> Jun Gao 高俊 > >> [email protected] > >> > >> > > > > -- > > > > DolphinScheduler(Incubator) PPMC > > Jun Gao 高俊 > > [email protected] > > > > > > -- > > DolphinScheduler(Incubator) PPMC > Jun Gao 高俊 > [email protected] -- DolphinScheduler(Incubator) PPMC Jun Gao 高俊 [email protected]
