I think a common plugin module is necessary, but I don't recommend putting all the plugins in the root directory, which looks complicated. We can have ds-plugin module in root with many sub modules, such as alert-plugin / task-plugin /...
hgaol <dhan...@hotmail.com> 于2020年5月13日周三 下午8:24写道： > Hi @Jun Gao, > > I went through the link you sent, and I agree that if the job was plugged > in will be better. But I’m not quite familiar with DS job currently, so I > may not able to answer your question about how to add a ClickHouse plugin. > > I’ve written interfaces about Alert plugin to support different alert > type. If you want to implement alert plugin, you can refer to the PR > mentioned before. > > For job plugin, I think you can leverage the plugin design and enhance it > to support job plugin. You may need to define a JobProvider, a JobPlugin > interface, option with a AbstractJobPlugin. > > Currently, this PR is about Alert plugin. For job plugin or other plugin > design, I think it may be many topics and details to discuss. > > > Thanks, > Han Gao > > > On May 13, 2020, at 10:19, JUN GAO <gaojun2...@gmail.com> wrote: > > In addition to the job can be plugged in, is there any other content that > can be plugged in? If so, we need to consider the structural design in > advance, because this is a very important design transformation, we can > spend more time to think more > > JUN GAO <gaojun2...@gmail.com> 于2020年5月13日周三 上午10:11写道： > >> @hgaol <dhan...@hotmail.com> There are some discuss about DS Plugin >> . But this is an early design and can only be used as a reference . I hope >> it can help your design. >> https://github.com/apache/incubator-dolphinscheduler/issues/201 >> >> I see the model `dolphinscheduler-plugin-api` . When I want to add a >> ClickHouse Job Plugin, should I create a new module called >> plugin-clickhouse-job directly in the root directory of the project >> (because I don’t know if only the job can do plugin)? Or we create a moudle >> called ds-job-plugin in the root directory, and then create a moudle called >> clickhouse-job under this moudle? Can you make a standardized design in >> this regard? Everyone can follow this design afterwards. >> >> @leon bao @Gang li Do you guys have any ideas ？ >> >> >> >> hgaol <dhan...@hotmail.com> 于2020年5月12日周二 下午9:33写道： >> >>> Hi @Gang Li, I agree with you that the plug-in module should be a common >>> module for all plug-in design. Currently I just create the SPI and plugin >>> interface for alert plug-in. If need more type for future, it’s very easy >>> to define other plugins. >>> >>> Currently the alert service can detect the plug-in jars and load it. But >>> I didn’t implement how to collaborate with front-end about the alert types >>> API and how many properties a plugin should have yet. For example, if we >>> add a webhook plugin, we should have the http method(form select), url(form >>> text), etc… So the plugin interface may change to achieve this goal. >>> >>> Hi @leon bao, as mentioned above, I agree to make the plugin module as a >>> common module for all plugin design, is that what you mean? >>> >>> > why don't we put the alert plug-in as an alert module? >>> I’m not clear about it, do you mean we could just put alert plugin in >>> alert module? I decide to put frequently used plugins in alert module, as >>> for others like SMS(which may has different api), make it as a plugin jar >>> file. >>> >>> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient. >>> Developers just need to implement the interfaces in SPI, which are in >>> `dolphinscheduler-plugin-api` module. >>> >>> Hi all, thanks for your careful review! Here is some link. >>> >>> PR about plugin management. >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2572> >>> PR about how to develop a simple plugin. >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593> >>> README about how to develop a simple plugin. >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07> >>> >>> Thanks in advance! >>> >>> Thanks, >>> hgaol >>> >>> On May 12, 2020, at 18:57, JUN GAO <gaojun2...@gmail.com> wrote: >>> >>> we can refer to presto plug-in, etc. >>> 1. Design the SPI and define the interface that the plugin needs to >>> implement >>> 2. All plugins create new models to implement the interface in SPI. >>> The benefits are as follows: >>> 1. For developers, they only need to care about the interface in SPI, >>> and do not need to care about other people's code, so it is easier to get >>> started developing plug-ins. >>> 2. For code reviewers, to improve the efficiency of review, independent >>> models will be more easily reviewed. If there is a problem with the code, >>> it is easy to roll back. >>> 3. For users, there are more flexible options. For example, some people >>> only want to use the A plug-in but not the B plug-in. >>> >>> Thank you ! >>> >>> lgcareer2...@outlook.com <lgcareer2...@outlook.com> 于2020年5月12日周二 >>> 下午6:31写道： >>> >>>> Hi Gao, >>>> >>>> May be I didn't express clearly,I think the dolphinscheduler-plugin >>>> module is a common module for all plugin design, >>>> It's not neccessarily to put the implement of a plugin in the >>>> dolphinscheduler-plugin module. >>>> >>>> I also agree with the following way your said,I see the pr you submited >>>> didn't match it now,Will you give a deep design about the following way? >>>> >>>> > A better way is that user do not need to change the source code to >>>> support new alert method. Instead, they just need to implement some >>>> interface, and compliance with the convention, then package all(include >>>> dependencies) in a single jar file, and put the jar file in some specific >>>> place, such as $DS_HOME/plugins. After that, when start Alert service, the >>>> plugins can be recognized and loaded. So user can choose the alert method >>>> in the portal. >>>> >>>> >>>> >>>> DolphinScheduler(Incubator) PPMC >>>> Gang Li 李岗 >>>> >>>> lgcareer2...@outlook.com<mailto:lgcareer2...@outlook.com> >>>> >>>> From: hgaol >>>> Date: 2020-05-11 19:00 >>>> To: lgcareer2...@outlook.com >>>> CC: dev >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >>>> Hi Gang, >>>> >>>> I agree with that making examples as a pure folder, thanks for your >>>> suggestion. >>>> >>>> For dolphinscheduler-plugin-api, do you mean it’s better to rename this >>>> module? I’m not clear about ‘all plugin function need to be implemented >>>> here’, do you mean we should put implemented plugin here? >>>> >>>> BTW, I’ve write a simple doc about how to implement an alert plugin. >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 >>>> >>>> Also, I think for some frequently used plugins(email/webhook/...), we >>>> could just implement in alert service. Here is the former PR about it. >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 >>>> >>>> Thanks for your careful review! >>>> >>>> Thanks, >>>> hgaol >>>> >>>> On May 11, 2020, at 18:09, lgcareer2...@outlook.com wrote: >>>> >>>> Hi Gao, >>>> >>>> I agree with the following you said,If this is a plugin module,all the >>>> plugin function need to be implemented here. >>>> Now the name is dolphinscheduler-plugin-api,I think we can change the >>>> module name, Such as `dolphinscheduler-plugin`. >>>> > I think dolphinscheduler-plugin-api could not only support alert >>>> plugin, but also support java job task for future. That’s another topic. >>>> >>>> Yes,I think it just a folder,not a sub-module. >>>> We can also look at other developers' comments on this. >>>> >Do you mean the examples module should not be a sub-module in >>>> dolphinscheduler project pom file, just a pure folder? >>>> >>>> >>>> >>>> DolphinScheduler(Incubator) PPMC >>>> Gang Li 李岗 >>>> >>>> lgcareer2...@outlook.com<mailto:lgcareer2...@outlook.com> >>>> >>>> From: hgaol >>>> Date: 2020-05-11 17:45 >>>> To: dev; lgcareer2019 >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >>>> Hi Gang, >>>> >>>> Thanks for your suggestion. >>>> 1.Whether the function of module dolphinscheduler-plugin-api can be >>>> puted in the alert module. >>>> [hgaol] The customer should depend on this module to implement their >>>> own plugin. If put it into alert service, customer should put >>>> `dolphinscheduler-alert` as dependence, which is too heavy for a plugin. >>>> >>>> Also, I think dolphinscheduler-plugin-api could not only support alert >>>> plugin, but also support java job task for future. That’s another topic. >>>> >>>> 2. I think the examples module can be a folder. >>>> [hgaol] Do you mean the examples module should not be a sub-module in >>>> dolphinscheduler project pom file, just a pure folder? >>>> >>>> Thanks, >>>> hgaol >>>> >>>> On May 11, 2020, at 11:51, lgcareer2...@outlook.com wrote: >>>> >>>> Hi hgaol, >>>> Sorry for not replying in time. >>>> I saw you added the module dolphinscheduler-plugin-api and examples >>>> module,here are my two points want to discuss. >>>> >>>> 1、Whether the function of module dolphinscheduler-plugin-api can be >>>> puted in the alert module. >>>> 2、I think the examples module can be a folder. >>>> >>>> >>>> >>>> >>>> >>>> >>>> DolphinScheduler(Incubator) PPMC >>>> Gang Li 李岗 >>>> >>>> lgcareer2...@outlook.com<mailto:lgcareer2...@outlook.com> >>>> >>>> From: hgaol >>>> Date: 2020-04-29 23:37 >>>> To: dev >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service >>>> Hi folks, >>>> >>>> I’ve created a PR for this feature, here is the link >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 < >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572> >>>> >>>> Feel free to let me know if there is any suggestion. >>>> >>>> Thanks, >>>> Han Gao >>>> >>>> On Apr 28, 2020, at 15:25, xingchun.chen <xingchun.c...@qq.com> wrote: >>>> >>>> It sounds very good, looking forward to your contribution >>>> >>>> >>>> best wish! >>>> >>>> >>>> ------------------ 原始邮件 ------------------ >>>> 发件人: "hgaol"<dhan...@hotmail.com>; >>>> 发送时间: 2020年4月28日(星期二) 下午3:11 >>>> 收件人: "dev"<dev@dolphinscheduler.apache.org>; >>>> >>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service >>>> >>>> >>>> >>>> Hi folks, >>>> >>>> I think it will be useful for Alert service supporting the plugin >>>> management. I’ve create an issue in Github and described about it, here < >>>> https://github.com/apache/incubator-dolphinscheduler/issues/2553> >>>> is the link. Copy here for convenience. >>>> >>>> > I think the plugin feature is needed by the customers, because >>>> different user may have different environment. For example, I want to use >>>> SMS to send alert information, but the SMS API is provided by some system >>>> we don't know before, and our DS has already deployed. In this case, I just >>>> want to implement the plugin code, and don't change the source code. >>>> > >>>> > Another benefits are developers can implement different plugin, >>>> like slack, wechat, telegram, github issue, and etc... >>>> > >>>> > Describe the solution you'd like >>>> > A clear and concise description of what you want to happen. >>>> > >>>> > A better way is that user do not need to change the source code to >>>> support new alert method. Instead, they just need to implement some >>>> interface, and compliance with the convention, then package all(include >>>> dependencies) in a single jar file, and put the jar file in some specific >>>> place, such as $DS_HOME/plugins. After that, when start Alert service, the >>>> plugins can be recognized and loaded. So user can choose the alert method >>>> in the portal. >>>> > >>>> > Currently, I've done some investigations, and if this feature is >>>> accepted, I think maybe I can implement it. >>>> > >>>> > Firstly, I'd better to adapt the current function, which is send >>>> mail and something like that. >>>> > >>>> > Then, we can add some more features, such as Alert handler >>>> instance, which is created to describe the alert >>>> information(email/slack/wechat/..., to whom, the title, content, and etc…). >>>> > >>>> > BTW, email and webhook are used frequently. I think they can be >>>> the default plugin, and written in the source code, instead of a jar file. >>>> >>>> Thanks, >>>> hgaol >>>> >>>> >>> >>> -- >>> >>> DolphinScheduler(Incubator) PPMC >>> Jun Gao 高俊 >>> gaojun2...@gmail.com >>> >>> >>> >> >> -- >> >> DolphinScheduler(Incubator) PPMC >> Jun Gao 高俊 >> gaojun2...@gmail.com >> >> > > -- > > DolphinScheduler(Incubator) PPMC > Jun Gao 高俊 > gaojun2...@gmail.com > > > -- DolphinScheduler(Incubator) PPMC BaoLiang 鲍亮 leon...@apache.org
