I agree with you, I think a dolphinscheduler-plugin module and sub module under dolphinscheduler-plugin will be better
Best Regards --------------- DolphinScheduler(Incubator) PPMC Lidong Dai 代立冬 dailidon...@gmail.com --------------- leon bao <leon...@apache.org> 于2020年5月14日周四 下午7:18写道： > I think a common plugin module is necessary, but I don't recommend putting > all the plugins in the root directory, which looks complicated. > We can have ds-plugin module in root with many sub modules, such as > alert-plugin / task-plugin /... > > hgaol <dhan...@hotmail.com> 于2020年5月13日周三 下午8:24写道： > > > Hi @Jun Gao, > > > > I went through the link you sent, and I agree that if the job was plugged > > in will be better. But I’m not quite familiar with DS job currently, so I > > may not able to answer your question about how to add a ClickHouse > plugin. > > > > I’ve written interfaces about Alert plugin to support different alert > > type. If you want to implement alert plugin, you can refer to the PR > > mentioned before. > > > > For job plugin, I think you can leverage the plugin design and enhance it > > to support job plugin. You may need to define a JobProvider, a JobPlugin > > interface, option with a AbstractJobPlugin. > > > > Currently, this PR is about Alert plugin. For job plugin or other plugin > > design, I think it may be many topics and details to discuss. > > > > > > Thanks, > > Han Gao > > > > > > On May 13, 2020, at 10:19, JUN GAO <gaojun2...@gmail.com> wrote: > > > > In addition to the job can be plugged in, is there any other content that > > can be plugged in? If so, we need to consider the structural design in > > advance, because this is a very important design transformation, we can > > spend more time to think more > > > > JUN GAO <gaojun2...@gmail.com> 于2020年5月13日周三 上午10:11写道： > > > >> @hgaol <dhan...@hotmail.com> There are some discuss about DS Plugin > >> . But this is an early design and can only be used as a reference . I > hope > >> it can help your design. > >> https://github.com/apache/incubator-dolphinscheduler/issues/201 > >> > >> I see the model `dolphinscheduler-plugin-api` . When I want to add a > >> ClickHouse Job Plugin, should I create a new module called > >> plugin-clickhouse-job directly in the root directory of the project > >> (because I don’t know if only the job can do plugin)? Or we create a > moudle > >> called ds-job-plugin in the root directory, and then create a moudle > called > >> clickhouse-job under this moudle? Can you make a standardized design in > >> this regard? Everyone can follow this design afterwards. > >> > >> @leon bao @Gang li Do you guys have any ideas ？ > >> > >> > >> > >> hgaol <dhan...@hotmail.com> 于2020年5月12日周二 下午9:33写道： > >> > >>> Hi @Gang Li, I agree with you that the plug-in module should be a > common > >>> module for all plug-in design. Currently I just create the SPI and > plugin > >>> interface for alert plug-in. If need more type for future, it’s very > easy > >>> to define other plugins. > >>> > >>> Currently the alert service can detect the plug-in jars and load it. > But > >>> I didn’t implement how to collaborate with front-end about the alert > types > >>> API and how many properties a plugin should have yet. For example, if > we > >>> add a webhook plugin, we should have the http method(form select), > url(form > >>> text), etc… So the plugin interface may change to achieve this goal. > >>> > >>> Hi @leon bao, as mentioned above, I agree to make the plugin module as > a > >>> common module for all plugin design, is that what you mean? > >>> > >>> > why don't we put the alert plug-in as an alert module? > >>> I’m not clear about it, do you mean we could just put alert plugin in > >>> alert module? I decide to put frequently used plugins in alert module, > as > >>> for others like SMS(which may has different api), make it as a plugin > jar > >>> file. > >>> > >>> Hi @Jun Gao, yes! I also use the SPI feature, it is quite convenient. > >>> Developers just need to implement the interfaces in SPI, which are in > >>> `dolphinscheduler-plugin-api` module. > >>> > >>> Hi all, thanks for your careful review! Here is some link. > >>> > >>> PR about plugin management. > >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2572> > >>> PR about how to develop a simple plugin. > >>> <https://github.com/apache/incubator-dolphinscheduler/pull/2593> > >>> README about how to develop a simple plugin. > >>> < > https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 > > > >>> > >>> Thanks in advance! > >>> > >>> Thanks, > >>> hgaol > >>> > >>> On May 12, 2020, at 18:57, JUN GAO <gaojun2...@gmail.com> wrote: > >>> > >>> we can refer to presto plug-in, etc. > >>> 1. Design the SPI and define the interface that the plugin needs to > >>> implement > >>> 2. All plugins create new models to implement the interface in SPI. > >>> The benefits are as follows: > >>> 1. For developers, they only need to care about the interface in SPI, > >>> and do not need to care about other people's code, so it is easier to > get > >>> started developing plug-ins. > >>> 2. For code reviewers, to improve the efficiency of review, independent > >>> models will be more easily reviewed. If there is a problem with the > code, > >>> it is easy to roll back. > >>> 3. For users, there are more flexible options. For example, some people > >>> only want to use the A plug-in but not the B plug-in. > >>> > >>> Thank you ! > >>> > >>> lgcareer2...@outlook.com <lgcareer2...@outlook.com> 于2020年5月12日周二 > >>> 下午6:31写道： > >>> > >>>> Hi Gao, > >>>> > >>>> May be I didn't express clearly,I think the dolphinscheduler-plugin > >>>> module is a common module for all plugin design, > >>>> It's not neccessarily to put the implement of a plugin in the > >>>> dolphinscheduler-plugin module. > >>>> > >>>> I also agree with the following way your said,I see the pr you > submited > >>>> didn't match it now,Will you give a deep design about the following > way? > >>>> > >>>> > A better way is that user do not need to change the source code to > >>>> support new alert method. Instead, they just need to implement some > >>>> interface, and compliance with the convention, then package > all(include > >>>> dependencies) in a single jar file, and put the jar file in some > specific > >>>> place, such as $DS_HOME/plugins. After that, when start Alert > service, the > >>>> plugins can be recognized and loaded. So user can choose the alert > method > >>>> in the portal. > >>>> > >>>> > >>>> > >>>> DolphinScheduler(Incubator) PPMC > >>>> Gang Li 李岗 > >>>> > >>>> lgcareer2...@outlook.com<mailto:lgcareer2...@outlook.com> > >>>> > >>>> From: hgaol > >>>> Date: 2020-05-11 19:00 > >>>> To: lgcareer2...@outlook.com > >>>> CC: dev > >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service > >>>> Hi Gang, > >>>> > >>>> I agree with that making examples as a pure folder, thanks for your > >>>> suggestion. > >>>> > >>>> For dolphinscheduler-plugin-api, do you mean it’s better to rename > this > >>>> module? I’m not clear about ‘all plugin function need to be > implemented > >>>> here’, do you mean we should put implemented plugin here? > >>>> > >>>> BTW, I’ve write a simple doc about how to implement an alert plugin. > >>>> > https://github.com/apache/incubator-dolphinscheduler/pull/2593/files#diff-0bdec77746c7a907bd4fca67469e0a07 > >>>> > >>>> Also, I think for some frequently used plugins(email/webhook/...), we > >>>> could just implement in alert service. Here is the former PR about it. > >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 > >>>> > >>>> Thanks for your careful review! > >>>> > >>>> Thanks, > >>>> hgaol > >>>> > >>>> On May 11, 2020, at 18:09, lgcareer2...@outlook.com wrote: > >>>> > >>>> Hi Gao, > >>>> > >>>> I agree with the following you said,If this is a plugin module,all the > >>>> plugin function need to be implemented here. > >>>> Now the name is dolphinscheduler-plugin-api,I think we can change the > >>>> module name, Such as `dolphinscheduler-plugin`. > >>>> > I think dolphinscheduler-plugin-api could not only support alert > >>>> plugin, but also support java job task for future. That’s another > topic. > >>>> > >>>> Yes,I think it just a folder,not a sub-module. > >>>> We can also look at other developers' comments on this. > >>>> >Do you mean the examples module should not be a sub-module in > >>>> dolphinscheduler project pom file, just a pure folder? > >>>> > >>>> > >>>> > >>>> DolphinScheduler(Incubator) PPMC > >>>> Gang Li 李岗 > >>>> > >>>> lgcareer2...@outlook.com<mailto:lgcareer2...@outlook.com> > >>>> > >>>> From: hgaol > >>>> Date: 2020-05-11 17:45 > >>>> To: dev; lgcareer2019 > >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service > >>>> Hi Gang, > >>>> > >>>> Thanks for your suggestion. > >>>> 1.Whether the function of module dolphinscheduler-plugin-api can be > >>>> puted in the alert module. > >>>> [hgaol] The customer should depend on this module to implement their > >>>> own plugin. If put it into alert service, customer should put > >>>> `dolphinscheduler-alert` as dependence, which is too heavy for a > plugin. > >>>> > >>>> Also, I think dolphinscheduler-plugin-api could not only support alert > >>>> plugin, but also support java job task for future. That’s another > topic. > >>>> > >>>> 2. I think the examples module can be a folder. > >>>> [hgaol] Do you mean the examples module should not be a sub-module in > >>>> dolphinscheduler project pom file, just a pure folder? > >>>> > >>>> Thanks, > >>>> hgaol > >>>> > >>>> On May 11, 2020, at 11:51, lgcareer2...@outlook.com wrote: > >>>> > >>>> Hi hgaol, > >>>> Sorry for not replying in time. > >>>> I saw you added the module dolphinscheduler-plugin-api and examples > >>>> module,here are my two points want to discuss. > >>>> > >>>> 1、Whether the function of module dolphinscheduler-plugin-api can be > >>>> puted in the alert module. > >>>> 2、I think the examples module can be a folder. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> DolphinScheduler(Incubator) PPMC > >>>> Gang Li 李岗 > >>>> > >>>> lgcareer2...@outlook.com<mailto:lgcareer2...@outlook.com> > >>>> > >>>> From: hgaol > >>>> Date: 2020-04-29 23:37 > >>>> To: dev > >>>> Subject: Re: [PROPOSAL][Feature] Support plugin in Alert service > >>>> Hi folks, > >>>> > >>>> I’ve created a PR for this feature, here is the link > >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572 < > >>>> https://github.com/apache/incubator-dolphinscheduler/pull/2572> > >>>> > >>>> Feel free to let me know if there is any suggestion. > >>>> > >>>> Thanks, > >>>> Han Gao > >>>> > >>>> On Apr 28, 2020, at 15:25, xingchun.chen <xingchun.c...@qq.com> > wrote: > >>>> > >>>> It sounds very good, looking forward to your contribution > >>>> > >>>> > >>>> best wish! > >>>> > >>>> > >>>> ------------------ 原始邮件 ------------------ > >>>> 发件人: "hgaol"<dhan...@hotmail.com>; > >>>> 发送时间: 2020年4月28日(星期二) 下午3:11 > >>>> 收件人: "dev"<dev@dolphinscheduler.apache.org>; > >>>> > >>>> 主题: [PROPOSAL][Feature] Support plugin in Alert service > >>>> > >>>> > >>>> > >>>> Hi folks, > >>>> > >>>> I think it will be useful for Alert service supporting the plugin > >>>> management. I’ve create an issue in Github and described about it, > here < > >>>> https://github.com/apache/incubator-dolphinscheduler/issues/2553> > >>>> is the link. Copy here for convenience. > >>>> > >>>> > I think the plugin feature is needed by the customers, because > >>>> different user may have different environment. For example, I want to > use > >>>> SMS to send alert information, but the SMS API is provided by some > system > >>>> we don't know before, and our DS has already deployed. In this case, > I just > >>>> want to implement the plugin code, and don't change the source code. > >>>> > > >>>> > Another benefits are developers can implement different plugin, > >>>> like slack, wechat, telegram, github issue, and etc... > >>>> > > >>>> > Describe the solution you'd like > >>>> > A clear and concise description of what you want to happen. > >>>> > > >>>> > A better way is that user do not need to change the source code > to > >>>> support new alert method. Instead, they just need to implement some > >>>> interface, and compliance with the convention, then package > all(include > >>>> dependencies) in a single jar file, and put the jar file in some > specific > >>>> place, such as $DS_HOME/plugins. After that, when start Alert > service, the > >>>> plugins can be recognized and loaded. So user can choose the alert > method > >>>> in the portal. > >>>> > > >>>> > Currently, I've done some investigations, and if this feature is > >>>> accepted, I think maybe I can implement it. > >>>> > > >>>> > Firstly, I'd better to adapt the current function, which is send > >>>> mail and something like that. > >>>> > > >>>> > Then, we can add some more features, such as Alert handler > >>>> instance, which is created to describe the alert > >>>> information(email/slack/wechat/..., to whom, the title, content, and > etc…). > >>>> > > >>>> > BTW, email and webhook are used frequently. I think they can be > >>>> the default plugin, and written in the source code, instead of a jar > file. > >>>> > >>>> Thanks, > >>>> hgaol > >>>> > >>>> > >>> > >>> -- > >>> > >>> DolphinScheduler(Incubator) PPMC > >>> Jun Gao 高俊 > >>> gaojun2...@gmail.com > >>> > >>> > >>> > >> > >> -- > >> > >> DolphinScheduler(Incubator) PPMC > >> Jun Gao 高俊 > >> gaojun2...@gmail.com > >> > >> > > > > -- > > > > DolphinScheduler(Incubator) PPMC > > Jun Gao 高俊 > > gaojun2...@gmail.com > > > > > > > > -- > DolphinScheduler(Incubator) PPMC > BaoLiang 鲍亮 > leon...@apache.org >
