Thanks for the reply, if everyone agrees with this solution, I can create a jira and implement the code. I suggest that can define the interface and framework code first. If you are interested, you can review it. If there is no problem, then implement the complete code. @ShaoFeng Shi, @zhoukang, @Xiaoxiang Yu
---------------------------------------- weibin0516 [email protected] http://rrd.me/fT6RW ---------------------------------------- > 在 2020年2月5日，下午11:19，Xiaoxiang Yu <[email protected]> 写道： > > Hi codingforfun and zhoukang, > I am agreed with codingforfun. Event, Listener, and ListenerBus, these > components should be good abstraction/design for a monitor framework. > Recently I am checking source code of System Cube and Dashboard Service > in apache kylin, I have find some exist abstraction of these > components(Event, Listener, and ListenerBus). > We can find them under core-mertics module > (https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib). > > <https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib).> > Maybe we can implement the webhook feature by extending this metrics > framework other than reinventing the wheel. I am not very sure am I right but > I guess it should works. > It is a draft version of and I extend some basic interface under > core-mertics module, I think it not perfect at the moment, this is my patch > https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea > > <https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea> > . I supposed my draft is intended to the beginning of the conversation, not > the final implementation. If you have any comments, please let me know, thank > you in advance. > > To test it, use the following config in kylin.properties and > kylinMetrics.xml. > > 1. kylin.properties > kylin.metrics.reporter-job-enabled=true > kylin.metrics.monitor-enabled=true > > 2. kylinMetrics.xml > <beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xmlns="http://www.springframework.org/schema/beans"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-3.1.xsd";> > > <description>Kylin Metrics Related Configuration > (SystemCube)</description> > > <!-- A Reservoir which don't staged metrics message at all, emit it in no > time. Maybe good for debug purpose.--> > <bean id="instantReservoir" > class="org.apache.kylin.metrics.lib.impl.InstantReservoir"/> > > <!-- A Reservoir which staged metrics message in memory, and emit them in > fixed rate. --> > <bean id="blockingReservoir" > class="org.apache.kylin.metrics.lib.impl.BlockingReservoir"> > <!-- minReportSize, only if currently count of staged message exceed > minReportSize, will Reservoir try to write message--> > <constructor-arg index="0"> > <value>100</value> > </constructor-arg> > > <!-- maxReportSize, max size of report in one time --> > <constructor-arg index="1"> > <value>500</value> > </constructor-arg> > > <!-- minReportTime, min duration(in minute) between two report > action--> > <constructor-arg index="2"> > <value>10</value> > </constructor-arg> > </bean> > > <bean id="hiveSink" > class="org.apache.kylin.metrics.lib.impl.hive.HiveSink"/> > <bean id="kafkaSink" > class="org.apache.kylin.metrics.lib.impl.kafka.KafkaSink"/> > > <bean id="initMetricsManager" > class="org.springframework.beans.factory.config.MethodInvokingFactoryBean"> > <property name="targetClass" > value="org.apache.kylin.metrics.MetricsManager"/> > <property name="targetMethod" value="initMetricsManager"/> > <property name="arguments"> > <list> > <!-- > Sink of System Cube. > --> > <ref bean="hiveSink"/> > > <!-- > Bind properties for each ActiveReservoirReporter. > --> > <map key-type="org.apache.kylin.metrics.lib.ActiveReservoir" > value-type="java.util.List"> > > <!-- Each ActiveReservoir can have multi > ReservoirReporter --> > <entry key-ref="instantReservoir"> > <list> > <bean class="org.apache.kylin.common.util.Pair"> > <!-- Implementation of ReservoirReporter--> > <property name="first" > > value="org.apache.kylin.metrics.lib.impl.kafka.KafkaReservoirReporter"/> > <!-- Properties for specific > ReservoirReporter--> > <property name="second"> > <props> > <prop > key="bootstrap.servers">cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092</prop> > </props> > </property> > </bean> > > <bean class="org.apache.kylin.common.util.Pair"> > <!-- Implementation of ReservoirReporter--> > <property name="first" > > value="org.apache.kylin.metrics.lib.impl.callback.CallbackActiveReservoirReporter"/> > <!-- Properties for specific > ReservoirReporter--> > <property name="second"> > <props> > <prop > key="callback.url">localhost:8888</prop> > </props> > </property> > </bean> > </list> > </entry> > </map> > </list> > </property> > </bean> > </beans> > > > > > -- > Best wishes to you ! > From ：Xiaoxiang Yu > > At 2020-02-05 10:03:13, "codingforfun" <[email protected]> wrote: > > I think we can refer to the implementation of spark listener, a few key > points are as follows: > 1. Define interface ListenerEvent, which can implement specific events. > Examples of which build cubes are CubeBuildStart, CubeBuildStepXStart, > CubeBuildStepXEnd, CubeBuildEnd. These specific classes can contain a lot of > information, such as owner, success/failed, error msg, resource occupation, > etc > 2. Define interface Listener, which contains the monitoring of all events. > When the user defines the Listener, some events can be selectively > implemented as needed, for example, onCubeBuildStart and CubeBuildEnd can be > implemented to obtain the time cost. > 3. Define a ListenerBus that provides the ability to register listeners(by > configured custom listener class names), receive events, and dispatch events > to the appropriate Listener in the background > > > > 在 2020年2月4日 11:17，ShaoFeng Shi <[email protected]>写道： > Good proposal. Is there some standard and popular framework for this? We can > integrate with the best solution there. > > Best regards, > > Shaofeng Shi 史少锋 > Apache Kylin PMC > Email: [email protected] <mailto:[email protected]> > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html > <https://kylin.apache.org/docs/gettingstarted/faq.html> > Join Kylin user mail group: [email protected] > <mailto:[email protected]> > Join Kylin dev mail group: [email protected] > <mailto:[email protected]> > > > > > Liukaige <[email protected] <mailto:[email protected]>> 于2020年1月25日周六 > 上午1:01写道： > Totally agreed. And this feature can be integrated with ETL scheduling, data > governance, approval flow etc. Brilliant idea. If you need any help, count me > in. > > 朱卫斌 <[email protected] <mailto:[email protected]>> 于2020年1月20日周一 > 下午10:10写道： > I think there should be a set of kylin's event, metrics mechanism, > event-driven, provide event, metrics interfaces, we can implement specific > listeners in the form of plugins. For example, we can implement dingtalk > plugin, SMS plugin or any other plugin, which has a high Flexibility (Not > only do notifications, but also do many things, such as unifying and even > interfering with the task.). We can refer to the design and implementation of > spark event and metrics. > I think this is very valuable, we can do it together. > > > > weibin0516 > [email protected] > Best wishes ! > > <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=weibin0516&uid=codingforfun%40126.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fwzpmmc%2F54c20faa3a1910ad49f4a5f1965fba47.jpg&items=%5B%22codingforfun%40126.com%22%2C%22Best+wishes+%21%22%5D> > 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 > > On 01/21/2020 10:57，Xiaoxiang Yu<[email protected]> <mailto:[email protected]> > wrote： > It looks good, and it should be useful for IT team, please go ahead! > > > -- > Best wishes to you ! > From ：Xiaoxiang Yu > > At 2020-01-20 19:58:40, "Zhou Kang" <[email protected] > <mailto:[email protected]>> wrote: > > Hi Kylin users & developers: > > > > Many apps support webhook, it is one way that apps can send automated > messages or information to other apps. > > Use webhook, I think we can send messages to chat tools（slack, > dingding）, send sms , trigger another workflow. > > Do we need to add webhook to Kylin? Such as when cubing job finished. > > What do you think about this, and which is the better way in your > environment ? > > > > -- > Best regards, > > Kaige Liu(刘凯歌) > > "Do small things with great love."
