Agree with xiaoxiang’s solution. I'm looking forward to this feature.
> 在 2020年2月6日，18:49，weibin zhu <[email protected]> 写道：
>
> Thanks for the reply, if everyone agrees with this solution, I can create a
> jira and implement the code. I suggest that can define the interface and
> framework code first. If you are interested, you can review it. If there is
> no problem, then implement the complete code.
> @ShaoFeng Shi, @zhoukang, @Xiaoxiang Yu
>
> ----------------------------------------
> weibin0516
>
> [email protected] <mailto:[email protected]>
> http://rrd.me/fT6RW
> ----------------------------------------
>
>> 在 2020年2月5日，下午11:19，Xiaoxiang Yu <[email protected] <mailto:[email protected]>>
>> 写道：
>>
>> Hi codingforfun and zhoukang,
>> I am agreed with codingforfun. Event, Listener, and ListenerBus, these
>> components should be good abstraction/design for a monitor framework.
>> Recently I am checking source code of System Cube and Dashboard Service
>> in apache kylin, I have find some exist abstraction of these
>> components(Event, Listener, and ListenerBus).
>> We can find them under core-mertics module
>> (https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib).
>>
>> <https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib).>
>> Maybe we can implement the webhook feature by extending this metrics
>> framework other than reinventing the wheel. I am not very sure am I right
>> but I guess it should works.
>> It is a draft version of and I extend some basic interface under
>> core-mertics module, I think it not perfect at the moment, this is my patch
>> https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea
>>
>> <https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea>
>> . I supposed my draft is intended to the beginning of the conversation, not
>> the final implementation. If you have any comments, please let me know,
>> thank you in advance.
>>
>> To test it, use the following config in kylin.properties and
>> kylinMetrics.xml.
>>
>> 1. kylin.properties
>> kylin.metrics.reporter-job-enabled=true
>> kylin.metrics.monitor-enabled=true
>>
>> 2. kylinMetrics.xml
>> <beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance
>> <http://www.w3.org/2001/XMLSchema-instance>"
>> xmlns="http://www.springframework.org/schema/beans
>> <http://www.springframework.org/schema/beans>"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> <http://www.springframework.org/schema/beans>
>> http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
>> <http://www.springframework.org/schema/beans/spring-beans-3.1.xsd>">
>>
>> <description>Kylin Metrics Related Configuration
>> (SystemCube)</description>
>>
>> <!-- A Reservoir which don't staged metrics message at all, emit it in
>> no time. Maybe good for debug purpose.-->
>> <bean id="instantReservoir"
>> class="org.apache.kylin.metrics.lib.impl.InstantReservoir"/>
>>
>> <!-- A Reservoir which staged metrics message in memory, and emit them
>> in fixed rate. -->
>> <bean id="blockingReservoir"
>> class="org.apache.kylin.metrics.lib.impl.BlockingReservoir">
>> <!-- minReportSize, only if currently count of staged message exceed
>> minReportSize, will Reservoir try to write message-->
>> <constructor-arg index="0">
>> <value>100</value>
>> </constructor-arg>
>>
>> <!-- maxReportSize, max size of report in one time -->
>> <constructor-arg index="1">
>> <value>500</value>
>> </constructor-arg>
>>
>> <!-- minReportTime, min duration(in minute) between two report
>> action-->
>> <constructor-arg index="2">
>> <value>10</value>
>> </constructor-arg>
>> </bean>
>>
>> <bean id="hiveSink"
>> class="org.apache.kylin.metrics.lib.impl.hive.HiveSink"/>
>> <bean id="kafkaSink"
>> class="org.apache.kylin.metrics.lib.impl.kafka.KafkaSink"/>
>>
>> <bean id="initMetricsManager"
>> class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
>> <property name="targetClass"
>> value="org.apache.kylin.metrics.MetricsManager"/>
>> <property name="targetMethod" value="initMetricsManager"/>
>> <property name="arguments">
>> <list>
>> <!--
>> Sink of System Cube.
>> -->
>> <ref bean="hiveSink"/>
>>
>> <!--
>> Bind properties for each ActiveReservoirReporter.
>> -->
>> <map key-type="org.apache.kylin.metrics.lib.ActiveReservoir"
>> value-type="java.util.List">
>>
>> <!-- Each ActiveReservoir can have multi
>> ReservoirReporter -->
>> <entry key-ref="instantReservoir">
>> <list>
>> <bean class="org.apache.kylin.common.util.Pair">
>> <!-- Implementation of ReservoirReporter-->
>> <property name="first"
>>
>> value="org.apache.kylin.metrics.lib.impl.kafka.KafkaReservoirReporter"/>
>> <!-- Properties for specific
>> ReservoirReporter-->
>> <property name="second">
>> <props>
>> <prop
>> key="bootstrap.servers">cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092</prop>
>> </props>
>> </property>
>> </bean>
>>
>> <bean class="org.apache.kylin.common.util.Pair">
>> <!-- Implementation of ReservoirReporter-->
>> <property name="first"
>>
>> value="org.apache.kylin.metrics.lib.impl.callback.CallbackActiveReservoirReporter"/>
>> <!-- Properties for specific
>> ReservoirReporter-->
>> <property name="second">
>> <props>
>> <prop
>> key="callback.url">localhost:8888</prop>
>> </props>
>> </property>
>> </bean>
>> </list>
>> </entry>
>> </map>
>> </list>
>> </property>
>> </bean>
>> </beans>
>>
>>
>>
>>
>> --
>> Best wishes to you !
>> From ：Xiaoxiang Yu
>>
>> At 2020-02-05 10:03:13, "codingforfun" <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> I think we can refer to the implementation of spark listener, a few key
>> points are as follows:
>> 1. Define interface ListenerEvent, which can implement specific events.
>> Examples of which build cubes are CubeBuildStart, CubeBuildStepXStart,
>> CubeBuildStepXEnd, CubeBuildEnd. These specific classes can contain a lot of
>> information, such as owner, success/failed, error msg, resource occupation,
>> etc
>> 2. Define interface Listener, which contains the monitoring of all events.
>> When the user defines the Listener, some events can be selectively
>> implemented as needed, for example, onCubeBuildStart and CubeBuildEnd can be
>> implemented to obtain the time cost.
>> 3. Define a ListenerBus that provides the ability to register listeners(by
>> configured custom listener class names), receive events, and dispatch events
>> to the appropriate Listener in the background
>>
>>
>>
>> 在 2020年2月4日 11:17，ShaoFeng Shi <[email protected]
>> <mailto:[email protected]>>写道：
>> Good proposal. Is there some standard and popular framework for this? We can
>> integrate with the best solution there.
>>
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC
>> Email: [email protected] <mailto:[email protected]>
>>
>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> <https://kylin.apache.org/docs/gettingstarted/faq.html>
>> Join Kylin user mail group: [email protected]
>> <mailto:[email protected]>
>> Join Kylin dev mail group: [email protected]
>> <mailto:[email protected]>
>>
>>
>>
>>
>> Liukaige <[email protected] <mailto:[email protected]>> 于2020年1月25日周六
>> 上午1:01写道：
>> Totally agreed. And this feature can be integrated with ETL scheduling, data
>> governance, approval flow etc. Brilliant idea. If you need any help, count
>> me in.
>>
>> 朱卫斌 <[email protected] <mailto:[email protected]>> 于2020年1月20日周一
>> 下午10:10写道：
>> I think there should be a set of kylin's event, metrics mechanism,
>> event-driven, provide event, metrics interfaces, we can implement specific
>> listeners in the form of plugins. For example, we can implement dingtalk
>> plugin, SMS plugin or any other plugin, which has a high Flexibility (Not
>> only do notifications, but also do many things, such as unifying and even
>> interfering with the task.). We can refer to the design and implementation
>> of spark event and metrics.
>> I think this is very valuable, we can do it together.
>>
>>
>>
>> weibin0516
>> [email protected]
>> Best wishes !
>>
>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=weibin0516&uid=codingforfun%40126.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fwzpmmc%2F54c20faa3a1910ad49f4a5f1965fba47.jpg&items=%5B%22codingforfun%40126.com%22%2C%22Best+wishes+%21%22%5D>
>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
>>
>> On 01/21/2020 10:57，Xiaoxiang Yu<[email protected]> <mailto:[email protected]>
>> wrote：
>> It looks good, and it should be useful for IT team, please go ahead!
>>
>>
>> --
>> Best wishes to you !
>> From ：Xiaoxiang Yu
>>
>> At 2020-01-20 19:58:40, "Zhou Kang" <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hi Kylin users & developers:
>>
>>
>>
>> Many apps support webhook, it is one way that apps can send automated
>> messages or information to other apps.
>>
>> Use webhook, I think we can send messages to chat tools（slack,
>> dingding）, send sms , trigger another workflow.
>>
>> Do we need to add webhook to Kylin? Such as when cubing job finished.
>>
>> What do you think about this, and which is the better way in your
>> environment ?
>>
>>
>>
>> --
>> Best regards,
>>
>> Kaige Liu(刘凯歌)
>>
>> "Do small things with great love."
>
