Hi codingforfun and zhoukang,
I am agreed with codingforfun. Event, Listener, and ListenerBus, these
components should be good abstraction/design for a monitor framework.
Recently I am checking source code of System Cube and Dashboard Service in
apache kylin, I have find some exist abstraction of these components(Event,
Listener, and ListenerBus).
We can find them under core-mertics module
(https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib).
Maybe we can implement the webhook feature by extending this metrics framework
other than reinventing the wheel. I am not very sure am I right but I guess it
should works.
It is a draft version of and I extend some basic interface under
core-mertics module, I think it not perfect at the moment, this is my patch
https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea
. I supposed my draft is intended to the beginning of the conversation, not
the final implementation. If you have any comments, please let me know, thank
you in advance.
To test it, use the following config in kylin.properties and
kylinMetrics.xml.
1. kylin.properties
kylin.metrics.reporter-job-enabled=true
kylin.metrics.monitor-enabled=true
2. kylinMetrics.xml
<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns="http://www.springframework.org/schema/beans";
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.1.xsd";>
<description>Kylin Metrics Related Configuration (SystemCube)</description>
<!-- A Reservoir which don't staged metrics message at all, emit it in no
time. Maybe good for debug purpose.-->
<bean id="instantReservoir"
class="org.apache.kylin.metrics.lib.impl.InstantReservoir"/>
<!-- A Reservoir which staged metrics message in memory, and emit them in
fixed rate. -->
<bean id="blockingReservoir"
class="org.apache.kylin.metrics.lib.impl.BlockingReservoir">
<!-- minReportSize, only if currently count of staged message exceed
minReportSize, will Reservoir try to write message-->
<constructor-arg index="0">
<value>100</value>
</constructor-arg>
<!-- maxReportSize, max size of report in one time -->
<constructor-arg index="1">
<value>500</value>
</constructor-arg>
<!-- minReportTime, min duration(in minute) between two report action-->
<constructor-arg index="2">
<value>10</value>
</constructor-arg>
</bean>
<bean id="hiveSink"
class="org.apache.kylin.metrics.lib.impl.hive.HiveSink"/>
<bean id="kafkaSink"
class="org.apache.kylin.metrics.lib.impl.kafka.KafkaSink"/>
<bean id="initMetricsManager"
class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
<property name="targetClass"
value="org.apache.kylin.metrics.MetricsManager"/>
<property name="targetMethod" value="initMetricsManager"/>
<property name="arguments">
<list>
<!--
Sink of System Cube.
-->
<ref bean="hiveSink"/>
<!--
Bind properties for each ActiveReservoirReporter.
-->
<map key-type="org.apache.kylin.metrics.lib.ActiveReservoir"
value-type="java.util.List">
<!-- Each ActiveReservoir can have multi ReservoirReporter
-->
<entry key-ref="instantReservoir">
<list>
<bean class="org.apache.kylin.common.util.Pair">
<!-- Implementation of ReservoirReporter-->
<property name="first"
value="org.apache.kylin.metrics.lib.impl.kafka.KafkaReservoirReporter"/>
<!-- Properties for specific
ReservoirReporter-->
<property name="second">
<props>
<prop
key="bootstrap.servers">cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092</prop>
</props>
</property>
</bean>
<bean class="org.apache.kylin.common.util.Pair">
<!-- Implementation of ReservoirReporter-->
<property name="first"
value="org.apache.kylin.metrics.lib.impl.callback.CallbackActiveReservoirReporter"/>
<!-- Properties for specific
ReservoirReporter-->
<property name="second">
<props>
<prop
key="callback.url">localhost:8888</prop>
</props>
</property>
</bean>
</list>
</entry>
</map>
</list>
</property>
</bean>
</beans>
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2020-02-05 10:03:13, "codingforfun" <[email protected]> wrote:
I think we can refer to the implementation of spark listener, a few key points
are as follows:
1. Define interface ListenerEvent, which can implement specific events.
Examples of which build cubes are CubeBuildStart, CubeBuildStepXStart,
CubeBuildStepXEnd, CubeBuildEnd. These specific classes can contain a lot of
information, such as owner, success/failed, error msg, resource occupation, etc
2. Define interface Listener, which contains the monitoring of all events. When
the user defines the Listener, some events can be selectively implemented as
needed, for example, onCubeBuildStart and CubeBuildEnd can be implemented to
obtain the time cost.
3. Define a ListenerBus that provides the ability to register listeners(by
configured custom listener class names), receive events, and dispatch events to
the appropriate Listener in the background
在 2020年2月4日 11:17，ShaoFeng Shi <[email protected]>写道：
Good proposal. Is there some standard and popular framework for this? We can
integrate with the best solution there.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: [email protected]
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: [email protected]
Join Kylin dev mail group: [email protected]
Liukaige <[email protected]> 于2020年1月25日周六 上午1:01写道：
Totally agreed. And this feature can be integrated with ETL scheduling, data
governance, approval flow etc. Brilliant idea. If you need any help, count me
in.
朱卫斌 <[email protected]> 于2020年1月20日周一 下午10:10写道：
I think there should be a set of kylin's event, metrics mechanism,
event-driven, provide event, metrics interfaces, we can implement specific
listeners in the form of plugins. For example, we can implement dingtalk
plugin, SMS plugin or any other plugin, which has a high Flexibility (Not only
do notifications, but also do many things, such as unifying and even
interfering with the task.). We can refer to the design and implementation of
spark event and metrics.
I think this is very valuable, we can do it together.
| |
weibin0516
|
|
[email protected]
Best wishes !
|
签名由网易邮箱大师定制
On 01/21/2020 10:57，Xiaoxiang Yu<[email protected]> wrote：
It looks good, and it should be useful for IT team, please go ahead!
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2020-01-20 19:58:40, "Zhou Kang" <[email protected]> wrote:
Hi Kylin users & developers:
Many apps support webhook, it is one way that apps can send automated
messages or information to other apps.
Use webhook, I think we can send messages to chat tools（slack,
dingding）, send sms , trigger another workflow.
Do we need to add webhook to Kylin? Such as when cubing job finished.
What do you think about this, and which is the better way in your
environment ?
--
Best regards,
Kaige Liu(刘凯歌)
"Do small things with great love."
