this is very cool stuff, looking forward for the first release. Best Regards! ---------------------
Luke Han On Thu, Feb 6, 2020 at 7:27 PM Yaqian Zhang <[email protected]> wrote: > Agree with xiaoxiang’s solution. I'm looking forward to this feature. > > 在 2020年2月6日，18:49，weibin zhu <[email protected]> 写道： > > Thanks for the reply, if everyone agrees with this solution, I can create > a jira and implement the code. I suggest that can define the interface and > framework code first. If you are interested, you can review it. If there is > no problem, then implement the complete code. > @ShaoFeng Shi, @zhoukang, @Xiaoxiang Yu > > ---------------------------------------- > weibin0516 > > [email protected] > http://rrd.me/fT6RW > ---------------------------------------- > > 在 2020年2月5日，下午11:19，Xiaoxiang Yu <[email protected]> 写道： > > Hi codingforfun and zhoukang, > I am agreed with codingforfun. Event, Listener, and ListenerBus, these > components should be good abstraction/design for a monitor framework. > Recently I am checking source code of System Cube and Dashboard > Service in apache kylin, I have find some exist abstraction of these > components(Event, Listener, and ListenerBus). > We can find them under core-mertics module ( > https://github.com/apache/kylin/tree/master/core-metrics/src/main/java/org/apache/kylin/metrics/lib). > Maybe > we can implement the webhook feature by extending this metrics framework > other than reinventing the wheel. I am not very sure am I right but I guess > it should works. > It is a draft version of and I extend some basic interface > under core-mertics module, I think it not perfect at the moment, this is my > patch > https://github.com/hit-lacus/kylin/commit/db5b52ca81fd20b4bdd3328c90739373eba2adea > . > I supposed my draft is intended to the beginning of the conversation, not > the final implementation. If you have any comments, please let me know, > thank you in advance. > > To test it, use the following config in kylin.properties and > kylinMetrics.xml. > > 1. kylin.properties > kylin.metrics.reporter-job-enabled=true > kylin.metrics.monitor-enabled=true > > 2. kylinMetrics.xml > <beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xmlns="http://www.springframework.org/schema/beans"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > > http://www.springframework.org/schema/beans/spring-beans-3.1.xsd";> > > <description>Kylin Metrics Related Configuration > (SystemCube)</description> > > <!-- A Reservoir which don't staged metrics message at all, emit it in > no time. Maybe good for debug purpose.--> > <bean id="instantReservoir" > class="org.apache.kylin.metrics.lib.impl.InstantReservoir"/> > > <!-- A Reservoir which staged metrics message in memory, and emit them > in fixed rate. --> > <bean id="blockingReservoir" > class="org.apache.kylin.metrics.lib.impl.BlockingReservoir"> > <!-- minReportSize, only if currently count of staged message > exceed minReportSize, will Reservoir try to write message--> > <constructor-arg index="0"> > <value>100</value> > </constructor-arg> > > <!-- maxReportSize, max size of report in one time --> > <constructor-arg index="1"> > <value>500</value> > </constructor-arg> > > <!-- minReportTime, min duration(in minute) between two report > action--> > <constructor-arg index="2"> > <value>10</value> > </constructor-arg> > </bean> > > <bean id="hiveSink" > class="org.apache.kylin.metrics.lib.impl.hive.HiveSink"/> > <bean id="kafkaSink" > class="org.apache.kylin.metrics.lib.impl.kafka.KafkaSink"/> > > <bean id="initMetricsManager" > class="org.springframework.beans.factory.config.MethodInvokingFactoryBean"> > <property name="targetClass" > value="org.apache.kylin.metrics.MetricsManager"/> > <property name="targetMethod" value="initMetricsManager"/> > <property name="arguments"> > <list> > <!-- > Sink of System Cube. > --> > <ref bean="hiveSink"/> > > <!-- > Bind properties for each ActiveReservoirReporter. > --> > <map > key-type="org.apache.kylin.metrics.lib.ActiveReservoir" > value-type="java.util.List"> > > <!-- Each ActiveReservoir can have multi > ReservoirReporter --> > <entry key-ref="instantReservoir"> > <list> > <bean > class="org.apache.kylin.common.util.Pair"> > <!-- Implementation of ReservoirReporter--> > <property name="first" > > value="org.apache.kylin.metrics.lib.impl.kafka.KafkaReservoirReporter"/> > <!-- Properties for specific > ReservoirReporter--> > <property name="second"> > <props> > <prop > key="bootstrap.servers">cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092</prop> > </props> > </property> > </bean> > > <bean > class="org.apache.kylin.common.util.Pair"> > <!-- Implementation of ReservoirReporter--> > <property name="first" > > value="org.apache.kylin.metrics.lib.impl.callback.CallbackActiveReservoirReporter"/> > <!-- Properties for specific > ReservoirReporter--> > <property name="second"> > <props> > <prop > key="callback.url">localhost:8888</prop> > </props> > </property> > </bean> > </list> > </entry> > </map> > </list> > </property> > </bean> > </beans> > > > > > -- > *Best wishes to you ! * > *From ：**Xiaoxiang Yu* > > At 2020-02-05 10:03:13, "codingforfun" <[email protected]> wrote: > > I think we can refer to the implementation of spark listener, a few key > points are as follows: > 1. Define interface ListenerEvent, which can implement specific events. > Examples of which build cubes are CubeBuildStart, CubeBuildStepXStart, > CubeBuildStepXEnd, CubeBuildEnd. These specific classes can contain a lot > of information, such as owner, success/failed, error msg, resource > occupation, etc > 2. Define interface Listener, which contains the monitoring of all events. > When the user defines the Listener, some events can be selectively > implemented as needed, for example, onCubeBuildStart and CubeBuildEnd can > be implemented to obtain the time cost. > 3. Define a ListenerBus that provides the ability to register listeners(by > configured custom listener class names), receive events, and dispatch > events to the appropriate Listener in the background > > > > 在 2020年2月4日 11:17，ShaoFeng Shi <[email protected]>写道： > > Good proposal. Is there some standard and popular framework for this? We > can integrate with the best solution there. > > Best regards, > > Shaofeng Shi 史少锋 > Apache Kylin PMC > Email: [email protected] > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html > Join Kylin user mail group: [email protected] > Join Kylin dev mail group: [email protected] > > > > > Liukaige <[email protected]> 于2020年1月25日周六 上午1:01写道： > >> Totally agreed. And this feature can be integrated with ETL scheduling, >> data governance, approval flow etc. Brilliant idea. If you need any help, >> count me in. >> >> 朱卫斌 <[email protected]> 于2020年1月20日周一 下午10:10写道： >> >>> I think there should be a set of kylin's event, metrics mechanism, >>> event-driven, provide event, metrics interfaces, we can implement specific >>> listeners in the form of plugins. For example, we can implement dingtalk >>> plugin, SMS plugin or any other plugin, which has a high Flexibility (Not >>> only do notifications, but also do many things, such as unifying and even >>> interfering with the task.). We can refer to the design and implementation >>> of spark event and metrics. >>> I think this is very valuable, we can do it together. >>> >>> weibin0516 >>> [email protected] >>> Best wishes ! >>> >>> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=weibin0516&uid=codingforfun%40126.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fwzpmmc%2F54c20faa3a1910ad49f4a5f1965fba47.jpg&items=%5B%22codingforfun%40126.com%22%2C%22Best+wishes+%21%22%5D> >>> 签名由 网易邮箱大师 <https://mail.163.com/dashi/dlpro.html?from=mail81> 定制 >>> >>> On 01/21/2020 10:57，Xiaoxiang Yu<[email protected]> <[email protected]> >>> wrote： >>> >>> It looks good, and it should be useful for IT team, please go ahead! >>> >>> >>> -- >>> *Best wishes to you ! * >>> *From ：**Xiaoxiang Yu* >>> >>> At 2020-01-20 19:58:40, "Zhou Kang" <[email protected]> wrote: >>> >>> Hi Kylin users & developers: >>> >>> >>> >>> Many apps support webhook, it is one way that apps can send >>> automated messages or information to other apps. >>> >>> Use webhook, I think we can send messages to chat tools（slack, >>> dingding）, send sms , trigger another workflow. >>> >>> Do we need to add webhook to Kylin? Such as when cubing job >>> finished. >>> >>> What do you think about this, and which is the better way in your >>> environment ? >>> >>> >> >> -- >> Best regards, >> >> Kaige Liu(刘凯歌) >> >> *"Do small things with great love." * >> > > >
