1. Make sure HDFS_AUDIT_LOG_MONITOR_APP_SANDBOX writes to the right Kafka topic 2. Define a policy based on the stream installed by HDFS_AUDIT_LOG_MONITOR_APP_SANDBOX. Alert topology would proactively read data from topics unless policies are defined 3. After define policies, check coordinator scheduled metadata, which triggers alert engine topology GET /rest/metadata/schedulestates And check SpoutSpecs part, to avoid coordinator scheduler does not work, you can also manually do it by sending a RESTful request POST /rest/metadata/schedulestates
Regards, Qingwen Zhao | Apache Eagle Desk: 20998167 | Cell: +86 18818215025 On 5/16/17, 11:19 AM, "lpn1509" <[email protected]> wrote: >HI > > >这个拓扑 没有获取到topic， 这个是如何配置的 ？ > >ALERT_UNIT_TOPOLOGY_APP_SANDBOX ><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.64. >23%3A8080%2Ftopology.html%3Fid%3DALERT_UNIT_TOPOLOGY_APP_SANDBOX-23-149490 >0983&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a7bc >5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata=WE >1RWmY9CAgDrc4EJy7fsvuFyOe%2F9aaNW201gKwzEkE%3D&reserved=0> > >2017-05-16 10:57:30.849 o.a.e.a.e.c.i.ZKMetadataChangeNotifyService >[INFO] Metadata changed isValueVersionId: true, value: >spec_version_1494903450729 >2017-05-16 10:57:30.850 o.a.e.a.s.MetadataServiceClientImpl [INFO] >Requesting >https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.64.2 >3%3A9090%2Frest%2Fmetadata%2Fschedulestates%2Fspec_version_1494903450729&d >ata=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a7bc5%7C46 >326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata=0vuZ5Nab >tU6Db5rAG8ZA6IMIaaoRMGFbGmmOlvYYqzA%3D&reserved=0 >2017-05-16 10:57:30.852 o.a.e.a.e.c.i.ZKMetadataChangeNotifyService >[INFO] Metadata changed isValueVersionId: true, value: >spec_version_1494903450729 >2017-05-16 10:57:30.853 o.a.e.a.s.MetadataServiceClientImpl [INFO] >Requesting >https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.64.2 >3%3A9090%2Frest%2Fmetadata%2Fschedulestates%2Fspec_version_1494903450729&d >ata=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a7bc5%7C46 >326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata=0vuZ5Nab >tU6Db5rAG8ZA6IMIaaoRMGFbGmmOlvYYqzA%3D&reserved=0 >2017-05-16 10:57:30.869 o.a.e.a.e.c.i.ZKMetadataChangeNotifyService >[INFO] Metadata changed isValueVersionId: true, value: >spec_version_1494903450729 >2017-05-16 10:57:30.869 o.a.e.a.s.MetadataServiceClientImpl [INFO] >Requesting >https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.64.2 >3%3A9090%2Frest%2Fmetadata%2Fschedulestates%2Fspec_version_1494903450729&d >ata=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a7bc5%7C46 >326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata=0vuZ5Nab >tU6Db5rAG8ZA6IMIaaoRMGFbGmmOlvYYqzA%3D&reserved=0 >2017-05-16 10:57:30.894 o.a.e.a.e.c.i.ZKMetadataChangeNotifyService >[INFO] Metadata changed isValueVersionId: true, value: >spec_version_1494903450729 >2017-05-16 10:57:30.894 o.a.e.a.s.MetadataServiceClientImpl [INFO] >Requesting >https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.64.2 >3%3A9090%2Frest%2Fmetadata%2Fschedulestates%2Fspec_version_1494903450729&d >ata=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a7bc5%7C46 >326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata=0vuZ5Nab >tU6Db5rAG8ZA6IMIaaoRMGFbGmmOlvYYqzA%3D&reserved=0 >2017-05-16 10:57:30.917 o.a.e.a.e.r.AlertBolt [INFO] no new policy with >AlertBoltSpec >version:spec_version_1494903450729-topo:ALERT_UNIT_TOPOLOGY_APP_SANDBOX, >boltPolicyIdsMap {}, > publishPartitions for this bolt alertBolt0 >2017-05-16 10:57:30.918 o.a.e.a.e.s.CorrelationSpout [INFO] new metadata >is updated >version:spec_version_1494903450729-topo:ALERT_UNIT_TOPOLOGY_APP_SANDBOX >2017-05-16 10:57:30.919 o.a.e.a.e.s.CorrelationSpout [INFO] Topics were >added=[], removed=[], modified=[] 为什么这里是空的 >2017-05-16 10:57:30.919 o.a.e.a.e.s.CorrelationSpout [INFO] after >CorrelationSpout reloads, 0 kafkaSpouts are generated for 0 topics > > > >HDFS_AUDIT_LOG_MONITOR_APP_SANDBOX ><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.64. >23%3A8080%2Ftopology.html%3Fid%3DHDFS_AUDIT_LOG_MONITOR_APP_SANDBOX-22-149 >4900978&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a >7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata >=q0dAeVF6NgbuzdeEl2pR9DtJbyc%2BYTlannB45vGQ89M%3D&reserved=0> > >而这个拓扑是正常在执行 ； > >SA|李平|lpn1509 >在线游戏事业部-技术部-网管组 >phone： 18819381175 >email：[email protected] > > > > >> 在 2017年5月15日，11:11，lpn1509 <[email protected]> 写道： >> >> >> HI >> >> it is for what ? 下面这个用来干嘛的？ 怎么用？ document has nothing about it ~~~ >> >> <PastedGraphic-1.tiff>SA|李平|lpn1509 >> 在线游戏事业部-技术部-网管组 >> phone： 18819381175 >> email：[email protected] >><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcorp.net >>ease.com%2F&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d >>49c0a7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773 >>&sdata=psd3ufSlpC3yAIzpxgPCVI%2BcuTI7ZbzTnRStykV%2BOy0%3D&reserved=0> >> >> >> >> >>> 在 2017年5月15日，10:14，lpn1509 <[email protected] >>><mailto:[email protected]>> 写道： >>> >>> alter spout 拿不到数据 ~ 怎么处理？ >>> >>> 2017-05-15 09:37:51.870 o.a.e.a.e.c.i.ZKMetadataChangeNotifyService >>>[INFO] Metadata changed isValueVersionId: true, value: >>>spec_version_1494559752711 >>> 2017-05-15 09:37:51.871 o.a.e.a.s.MetadataServiceClientImpl [INFO] >>>Requesting >>>https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.64 >>>.23%3A9090%2Frest%2Fmetadata%2Fschedulestates%2Fspec_version_14945597527 >>>11&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a7bc >>>5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata= >>>4X4dvT1PTCUdl2fZ2ANT8s53yKbcWnri7sRa4OmfNso%3D&reserved=0 >>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.6 >>>4.23%3A9090%2Frest%2Fmetadata%2Fschedulestates%2Fspec_version_1494559752 >>>711&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a7b >>>c5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata >>>=4X4dvT1PTCUdl2fZ2ANT8s53yKbcWnri7sRa4OmfNso%3D&reserved=0> >>> 2017-05-15 09:37:52.004 o.a.e.a.s.MetadataServiceClientImpl [WARN] >>>List one entity failed, ignored and continue, >>>path:/metadata/schedulestates/spec_version_1494559752711 >>> com.sun.jersey.api.client.UniformInterfaceException: GET >>>https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.64 >>>.23%3A9090%2Frest%2Fmetadata%2Fschedulestates%2Fspec_version_14945597527 >>>11&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a7bc >>>5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata= >>>4X4dvT1PTCUdl2fZ2ANT8s53yKbcWnri7sRa4OmfNso%3D&reserved=0 >>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.63.6 >>>4.23%3A9090%2Frest%2Fmetadata%2Fschedulestates%2Fspec_version_1494559752 >>>711&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0a7b >>>c5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836773&sdata >>>=4X4dvT1PTCUdl2fZ2ANT8s53yKbcWnri7sRa4OmfNso%3D&reserved=0> returned a >>>response status of 204 No Content >>> at >>>com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:6 >>>09) ~[stormjar.jar:?] >>> at >>>com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:5 >>>86) ~[stormjar.jar:?] >>> at >>>org.apache.eagle.alert.service.MetadataServiceClientImpl.listOne(Metadat >>>aServiceClientImpl.java:182) [stormjar.jar:?] >>> at >>>org.apache.eagle.alert.service.MetadataServiceClientImpl.getVersionedSpe >>>c(MetadataServiceClientImpl.java:166) [stormjar.jar:?] >>> at >>>org.apache.eagle.alert.engine.coordinator.impl.ZKMetadataChangeNotifySer >>>vice.onNewConfig(ZKMetadataChangeNotifyService.java:115) >>>[stormjar.jar:?] >>> at >>>org.apache.eagle.alert.engine.coordinator.impl.ZKMetadataChangeNotifySer >>>vice.activateFetchMetaData(ZKMetadataChangeNotifyService.java:77) >>>[stormjar.jar:?] >>> at >>>org.apache.eagle.alert.engine.runner.AbstractStreamBolt.prepare(Abstract >>>StreamBolt.java:86) [stormjar.jar:?] >>> at >>>backtype.storm.daemon.executor$fn__5694$fn__5707.invoke(executor.clj:757 >>>) [storm-core-0.10.0.jar:0.10.0] >>> at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477) >>>[storm-core-0.10.0.jar:0.10.0] >>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] >>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102] >>> >>> >>> SA|李平|lpn1509 >>> 在线游戏事业部-技术部-网管组 >>> phone： 18819381175 >>> email：[email protected] >>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcorp.ne >>>tease.com%2F&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff0 >>>8d49c0a7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218836 >>>773&sdata=psd3ufSlpC3yAIzpxgPCVI%2BcuTI7ZbzTnRStykV%2BOy0%3D&reserved=0> >>> >>> >>> >>> >>>> 在 2017年5月15日，09:51，lpn1509 <[email protected] >>>><mailto:[email protected]>> 写道： >>>> >>>> HI , 关于 0.5.0 版本的eagle ， 其中的alter的模块如何指定读取哪一个TOPIC呢？ >>>> >>>> >>>> >>>> SA|李平|lpn1509 >>>> 在线游戏事业部-技术部-网管组 >>>> phone： 18819381175 >>>> email：[email protected] >>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcorp.n >>>>etease.com%2F&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abf >>>>f08d49c0a7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016218 >>>>836773&sdata=psd3ufSlpC3yAIzpxgPCVI%2BcuTI7ZbzTnRStykV%2BOy0%3D&reserve >>>>d=0> >>>> >>>> >>>> >>>> >>>>> 在 2017年5月12日，15:13，lpn1509 <[email protected] >>>>><mailto:[email protected]>> 写道： >>>>> >>>>> HI >>>>> >>>>> 麻烦指导一下， 关于 0.5.0 版本的eagle.conf >>>>> >>>>> 关于一下这部分， 我不想用hbase ， 而是选择jdbc方式，选择这种方式， 该如何配置呢？ >>>>> >>>>> >>>>> storage { >>>>> # storage type: ["hbase","jdbc"] >>>>> # default is "hbase" >>>>> type = "jdbc" >>>>> jdbc { >>>>> username = "eagle" >>>>> password = "eagle" >>>>> driverClassName = com.mysql.jdbc.Driver >>>>> database = "eagle" >>>>> connection = "jdbc:mysql://10.120.165.104:3306/eagle" >>>>> } >>>>> >>>>> SA|李平|lpn1509 >>>>> 在线游戏事业部-技术部-网管组 >>>>> phone： 18819381175 >>>>> email：[email protected] >>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcorp. >>>>>netease.com%2F&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2a >>>>>bff08d49c0a7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305016 >>>>>218836773&sdata=psd3ufSlpC3yAIzpxgPCVI%2BcuTI7ZbzTnRStykV%2BOy0%3D&res >>>>>erved=0> >>>>> >>>>> >>>>> >>>>> >>>>>> 在 2017年5月8日，16:13，lpn1509 <[email protected] >>>>>><mailto:[email protected]>> 写道： >>>>>> >>>>>> >>>>>> HI , >>>>>> >>>>>> For the job analysis, can I use 0.4.0 to do it ? >>>>>> >>>>>> or >>>>>> >>>>>> can i use 0.5.0 for hadoop which is based on java 1.7 >>>>>> >>>>>> >>>>>> eagle的0.5.0版本， 我是否可以用来监控 基于java 1.7 的hadoop集群呢？ 目前我们 的hadoop集群都是基于 >>>>>>1.7 的java的 >>>>>> >>>>>> SA|李平|lpn1509 >>>>>> 在线游戏事业部-技术部-网管组 >>>>>> phone： 18819381175 >>>>>> email：[email protected] >>>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcorp >>>>>>.netease.com%2F&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b >>>>>>2abff08d49c0a7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C636305 >>>>>>016218836773&sdata=psd3ufSlpC3yAIzpxgPCVI%2BcuTI7ZbzTnRStykV%2BOy0%3D >>>>>>&reserved=0> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> 在 2017年5月6日，00:30，Edward Zhang <[email protected] >>>>>>><mailto:[email protected]>> 写道： >>>>>>> >>>>>>> Hi LPN, >>>>>>> >>>>>>> Please kindly subscribe to eagle dev mail list, so you can receive >>>>>>>message from eagle community. >>>>>>> >>>>>>> Thanks >>>>>>> Edward >>>>>>> >>>>>>> >>>>>>> ---------- Forwarded message ---------- >>>>>>> From: Liangfei Su <[email protected] >>>>>>><mailto:[email protected]>> >>>>>>> Date: 2017-05-04 23:22 GMT-07:00 >>>>>>> Subject: Re: eage问题咨询 >>>>>>> To: [email protected] <mailto:[email protected]> >>>>>>> >>>>>>> >>>>>>> Thanks for the interest. >>>>>>> >>>>>>> For the job analysis, you might follow the instruction by Jijun >>>>>>>for eagle >>>>>>> 0.5.0 snapshot. >>>>>>> >>>>>>> Thanks >>>>>>> Ralph >>>>>>> >>>>>>> >>>>>>> 2017-05-05 10:55 GMT+08:00 lpn1509 <[email protected] >>>>>>><mailto:[email protected]>>: >>>>>>> >>>>>>> > >>>>>>> > HI ， 试用了eagle 0.4.0版本 >>>>>>> > >>>>>>> > 实现了namenode的审计日志的检测功能， >>>>>>> > >>>>>>> > 但是我比较关心的是eagle在集群任务分析、负载分析上的实现， 官网没有这方便的具体的指 引， 另外官网的指引与0.4.0版本不太一 >>>>>>>样 >>>>>>> > >>>>>>> > >>>>>>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi >>>>>>>ki.apache.org%2Fconfluence%2Fdisplay%2FEAG%2FHadoop%2BJMX%2BMonitori >>>>>>>ng&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0 >>>>>>>a7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C63630501621883677 >>>>>>>3&sdata=ultjSfhH%2F7MarRgy6dJqZsuVJ1KtBd5IQpywr816F9o%3D&reserved=0 >>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw >>>>>>>iki.apache.org%2Fconfluence%2Fdisplay%2FEAG%2FHadoop%2BJMX%2BMonitor >>>>>>>ing&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c >>>>>>>0a7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C6363050162188367 >>>>>>>73&sdata=ultjSfhH%2F7MarRgy6dJqZsuVJ1KtBd5IQpywr816F9o%3D&reserved=0 >>>>>>>> < >>>>>>> > >>>>>>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi >>>>>>>ki.apache.org%2Fconfluence%2Fdisplay%2FEAG%2FHadoop%2BJMX%2BMonitori >>>>>>>ng&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c0 >>>>>>>a7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C63630501621883677 >>>>>>>3&sdata=ultjSfhH%2F7MarRgy6dJqZsuVJ1KtBd5IQpywr816F9o%3D&reserved=0 >>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw >>>>>>>iki.apache.org%2Fconfluence%2Fdisplay%2FEAG%2FHadoop%2BJMX%2BMonitor >>>>>>>ing&data=02%7C01%7Cqingwzhao%40ebay.com%7C32cb882952f245b2abff08d49c >>>>>>>0a7bc5%7C46326bff992841a0baca17c16c94ea99%7C1%7C0%7C6363050162188467 >>>>>>>82&sdata=nJXics%2BVINR7P0AziRALVLqqNenoFPEmtZkXqYbQIPM%3D&reserved=0 >>>>>>>>> >>>>>>> > >>>>>>> > >>>>>>> > 是否可以给我一些指引资料呢？ >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > SA|李平|lpn1509 >>>>>>> > 在线游戏事业部-技术部-网管组 >>>>>>> > phone： 18819381175 >>>>>>> > email：[email protected] <mailto:[email protected]> >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> >>>>>> >>>>> >>>> >>> >> >
