Hi Edward, In the new architecture, is the storage(hbase) taken off? How do the adaptors store the data? for example, hadoop namenode metrics.
Best Regards, Qingwen Zhao | 赵晴雯 Edward Zhang <[email protected]> 于2018年10月12日周五 上午10:49写道： > Hi Eaglers, > > I would like to start some discussion about architecture improvement for > Apache Eagle based on community experience and feedback. The improvement is > targeted for simplifying installation and development of Apache Eagle. > > Eagle's main responsibility is to report abnormality instantly by applying > policies on streaming data. Eagle consists of two major components, Policy > Engine and Adaptors. Policy Engine is a standalone application which > provides REST API to manage policy lifecycle for different data sources and > provides runtime to evaluate policy on streaming data. Adaptors are those > applications which fetch/process log/metrics from outside and send data to > policy engine for alerting purpose. > > But right now in Eagle code base, it is not clearly focusing on the two > components. For example the current source code includes map/reduce > job/task log retrieval/cleanup/analysis which is very useful but probably > Eagle only needs the portion of data retrieval/cleanup part and so data can > be streamed into policy engine for alerting purpose. For job/task analysis > part, it can be maintained in other project. > > First let me list the main modules Eagle source code consists of. > - eagle core > - policy engine (coordinator, runtime, and web) > - monitor application management > - eagle query framework - for querying time series data from hbase > - eagle adaptors > - gc log fetch/processing and alerting > - metric fetch/processing and alerting, including name node, data > node, hbase etc. > - jpm: job performance management. > - haoop yarn queue statistics fetch/processing > - hadoop mapreduce history job log processing > - hadoop mapreduce running job processing > - spark history job log processing > - spark running job processing > - jpm web application > - hadoop job analyzer > - security monitoring > - hdfs audit log fetch/processing > - hdfs auth log fetch/processing > - hbase audit log fetch/processing > - hive log fetch/processing > - maprfs audit log fetch/processing > - oozie audit log fetch/processing > - hadoop topology stats fetch/processing > - eagle server > > It is very obvious that it is not scale for Eagle community to maintain so > large amount of monitoring adaptors especially when Hadoop/Spark versions > are evolving pretty fast. > > My suggestion is Eagle ONLY focus on policy engine and some default > important adaptors but remove/separate some unrelated functionalities. For > policy engine, it would be nice if it can run on popular streaming engine > besides Apache Storm so that it can be easily deployed for community users. > For default important adaptors, I may suggest Eagle have ONLY HDFS audit > log, Hadoop running job, Spark running job, HDFS namenode metrics etc. For > unrelated functionalities, we can either remove them from Eagle code base > or separate them into standalone executable if that is still really needed > under Apache Eagle monitoring umbrella by community. > > So the proposed Eagle code base would be like: > - policy engine > - coordinator > - runtime > - web > - adaptors > - hdfs audit log > - Hadoop running job > - Spark running job > - HDFS namenode metrics > - Hadoop yarn queue metrics > - extensions (some non default adaptors contributed by community) > - executables (standalone executables which are legacy) > > It would be great if you can provide more feedback on this discussion. > > (By the way, I also had a lot of discussion with Hao, Chen, Eagle PMC > member and core developer about this topic based on his experience of > engaging Eagle users.) > > Thanks > Edward >
