[
https://issues.apache.org/jira/browse/EAGLE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hao Chen updated EAGLE-5:
-------------------------
Description:
We should decouple the core framework and features from security monitoring
case, so that we could bring the scalability, real-time alerting and powerful
policy syntax to more general-purpose cases like any kind of event/metrics
real-time monitoring. As the first step, we could split out some core parts as
a minimal general-purpose monitoring engine.
Use Cases
Currently Eagle supports very complex data processing pipeline for hadoop
audit/security logs, but I think we reuse some valuable components in Eagle:
1) distributed policy engine
2) highly abstracted streaming program API
3) user-friendly policy & alert management UI
for more general cases like what traditional monitoring produces do.
Use Case One: For example, as to ops team like DBA, Hardware or Cloud team,
lots of users just would assume that given monitoring data format is known like
typical time series data points, in such case, user just need to tell Eagle
what's the stream schema and preprocess the data into kafka with external
program like scripts or agents, and Eagle could provide a generic topology to
monitor the stream without any programming, just like most traditional
monitoring products' paradigm.
Use Case Two: Some advanced users with development skill may want to easily
use Eagle streaming program API for process complex monitoring data like some
complex logs, connect to Eagle's metadata engine for managing policy in UI, and
execute the policy in eagle distributed policy engine in real-time.
Design
So that we need to do following works:
1. Implement a generic pipeline topology as starting like: Kafka ->
EventParser(JSON) -> Metadata Manager -> PolicyEngine which could be reused for
lots of simple use cases like metrics monitoring.
2. Allow to import or design stream schema in UI. Today, we only assume the
stream schema is already defined in databases, but as to most general cases, we
should allow to define stream schema by eagle tool like UI for more generic
purpose.
3. For advanced users like developers, we should make our streaming framework
more easy for use. One of the most critical parts is developers has to define
stream schema in database, and write code independently. In fact, for most
cases like hadoop security monitoring, the schema will never change
independently, in such case, we should even define the stream schema in code,
even we could also define policy inline as well, so we could run the eagle
monitoring engine without metadata store (hbase).
was:We should decouple the core framework and features from security
monitoring case, so that we could bring the scalability, real-time alerting and
powerful policy syntax to more general-purpose cases like any kind of
event/metrics real-time monitoring. As the first step, we could split out some
core parts as a minimal general-purpose monitoring engine.
> Minimal general-purpose monitoring engine
> ------------------------------------------
>
> Key: EAGLE-5
> URL: https://issues.apache.org/jira/browse/EAGLE-5
> Project: Eagle
> Issue Type: New Feature
> Reporter: Hao Chen
>
> We should decouple the core framework and features from security monitoring
> case, so that we could bring the scalability, real-time alerting and powerful
> policy syntax to more general-purpose cases like any kind of event/metrics
> real-time monitoring. As the first step, we could split out some core parts
> as a minimal general-purpose monitoring engine.
> Use Cases
> Currently Eagle supports very complex data processing pipeline for hadoop
> audit/security logs, but I think we reuse some valuable components in Eagle:
> 1) distributed policy engine
> 2) highly abstracted streaming program API
> 3) user-friendly policy & alert management UI
> for more general cases like what traditional monitoring produces do.
> Use Case One: For example, as to ops team like DBA, Hardware or Cloud team,
> lots of users just would assume that given monitoring data format is known
> like typical time series data points, in such case, user just need to tell
> Eagle what's the stream schema and preprocess the data into kafka with
> external program like scripts or agents, and Eagle could provide a generic
> topology to monitor the stream without any programming, just like most
> traditional monitoring products' paradigm.
> Use Case Two: Some advanced users with development skill may want to easily
> use Eagle streaming program API for process complex monitoring data like some
> complex logs, connect to Eagle's metadata engine for managing policy in UI,
> and execute the policy in eagle distributed policy engine in real-time.
> Design
> So that we need to do following works:
> 1. Implement a generic pipeline topology as starting like: Kafka ->
> EventParser(JSON) -> Metadata Manager -> PolicyEngine which could be reused
> for lots of simple use cases like metrics monitoring.
> 2. Allow to import or design stream schema in UI. Today, we only assume the
> stream schema is already defined in databases, but as to most general cases,
> we should allow to define stream schema by eagle tool like UI for more
> generic purpose.
> 3. For advanced users like developers, we should make our streaming framework
> more easy for use. One of the most critical parts is developers has to define
> stream schema in database, and write code independently. In fact, for most
> cases like hadoop security monitoring, the schema will never change
> independently, in such case, we should even define the stream schema in code,
> even we could also define policy inline as well, so we could run the eagle
> monitoring engine without metadata store (hbase).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
