Thanks for the valuable comments, you are right in the high-level observations :-)
(I participated some offline discussion on this proposal) I think this proposal is based on the requirements that Eagle not only monitors security events from hadoop but also monitors security events from other data source, for example cassandra and even more requirements are from hadoop native metric monitoring. In last 2 months, when we want to onboard new diverse datasources(for example mongo db metrics, hadoop native metrics etc.), we find it is impossible for user(mostly operations team) to understand storm topology or write code for very simple metrics ingestion/alert rules. For monitoring perspective, user usually wants metric/log onboarding is as simple as turn-key operation. It is possible for Eagle developers to develop applications for each data source, but it might be better for Eagle to support logs/metrics with some general schema. User just needs some configurations to get new data source flow into Eagle and create policy on-the-fly against the streaming data. Maybe I am wrong, but Eagle looks is an application to Storm framework, but Eagle would be a framework to monitoring applications e.g. security or other data activity. The point of Eagle to separate application and framework is very correct. In Eagle source code, the application code and framework code are separated from the beginning. For those features like policy restore after machine fails, DSL, aggregation etc, I think Eagle team should look for contributing back to Storm if people agree that is what stream framework should have. We also found that Eagle monitoring uses streaming framework but streaming framework is not customized for monitoring. The gap between monitoring platform and streaming framework has to be filled to make sure monitoring is reliable. For example compared to real-time streaming analytics, monitoring does not want to have any false alert or missing alert especially for security event, which requires more processing semantics than popular streaming framework provides. We will actively explore help from streaming projects. Thanks Edward On Mon, Jan 11, 2016 at 10:55 AM, Julian Hyde <[email protected]> wrote: > Can I make a high-level observation? (And, although I’m a mentor of this > project, I’m not speaking as a mentor, just someone who has built various > database and streaming systems over the years.) > > I’ve noticed that Eagle is taking on several problems that — > architecturally speaking — should be part of the underlying streaming > system. This topology manager and also, a DSL declarative streaming > queries, and making sure that streaming queries continue where they left > off, even in the presence of failures of individual stream-processing nodes. > > These are very hard distributed systems problems, and they are horizontal > problems that have nothing to do with Eagle’s problem domain (security). It > would be analogous to an application that is selling concert tickets > deciding to develop HBase as part of the application. > > If the Eagle community wants to solve these problems, that’s awesome, and > you should go for it. Apache projects are great at pulling in people with > diverse skills and when they gather momentum they can build some amazing > technology. But I think Eagle should consider putting more architectural > separation between your stream management technology and your application. > You could do that by building separate modules (and testing them > independently). Or you could contribute the functionality you need to the > underlying system (e.g. Storm/Nimbus). Your project will run much more > smoothly if you call out the hard problems you are trying to solve. > > And, as a side benefit, projects that have nothing to do with Hadoop > security will be able to use (and test, and bug-fix) the technology you are > developing. > > Julian > > > > On Jan 11, 2016, at 8:57 AM, Hao Chen <[email protected]> wrote: > > > > Currently eagle is requiring user to manually manage topologies > completely > > independent of eagle components, which is not very smooth for the user > and > > management experience end-to-end from on-boarding datasource, starting > > topologies, defining policy and also monitoring policy and execution > > status, so how do you think we manage everything in single place and > > dynamically manage topology lifecycle like > > starting/stopping/status/monitoring as well policy > > creation/modification/monitoring all in eagle ui only? So that user don't > > need to touch storm anymore except specifying where the nimbus is when > > setting up eagle. > > > > > > -- > > > > Hao > >
