Can I make a high-level observation? (And, although I’m a mentor of this project, I’m not speaking as a mentor, just someone who has built various database and streaming systems over the years.)
I’ve noticed that Eagle is taking on several problems that — architecturally speaking — should be part of the underlying streaming system. This topology manager and also, a DSL declarative streaming queries, and making sure that streaming queries continue where they left off, even in the presence of failures of individual stream-processing nodes. These are very hard distributed systems problems, and they are horizontal problems that have nothing to do with Eagle’s problem domain (security). It would be analogous to an application that is selling concert tickets deciding to develop HBase as part of the application. If the Eagle community wants to solve these problems, that’s awesome, and you should go for it. Apache projects are great at pulling in people with diverse skills and when they gather momentum they can build some amazing technology. But I think Eagle should consider putting more architectural separation between your stream management technology and your application. You could do that by building separate modules (and testing them independently). Or you could contribute the functionality you need to the underlying system (e.g. Storm/Nimbus). Your project will run much more smoothly if you call out the hard problems you are trying to solve. And, as a side benefit, projects that have nothing to do with Hadoop security will be able to use (and test, and bug-fix) the technology you are developing. Julian > On Jan 11, 2016, at 8:57 AM, Hao Chen <[email protected]> wrote: > > Currently eagle is requiring user to manually manage topologies completely > independent of eagle components, which is not very smooth for the user and > management experience end-to-end from on-boarding datasource, starting > topologies, defining policy and also monitoring policy and execution > status, so how do you think we manage everything in single place and > dynamically manage topology lifecycle like > starting/stopping/status/monitoring as well policy > creation/modification/monitoring all in eagle ui only? So that user don't > need to touch storm anymore except specifying where the nimbus is when > setting up eagle. > > > -- > > Hao
