Comments inline below.
On Thu, May 3, 2018 at 3:25 PM, Ryan Merriman <merrim...@gmail.com> wrote: > Otto, > > I'm assuming just adding it to the Alerts UI is less work but I wouldn't be > strongly opposed to it being it's own UI. What are the reasons for doing > that? > > I don't know that we should split them up. It seems like a sub-section or wizard or some such would be useful here. The use cases I've seen/heard around PCAP often started with an infosec analyst doing a search on alerts that followed with them going to query pcap data corresponding to the threats they're investigating. Maybe we should emphasize streamlining this experience? > Mike, > > On using metron-api: > > 1. I'm making an assumption about it not being used much. Maybe it > still works without issue. I agree, we'll have to test anything we > build > so this is a minor issue. > 2. Updating metron-api to be asynchronous is a requirement in my opinion > Yes, I agree this is reasonable to do. > 3. The MPack work is the major drawback for me. We're essentially > creating a brand new Metron component. There are a lot of examples we > can > draw from but it's going to be a large chunk of new MPack code to > maintain > and MPack development has been painful in the past. I think it will > include: > 1. Creating a start script > 2. Creating master.py and commands.py scripts for managing the > application lifecycle, service checks, etc > 3. Creating an -env.xml file for exposing properties in Ambari > 4. Adding the component to the various MPack files > (metron_theme.json, metainfo.xml, service_advisor.py, etc.) > Awesome - this is exactly what I/we need to understand your vision for this and other features, and weigh the pros and cons. > 4. Our Storm topologies are completely different use cases and much more > complex so I don't understand the comparison. But if you prefer this > coding style then I think this is a minor issue as well. > > I still don't understand what the specific code style is in Pcap that is problematic. Even if I might disagree with you (haha, it could be like arguing spaces vs tabs), you called it out specifically, and I want to understand your position and reasons. I might agree with you, I might not, but I do want to understand the point that's being made regardless. Is it formatting? Class, interface, and package structure? Esoteric names? Documentation? Tabs vs spaces, lol? We have the ability to change any of these things, and they don't necessarily (probably shouldn't) be done inside of this feature. The reason I pointed out the Storm topology pieces is because they are indeed complex, and I think they can be greatly simplified. The unified enrichment topology is one such example that came through testing, but there are other improvements obtained simply by taking a holistic view of what's there and introducing simplifying refactorings. Same thing we can do here with Pcap, if it seems useful. > On micro-services: > > 1. Our REST service already includes a lot of dependencies and is > difficult to manage in it's current state. I just went through this on > https://github.com/apache/metron/pull/1008. It was painful. When we > tried to include mapreduce and yarn dependencies it became what seemed > like > an endless NoSuchMethod, NoClassDef and similar errors. Even if we can > get > it to work it's going to make managing our REST service that much harder > than it already is. I think the shaded jars are the source of all this > trouble and I agree it would be nice to improve our architecture in this > area. However I don't think it's a simple fix and now we're getting > into > the "will likely take a long time to plan and implement" concern. If > anyone has ideas on how to solve our shaded jar challenge I would be all > for it. > That's why I gave my recommendation about managing external dependencies as a special module. I do see in the PR you cited that, surprise surprise, Jackson is in the list. I know you're saying "take my word for it, I did a POC and tried it. It didn't work. Throw that bathwater OUT." I'm just asking that you at least share the pom files and any other specifics of the issue so that the community can A) see what the issues and hurdles are so that we can also weigh the trade-offs that you seem to have already decided on B) help, and C) have a record of that we can refer to for the next 10 times we go through the exact same thing. This is useful information for everyone for more than just this feature. If we set precedent here by punting without a strict and clear set of reasons, we'll literally do it for every other feature that adds new dependencies going forward. I don't think we should manage our architecture and features in this manner. 2. All the MPack work listed above would also be required here. A > micro-services pattern is a significant shift and can't even give you > concrete examples of what exactly we would have to do. We would need > to go > through extensive design and planning to even get to that point. > So everyone is on the same page, does this capture your meaning by micro-services? - https://martinfowler.com/articles/microservices.html. I think we're missing a 4th option that is lumped in with the micro-services option, which is to simply expose the functionality that kicks off the mapreduce job from our REST API. It's literally relocating a fair amount of the metron-api project. This brings us back to point #1 about the dependency issues, which I am not sufficiently convinced is an insurmountable problem yet. At any rate, if we did that, we'd be saving all the work you're concerned about in the MPack. And we'd be deprecating and removing code and simplifying the code base. And we'd be making the same compromise about not taking on a massive micro-services feature. I agree with you - micro-services isn't necessary to get this feature rolling. 3. It would be a branch new component. See above plus any new > infrastructure we would need (web server/proxy, service discovery, etc) > > But not if we didn't split it out as a micro-service, per my points above. On pcap-query: > > 1. I don't recall any users or customers directly using metron-api but > if you say so I believe you :) > Are there many folks using the pcap_query.sh? I haven't noticed anything on the mailing lists in quite a while, but I might have missed it. My point here was that the testing effort seems about the same. I want to distill our pros and cons down to a very concise list and e2e testing would be an item I would probably remove from the list of reasons for any particular solution. 2. As I understand it the pcap topology and pcap query are somewhat > decoupled. Maybe location of pcap files would be shared? MPack work > here > is likely to include adding a couple properties and moving some around > so > they can be shared. Deciding between Ambari and global config would be > similar to properties we add to any component. > > Yeah, file location is the first one that came to mind. We're hashing out an architecture and short and long-term solutions and level of effort in this DISCUSS. Since you brought up effort around MPack I thought it made sense to get an idea of what we'd be needing for a reasonable solution in each of the proposed solutions. I think you may be underestimating how difficult it's going to be to solve > our dependency problem. Or maybe it's me that is overestimating it :) It > could be something we experiment with before we start on the pcap work. > There is major upside and it would benefit the whole project. But until > then we can't fit anymore more screwdrivers in the toolbox. For me the > only reasonable options are to use the existing metron-api as it's own > separate service or call out to the pcap_query.sh script from our existing > REST app. I could go either way really. I'm just not excited about all > the MPack code we have to write for a new component. Maybe it won't be > that bad. > > I literally don't have a clue about the dependency problem size, other than having solved other dependency issues in the project. It was painful, but doable. Share your findings in a way that the community can clearly and concisely understand and reproduce the POC problems, and I think we will be much better equipped to make an informed decision. > On Thu, May 3, 2018 at 2:50 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > First thought is why the Alerts-UI and Not a dedicated Query UI? > > > > > > On May 3, 2018 at 14:36:04, Ryan Merriman (merrim...@gmail.com) wrote: > > > > We are planning on adding the pcap query feature to the Alerts UI. Before > > we start this work, I think it is important to get community buy in on > the > > architectural approach. There are a couple different options. > > > > One option is to leverage the existing metron-api module that exposes > pcap > > queries through a REST service. The upsides are: > > > > - some work has already been done > > - it's part of our build so we know unit and integration tests pass > > > > The downsides are: > > > > - It hasn't been used in a while and will need some end to end testing > > to make sure it still functions properly > > - It is synchronous and will block the UI, using up the limited number > > of concurrent connections available in a browser > > - It will require significant MPack work to properly set it up on install > > - It is a legacy module from OpenSOC and coding style is significantly > > different > > > > Another option would be moving to a micro-services architecture. We have > > experimented with a proof of concept and found it was too hard to add > this > > feature into our existing REST services because of all the dependencies > > that must coexist in the same application. The upsides are: > > > > - Would provide a platform for future Batch/MR/YARN type features > > - There would be fewer technical compromises since we are building it > > from the ground up > > > > The downsides are: > > > > - Will require the most effort and will likely take a long time to plan > > and implement > > - Like the previous option, will require significant MPack work > > > > A third option would be to add an endpoint to our existing REST service > > that delegates to the pcap_query.sh script through the Java Process > class. > > The upsides to this approach are: > > > > - We know the pcap_query.sh script works and would require minimal > > changes > > - Minimal MPack work is required since our REST service is already > > included > > > > The downsides are: > > > > - Does not set us up to easily add other batch-oriented features in the > > future > > - OS-level security becomes a concern since we are delegating to a > > script in a separate process > > > > I feel like ultimately we want to transition to a micro-services > > architecture because it will provide more flexibility and make it easier > > to > > grow our set of features. But in the meantime, wrapping the pcap_query.sh > > script would allow us to add this feature with less work and fewer lines > > of > > code. If and when we decide to deploy a separate REST application for > > batch features, the UI portion would require minimal changes. > > > > What does everyone think? > > > > >
