Yes, completely agreed. We're on the same page. On Thu, May 3, 2018 at 7:50 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:
> I think my point is that maybe we should have a discuss about: > > * PCAP UI, goals etc > * Where it would live and why, what that would mean etc > * Backend ( this original mail ) > > > > On May 3, 2018 at 18:34:00, Michael Miklavcic (michael.miklav...@gmail.com) > wrote: > > Otto, what are you and your customers finding useful and/or difficult from > a split management/alerts UI perspective? It might help us to restate the > original scope and intent around maintaining separate management and alert > UI's, to your point about "contrary to previous direction." I personally > don't have a strong position on this other than 1) management is a > different feature set from drilling into threat intel, yet many apps still > have their management UI combined with the end user experience and 2) we > should probably consider pcap in context of a workflow with alerts. > > On Thu, May 3, 2018 at 4:19 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t the > > alerts ui anymore. > > > > It is becoming more of a “composite” app, with multiple feature ui’s > > together. I didn’t think that > > was what we were going for, thus the config ui and the alert ui. > > > > Just adding disparate thing as ‘new tabs’ to a ui may be expedient but > it > > seems contrary to > > our previous direction. > > > > There are a few things to consider if we are going to start moving > > everything into Alerts Ui aren’t there? > > > > It may be a better road to bring it in on it’s own like the alerts ui > > effort, so it can be released with ‘qualifiers’ and tested with > > the right expectations without effecting the Alerts UI. > > > > > > > > On May 3, 2018 at 17:25:54, Ryan Merriman (merrim...@gmail.com) wrote: > > > > Otto, > > > > I'm assuming just adding it to the Alerts UI is less work but I wouldn't > be > > strongly opposed to it being it's own UI. What are the reasons for doing > > that? > > > > Mike, > > > > On using metron-api: > > > > 1. I'm making an assumption about it not being used much. Maybe it > > still works without issue. I agree, we'll have to test anything we build > > so this is a minor issue. > > 2. Updating metron-api to be asynchronous is a requirement in my opinion > > 3. The MPack work is the major drawback for me. We're essentially > > creating a brand new Metron component. There are a lot of examples we > can > > draw from but it's going to be a large chunk of new MPack code to > maintain > > and MPack development has been painful in the past. I think it will > > include: > > 1. Creating a start script > > 2. Creating master.py and commands.py scripts for managing the > > application lifecycle, service checks, etc > > 3. Creating an -env.xml file for exposing properties in Ambari > > 4. Adding the component to the various MPack files > > (metron_theme.json, metainfo.xml, service_advisor.py, etc.) > > 4. Our Storm topologies are completely different use cases and much more > > complex so I don't understand the comparison. But if you prefer this > > coding style then I think this is a minor issue as well. > > > > On micro-services: > > > > 1. Our REST service already includes a lot of dependencies and is > > difficult to manage in it's current state. I just went through this on > > https://github.com/apache/metron/pull/1008. It was painful. When we > > tried to include mapreduce and yarn dependencies it became what seemed > like > > an endless NoSuchMethod, NoClassDef and similar errors. Even if we can > get > > it to work it's going to make managing our REST service that much harder > > than it already is. I think the shaded jars are the source of all this > > trouble and I agree it would be nice to improve our architecture in this > > area. However I don't think it's a simple fix and now we're getting into > > the "will likely take a long time to plan and implement" concern. If > > anyone has ideas on how to solve our shaded jar challenge I would be all > > for it. > > 2. All the MPack work listed above would also be required here. A > > micro-services pattern is a significant shift and can't even give you > > concrete examples of what exactly we would have to do. We would need to > go > > through extensive design and planning to even get to that point. > > 3. It would be a branch new component. See above plus any new > > infrastructure we would need (web server/proxy, service discovery, etc) > > > > On pcap-query: > > > > 1. I don't recall any users or customers directly using metron-api but > > if you say so I believe you :) > > 2. As I understand it the pcap topology and pcap query are somewhat > > decoupled. Maybe location of pcap files would be shared? MPack work here > > is likely to include adding a couple properties and moving some around > so > > they can be shared. Deciding between Ambari and global config would be > > similar to properties we add to any component. > > > > I think you may be underestimating how difficult it's going to be to > solve > > our dependency problem. Or maybe it's me that is overestimating it :) It > > could be something we experiment with before we start on the pcap work. > > There is major upside and it would benefit the whole project. But until > > then we can't fit anymore more screwdrivers in the toolbox. For me the > > only reasonable options are to use the existing metron-api as it's own > > separate service or call out to the pcap_query.sh script from our > existing > > REST app. I could go either way really. I'm just not excited about all > > the MPack code we have to write for a new component. Maybe it won't be > > that bad. > > > > On Thu, May 3, 2018 at 2:50 PM, Otto Fowler <ottobackwa...@gmail.com> > > wrote: > > > > > First thought is why the Alerts-UI and Not a dedicated Query UI? > > > > > > > > > On May 3, 2018 at 14:36:04, Ryan Merriman (merrim...@gmail.com) > wrote: > > > > > > We are planning on adding the pcap query feature to the Alerts UI. > Before > > > we start this work, I think it is important to get community buy in on > > the > > > architectural approach. There are a couple different options. > > > > > > One option is to leverage the existing metron-api module that exposes > > pcap > > > queries through a REST service. The upsides are: > > > > > > - some work has already been done > > > - it's part of our build so we know unit and integration tests pass > > > > > > The downsides are: > > > > > > - It hasn't been used in a while and will need some end to end testing > > > to make sure it still functions properly > > > - It is synchronous and will block the UI, using up the limited number > > > of concurrent connections available in a browser > > > - It will require significant MPack work to properly set it up on > install > > > - It is a legacy module from OpenSOC and coding style is significantly > > > different > > > > > > Another option would be moving to a micro-services architecture. We > have > > > experimented with a proof of concept and found it was too hard to add > > this > > > feature into our existing REST services because of all the > dependencies > > > that must coexist in the same application. The upsides are: > > > > > > - Would provide a platform for future Batch/MR/YARN type features > > > - There would be fewer technical compromises since we are building it > > > from the ground up > > > > > > The downsides are: > > > > > > - Will require the most effort and will likely take a long time to > plan > > > and implement > > > - Like the previous option, will require significant MPack work > > > > > > A third option would be to add an endpoint to our existing REST > service > > > that delegates to the pcap_query.sh script through the Java Process > > class. > > > The upsides to this approach are: > > > > > > - We know the pcap_query.sh script works and would require minimal > > > changes > > > - Minimal MPack work is required since our REST service is already > > > included > > > > > > The downsides are: > > > > > > - Does not set us up to easily add other batch-oriented features in > the > > > future > > > - OS-level security becomes a concern since we are delegating to a > > > script in a separate process > > > > > > I feel like ultimately we want to transition to a micro-services > > > architecture because it will provide more flexibility and make it > easier > > > to > > > grow our set of features. But in the meantime, wrapping the > pcap_query.sh > > > script would allow us to add this feature with less work and fewer > lines > > > of > > > code. If and when we decide to deploy a separate REST application for > > > batch features, the UI portion would require minimal changes. > > > > > > What does everyone think? > > > > > > > > > >
