Tabs vs spaces was a Silicon Valley joke, man :-) On Thu, May 3, 2018, 8:42 PM Ryan Merriman <merrim...@gmail.com> wrote:
> Mike, > > I never said there was anything problematic in metron-api, just that is was > inconsistent with the rest of Metron. There is work involved in making it > consistent which is why I listed it as a downside. I'm less concerned with > whether we use tabs or spaces but that we use one or the other. > > I apologize for not making this clearer in my original message, but I did > not lead the POC development. My involvement was helping troubleshoot > issues they ran into and answering questions about Metron in general. I've > shared with you the information that I have which is my observations about > the types of issues they ran into. I don't have a branch or pom file you > can experiment with. I will reach out to that person and see if they are > able to share the exact errors they hit. Also, the "trade-offs that you > seem to have already decided on" is not based on a specific issue or > challenge they faced in the POC. It's based off of the past couple years > of working on our REST module and the reoccurring challenges and patterns I > see over a period of time. > > Otto, > > Makes sense to me. I will start the other threads. > > On Thu, May 3, 2018 at 8:50 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > I think my point is that maybe we should have a discuss about: > > > > * PCAP UI, goals etc > > * Where it would live and why, what that would mean etc > > * Backend ( this original mail ) > > > > > > > > On May 3, 2018 at 18:34:00, Michael Miklavcic ( > michael.miklav...@gmail.com) > > wrote: > > > > Otto, what are you and your customers finding useful and/or difficult > from > > a split management/alerts UI perspective? It might help us to restate the > > original scope and intent around maintaining separate management and > alert > > UI's, to your point about "contrary to previous direction." I personally > > don't have a strong position on this other than 1) management is a > > different feature set from drilling into threat intel, yet many apps > still > > have their management UI combined with the end user experience and 2) we > > should probably consider pcap in context of a workflow with alerts. > > > > On Thu, May 3, 2018 at 4:19 PM, Otto Fowler <ottobackwa...@gmail.com> > > wrote: > > > > > If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t > the > > > alerts ui anymore. > > > > > > It is becoming more of a “composite” app, with multiple feature ui’s > > > together. I didn’t think that > > > was what we were going for, thus the config ui and the alert ui. > > > > > > Just adding disparate thing as ‘new tabs’ to a ui may be expedient but > > it > > > seems contrary to > > > our previous direction. > > > > > > There are a few things to consider if we are going to start moving > > > everything into Alerts Ui aren’t there? > > > > > > It may be a better road to bring it in on it’s own like the alerts ui > > > effort, so it can be released with ‘qualifiers’ and tested with > > > the right expectations without effecting the Alerts UI. > > > > > > > > > > > > On May 3, 2018 at 17:25:54, Ryan Merriman (merrim...@gmail.com) wrote: > > > > > > Otto, > > > > > > I'm assuming just adding it to the Alerts UI is less work but I > wouldn't > > be > > > strongly opposed to it being it's own UI. What are the reasons for > doing > > > that? > > > > > > Mike, > > > > > > On using metron-api: > > > > > > 1. I'm making an assumption about it not being used much. Maybe it > > > still works without issue. I agree, we'll have to test anything we > build > > > so this is a minor issue. > > > 2. Updating metron-api to be asynchronous is a requirement in my > opinion > > > 3. The MPack work is the major drawback for me. We're essentially > > > creating a brand new Metron component. There are a lot of examples we > > can > > > draw from but it's going to be a large chunk of new MPack code to > > maintain > > > and MPack development has been painful in the past. I think it will > > > include: > > > 1. Creating a start script > > > 2. Creating master.py and commands.py scripts for managing the > > > application lifecycle, service checks, etc > > > 3. Creating an -env.xml file for exposing properties in Ambari > > > 4. Adding the component to the various MPack files > > > (metron_theme.json, metainfo.xml, service_advisor.py, etc.) > > > 4. Our Storm topologies are completely different use cases and much > more > > > complex so I don't understand the comparison. But if you prefer this > > > coding style then I think this is a minor issue as well. > > > > > > On micro-services: > > > > > > 1. Our REST service already includes a lot of dependencies and is > > > difficult to manage in it's current state. I just went through this on > > > https://github.com/apache/metron/pull/1008. It was painful. When we > > > tried to include mapreduce and yarn dependencies it became what seemed > > like > > > an endless NoSuchMethod, NoClassDef and similar errors. Even if we can > > get > > > it to work it's going to make managing our REST service that much > harder > > > than it already is. I think the shaded jars are the source of all this > > > trouble and I agree it would be nice to improve our architecture in > this > > > area. However I don't think it's a simple fix and now we're getting > into > > > the "will likely take a long time to plan and implement" concern. If > > > anyone has ideas on how to solve our shaded jar challenge I would be > all > > > for it. > > > 2. All the MPack work listed above would also be required here. A > > > micro-services pattern is a significant shift and can't even give you > > > concrete examples of what exactly we would have to do. We would need to > > go > > > through extensive design and planning to even get to that point. > > > 3. It would be a branch new component. See above plus any new > > > infrastructure we would need (web server/proxy, service discovery, etc) > > > > > > On pcap-query: > > > > > > 1. I don't recall any users or customers directly using metron-api but > > > if you say so I believe you :) > > > 2. As I understand it the pcap topology and pcap query are somewhat > > > decoupled. Maybe location of pcap files would be shared? MPack work > here > > > is likely to include adding a couple properties and moving some around > > so > > > they can be shared. Deciding between Ambari and global config would be > > > similar to properties we add to any component. > > > > > > I think you may be underestimating how difficult it's going to be to > > solve > > > our dependency problem. Or maybe it's me that is overestimating it :) > It > > > could be something we experiment with before we start on the pcap work. > > > There is major upside and it would benefit the whole project. But until > > > then we can't fit anymore more screwdrivers in the toolbox. For me the > > > only reasonable options are to use the existing metron-api as it's own > > > separate service or call out to the pcap_query.sh script from our > > existing > > > REST app. I could go either way really. I'm just not excited about all > > > the MPack code we have to write for a new component. Maybe it won't be > > > that bad. > > > > > > On Thu, May 3, 2018 at 2:50 PM, Otto Fowler <ottobackwa...@gmail.com> > > > wrote: > > > > > > > First thought is why the Alerts-UI and Not a dedicated Query UI? > > > > > > > > > > > > On May 3, 2018 at 14:36:04, Ryan Merriman (merrim...@gmail.com) > > wrote: > > > > > > > > We are planning on adding the pcap query feature to the Alerts UI. > > Before > > > > we start this work, I think it is important to get community buy in > on > > > the > > > > architectural approach. There are a couple different options. > > > > > > > > One option is to leverage the existing metron-api module that exposes > > > pcap > > > > queries through a REST service. The upsides are: > > > > > > > > - some work has already been done > > > > - it's part of our build so we know unit and integration tests pass > > > > > > > > The downsides are: > > > > > > > > - It hasn't been used in a while and will need some end to end > testing > > > > to make sure it still functions properly > > > > - It is synchronous and will block the UI, using up the limited > number > > > > of concurrent connections available in a browser > > > > - It will require significant MPack work to properly set it up on > > install > > > > - It is a legacy module from OpenSOC and coding style is > significantly > > > > different > > > > > > > > Another option would be moving to a micro-services architecture. We > > have > > > > experimented with a proof of concept and found it was too hard to add > > > this > > > > feature into our existing REST services because of all the > > dependencies > > > > that must coexist in the same application. The upsides are: > > > > > > > > - Would provide a platform for future Batch/MR/YARN type features > > > > - There would be fewer technical compromises since we are building it > > > > from the ground up > > > > > > > > The downsides are: > > > > > > > > - Will require the most effort and will likely take a long time to > > plan > > > > and implement > > > > - Like the previous option, will require significant MPack work > > > > > > > > A third option would be to add an endpoint to our existing REST > > service > > > > that delegates to the pcap_query.sh script through the Java Process > > > class. > > > > The upsides to this approach are: > > > > > > > > - We know the pcap_query.sh script works and would require minimal > > > > changes > > > > - Minimal MPack work is required since our REST service is already > > > > included > > > > > > > > The downsides are: > > > > > > > > - Does not set us up to easily add other batch-oriented features in > > the > > > > future > > > > - OS-level security becomes a concern since we are delegating to a > > > > script in a separate process > > > > > > > > I feel like ultimately we want to transition to a micro-services > > > > architecture because it will provide more flexibility and make it > > easier > > > > to > > > > grow our set of features. But in the meantime, wrapping the > > pcap_query.sh > > > > script would allow us to add this feature with less work and fewer > > lines > > > > of > > > > code. If and when we decide to deploy a separate REST application for > > > > batch features, the UI portion would require minimal changes. > > > > > > > > What does everyone think? > > > > > > > > > > > > > > > >
