I know, I was running with it :)
> On May 3, 2018, at 10:21 PM, Michael Miklavcic <michael.miklav...@gmail.com> > wrote: > > Tabs vs spaces was a Silicon Valley joke, man :-) > >> On Thu, May 3, 2018, 8:42 PM Ryan Merriman <merrim...@gmail.com> wrote: >> >> Mike, >> >> I never said there was anything problematic in metron-api, just that is was >> inconsistent with the rest of Metron. There is work involved in making it >> consistent which is why I listed it as a downside. I'm less concerned with >> whether we use tabs or spaces but that we use one or the other. >> >> I apologize for not making this clearer in my original message, but I did >> not lead the POC development. My involvement was helping troubleshoot >> issues they ran into and answering questions about Metron in general. I've >> shared with you the information that I have which is my observations about >> the types of issues they ran into. I don't have a branch or pom file you >> can experiment with. I will reach out to that person and see if they are >> able to share the exact errors they hit. Also, the "trade-offs that you >> seem to have already decided on" is not based on a specific issue or >> challenge they faced in the POC. It's based off of the past couple years >> of working on our REST module and the reoccurring challenges and patterns I >> see over a period of time. >> >> Otto, >> >> Makes sense to me. I will start the other threads. >> >> On Thu, May 3, 2018 at 8:50 PM, Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>> I think my point is that maybe we should have a discuss about: >>> >>> * PCAP UI, goals etc >>> * Where it would live and why, what that would mean etc >>> * Backend ( this original mail ) >>> >>> >>> >>> On May 3, 2018 at 18:34:00, Michael Miklavcic ( >> michael.miklav...@gmail.com) >>> wrote: >>> >>> Otto, what are you and your customers finding useful and/or difficult >> from >>> a split management/alerts UI perspective? It might help us to restate the >>> original scope and intent around maintaining separate management and >> alert >>> UI's, to your point about "contrary to previous direction." I personally >>> don't have a strong position on this other than 1) management is a >>> different feature set from drilling into threat intel, yet many apps >> still >>> have their management UI combined with the end user experience and 2) we >>> should probably consider pcap in context of a workflow with alerts. >>> >>> On Thu, May 3, 2018 at 4:19 PM, Otto Fowler <ottobackwa...@gmail.com> >>> wrote: >>> >>>> If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t >> the >>>> alerts ui anymore. >>>> >>>> It is becoming more of a “composite” app, with multiple feature ui’s >>>> together. I didn’t think that >>>> was what we were going for, thus the config ui and the alert ui. >>>> >>>> Just adding disparate thing as ‘new tabs’ to a ui may be expedient but >>> it >>>> seems contrary to >>>> our previous direction. >>>> >>>> There are a few things to consider if we are going to start moving >>>> everything into Alerts Ui aren’t there? >>>> >>>> It may be a better road to bring it in on it’s own like the alerts ui >>>> effort, so it can be released with ‘qualifiers’ and tested with >>>> the right expectations without effecting the Alerts UI. >>>> >>>> >>>> >>>> On May 3, 2018 at 17:25:54, Ryan Merriman (merrim...@gmail.com) wrote: >>>> >>>> Otto, >>>> >>>> I'm assuming just adding it to the Alerts UI is less work but I >> wouldn't >>> be >>>> strongly opposed to it being it's own UI. What are the reasons for >> doing >>>> that? >>>> >>>> Mike, >>>> >>>> On using metron-api: >>>> >>>> 1. I'm making an assumption about it not being used much. Maybe it >>>> still works without issue. I agree, we'll have to test anything we >> build >>>> so this is a minor issue. >>>> 2. Updating metron-api to be asynchronous is a requirement in my >> opinion >>>> 3. The MPack work is the major drawback for me. We're essentially >>>> creating a brand new Metron component. There are a lot of examples we >>> can >>>> draw from but it's going to be a large chunk of new MPack code to >>> maintain >>>> and MPack development has been painful in the past. I think it will >>>> include: >>>> 1. Creating a start script >>>> 2. Creating master.py and commands.py scripts for managing the >>>> application lifecycle, service checks, etc >>>> 3. Creating an -env.xml file for exposing properties in Ambari >>>> 4. Adding the component to the various MPack files >>>> (metron_theme.json, metainfo.xml, service_advisor.py, etc.) >>>> 4. Our Storm topologies are completely different use cases and much >> more >>>> complex so I don't understand the comparison. But if you prefer this >>>> coding style then I think this is a minor issue as well. >>>> >>>> On micro-services: >>>> >>>> 1. Our REST service already includes a lot of dependencies and is >>>> difficult to manage in it's current state. I just went through this on >>>> https://github.com/apache/metron/pull/1008. It was painful. When we >>>> tried to include mapreduce and yarn dependencies it became what seemed >>> like >>>> an endless NoSuchMethod, NoClassDef and similar errors. Even if we can >>> get >>>> it to work it's going to make managing our REST service that much >> harder >>>> than it already is. I think the shaded jars are the source of all this >>>> trouble and I agree it would be nice to improve our architecture in >> this >>>> area. However I don't think it's a simple fix and now we're getting >> into >>>> the "will likely take a long time to plan and implement" concern. If >>>> anyone has ideas on how to solve our shaded jar challenge I would be >> all >>>> for it. >>>> 2. All the MPack work listed above would also be required here. A >>>> micro-services pattern is a significant shift and can't even give you >>>> concrete examples of what exactly we would have to do. We would need to >>> go >>>> through extensive design and planning to even get to that point. >>>> 3. It would be a branch new component. See above plus any new >>>> infrastructure we would need (web server/proxy, service discovery, etc) >>>> >>>> On pcap-query: >>>> >>>> 1. I don't recall any users or customers directly using metron-api but >>>> if you say so I believe you :) >>>> 2. As I understand it the pcap topology and pcap query are somewhat >>>> decoupled. Maybe location of pcap files would be shared? MPack work >> here >>>> is likely to include adding a couple properties and moving some around >>> so >>>> they can be shared. Deciding between Ambari and global config would be >>>> similar to properties we add to any component. >>>> >>>> I think you may be underestimating how difficult it's going to be to >>> solve >>>> our dependency problem. Or maybe it's me that is overestimating it :) >> It >>>> could be something we experiment with before we start on the pcap work. >>>> There is major upside and it would benefit the whole project. But until >>>> then we can't fit anymore more screwdrivers in the toolbox. For me the >>>> only reasonable options are to use the existing metron-api as it's own >>>> separate service or call out to the pcap_query.sh script from our >>> existing >>>> REST app. I could go either way really. I'm just not excited about all >>>> the MPack code we have to write for a new component. Maybe it won't be >>>> that bad. >>>> >>>> On Thu, May 3, 2018 at 2:50 PM, Otto Fowler <ottobackwa...@gmail.com> >>>> wrote: >>>> >>>>> First thought is why the Alerts-UI and Not a dedicated Query UI? >>>>> >>>>> >>>>> On May 3, 2018 at 14:36:04, Ryan Merriman (merrim...@gmail.com) >>> wrote: >>>>> >>>>> We are planning on adding the pcap query feature to the Alerts UI. >>> Before >>>>> we start this work, I think it is important to get community buy in >> on >>>> the >>>>> architectural approach. There are a couple different options. >>>>> >>>>> One option is to leverage the existing metron-api module that exposes >>>> pcap >>>>> queries through a REST service. The upsides are: >>>>> >>>>> - some work has already been done >>>>> - it's part of our build so we know unit and integration tests pass >>>>> >>>>> The downsides are: >>>>> >>>>> - It hasn't been used in a while and will need some end to end >> testing >>>>> to make sure it still functions properly >>>>> - It is synchronous and will block the UI, using up the limited >> number >>>>> of concurrent connections available in a browser >>>>> - It will require significant MPack work to properly set it up on >>> install >>>>> - It is a legacy module from OpenSOC and coding style is >> significantly >>>>> different >>>>> >>>>> Another option would be moving to a micro-services architecture. We >>> have >>>>> experimented with a proof of concept and found it was too hard to add >>>> this >>>>> feature into our existing REST services because of all the >>> dependencies >>>>> that must coexist in the same application. The upsides are: >>>>> >>>>> - Would provide a platform for future Batch/MR/YARN type features >>>>> - There would be fewer technical compromises since we are building it >>>>> from the ground up >>>>> >>>>> The downsides are: >>>>> >>>>> - Will require the most effort and will likely take a long time to >>> plan >>>>> and implement >>>>> - Like the previous option, will require significant MPack work >>>>> >>>>> A third option would be to add an endpoint to our existing REST >>> service >>>>> that delegates to the pcap_query.sh script through the Java Process >>>> class. >>>>> The upsides to this approach are: >>>>> >>>>> - We know the pcap_query.sh script works and would require minimal >>>>> changes >>>>> - Minimal MPack work is required since our REST service is already >>>>> included >>>>> >>>>> The downsides are: >>>>> >>>>> - Does not set us up to easily add other batch-oriented features in >>> the >>>>> future >>>>> - OS-level security becomes a concern since we are delegating to a >>>>> script in a separate process >>>>> >>>>> I feel like ultimately we want to transition to a micro-services >>>>> architecture because it will provide more flexibility and make it >>> easier >>>>> to >>>>> grow our set of features. But in the meantime, wrapping the >>> pcap_query.sh >>>>> script would allow us to add this feature with less work and fewer >>> lines >>>>> of >>>>> code. If and when we decide to deploy a separate REST application for >>>>> batch features, the UI portion would require minimal changes. >>>>> >>>>> What does everyone think? >>>>> >>>>> >>>> >>> >>> >>
