Re: [DISCUSS] Pcap panel architecture

Yes there will be an admin role that can read and delete all. On Fri, May 11, 2018 at 4:11 PM, Otto Fowler wrote: > Do we at least require a admin/super user? See’s all the queues and jobs? > > > On May 11, 2018 at 17:03:34, Ryan Merriman (merrim...@gmail.com) wrote: >

Re: [DISCUSS] Pcap panel architecture

Thanks everyone for the input and feedback. I will attempt to summarize so we can come to a consensus and get this tasked out. The following endpoints will be included: - GET /api/v1/pcap/metadata?basePath - This endpoint will return metadata of pcap data stored in HDFS. This would

Re: [DISCUSS] Pcap panel architecture

At the very least there needs to be the ability to share downloaded PCAPs with other users and/or have roles that can see all pcaps. A platform engineer may want to clean up old pcaps after x time, or a manger may ask an analyst to find all of the traffic that exhibits xyz behavior, dump a pcap,

Re: [DISCUSS] Pcap panel architecture

Mike, I believe the /pcapGetter/getPcapsByIdentifiers endpoint exposes the fixed query option which we have covered. I agree with you that deprecating the metron-api module should be a goal of this feature. On Wed, May 9, 2018 at 1:36 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote:

Re: [DISCUSS] Pcap panel architecture

Security is another important topic related to our pcap architecture. This may spill over into a more general, system-wide discussion and we can start a separate thread for that if necessary. I'm assuming we want to manage pcap queries by user. One important question is which user do we use to

Re: [DISCUSS] Pcap panel architecture

This looks like a pretty good start Ryan. Does the metadata endpoint cover this https://github.com/apache/metron/tree/master/ metron-platform/metron-api#the-pcapgettergetpcapsbyidentifiers-endpoint from the original metron-api? If so, then we would be able to deprecate the existing metron-api

Re: [DISCUSS] Pcap panel architecture

Thanks for the feedback Jon. I'm am not as familiar with BPF filtering as you probably are. Do you have an idea of much effort would be involved in implementing this? I suspect this would be another PcapFilter (

Re: [DISCUSS] Pcap panel architecture

This looks really great and gets me excited to maybe revisit some old conversations about PCAP capture in Metron. The only thing that I think it's missing is the ability to filter using bpf. I think the same thing can technically be accomplished by using packet_filter and I wouldn't throw a fit

Re: [DISCUSS] Pcap panel architecture

Now that we are confident we can run submit a MR job from our current REST application, is this the desired approach? Just want to confirm. Next I think we should map out what the REST interface will look like. Here are the endpoints I'm thinking about: GET /api/v1/pcap/metadata?basePath This

Re: [DISCUSS] Pcap panel architecture

Sweet! That's great news. The pom changes are a lot simpler than I expected. Very nice. On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman wrote: > Finally figured it out. Commit is here: > https://github.com/merrimanr/incubator-metron/commit/ >

Re: [DISCUSS] Pcap panel architecture

Finally figured it out. Commit is here: https://github.com/merrimanr/incubator-metron/commit/22fe5e9ff3c167b42ebeb7a9f1000753a409aff1 It came down to figuring out the right combination of maven dependencies and passing in the HDP version to REST as a Java system property. I also included some

Re: [DISCUSS] Pcap panel architecture

@Ryan - pulled your branch and experimented with a few things. In doing so, it dawned on me that by adding the yarn and hadoop classpath, you probably didn't introduce a new classpath issue, rather you probably just moved onto the next classpath issue, ie hbase per your exception about hbase jaxb.

Re: [DISCUSS] Pcap panel architecture

That would be a step closer to something more like a micro-service architecture. However, I would want to make sure we think about the operational complexity, and mpack implications of having another server installed and running somewhere on the cluster (also, ssl, kerberos, etc etc requirements

Re: [DISCUSS] Pcap panel architecture

+1 to having metron-api as it's own service and using a gateway type pattern. On Tue, May 8, 2018 at 8:13 AM, Otto Fowler wrote: > Why not have metron-api as it’s own service and use a ‘gateway’ type > pattern in rest? > > > On May 8, 2018 at 08:45:33, Ryan Merriman

Re: [DISCUSS] Pcap panel architecture

Why not have metron-api as it’s own service and use a ‘gateway’ type pattern in rest? On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com) wrote: Moving the yarn classpath command earlier in the classpath now gives this error: Caused by: java.lang.NoSuchMethodError:

Re: [DISCUSS] Pcap panel architecture

Moving the yarn classpath command earlier in the classpath now gives this error: Caused by: java.lang.NoSuchMethodError: javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String; I will experiment with other combinations, I suspect we will need finer-grain control over the order.

Re: [DISCUSS] Pcap panel architecture

What order did you add the hadoop or yarn classpath? The "shaded" package stands out to me in this name "org.apache.hadoop.hbase.*shaded* .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding those packages earlier on the classpath. I think that find command needs a "jar tvf",

Re: [DISCUSS] Pcap panel architecture

To explore the idea of merging metron-api into metron-rest and running pcap queries inside our REST application, I created a simple test here: https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test. A summary of what's included: - Added pcap as a dependency in the metron-rest

Re: [DISCUSS] Pcap panel architecture

I can imagine a new generic service(s) capability whose job ( pun intended ) is to abstract the submittal, tracking, and storage of results to yarn. It would be extended with storage providers, queue provider, possibly some set of policies or rather strategies. The pcap ‘report’ would be a

Re: [DISCUSS] Pcap panel architecture

RE: Tracking v. users The submittal and tracking can associate the submitter with the yarn job and track that, regardless of the yarn credentials. IE> if all submittals and monitoring are by the same yarn user ( Metron ) from a single or co-operative set of services, that service can maintain

Re: [DISCUSS] Pcap panel architecture

That sounds fine - I'd imagine we'd be looking to hit the classpath related problems asap when merging the modules. For the module, we just have a pom that supplies external dependencies. Rather than every metron module depending on Guava or Jackson directly, or via transitive dependencies, we

Re: [DISCUSS] Pcap panel architecture

Otto, your use case makes sense to me. We'll have to think about how to manage the user to job relationships. I'm assuming YARN jobs will be submitted as the metron service user so YARN won't keep track of this for us. Is that assumption correct? Do you have any ideas for doing that? Mike, I

Re: [DISCUSS] Pcap panel architecture

>From my response on the other thread, but applicable to the backend stuff: "The PCAP Query seems more like PCAP Report to me. You are generating a report based on parameters. That report is something that takes some time and external process to generate… ie you have to wait for it. I can

Re: [DISCUSS] Pcap panel architecture

I started a separate thread on Pcap UI considerations and user requirements at Otto's request. This should help us keep these two related but separate discussions focused. On Fri, May 4, 2018 at 7:19 AM, Michel Sumbul wrote: > Hello, > > > > (Youhouuu my first reply on

Re: [DISCUSS] Pcap panel architecture

Hello, (Youhouuu my first reply on this kind of mail chain^^) If I may, I would like to share my view on the following 3 points. - Backend: The current metron-api is totally seperate, it will be logic for me to have it at the same place as the others rest api. Especially when more security

Re: [DISCUSS] Pcap panel architecture

I know, I was running with it :) > On May 3, 2018, at 10:21 PM, Michael Miklavcic > wrote: > > Tabs vs spaces was a Silicon Valley joke, man :-) > >> On Thu, May 3, 2018, 8:42 PM Ryan Merriman wrote: >> >> Mike, >> >> I never said there was

Re: [DISCUSS] Pcap panel architecture

Tabs vs spaces was a Silicon Valley joke, man :-) On Thu, May 3, 2018, 8:42 PM Ryan Merriman wrote: > Mike, > > I never said there was anything problematic in metron-api, just that is was > inconsistent with the rest of Metron. There is work involved in making it >

Re: [DISCUSS] Pcap panel architecture

Mike, I never said there was anything problematic in metron-api, just that is was inconsistent with the rest of Metron. There is work involved in making it consistent which is why I listed it as a downside. I'm less concerned with whether we use tabs or spaces but that we use one or the other.

Re: [DISCUSS] Pcap panel architecture

Yes, completely agreed. We're on the same page. On Thu, May 3, 2018 at 7:50 PM, Otto Fowler wrote: > I think my point is that maybe we should have a discuss about: > > * PCAP UI, goals etc > * Where it would live and why, what that would mean etc > * Backend ( this

Re: [DISCUSS] Pcap panel architecture

I think my point is that maybe we should have a discuss about: * PCAP UI, goals etc * Where it would live and why, what that would mean etc * Backend ( this original mail ) On May 3, 2018 at 18:34:00, Michael Miklavcic (michael.miklav...@gmail.com) wrote: Otto, what are you and your customers

Re: [DISCUSS] Pcap panel architecture

Otto, what are you and your customers finding useful and/or difficult from a split management/alerts UI perspective? It might help us to restate the original scope and intent around maintaining separate management and alert UI's, to your point about "contrary to previous direction." I personally

Re: [DISCUSS] Pcap panel architecture

Comments inline below. On Thu, May 3, 2018 at 3:25 PM, Ryan Merriman wrote: > Otto, > > I'm assuming just adding it to the Alerts UI is less work but I wouldn't be > strongly opposed to it being it's own UI. What are the reasons for doing > that? > > I don't know that we

Re: [DISCUSS] Pcap panel architecture

If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t the alerts ui anymore. It is becoming more of a “composite” app, with multiple feature ui’s together. I didn’t think that was what we were going for, thus the config ui and the alert ui. Just adding disparate thing as ‘new

Re: [DISCUSS] Pcap panel architecture

Otto, I'm assuming just adding it to the Alerts UI is less work but I wouldn't be strongly opposed to it being it's own UI. What are the reasons for doing that? Mike, On using metron-api: 1. I'm making an assumption about it not being used much. Maybe it still works without issue. I

Re: [DISCUSS] Pcap panel architecture

First thought is why the Alerts-UI and Not a dedicated Query UI? On May 3, 2018 at 14:36:04, Ryan Merriman (merrim...@gmail.com) wrote: We are planning on adding the pcap query feature to the Alerts UI. Before we start this work, I think it is important to get community buy in on the

Re: [DISCUSS] Pcap panel architecture

Thanks for the write-up, Ryan. A few questions and comments. 1. metron-api 1. "It hasn't been used in a while and will need some end to end testing to make sure it still functions properly" > I was probably one of the last developers to touch this code a year or more ago -