Yes there will be an admin role that can read and delete all.
On Fri, May 11, 2018 at 4:11 PM, Otto Fowler
wrote:
> Do we at least require a admin/super user? See’s all the queues and jobs?
>
>
> On May 11, 2018 at 17:03:34, Ryan Merriman (merrim...@gmail.com) wrote:
>
> Thanks everyone for the
Do we at least require a admin/super user? See’s all the queues and jobs?
On May 11, 2018 at 17:03:34, Ryan Merriman (merrim...@gmail.com) wrote:
Thanks everyone for the input and feedback. I will attempt to summarize so
we can come to a consensus and get this tasked out.
The following endpoin
Thanks everyone for the input and feedback. I will attempt to summarize so
we can come to a consensus and get this tasked out.
The following endpoints will be included:
- GET /api/v1/pcap/metadata?basePath - This endpoint will return
metadata of pcap data stored in HDFS. This would includ
On the sharing and securing points, it seems like having the result of a
query run hang around in HDFS on a given URL would work, we can then use
Ranger policies (RBAC, or even ABAC later) to control access, this also
solves the page storage problem, and gives us a kind of two step approach,
both o
Don’t lose the use case for manually uploading PCAPS for analysis Jon.
On May 11, 2018 at 10:14:02, zeo...@gmail.com (zeo...@gmail.com) wrote:
I think baby steps are fine - admin gets access to all, otherwise you only
see your own pcaps, but we file a jira for a future add of API security,
which
I think baby steps are fine - admin gets access to all, otherwise you only
see your own pcaps, but we file a jira for a future add of API security,
which more mature SOCs that align with the Metron personas will need.
Jon
On Fri, May 11, 2018, 09:27 Ryan Merriman wrote:
> That's a good point Jo
That's a good point Jon. There are different levels of effort associated
with different options. If we want to allow pcaps to be shared with
specific users, we will need to introduce ACL security in our REST
application using something like the ACL capability that comes with Spring
Security or Ra
At the very least there needs to be the ability to share downloaded PCAPs
with other users and/or have roles that can see all pcaps. A platform
engineer may want to clean up old pcaps after x time, or a manger may ask
an analyst to find all of the traffic that exhibits xyz behavior, dump a
pcap, a
Mike, I believe the /pcapGetter/getPcapsByIdentifiers endpoint exposes the
fixed query option which we have covered. I agree with you that
deprecating the metron-api module should be a goal of this feature.
On Wed, May 9, 2018 at 1:36 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:
>
Security is another important topic related to our pcap architecture. This
may spill over into a more general, system-wide discussion and we can start
a separate thread for that if necessary.
I'm assuming we want to manage pcap queries by user. One important
question is which user do we use to s
This looks like a pretty good start Ryan. Does the metadata endpoint cover
this https://github.com/apache/metron/tree/master/
metron-platform/metron-api#the-pcapgettergetpcapsbyidentifiers-endpoint
from the original metron-api? If so, then we would be able to deprecate the
existing metron-api proje
Thanks for the feedback Jon. I'm am not as familiar with BPF filtering as
you probably are. Do you have an idea of much effort would be involved in
implementing this? I suspect this would be another PcapFilter (
https://github.com/apache/metron/blob/master/metron-platform/metron-pcap/src/main/ja
This looks really great and gets me excited to maybe revisit some old
conversations about PCAP capture in Metron. The only thing that I think
it's missing is the ability to filter using bpf. I think the same thing
can technically be accomplished by using packet_filter and I wouldn't throw
a fit i
Now that we are confident we can run submit a MR job from our current REST
application, is this the desired approach? Just want to confirm.
Next I think we should map out what the REST interface will look like.
Here are the endpoints I'm thinking about:
GET /api/v1/pcap/metadata?basePath
This e
Sweet! That's great news. The pom changes are a lot simpler than I
expected. Very nice.
On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman wrote:
> Finally figured it out. Commit is here:
> https://github.com/merrimanr/incubator-metron/commit/
> 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
>
> It came d
Finally figured it out. Commit is here:
https://github.com/merrimanr/incubator-metron/commit/22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
It came down to figuring out the right combination of maven dependencies
and passing in the HDP version to REST as a Java system property. I also
included some HD
@Ryan - pulled your branch and experimented with a few things. In doing so,
it dawned on me that by adding the yarn and hadoop classpath, you probably
didn't introduce a new classpath issue, rather you probably just moved onto
the next classpath issue, ie hbase per your exception about hbase jaxb.
That would be a step closer to something more like a micro-service
architecture. However, I would want to make sure we think about the
operational complexity, and mpack implications of having another server
installed and running somewhere on the cluster (also, ssl, kerberos, etc
etc requirements fo
+1 to having metron-api as it's own service and using a gateway type
pattern.
On Tue, May 8, 2018 at 8:13 AM, Otto Fowler wrote:
> Why not have metron-api as it’s own service and use a ‘gateway’ type
> pattern in rest?
>
>
> On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com) wrote:
Why not have metron-api as it’s own service and use a ‘gateway’ type
pattern in rest?
On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com) wrote:
Moving the yarn classpath command earlier in the classpath now gives this
error:
Caused by: java.lang.NoSuchMethodError:
javax.servlet.Ser
Moving the yarn classpath command earlier in the classpath now gives this
error:
Caused by: java.lang.NoSuchMethodError:
javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;
I will experiment with other combinations, I suspect we will need
finer-grain control over the order.
The
What order did you add the hadoop or yarn classpath? The "shaded" package
stands out to me in this name "org.apache.hadoop.hbase.*shaded*
.org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding
those packages earlier on the classpath.
I think that find command needs a "jar tvf", oth
To explore the idea of merging metron-api into metron-rest and running pcap
queries inside our REST application, I created a simple test here:
https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test. A
summary of what's included:
- Added pcap as a dependency in the metron-rest pom.xm
I can imagine a new generic service(s) capability whose job ( pun intended
) is to
abstract the submittal, tracking, and storage of results to yarn.
It would be extended with storage providers, queue provider, possibly some
set of policies or rather strategies.
The pcap ‘report’ would be a client
RE: Tracking v. users
The submittal and tracking can associate the submitter with the yarn job
and track that,
regardless of the yarn credentials.
IE> if all submittals and monitoring are by the same yarn user ( Metron )
from a single or
co-operative set of services, that service can maintain the
That sounds fine - I'd imagine we'd be looking to hit the classpath related
problems asap when merging the modules.
For the module, we just have a pom that supplies external dependencies.
Rather than every metron module depending on Guava or Jackson directly, or
via transitive dependencies, we spe
Otto, your use case makes sense to me. We'll have to think about how to
manage the user to job relationships. I'm assuming YARN jobs will be
submitted as the metron service user so YARN won't keep track of this for
us. Is that assumption correct? Do you have any ideas for doing that?
Mike, I c
>From my response on the other thread, but applicable to the backend stuff:
"The PCAP Query seems more like PCAP Report to me. You are generating a
report based on parameters.
That report is something that takes some time and external process to
generate… ie you have to wait for it.
I can almost
I started a separate thread on Pcap UI considerations and user requirements
at Otto's request. This should help us keep these two related but separate
discussions focused.
On Fri, May 4, 2018 at 7:19 AM, Michel Sumbul
wrote:
> Hello,
>
>
>
> (Youhouuu my first reply on this kind of mail chain^^
Hello,
(Youhouuu my first reply on this kind of mail chain^^)
If I may, I would like to share my view on the following 3 points.
- Backend:
The current metron-api is totally seperate, it will be logic for me to have
it at the same place as the others rest api. Especially when more security
I know, I was running with it :)
> On May 3, 2018, at 10:21 PM, Michael Miklavcic
> wrote:
>
> Tabs vs spaces was a Silicon Valley joke, man :-)
>
>> On Thu, May 3, 2018, 8:42 PM Ryan Merriman wrote:
>>
>> Mike,
>>
>> I never said there was anything problematic in metron-api, just that is w
Tabs vs spaces was a Silicon Valley joke, man :-)
On Thu, May 3, 2018, 8:42 PM Ryan Merriman wrote:
> Mike,
>
> I never said there was anything problematic in metron-api, just that is was
> inconsistent with the rest of Metron. There is work involved in making it
> consistent which is why I lis
Mike,
I never said there was anything problematic in metron-api, just that is was
inconsistent with the rest of Metron. There is work involved in making it
consistent which is why I listed it as a downside. I'm less concerned with
whether we use tabs or spaces but that we use one or the other.
Yes, completely agreed. We're on the same page.
On Thu, May 3, 2018 at 7:50 PM, Otto Fowler wrote:
> I think my point is that maybe we should have a discuss about:
>
> * PCAP UI, goals etc
> * Where it would live and why, what that would mean etc
> * Backend ( this original mail )
>
>
>
> On May
I think my point is that maybe we should have a discuss about:
* PCAP UI, goals etc
* Where it would live and why, what that would mean etc
* Backend ( this original mail )
On May 3, 2018 at 18:34:00, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:
Otto, what are you and your customers
Otto, what are you and your customers finding useful and/or difficult from
a split management/alerts UI perspective? It might help us to restate the
original scope and intent around maintaining separate management and alert
UI's, to your point about "contrary to previous direction." I personally
do
Comments inline below.
On Thu, May 3, 2018 at 3:25 PM, Ryan Merriman wrote:
> Otto,
>
> I'm assuming just adding it to the Alerts UI is less work but I wouldn't be
> strongly opposed to it being it's own UI. What are the reasons for doing
> that?
>
> I don't know that we should split them up.
If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t the
alerts ui anymore.
It is becoming more of a “composite” app, with multiple feature ui’s
together. I didn’t think that
was what we were going for, thus the config ui and the alert ui.
Just adding disparate thing as ‘new tabs
Otto,
I'm assuming just adding it to the Alerts UI is less work but I wouldn't be
strongly opposed to it being it's own UI. What are the reasons for doing
that?
Mike,
On using metron-api:
1. I'm making an assumption about it not being used much. Maybe it
still works without issue. I ag
First thought is why the Alerts-UI and Not a dedicated Query UI?
On May 3, 2018 at 14:36:04, Ryan Merriman (merrim...@gmail.com) wrote:
We are planning on adding the pcap query feature to the Alerts UI. Before
we start this work, I think it is important to get community buy in on the
architectu
Thanks for the write-up, Ryan. A few questions and comments.
1. metron-api
1. "It hasn't been used in a while and will need some end to end
testing to make sure it still functions properly" > I was probably
one of the last developers to touch this code a year or more ago
- fwi
41 matches
Mail list logo
