Re: [DISCUSS] Release?

[Michael Miklavcic]

Nick, I just created a Jira for this and included a permalink to the
original DISCUSS thread.

https://issues.apache.org/jira/browse/METRON-1550


On Wed, May 9, 2018 at 12:36 PM, Nick Allen  wrote:

> IMO, It would be nice to have, but I don't consider it a blocker for the
> release.  Of course, if its something that we can knock out soon (this
> week?), then there would be no reason not to include it.
>
> Did you create a JIRA for this one so we can track it?
>
> On Wed, May 9, 2018 at 2:16 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > One item we haven't gotten around to was redoing the index names to use a
> > metron_ prefix. I'm the one that pushed the original DISCUSS thread on
> > this, but haven't had a chance to advance it. Does anyone have any strong
> > opinions on it? I originally thought it made sense to include alongside
> the
> > other major ES and Solr changes.
> >
> > On Wed, May 9, 2018 at 12:13 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > I'm also a +1 on 0.5.0. This is a fairly big release.
> > >
> > > On Wed, May 9, 2018 at 12:05 PM, Nick Allen 
> wrote:
> > >
> > >> +1 to 0.5.0
> > >>
> > >> On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com 
> > >> wrote:
> > >>
> > >> > I agree that it's probably time (more likely, overdue) for a
> release.
> > >> > Based off of looking at all of those changes I would also suggest
> > going
> > >> to
> > >> > at least 0.5.x.
> > >> >
> > >> > It probably makes sense to take a look at Upgrading.md
> > >> >  (and
> > >> related
> > >> > docs) to make sure it's accurate as well.
> > >> >
> > >> > Jon
> > >> >
> > >> > On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
> > >> > michael.miklav...@gmail.com> wrote:
> > >> >
> > >> > > Good call - I thought that made our last release, but this would
> be
> > >> the
> > >> > 2nd
> > >> > > follow-on from when Nick originally posed a breakdown
> > >> > > 4 months ago METRON-939: Upgrade ElasticSearch and Kibana
> (mmiklavc
> > >> via
> > >> > > mmiklavc) closes apache/metron#840
> > >> > >
> > >> > >
> > >> > > https://lists.apache.org/thread.html/
> 01fb18dd0ee10845588c0c1a4b3f2f
> > >> > 36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
> > >> > >
> > >> > > On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com <
> zeo...@gmail.com
> > >
> > >> > > wrote:
> > >> > >
> > >> > > > We should also mention the Upgrade of ElasticSearch and Kibana
> > >> > > >
> > >> > > > Jon
> > >> > > >
> > >> > > > On Wed, May 9, 2018 at 12:49 PM Nick Allen 
> > >> wrote:
> > >> > > >
> > >> > > > > Oh, and also the Solr work that is currently in a feature
> > >> branch.  We
> > >> > > > would
> > >> > > > > have to get the work finished up and merged though.  Sounds
> like
> > >> we
> > >> > are
> > >> > > > > real close on that.
> > >> > > > >
> > >> > > > > On Wed, May 9, 2018 at 12:47 PM, Nick Allen <
> n...@nickallen.org
> > >
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > The next part of the conversation would be, what should the
> > >> version
> > >> > > > > number
> > >> > > > > > be?  To help with that, I have tried to summarize the
> changes
> > in
> > >> > the
> > >> > > > > > release.  Of course, this is going to be heavily biased
> > towards
> > >> my
> > >> > > own
> > >> > > > > > interests, so please feel free to chime in if I have missed
> > >> > anything.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >- Significant performance improvements in Parsing,
> > >> Enrichments,
> > >> > > and
> > >> > > > > >Indexing
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >- Support for Ubuntu installations
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >- Support for Storm 1.1 and HDP 2.6
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >- Event time processing in the Profiler
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >- Complex document support in the JSONMapParser
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >- Support for the Elasticsearch X-Pack
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >- Run Stellar in a Zeppelin Notebook
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >- Oodles of bug fixes and usability improvements
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Wed, May 9, 2018 at 12:14 PM, Nick Allen <
> > n...@nickallen.org
> > >> >
> > >> > > > wrote:
> > >> > > > > >
> > >> > > > > >> Something like this might be more digestible for these
> > >> purposes.
> > >> > > > > >>
> > >> > > > > >> $git log --pretty="%cr %s" tags/apache-metron-0.4.2-relea
> > >> se..HEAD
> > >> > > > > >>
> > >> > > > > >> 88 minutes ago METRON-1530 Default proxy config settings in
> > >> > > > > >> metron-contrib need to be updated (sardell via merrimanr)
> > >> closes
> > >> > > > > >> apache/metron#998
> > >> > > > > >> 5 days ago METRON-1545 Upgrade Spring and Spring Boot
> > >> (merrimanr)
> > >> > > > closes
> > >> >

Re: [DISCUSS] Pcap panel architecture

[zeo...@gmail.com]

At the very least there needs to be the ability to share downloaded PCAPs
with other users and/or have roles that can see all pcaps.  A platform
engineer may want to clean up old pcaps after x time, or a manger may ask
an analyst to find all of the traffic that exhibits xyz behavior, dump a
pcap, and then point him to it so the manager can review.  Since the
pcap may be huge, we wouldn't want to try to push people to sending it via
email, uploading to a file server, finding an external hard drive, etc.

Jon

On Thu, May 10, 2018 at 10:16 AM Ryan Merriman  wrote:

> Mike, I believe the /pcapGetter/getPcapsByIdentifiers endpoint exposes the
> fixed query option which we have covered.  I agree with you that
> deprecating the metron-api module should be a goal of this feature.
>
> On Wed, May 9, 2018 at 1:36 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > This looks like a pretty good start Ryan. Does the metadata endpoint
> cover
> > this https://github.com/apache/metron/tree/master/
> > metron-platform/metron-api#the-pcapgettergetpcapsbyidentifiers-endpoint
> > from the original metron-api? If so, then we would be able to deprecate
> the
> > existing metron-api project. If we later go to micro-services, a pcap
> > module would spin back into the fold, but it would probably look
> different
> > from metron-api.
> >
> > I commented on the UI thread, but to reiterate for the purpose of backend
> > functionality here I don't believe there is a way to "PAUSE" or "SUSPEND"
> > jobs. That said, I think GET /api/v1/pcap/stop/ is sufficient for
> > the job management operations.
> >
> > On Wed, May 9, 2018 at 11:00 AM, Ryan Merriman 
> > wrote:
> >
> > > Now that we are confident we can run submit a MR job from our current
> > REST
> > > application, is this the desired approach?  Just want to confirm.
> > >
> > > Next I think we should map out what the REST interface will look like.
> > > Here are the endpoints I'm thinking about:
> > >
> > > GET /api/v1/pcap/metadata?basePath
> > >
> > > This endpoint will return metadata of pcap data stored in HDFS.  This
> > would
> > > include pcap size, date ranges (how far back can I go), etc.  It would
> > > accept an optional HDFS basePath parameter for cases where pcap data is
> > > stored in multiple places and/or different from the default location.
> > >
> > > POST /api/v1/pcap/query
> > >
> > > This endpoint would accept a pcap request, submit a pcap query job, and
> > > return a job id.  The request would be an object containing the
> > parameters
> > > documented here:  https://github.com/apache/metron/tree/master/
> > > metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
> > > would be associated with a user that submits it.  An exception will be
> > > returned for violating constraints like too many queries submitted,
> query
> > > parameters out of limits, etc.
> > >
> > > GET /api/v1/pcap/status/
> > >
> > > This endpoint will return the status of a running job.  I imagine this
> is
> > > just a proxy to the YARN REST api.  We can discuss the implementation
> > > behind these endpoints later.
> > >
> > > GET /api/v1/pcap/stop/
> > >
> > > This endpoint would kill a running pcap job.  If the job has already
> > > completed this is a noop.
> > >
> > > GET /api/v1/pcap/list
> > >
> > > This endpoint will list a user's submitted pcap queries.  Items in the
> > list
> > > would contain job id, status (is it finished?), start/end time, and
> > number
> > > of pages.  Maybe there is some overlap with the status endpoint above
> and
> > > the status endpoint is not needed?
> > >
> > > GET /api/v1/pcap/pdml//
> > >
> > > This endpoint will return pcap results for the given page in pdml
> format
> > (
> > > https://wiki.wireshark.org/PDML).  Are there other formats we want to
> > > support?
> > >
> > > GET /api/v1/pcap/raw//
> > >
> > > This endpoint will allow a user to download raw pcap results for the
> > given
> > > page.
> > >
> > > DELETE /api/v1/pcap/
> > >
> > > This endpoint will delete pcap query results.  Not sure yet how this
> fits
> > > in with our broader cleanup strategy.
> > >
> > > This should get us started.  What did I miss and what would you change
> > > about these?  I did not include much detail related to security,
> cleanup
> > > strategy, or underlying implementation details but these are items we
> > > should discuss at some point.
> > >
> > > On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Sweet! That's great news. The pom changes are a lot simpler than I
> > > > expected. Very nice.
> > > >
> > > > On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman 
> > > wrote:
> > > >
> > > > > Finally figured it out.  Commit is here:
> > > > > https://github.com/merrimanr/incubator-metron/commit/
> > > > > 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
> > > > >
> > > > > It came down to figuring out the right combination of maven
> > > dependencies
> > > > > and passing i

Re: [DISCUSS] Release Manager

[Matt Foley]

Guys, heartfelt thanks for the appreciation.  I'm really sorry I can't keep 
doing it, currently buried by unrelated work.
Thanks,
--Matt

On 5/10/18, 10:34 AM, "Nick Allen"  wrote:

+1 to Justin

And many thanks to Matt and all his hard work.

On Thu, May 10, 2018 at 12:13 PM, Ryan Merriman  wrote:

> Yes +1 to Justin being RM.  Thank you for taking that on.
>
> On Thu, May 10, 2018 at 11:08 AM, Casey Stella  wrote:
>
> > I'm +1 to Justin being RM; he's going to have big shoes to fill with 
Matt
> > gone. ;) Also, if it wasn't obvious, deep and hearty thanks to Matt 
again
> > for being our RM.
> >
> > On Thu, May 10, 2018 at 12:06 PM Ryan Merriman 
> > wrote:
> >
> > > Thanks for all your help Matt.
> > >
> > > On Thu, May 10, 2018 at 10:53 AM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Thanks Matt for doing this for the community.
> > > >
> > > > Justin Leet as new lord commander of the Night's Watch? Aye, dilly,
> > > dilly.
> > > >
> > > > On Thu, May 10, 2018 at 9:07 AM, Justin Leet 
> > > > wrote:
> > > >
> > > > > I'd be happy to to volunteer to take over for a while.
> > > > >
> > > > > Thanks to Matt for all the help through the last couple releases!
> > > > >
> > > > > Justin
> > > > >
> > > > > On Thu, May 10, 2018 at 11:06 AM, Casey Stella  >
> > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Matt Foley, our esteemed Release manager for the last couple
> > > releases,
> > > > > has
> > > > > > asked to be relieved.  So, I'm calling on volunteers for the 
next
> > > > release
> > > > > > manager.  It should be a committer and there are a few things
> that
> > > > > require
> > > > > > a PMC member, I believe, but the release manager can ask for 
help
> > > from
> > > > a
> > > > > > PMC member.
> > > > > >
> > > > > > So, Matt's watch has ended, who wants to volunteer?
> > > > > >
> > > > > > Casey
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Release Manager

[Nick Allen]

+1 to Justin

And many thanks to Matt and all his hard work.

On Thu, May 10, 2018 at 12:13 PM, Ryan Merriman  wrote:

> Yes +1 to Justin being RM.  Thank you for taking that on.
>
> On Thu, May 10, 2018 at 11:08 AM, Casey Stella  wrote:
>
> > I'm +1 to Justin being RM; he's going to have big shoes to fill with Matt
> > gone. ;) Also, if it wasn't obvious, deep and hearty thanks to Matt again
> > for being our RM.
> >
> > On Thu, May 10, 2018 at 12:06 PM Ryan Merriman 
> > wrote:
> >
> > > Thanks for all your help Matt.
> > >
> > > On Thu, May 10, 2018 at 10:53 AM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Thanks Matt for doing this for the community.
> > > >
> > > > Justin Leet as new lord commander of the Night's Watch? Aye, dilly,
> > > dilly.
> > > >
> > > > On Thu, May 10, 2018 at 9:07 AM, Justin Leet 
> > > > wrote:
> > > >
> > > > > I'd be happy to to volunteer to take over for a while.
> > > > >
> > > > > Thanks to Matt for all the help through the last couple releases!
> > > > >
> > > > > Justin
> > > > >
> > > > > On Thu, May 10, 2018 at 11:06 AM, Casey Stella  >
> > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Matt Foley, our esteemed Release manager for the last couple
> > > releases,
> > > > > has
> > > > > > asked to be relieved.  So, I'm calling on volunteers for the next
> > > > release
> > > > > > manager.  It should be a committer and there are a few things
> that
> > > > > require
> > > > > > a PMC member, I believe, but the release manager can ask for help
> > > from
> > > > a
> > > > > > PMC member.
> > > > > >
> > > > > > So, Matt's watch has ended, who wants to volunteer?
> > > > > >
> > > > > > Casey
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Release Manager

[Ryan Merriman]

Yes +1 to Justin being RM.  Thank you for taking that on.

On Thu, May 10, 2018 at 11:08 AM, Casey Stella  wrote:

> I'm +1 to Justin being RM; he's going to have big shoes to fill with Matt
> gone. ;) Also, if it wasn't obvious, deep and hearty thanks to Matt again
> for being our RM.
>
> On Thu, May 10, 2018 at 12:06 PM Ryan Merriman 
> wrote:
>
> > Thanks for all your help Matt.
> >
> > On Thu, May 10, 2018 at 10:53 AM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Thanks Matt for doing this for the community.
> > >
> > > Justin Leet as new lord commander of the Night's Watch? Aye, dilly,
> > dilly.
> > >
> > > On Thu, May 10, 2018 at 9:07 AM, Justin Leet 
> > > wrote:
> > >
> > > > I'd be happy to to volunteer to take over for a while.
> > > >
> > > > Thanks to Matt for all the help through the last couple releases!
> > > >
> > > > Justin
> > > >
> > > > On Thu, May 10, 2018 at 11:06 AM, Casey Stella 
> > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Matt Foley, our esteemed Release manager for the last couple
> > releases,
> > > > has
> > > > > asked to be relieved.  So, I'm calling on volunteers for the next
> > > release
> > > > > manager.  It should be a committer and there are a few things that
> > > > require
> > > > > a PMC member, I believe, but the release manager can ask for help
> > from
> > > a
> > > > > PMC member.
> > > > >
> > > > > So, Matt's watch has ended, who wants to volunteer?
> > > > >
> > > > > Casey
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Release Manager

[Casey Stella]

I'm +1 to Justin being RM; he's going to have big shoes to fill with Matt
gone. ;) Also, if it wasn't obvious, deep and hearty thanks to Matt again
for being our RM.

On Thu, May 10, 2018 at 12:06 PM Ryan Merriman  wrote:

> Thanks for all your help Matt.
>
> On Thu, May 10, 2018 at 10:53 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Thanks Matt for doing this for the community.
> >
> > Justin Leet as new lord commander of the Night's Watch? Aye, dilly,
> dilly.
> >
> > On Thu, May 10, 2018 at 9:07 AM, Justin Leet 
> > wrote:
> >
> > > I'd be happy to to volunteer to take over for a while.
> > >
> > > Thanks to Matt for all the help through the last couple releases!
> > >
> > > Justin
> > >
> > > On Thu, May 10, 2018 at 11:06 AM, Casey Stella 
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Matt Foley, our esteemed Release manager for the last couple
> releases,
> > > has
> > > > asked to be relieved.  So, I'm calling on volunteers for the next
> > release
> > > > manager.  It should be a committer and there are a few things that
> > > require
> > > > a PMC member, I believe, but the release manager can ask for help
> from
> > a
> > > > PMC member.
> > > >
> > > > So, Matt's watch has ended, who wants to volunteer?
> > > >
> > > > Casey
> > > >
> > >
> >
>

Re: [DISCUSS] Release Manager

[Ryan Merriman]

Thanks for all your help Matt.

On Thu, May 10, 2018 at 10:53 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Thanks Matt for doing this for the community.
>
> Justin Leet as new lord commander of the Night's Watch? Aye, dilly, dilly.
>
> On Thu, May 10, 2018 at 9:07 AM, Justin Leet 
> wrote:
>
> > I'd be happy to to volunteer to take over for a while.
> >
> > Thanks to Matt for all the help through the last couple releases!
> >
> > Justin
> >
> > On Thu, May 10, 2018 at 11:06 AM, Casey Stella 
> wrote:
> >
> > > Hi All,
> > >
> > > Matt Foley, our esteemed Release manager for the last couple releases,
> > has
> > > asked to be relieved.  So, I'm calling on volunteers for the next
> release
> > > manager.  It should be a committer and there are a few things that
> > require
> > > a PMC member, I believe, but the release manager can ask for help from
> a
> > > PMC member.
> > >
> > > So, Matt's watch has ended, who wants to volunteer?
> > >
> > > Casey
> > >
> >
>

Re: [DISCUSS] Release Manager

[Michael Miklavcic]

Thanks Matt for doing this for the community.

Justin Leet as new lord commander of the Night's Watch? Aye, dilly, dilly.

On Thu, May 10, 2018 at 9:07 AM, Justin Leet  wrote:

> I'd be happy to to volunteer to take over for a while.
>
> Thanks to Matt for all the help through the last couple releases!
>
> Justin
>
> On Thu, May 10, 2018 at 11:06 AM, Casey Stella  wrote:
>
> > Hi All,
> >
> > Matt Foley, our esteemed Release manager for the last couple releases,
> has
> > asked to be relieved.  So, I'm calling on volunteers for the next release
> > manager.  It should be a committer and there are a few things that
> require
> > a PMC member, I believe, but the release manager can ask for help from a
> > PMC member.
> >
> > So, Matt's watch has ended, who wants to volunteer?
> >
> > Casey
> >
>

Re: [DISCUSS] Release Manager

[Justin Leet]

I'd be happy to to volunteer to take over for a while.

Thanks to Matt for all the help through the last couple releases!

Justin

On Thu, May 10, 2018 at 11:06 AM, Casey Stella  wrote:

> Hi All,
>
> Matt Foley, our esteemed Release manager for the last couple releases, has
> asked to be relieved.  So, I'm calling on volunteers for the next release
> manager.  It should be a committer and there are a few things that require
> a PMC member, I believe, but the release manager can ask for help from a
> PMC member.
>
> So, Matt's watch has ended, who wants to volunteer?
>
> Casey
>

[DISCUSS] Release Manager

[Casey Stella]

Hi All,

Matt Foley, our esteemed Release manager for the last couple releases, has
asked to be relieved.  So, I'm calling on volunteers for the next release
manager.  It should be a committer and there are a few things that require
a PMC member, I believe, but the release manager can ask for help from a
PMC member.

So, Matt's watch has ended, who wants to volunteer?

Casey

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Mike, I believe the /pcapGetter/getPcapsByIdentifiers endpoint exposes the
fixed query option which we have covered.  I agree with you that
deprecating the metron-api module should be a goal of this feature.

On Wed, May 9, 2018 at 1:36 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> This looks like a pretty good start Ryan. Does the metadata endpoint cover
> this https://github.com/apache/metron/tree/master/
> metron-platform/metron-api#the-pcapgettergetpcapsbyidentifiers-endpoint
> from the original metron-api? If so, then we would be able to deprecate the
> existing metron-api project. If we later go to micro-services, a pcap
> module would spin back into the fold, but it would probably look different
> from metron-api.
>
> I commented on the UI thread, but to reiterate for the purpose of backend
> functionality here I don't believe there is a way to "PAUSE" or "SUSPEND"
> jobs. That said, I think GET /api/v1/pcap/stop/ is sufficient for
> the job management operations.
>
> On Wed, May 9, 2018 at 11:00 AM, Ryan Merriman 
> wrote:
>
> > Now that we are confident we can run submit a MR job from our current
> REST
> > application, is this the desired approach?  Just want to confirm.
> >
> > Next I think we should map out what the REST interface will look like.
> > Here are the endpoints I'm thinking about:
> >
> > GET /api/v1/pcap/metadata?basePath
> >
> > This endpoint will return metadata of pcap data stored in HDFS.  This
> would
> > include pcap size, date ranges (how far back can I go), etc.  It would
> > accept an optional HDFS basePath parameter for cases where pcap data is
> > stored in multiple places and/or different from the default location.
> >
> > POST /api/v1/pcap/query
> >
> > This endpoint would accept a pcap request, submit a pcap query job, and
> > return a job id.  The request would be an object containing the
> parameters
> > documented here:  https://github.com/apache/metron/tree/master/
> > metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
> > would be associated with a user that submits it.  An exception will be
> > returned for violating constraints like too many queries submitted, query
> > parameters out of limits, etc.
> >
> > GET /api/v1/pcap/status/
> >
> > This endpoint will return the status of a running job.  I imagine this is
> > just a proxy to the YARN REST api.  We can discuss the implementation
> > behind these endpoints later.
> >
> > GET /api/v1/pcap/stop/
> >
> > This endpoint would kill a running pcap job.  If the job has already
> > completed this is a noop.
> >
> > GET /api/v1/pcap/list
> >
> > This endpoint will list a user's submitted pcap queries.  Items in the
> list
> > would contain job id, status (is it finished?), start/end time, and
> number
> > of pages.  Maybe there is some overlap with the status endpoint above and
> > the status endpoint is not needed?
> >
> > GET /api/v1/pcap/pdml//
> >
> > This endpoint will return pcap results for the given page in pdml format
> (
> > https://wiki.wireshark.org/PDML).  Are there other formats we want to
> > support?
> >
> > GET /api/v1/pcap/raw//
> >
> > This endpoint will allow a user to download raw pcap results for the
> given
> > page.
> >
> > DELETE /api/v1/pcap/
> >
> > This endpoint will delete pcap query results.  Not sure yet how this fits
> > in with our broader cleanup strategy.
> >
> > This should get us started.  What did I miss and what would you change
> > about these?  I did not include much detail related to security, cleanup
> > strategy, or underlying implementation details but these are items we
> > should discuss at some point.
> >
> > On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Sweet! That's great news. The pom changes are a lot simpler than I
> > > expected. Very nice.
> > >
> > > On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman 
> > wrote:
> > >
> > > > Finally figured it out.  Commit is here:
> > > > https://github.com/merrimanr/incubator-metron/commit/
> > > > 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
> > > >
> > > > It came down to figuring out the right combination of maven
> > dependencies
> > > > and passing in the HDP version to REST as a Java system property.  I
> > also
> > > > included some HDFS setup tasks.  I tested this in full dev and can
> now
> > > > successfully run a pcap query and get results.  All you should have
> to
> > do
> > > > is generate some pcap data first.
> > > >
> > > > On Tue, May 8, 2018 at 4:17 PM, Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > @Ryan - pulled your branch and experimented with a few things. In
> > doing
> > > > so,
> > > > > it dawned on me that by adding the yarn and hadoop classpath, you
> > > > probably
> > > > > didn't introduce a new classpath issue, rather you probably just
> > moved
> > > > onto
> > > > > the next classpath issue, ie hbase per your exception about hbase
> > jaxb.
> > > > > Anyhow, I put up a br

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Security is another important topic related to our pcap architecture.  This
may spill over into a more general, system-wide discussion and we can start
a separate thread for that if necessary.

I'm assuming we want to manage pcap queries by user.  One important
question is which user do we use to submit MR jobs?  Right now they are
submitted with the "metron" service user.  If we continue with this
approach (all jobs run as the metron service user) it will require less
Kerberos work and configuration but we will need to manage user to job
relationships in the REST layer.  It would also give us less flexibility
since all assets are permissioned for a single user.  If we decided to have
REST impersonate users and submit jobs that way there would be substantial
work (maybe?) to get to that point since we don't do it now.  We would need
to add LDAP authentication to REST, sync OS users with LDAP, and all the
other stuff that goes along with setting that up.  Maybe we want to do this
anyways in the future.  Has anyone done this before and has a clear
understanding of what's involved?  If this were the ideal approach we need
to think about how we get to that point.  Managing user to job
relationships in REST would be throwaway in that case since that
information would now be stored in YARN.

For authorization I'm assuming that each user should only have access to
information about their queries and query results.  Any actions like
downloading results or cleaning up queries (deleting results) would also be
limited to that user.  Does this sound reasonable?  Do we want to add an
admin role that can do everything?  Is there anything anyone else wants to
discuss with regards to authorization or security in general?

On Wed, May 9, 2018 at 1:22 PM, Ryan Merriman  wrote:

> Thanks for the feedback Jon.  I'm am not as familiar with BPF filtering as
> you probably are.  Do you have an idea of much effort would be involved in
> implementing this?  I suspect this would be another PcapFilter (
> https://github.com/apache/metron/blob/master/metron-
> platform/metron-pcap/src/main/java/org/apache/metron/pcap/
> filter/PcapFilter.java) similar to fixed and query so regardless of when
> we decide to add it our endpoint strategy should support 1 to n filters.
> Maybe we have an endpoint for each type of filter:
>
> POST /api/v1/pcap/fixed
> POST /api/v1/pcap/query
> POST /api/v1/pcap/bpf
>
> This would allow us to accept requests that are structured differently and
> specific to the type of filter.
>
> On Wed, May 9, 2018 at 12:32 PM, zeo...@gmail.com 
> wrote:
>
>> This looks really great and gets me excited to maybe revisit some old
>> conversations about PCAP capture in Metron.  The only thing that I think
>> it's missing is the ability to filter using bpf.  I think the same thing
>> can technically be accomplished by using packet_filter and I wouldn't
>> throw
>> a fit if that's considered a follow-on, but bpf is the standard language
>> that people who do packet munging for a living know.
>>
>> Jon
>>
>> On Wed, May 9, 2018 at 1:00 PM Ryan Merriman  wrote:
>>
>> > Now that we are confident we can run submit a MR job from our current
>> REST
>> > application, is this the desired approach?  Just want to confirm.
>> >
>> > Next I think we should map out what the REST interface will look like.
>> > Here are the endpoints I'm thinking about:
>> >
>> > GET /api/v1/pcap/metadata?basePath
>> >
>> > This endpoint will return metadata of pcap data stored in HDFS.  This
>> would
>> > include pcap size, date ranges (how far back can I go), etc.  It would
>> > accept an optional HDFS basePath parameter for cases where pcap data is
>> > stored in multiple places and/or different from the default location.
>> >
>> > POST /api/v1/pcap/query
>> >
>> > This endpoint would accept a pcap request, submit a pcap query job, and
>> > return a job id.  The request would be an object containing the
>> parameters
>> > documented here:  https://github.com/apache/metron/tree/master/
>> > metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
>> > would be associated with a user that submits it.  An exception will be
>> > returned for violating constraints like too many queries submitted,
>> query
>> > parameters out of limits, etc.
>> >
>> > GET /api/v1/pcap/status/
>> >
>> > This endpoint will return the status of a running job.  I imagine this
>> is
>> > just a proxy to the YARN REST api.  We can discuss the implementation
>> > behind these endpoints later.
>> >
>> > GET /api/v1/pcap/stop/
>> >
>> > This endpoint would kill a running pcap job.  If the job has already
>> > completed this is a noop.
>> >
>> > GET /api/v1/pcap/list
>> >
>> > This endpoint will list a user's submitted pcap queries.  Items in the
>> list
>> > would contain job id, status (is it finished?), start/end time, and
>> number
>> > of pages.  Maybe there is some overlap with the status endpoint above
>> and
>> > the status endpoint is not needed?
>> >
>> > GET /a

Re: [DISCUSS] Release?

[Michael Miklavcic]

If we're going to put Solr in the next release I think the index name
change can wait for that release as well.

On Thu, May 10, 2018 at 7:09 AM, Nick Allen  wrote:

> > I tend to like grouping the es changes into one release (i.e. include
> the index
> name change) and solr into another (next release).
>
> Is anyone willing to volunteer to do the work for the index name change?
>
> If there are no takers, I think we need to move on and cut a release.
>
>
>
>
> On Thu, May 10, 2018 at 8:43 AM, zeo...@gmail.com 
> wrote:
>
> > I tend to like grouping the es changes into one release (i.e. include the
> > index name change) and solr into another (next release).
> >
> > I think we go too long between releases myself and wouldn't be against
> > doing two releases just a couple of months apart.
> >
> > Jon
> >
> > On Wed, May 9, 2018, 14:47 Michael Miklavcic <
> michael.miklav...@gmail.com>
> > wrote:
> >
> > > I don't have a strong opinion. The ES upgrade alone is a massive
> feature.
> > > It could make it easier to include the index change I mentioned along
> > with
> > > Solr as a follow-up. I think if we did split, we could arguably start
> on
> > > the next release with Solr almost immediately.
> > >
> > > On Wed, May 9, 2018, 12:40 PM Nick Allen  wrote:
> > >
> > > > Simon brought up the idea of including the Solr enhancements
> (currently
> > > in
> > > > a feature branch) for the release.  What are people's opinions on
> this?
> > > Is
> > > > this something that is a blocker for the release?
> > > >
> > > > IMO, there is so much already in master waiting to be released that I
> > > don't
> > > > see a need to include it in the next release.  I'd be happy to see
> that
> > > > Solr work drive a follow-on release.
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 9, 2018 at 2:16 PM, Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > One item we haven't gotten around to was redoing the index names to
> > > use a
> > > > > metron_ prefix. I'm the one that pushed the original DISCUSS thread
> > on
> > > > > this, but haven't had a chance to advance it. Does anyone have any
> > > strong
> > > > > opinions on it? I originally thought it made sense to include
> > alongside
> > > > the
> > > > > other major ES and Solr changes.
> > > > >
> > > > > On Wed, May 9, 2018 at 12:13 PM, Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > > > I'm also a +1 on 0.5.0. This is a fairly big release.
> > > > > >
> > > > > > On Wed, May 9, 2018 at 12:05 PM, Nick Allen 
> > > > wrote:
> > > > > >
> > > > > >> +1 to 0.5.0
> > > > > >>
> > > > > >> On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com <
> > zeo...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I agree that it's probably time (more likely, overdue) for a
> > > > release.
> > > > > >> > Based off of looking at all of those changes I would also
> > suggest
> > > > > going
> > > > > >> to
> > > > > >> > at least 0.5.x.
> > > > > >> >
> > > > > >> > It probably makes sense to take a look at Upgrading.md
> > > > > >> > 
> > (and
> > > > > >> related
> > > > > >> > docs) to make sure it's accurate as well.
> > > > > >> >
> > > > > >> > Jon
> > > > > >> >
> > > > > >> > On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
> > > > > >> > michael.miklav...@gmail.com> wrote:
> > > > > >> >
> > > > > >> > > Good call - I thought that made our last release, but this
> > would
> > > > be
> > > > > >> the
> > > > > >> > 2nd
> > > > > >> > > follow-on from when Nick originally posed a breakdown
> > > > > >> > > 4 months ago METRON-939: Upgrade ElasticSearch and Kibana
> > > > (mmiklavc
> > > > > >> via
> > > > > >> > > mmiklavc) closes apache/metron#840
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f
> > > > > >> > 36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
> > > > > >> > >
> > > > > >> > > On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com <
> > > > zeo...@gmail.com
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > We should also mention the Upgrade of ElasticSearch and
> > Kibana
> > > > > >> > > >
> > > > > >> > > > Jon
> > > > > >> > > >
> > > > > >> > > > On Wed, May 9, 2018 at 12:49 PM Nick Allen <
> > > n...@nickallen.org>
> > > > > >> wrote:
> > > > > >> > > >
> > > > > >> > > > > Oh, and also the Solr work that is currently in a
> feature
> > > > > >> branch.  We
> > > > > >> > > > would
> > > > > >> > > > > have to get the work finished up and merged though.
> > Sounds
> > > > like
> > > > > >> we
> > > > > >> > are
> > > > > >> > > > > real close on that.
> > > > > >> > > > >
> > > > > >> > > > > On Wed, May 9, 2018 at 12:47 PM, Nick Allen <
> > > > n...@nickallen.org
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > The next part of the conversation would be, what
> should
> > > the
> > > > > >> ver

Re: [DISCUSS] Release?

[Nick Allen]

> I tend to like grouping the es changes into one release (i.e. include the 
> index
name change) and solr into another (next release).

Is anyone willing to volunteer to do the work for the index name change?

If there are no takers, I think we need to move on and cut a release.




On Thu, May 10, 2018 at 8:43 AM, zeo...@gmail.com  wrote:

> I tend to like grouping the es changes into one release (i.e. include the
> index name change) and solr into another (next release).
>
> I think we go too long between releases myself and wouldn't be against
> doing two releases just a couple of months apart.
>
> Jon
>
> On Wed, May 9, 2018, 14:47 Michael Miklavcic 
> wrote:
>
> > I don't have a strong opinion. The ES upgrade alone is a massive feature.
> > It could make it easier to include the index change I mentioned along
> with
> > Solr as a follow-up. I think if we did split, we could arguably start on
> > the next release with Solr almost immediately.
> >
> > On Wed, May 9, 2018, 12:40 PM Nick Allen  wrote:
> >
> > > Simon brought up the idea of including the Solr enhancements (currently
> > in
> > > a feature branch) for the release.  What are people's opinions on this?
> > Is
> > > this something that is a blocker for the release?
> > >
> > > IMO, there is so much already in master waiting to be released that I
> > don't
> > > see a need to include it in the next release.  I'd be happy to see that
> > > Solr work drive a follow-on release.
> > >
> > >
> > >
> > >
> > > On Wed, May 9, 2018 at 2:16 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > One item we haven't gotten around to was redoing the index names to
> > use a
> > > > metron_ prefix. I'm the one that pushed the original DISCUSS thread
> on
> > > > this, but haven't had a chance to advance it. Does anyone have any
> > strong
> > > > opinions on it? I originally thought it made sense to include
> alongside
> > > the
> > > > other major ES and Solr changes.
> > > >
> > > > On Wed, May 9, 2018 at 12:13 PM, Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > I'm also a +1 on 0.5.0. This is a fairly big release.
> > > > >
> > > > > On Wed, May 9, 2018 at 12:05 PM, Nick Allen 
> > > wrote:
> > > > >
> > > > >> +1 to 0.5.0
> > > > >>
> > > > >> On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com <
> zeo...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > I agree that it's probably time (more likely, overdue) for a
> > > release.
> > > > >> > Based off of looking at all of those changes I would also
> suggest
> > > > going
> > > > >> to
> > > > >> > at least 0.5.x.
> > > > >> >
> > > > >> > It probably makes sense to take a look at Upgrading.md
> > > > >> > 
> (and
> > > > >> related
> > > > >> > docs) to make sure it's accurate as well.
> > > > >> >
> > > > >> > Jon
> > > > >> >
> > > > >> > On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
> > > > >> > michael.miklav...@gmail.com> wrote:
> > > > >> >
> > > > >> > > Good call - I thought that made our last release, but this
> would
> > > be
> > > > >> the
> > > > >> > 2nd
> > > > >> > > follow-on from when Nick originally posed a breakdown
> > > > >> > > 4 months ago METRON-939: Upgrade ElasticSearch and Kibana
> > > (mmiklavc
> > > > >> via
> > > > >> > > mmiklavc) closes apache/metron#840
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f
> > > > >> > 36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
> > > > >> > >
> > > > >> > > On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com <
> > > zeo...@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > We should also mention the Upgrade of ElasticSearch and
> Kibana
> > > > >> > > >
> > > > >> > > > Jon
> > > > >> > > >
> > > > >> > > > On Wed, May 9, 2018 at 12:49 PM Nick Allen <
> > n...@nickallen.org>
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > > > Oh, and also the Solr work that is currently in a feature
> > > > >> branch.  We
> > > > >> > > > would
> > > > >> > > > > have to get the work finished up and merged though.
> Sounds
> > > like
> > > > >> we
> > > > >> > are
> > > > >> > > > > real close on that.
> > > > >> > > > >
> > > > >> > > > > On Wed, May 9, 2018 at 12:47 PM, Nick Allen <
> > > n...@nickallen.org
> > > > >
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > > The next part of the conversation would be, what should
> > the
> > > > >> version
> > > > >> > > > > number
> > > > >> > > > > > be?  To help with that, I have tried to summarize the
> > > changes
> > > > in
> > > > >> > the
> > > > >> > > > > > release.  Of course, this is going to be heavily biased
> > > > towards
> > > > >> my
> > > > >> > > own
> > > > >> > > > > > interests, so please feel free to chime in if I have
> > missed
> > > > >> > anything.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >- Significant performance improvements

Re: [DISCUSS] Release?

[zeo...@gmail.com]

I tend to like grouping the es changes into one release (i.e. include the
index name change) and solr into another (next release).

I think we go too long between releases myself and wouldn't be against
doing two releases just a couple of months apart.

Jon

On Wed, May 9, 2018, 14:47 Michael Miklavcic 
wrote:

> I don't have a strong opinion. The ES upgrade alone is a massive feature.
> It could make it easier to include the index change I mentioned along with
> Solr as a follow-up. I think if we did split, we could arguably start on
> the next release with Solr almost immediately.
>
> On Wed, May 9, 2018, 12:40 PM Nick Allen  wrote:
>
> > Simon brought up the idea of including the Solr enhancements (currently
> in
> > a feature branch) for the release.  What are people's opinions on this?
> Is
> > this something that is a blocker for the release?
> >
> > IMO, there is so much already in master waiting to be released that I
> don't
> > see a need to include it in the next release.  I'd be happy to see that
> > Solr work drive a follow-on release.
> >
> >
> >
> >
> > On Wed, May 9, 2018 at 2:16 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > One item we haven't gotten around to was redoing the index names to
> use a
> > > metron_ prefix. I'm the one that pushed the original DISCUSS thread on
> > > this, but haven't had a chance to advance it. Does anyone have any
> strong
> > > opinions on it? I originally thought it made sense to include alongside
> > the
> > > other major ES and Solr changes.
> > >
> > > On Wed, May 9, 2018 at 12:13 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > I'm also a +1 on 0.5.0. This is a fairly big release.
> > > >
> > > > On Wed, May 9, 2018 at 12:05 PM, Nick Allen 
> > wrote:
> > > >
> > > >> +1 to 0.5.0
> > > >>
> > > >> On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com 
> > > >> wrote:
> > > >>
> > > >> > I agree that it's probably time (more likely, overdue) for a
> > release.
> > > >> > Based off of looking at all of those changes I would also suggest
> > > going
> > > >> to
> > > >> > at least 0.5.x.
> > > >> >
> > > >> > It probably makes sense to take a look at Upgrading.md
> > > >> >  (and
> > > >> related
> > > >> > docs) to make sure it's accurate as well.
> > > >> >
> > > >> > Jon
> > > >> >
> > > >> > On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
> > > >> > michael.miklav...@gmail.com> wrote:
> > > >> >
> > > >> > > Good call - I thought that made our last release, but this would
> > be
> > > >> the
> > > >> > 2nd
> > > >> > > follow-on from when Nick originally posed a breakdown
> > > >> > > 4 months ago METRON-939: Upgrade ElasticSearch and Kibana
> > (mmiklavc
> > > >> via
> > > >> > > mmiklavc) closes apache/metron#840
> > > >> > >
> > > >> > >
> > > >> > >
> > https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f
> > > >> > 36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
> > > >> > >
> > > >> > > On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com <
> > zeo...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > We should also mention the Upgrade of ElasticSearch and Kibana
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > > On Wed, May 9, 2018 at 12:49 PM Nick Allen <
> n...@nickallen.org>
> > > >> wrote:
> > > >> > > >
> > > >> > > > > Oh, and also the Solr work that is currently in a feature
> > > >> branch.  We
> > > >> > > > would
> > > >> > > > > have to get the work finished up and merged though.  Sounds
> > like
> > > >> we
> > > >> > are
> > > >> > > > > real close on that.
> > > >> > > > >
> > > >> > > > > On Wed, May 9, 2018 at 12:47 PM, Nick Allen <
> > n...@nickallen.org
> > > >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > The next part of the conversation would be, what should
> the
> > > >> version
> > > >> > > > > number
> > > >> > > > > > be?  To help with that, I have tried to summarize the
> > changes
> > > in
> > > >> > the
> > > >> > > > > > release.  Of course, this is going to be heavily biased
> > > towards
> > > >> my
> > > >> > > own
> > > >> > > > > > interests, so please feel free to chime in if I have
> missed
> > > >> > anything.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >- Significant performance improvements in Parsing,
> > > >> Enrichments,
> > > >> > > and
> > > >> > > > > >Indexing
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >- Support for Ubuntu installations
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >- Support for Storm 1.1 and HDP 2.6
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >- Event time processing in the Profiler
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >- Complex document support in the JSONMapParser
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >- Support for the Elasticsearch X-Pack
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >- R