Assign METRON-1307 to Brian Hurley and close

[zeo...@gmail.com]

I'm unable to find Brian Hurley in the list of assignees, but he was the
one who contributed the fix[1].  Can someone assign and close this JIRA?
Thanks,

Jon

1:  https://github.com/apache/metron/pull/835
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

The way master's full-dev is set up right now is non optimal for the bro
plugin configuration, and I would like to complete the roadmap I outlined
in my discuss thread before a release.  I have a PR open against
Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
expect it will take me until end of next week at the latest to have the
rest of the work done to move us to 2.5.2, and to pin to a specific package
version.  At that point I'm comfortable with a release.

Jon

On Wed, Nov 15, 2017, 18:57 Matt Foley  wrote:

> I’ve been listening.  Looks like there are still a number of major issues
> to be committed first, right?
> The discussion on this thread constitutes sufficient engagement, I think,
> especially given the Subject line :-)
> Would the folks working on the 6 issues listed by Nick care to suggest a
> cut-off date by which they’ll probably have those fixes in?
> I’ll be happy to run the release process, if the community so wishes, as
> soon as those issues are committed.
>
> I guess I should, pro forma, send the list of commits already in since the
> last release.  I’ll do that today.
> Also, if anyone wishes to raise a hand and propose additional commits are
> needed, please do so on this thread.
> Thanks,
> --Matt
>
>
> On 11/15/17, 2:09 PM, "Casey Stella"  wrote:
>
> I'd say that if a release is this imminent that we had better notify
> the
> release manager who will make a release announcement, Nick.  Matt, are
> you
> tuning in to this?
>
> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen 
> wrote:
>
> > Hi Guys -
> >
> > I want to follow-up on this discussion.  It sounds like most people
> are in
> > agreement with the general approach.
> >
> > A lot of people have been working hard on Metaalerts and
> Elasticsearch.  I
> > have checked-in with those doing the heavy lifting and have compiled
> a more
> > detailed plan based on where we are at now.  To the best of my
> knowledge
> > here is the plan of attack for finishing out this effort.
> >
> >   (1) First, METRON-1289 needs to go in.  This one was a fairly big
> effort
> > and I am hearing that we are pretty close.
> >
> >   (2) METRON-1294 fixes an issue in how field types are looked-up.
> >
> >   (3) METRON-1290 is next.  While this may have been fixed in
> M-1289, there
> > may be some test cases we want from this PR.
> >
> >   (4) METRON-1301 addresses a problem with the sorting logic.
> >
> >   (5) METRON-1291 fixes an issue with escalation of metaalerts.
> >
> >   (6) That leads us to Raghu's UI work in METRON-1252.  This
> introduces the
> > UI bits that depend on all the previous backend work.
> >
> >   (7) At this point, we should have our best effort at running
> Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> >   (8) After we cut the release, we can introduce the work for ES 5.x
> in
> > METRON-939.  I know we will need lots of help testing and reviewing
> this
> > one.
> >
> > Please correct me if I am wrong.  I will try and send out updates as
> we
> > make progress.
> >
> >
> >
> >
> >
> > On Mon, Nov 6, 2017 at 1:03 PM, zeo...@gmail.com 
> wrote:
> >
> > > I agree, I think it's very reasonable to move in line with Nick's
> > > proposal.  I would also suggest that we outline what the target
> versions
> > > would be to add in the METRON-777 components, since it has been
> > functional
> > > for a very long time but not reviewed and has some really rockstar
> > > improvements.
> > >
> > > Jon
> > >
> > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> ottobackwa...@gmail.com>
> > > wrote:
> > >
> > > > I think the ES cutover should be the start of the 0.5.x series,
> and we
> > > > continue on with 0.4.x for the
> > > > metadata improvements etc.  We could chose to focus 0.5.x’s first
> > > releases
> > > > on not only ES but
> > > > getting a handle on kibana and the mpack situation as well.
> > > >
> > > >
> > > >
> > > >
> > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
> > > > michael.miklav...@gmail.com) wrote:
> > > &g

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

My PR is to turn it into a package containing a plugin*

On Thu, Nov 16, 2017, 08:01 zeo...@gmail.com  wrote:

> The way master's full-dev is set up right now is non optimal for the bro
> plugin configuration, and I would like to complete the roadmap I outlined
> in my discuss thread before a release.  I have a PR open against
> Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
> expect it will take me until end of next week at the latest to have the
> rest of the work done to move us to 2.5.2, and to pin to a specific package
> version.  At that point I'm comfortable with a release.
>
> Jon
>
> On Wed, Nov 15, 2017, 18:57 Matt Foley  wrote:
>
>> I’ve been listening.  Looks like there are still a number of major issues
>> to be committed first, right?
>> The discussion on this thread constitutes sufficient engagement, I think,
>> especially given the Subject line :-)
>> Would the folks working on the 6 issues listed by Nick care to suggest a
>> cut-off date by which they’ll probably have those fixes in?
>> I’ll be happy to run the release process, if the community so wishes, as
>> soon as those issues are committed.
>>
>> I guess I should, pro forma, send the list of commits already in since
>> the last release.  I’ll do that today.
>> Also, if anyone wishes to raise a hand and propose additional commits are
>> needed, please do so on this thread.
>> Thanks,
>> --Matt
>>
>>
>> On 11/15/17, 2:09 PM, "Casey Stella"  wrote:
>>
>> I'd say that if a release is this imminent that we had better notify
>> the
>> release manager who will make a release announcement, Nick.  Matt,
>> are you
>> tuning in to this?
>>
>> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen 
>> wrote:
>>
>> > Hi Guys -
>> >
>> > I want to follow-up on this discussion.  It sounds like most people
>> are in
>> > agreement with the general approach.
>> >
>> > A lot of people have been working hard on Metaalerts and
>> Elasticsearch.  I
>> > have checked-in with those doing the heavy lifting and have
>> compiled a more
>> > detailed plan based on where we are at now.  To the best of my
>> knowledge
>> > here is the plan of attack for finishing out this effort.
>> >
>> >   (1) First, METRON-1289 needs to go in.  This one was a fairly big
>> effort
>> > and I am hearing that we are pretty close.
>> >
>> >   (2) METRON-1294 fixes an issue in how field types are looked-up.
>> >
>> >   (3) METRON-1290 is next.  While this may have been fixed in
>> M-1289, there
>> > may be some test cases we want from this PR.
>> >
>> >   (4) METRON-1301 addresses a problem with the sorting logic.
>> >
>> >   (5) METRON-1291 fixes an issue with escalation of metaalerts.
>> >
>> >   (6) That leads us to Raghu's UI work in METRON-1252.  This
>> introduces the
>> > UI bits that depend on all the previous backend work.
>> >
>> >   (7) At this point, we should have our best effort at running
>> Metaalerts
>> > on Elasticsearch 2.x. I propose that we cut a release here.
>> >
>> >   (8) After we cut the release, we can introduce the work for ES
>> 5.x in
>> > METRON-939.  I know we will need lots of help testing and reviewing
>> this
>> > one.
>> >
>> > Please correct me if I am wrong.  I will try and send out updates
>> as we
>> > make progress.
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Nov 6, 2017 at 1:03 PM, zeo...@gmail.com 
>> wrote:
>> >
>> > > I agree, I think it's very reasonable to move in line with Nick's
>> > > proposal.  I would also suggest that we outline what the target
>> versions
>> > > would be to add in the METRON-777 components, since it has been
>> > functional
>> > > for a very long time but not reviewed and has some really rockstar
>> > > improvements.
>> > >
>> > > Jon
>> > >
>> > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
>> ottobackwa...@gmail.com>
>> > > wrote:
>> > >
>> > > > I think the ES cutover should be the start of the 0.5.x series,
>> and 

Re: [DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

I would suggest that we institute a release procedure for the package
itself, but I don't think it necessarily has to line up with metron
releases (happy to be persuaded otherwise).  Then we can just link metron
to metron-bro-plugin-kafka by pointing to specific
metron-bro-plugin-kafka releases (git tags

).

Right now, full-dev spins up against the
apache/metron-bro-plugin-kafka master branch, which is not a good idea to
have in place for an upcoming release.  That is the crux of why I think we
need to finalize the move to bro 2.5.2 and the plugin packaging before our
next release (working on it as we speak).

Jon

On Thu, Nov 16, 2017 at 9:10 AM Nick Allen  wrote:

> The code of the 'Kafka Plugin for Bro' is now maintained in the external
> repository that we set up a while back.  Do we need to change anything in
> the release procedure to account for this?
>
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

The problem is that we're currently pinning to master and if something in
the plugin breaks backward compatibility, our prior releases will be
perpetually broken, which is why I suggest it blocks the upcoming release.

The alternative would be to update ansible to pin to a specific commit or
to make a release in apache/metron-bro-plugin-kafka sooner rather than
later and pin to its branch/tag.  That feels like a waste of time though,
as the 2.5.2 upgrade is somewhat trivial.

Jon

On Thu, Nov 16, 2017 at 10:14 AM Nick Allen  wrote:

> While I think that the Bro work is super valuable, Jon, I am not sure that
> we need to block the next release for it.  In my mind, the "theme" of the
> next release is Metaalerts running on ES 2.x.  I'd prefer to just stick
> with that.
>
> I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs merged
> and start cutting release candidates ASAP.  That could be possible by end
> of week based on the "big one" (METRON-1289) getting merged in last night.
>
> Of course, if you happen to get all the Bro work done in time, then
> definitely let's include it in the next release.  But I am wary of blocking
> the release for that work.  No need for you to rush through it.
>
> Just one man's opinion.  Would like to hear feedback from more of the
> community.
>
>
>
> On Thu, Nov 16, 2017 at 8:01 AM, zeo...@gmail.com 
> wrote:
>
> > The way master's full-dev is set up right now is non optimal for the bro
> > plugin configuration, and I would like to complete the roadmap I outlined
> > in my discuss thread before a release.  I have a PR open against
> > Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
> > expect it will take me until end of next week at the latest to have the
> > rest of the work done to move us to 2.5.2, and to pin to a specific
> package
> > version.  At that point I'm comfortable with a release.
> >
> > Jon
> >
> > On Wed, Nov 15, 2017, 18:57 Matt Foley  wrote:
> >
> > > I’ve been listening.  Looks like there are still a number of major
> issues
> > > to be committed first, right?
> > > The discussion on this thread constitutes sufficient engagement, I
> think,
> > > especially given the Subject line :-)
> > > Would the folks working on the 6 issues listed by Nick care to suggest
> a
> > > cut-off date by which they’ll probably have those fixes in?
> > > I’ll be happy to run the release process, if the community so wishes,
> as
> > > soon as those issues are committed.
> > >
> > > I guess I should, pro forma, send the list of commits already in since
> > the
> > > last release.  I’ll do that today.
> > > Also, if anyone wishes to raise a hand and propose additional commits
> are
> > > needed, please do so on this thread.
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 11/15/17, 2:09 PM, "Casey Stella"  wrote:
> > >
> > > I'd say that if a release is this imminent that we had better
> notify
> > > the
> > > release manager who will make a release announcement, Nick.  Matt,
> > are
> > > you
> > > tuning in to this?
> > >
> > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen 
> > > wrote:
> > >
> > > > Hi Guys -
> > > >
> > > > I want to follow-up on this discussion.  It sounds like most
> people
> > > are in
> > > > agreement with the general approach.
> > > >
> > > > A lot of people have been working hard on Metaalerts and
> > > Elasticsearch.  I
> > > > have checked-in with those doing the heavy lifting and have
> > compiled
> > > a more
> > > > detailed plan based on where we are at now.  To the best of my
> > > knowledge
> > > > here is the plan of attack for finishing out this effort.
> > > >
> > > >   (1) First, METRON-1289 needs to go in.  This one was a fairly
> big
> > > effort
> > > > and I am hearing that we are pretty close.
> > > >
> > > >   (2) METRON-1294 fixes an issue in how field types are
> looked-up.
> > > >
> > > >   (3) METRON-1290 is next.  While this may have been fixed in
> > > M-1289, there
> > > > may be some test cases we want from this PR.
> > > >
> > > >   (4) METRON-1301 addresses a problem with the sortin

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

I expect a few version changes up front to add some new features to the
package (0.1 for the initial release, 0.{2..n} for some new features, 1.0
when we stabilize) but after that it will probably only be updated to
follow kafka/librdkafka updates.

Jon

On Thu, Nov 16, 2017 at 10:10 AM Otto Fowler 
wrote:

> How often to we expect to change this?  If it is effectively pinned then a
> release process is not that bad.
>
>
> On November 16, 2017 at 10:06:53, Nick Allen (n...@nickallen.org) wrote:
>
> >
> > I would suggest that we institute a release procedure for the package
> > itself, but I don't think it necessarily has to line up with metron
> > releases (happy to be persuaded otherwise). Then we can just link metron
> > to metron-bro-plugin-kafka by pointing to specific
> > metron-bro-plugin-kafka releases (git tags
> > <
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> > versioning>
> > ).
> > Right now, full-dev spins up against the
> > apache/metron-bro-plugin-kafka master branch, which is not a good idea to
> > have in place for an upcoming release. That is the crux of why I think we
> > need to finalize the move to bro 2.5.2 and the plugin packaging before
> our
> > next release (working on it as we speak).
> > Jon
>
>
> ​I replayed Jon's comments from the other thread above.​
>
> My initial thought, is that I would not want to manage two separate release
> processes. I don't want to have a roll call, cut release candidates and
> test both.
>
> I was thinking we would just need to change some of the behind-the-scenes
> processes handled by the release manager. This is one area where I had
> thought using a submodule in Git would help.
>
>
>
>
>
>
>
>
>
> On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen  wrote:
>
> > + Restarting the thread to include mentors.
> >
> > The code of the 'Kafka Plugin for Bro' is now maintained in the external
> > repository that we set up a while back.
> >
> > - Metron Core: git://git.apache.org/metron.git
> > - Kafka Plugin for Bro: git://git.apache.org/
> > metron-bro-plugin-kafka.git
> >
> > (Q) Do we need to change anything in the release procedure to account for
> > this?
> >
>
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

For #3 we would need people to recursively clone our repo to pull in
submodules properly.  I work with other projects that do that and it is
perpetually an issue for new users.  I know it doesn't seem like a big
deal, but it is definitely a deterrent to some who forget this step or use
old instructions that don't include it.

I'm interested to see how other apache projects handle releases for
multiple repos.  I honestly think we can have a much lower bar for a
release on a secondary repo, but I don't know if that's acceptable.

Regarding 4, we could also fix forward.  I already have a solution that
could be very slightly repurposed in my full-dev testing instructions here
<https://github.com/apache/metron-bro-plugin-kafka/pull/3>.

Yes, I still prefer #5 but I'll stop talking so much and let also chime
in.  =)

Jon

On Thu, Nov 16, 2017 at 10:40 AM Nick Allen  wrote:

> > (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
> Ansible to pin to that release.
>
> The problem with this approach, as I see it, is that it commits us to
> having separate releases for the Bro plugin.  Which I am not sure if or how
> long it will take to come to a consensus on that.  We also would need to
> invent that release process rather quickly.
>
> > (4) Revert PR #837 because as you pointed out this approach does not work
> well with releases.  That would give us enough time to come up with a
> sensible approach and do that.
>
> I am kind of leaning towards (4) right now.  The approach taken in this PR,
> doesn't work.  Let's just revert it and give ourselves some time to come up
> with an approach that does work and has consensus.
>
>
>
>
>
>
>
> On Thu, Nov 16, 2017 at 10:37 AM, Nick Allen  wrote:
>
> > Just to layout some other alternatives...
> >
> > (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
> > Ansible to pin to that release.
> >
> > (2) [Jon] Update Ansible to pin to a specific commit.
> >
> > (3) Wouldn't the submodule approach solve this?  I imagine that if we use
> > a submodule, then all of the Bro plugin code will be contained directly
> in
> > each Metron release.  We would just need to change that small bit of
> > Ansible code so that it uses the code directly (like before) instead of
> > going to Git and checking it out.
> >
> > (4) Revert PR #837 because as you pointed out this approach does not work
> > well with releases.  That would give us enough time to come up with a
> > sensible approach and do that.
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 16, 2017 at 10:21 AM, zeo...@gmail.com 
> > wrote:
> >
> >> The problem is that we're currently pinning to master and if something
> in
> >> the plugin breaks backward compatibility, our prior releases will be
> >> perpetually broken, which is why I suggest it blocks the upcoming
> release.
> >>
> >> The alternative would be to update ansible to pin to a specific commit
> or
> >> to make a release in apache/metron-bro-plugin-kafka sooner rather than
> >> later and pin to its branch/tag.  That feels like a waste of time
> though,
> >> as the 2.5.2 upgrade is somewhat trivial.
> >>
> >> Jon
> >>
> >> On Thu, Nov 16, 2017 at 10:14 AM Nick Allen  wrote:
> >>
> >> > While I think that the Bro work is super valuable, Jon, I am not sure
> >> that
> >> > we need to block the next release for it.  In my mind, the "theme" of
> >> the
> >> > next release is Metaalerts running on ES 2.x.  I'd prefer to just
> stick
> >> > with that.
> >> >
> >> > I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs
> merged
> >> > and start cutting release candidates ASAP.  That could be possible by
> >> end
> >> > of week based on the "big one" (METRON-1289) getting merged in last
> >> night.
> >> >
> >> > Of course, if you happen to get all the Bro work done in time, then
> >> > definitely let's include it in the next release.  But I am wary of
> >> blocking
> >> > the release for that work.  No need for you to rush through it.
> >> >
> >> > Just one man's opinion.  Would like to hear feedback from more of the
> >> > community.
> >> >
> >> >
> >> >
> >> > On Thu, Nov 16, 2017 at 8:01 AM, zeo...@gmail.com 
> >> > wrote:
> >> >
> >> > > The way master's full-dev is 

master full-dev issues?

[zeo...@gmail.com]

Anybody else having issues spinning up full-dev?  I'm consistently failing
on the Metron Alerts UI install.  Spun it up fine yesterday for my other
testing.

2017-11-16 17:57:41,772 - Execution of '/usr/bin/yum -d 0 -e 0 -y
install metron-common' returned 1. Error: Nothing to do


https://gist.github.com/JonZeolla/3923f93a3e31c8064c4d77db89431106

Jon
-- 

Jon

Re: master full-dev issues?

[zeo...@gmail.com]

Nevermind, user error.

Jon

On Thu, Nov 16, 2017, 13:00 zeo...@gmail.com  wrote:

> Anybody else having issues spinning up full-dev?  I'm consistently failing
> on the Metron Alerts UI install.  Spun it up fine yesterday for my other
> testing.
>
> 2017-11-16 17:57:41,772 - Execution of '/usr/bin/yum -d 0 -e 0 -y install 
> metron-common' returned 1. Error: Nothing to do
>
>
> https://gist.github.com/JonZeolla/3923f93a3e31c8064c4d77db89431106
>
> Jon
> --
>
> Jon
>
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

gt;>METRON-1185: Stellar REPL does not work on a kerberized cluster when
> >> calling functions interacting with HBase closes
> apache/incubator-metron#755
> >>METRON-1186: Profiler Functions use classutils from shaded storm
> >> closes apache/incubator-metron#758
> >>METRON-1173: Fix pointers to old stellar docs closes
> >> apache/incubator-metron#746
> >>METRON-1179: Make STATS_ADD to take a list closes
> >> apache/incubator-metron#750
> >>METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list
> >> and not require a port closes apache/incubator-metron#751
> >>METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
> >> apache/metron#753
> >>METRON-1177 Stale running topologies seen post-kerberization and
> cause
> >> exceptions (nickwallen) closes apache/metron#748
> >>METRON-1158 Build backend for grouping alerts into meta alerts
> >> (justinleet) closes apache/metron#734
> >>METRON-1146: Add ability to parse JSON string into JSONObject for
> >> stellar closes apache/incubator-metron#727
> >>METRON-1176 REST: HDFS Service should support setting permissions on
> >> files when writing (ottobackwards) closes apache/metron#749
> >>METRON-1114 Add group by capabilities to search REST endpoint
> >> (merrimanr) closes apache/metron#702
> >>METRON-1167 Define Session Specific Global Configuration Values in
> the
> >> REPL (nickwallen) closes apache/metron#740
> >>METRON-1171: Better validation for the SUBSTRING stellar function
> >> closes apache/incubator-metron#745
> >>
> >>
> >>
> >> On 11/17/17, 11:59 AM, "Nick Allen"  wrote:
> >>
> >>I just wanted to send an update on where we are at.  We've gotten a
> lot
> >>done here recently as you can see below.
> >>
> >>  ✓ DONE (1) First, METRON-1289 needs to go in.  This one was a
> fairly
> >> big
> >>effort and I am hearing that we are pretty close.
> >>
> >>  ✓ DONE (2) METRON-1294 fixes an issue in how field types are
> >> looked-up.
> >>
> >>  ✓ DONE (3) METRON-1290 is next.  While this may have been fixed in
> >>M-1289, there may be some test cases we want from this PR.
> >>
> >>  ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
> >>
> >>  ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> metaalerts.
> >>
> >>  (6) That leads us to Raghu's UI work in METRON-1252.  This
> >> introduces the
> >>UI bits that depend on all the previous backend work.
> >>
> >>  (7) At this point, we should have our best effort at running
> >> Metaalerts
> >>on Elasticsearch 2.x. I propose that we cut a release here.
> >>
> >>  (8) After we cut the release, we can introduce the work for ES 5.x
> in
> >>METRON-939.  I know we will need lots of help testing and reviewing
> >> this
> >>one.
> >>
> >>
> >>
> >>We also have an outstanding question that needs resolved BEFORE we
> >>release.  We need to come to a consensus on how to release having
> >> moved our
> >>Bro Plugin to a separate repo.  I don't think we've heard from
> >> everyone on
> >>this.  I'd urge everyone to chime in so we can choose a path forward.
> >>
> >>If anyone is totally confused in regards to that discussion, I can
> try
> >> and
> >>send an options summary again as a separate discuss thread.  The
> >> original
> >>chain was somewhere around here [1].
> >>
> >>[1]
> >>https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e9
> >> 23a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
> >>
> >>
> >>
> >>On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen 
> >> wrote:
> >>
> >>> Hi Guys -
> >>>
> >>> I want to follow-up on this discussion.  It sounds like most people
> >> are in
> >>> agreement with the general approach.
> >>>
> >>> A lot of people have been working hard on Metaalerts and
> >> Elasticsearch.  I
> >>> have checked-in with those doing the heavy lifting and have compiled
> >> a more
> >>> detailed plan based on where we are at now.  To the best of my
> >> k

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

I propose that we coordinate the review of METRON-1329
<https://github.com/apache/metron-bro-plugin-kafka/pull/4> and METRON-1313
<https://github.com/apache/metron/pull/847>, then merge METRON-1329, pursue
a 0.1 release for apache/metron-bro-plugin-kafka, and then finalize
METRON-1313 <https://github.com/apache/metron/pull/847> prior to the
upcoming apache/metron release.  At that point the roadmap outlined here
<https://lists.apache.org/thread.html/286231d11fcb8349e0378d281a08d197876f80122056479156643104@%3Cdev.metron.apache.org%3E>
will be complete, and both of the issues you outlined below would be
addressed.

I did test making a bro package release of 0.4.2.0, and we can do that, but
the related (nested) plugin version must be major.minor, so it would need
to be something like 0.4 (or, my preference, 0.1).  I would rather keep the
package and plugin versions in line with each other than try to line the
package version up with apache/metron versions, and have the plugin version
out of whack (which is what people would see when checking to see if this
package was properly installed - see the version here
<https://github.com/apache/metron-bro-plugin-kafka/blob/master/tests/Baseline/kafka.show-plugin/output>
).

Let me know what you all think - happy to discuss further.

Jon

On Thu, Nov 16, 2017 at 1:26 PM Matt Foley  wrote:

> There’s two issues, I think:
> (1) We’d like to be able to version and evolve the main body of Metron and
> the metron-bro-plugin-kafka separately.
> (2) We want to be able to assure that each release of Metron has a
> known-working version of metron-bro-plugin-kafka
>
> At a very simple level, we can achieve these by
> (1) Don’t put in version-checking constraints, or make those constraints
> very loose and of a form that assumes forward-compatibility, ie “version of
> metron-bro-plugin-kafka must be AT LEAST ‘x’ to be used with version ‘y’ of
> Metron.”
> (2) But we should still do an archive branch of metron-bro-plugin-kafka
> when we make a release branch of Metron, unless the metron-bro-plugin-kafka
> has zero changes since the last Metron release.
>
> I would recommend that we use a 4-element version for
> metron-bro-plugin-kafka, and at least the first 3 elements should be equal
> to the corresponding Metron release version.  This isn’t necessary, but it
> makes things easy to keep track of.  That still leaves room for necessary
> patches on a given release line.
>
> If you prefer other approaches, please propose.  When we reach consensus,
> I can edit the Release Process to document it.
> Cheers,
> --Matt
>
> On 11/16/17, 7:22 AM, "zeo...@gmail.com"  wrote:
>
> I expect a few version changes up front to add some new features to the
> package (0.1 for the initial release, 0.{2..n} for some new features,
> 1.0
> when we stabilize) but after that it will probably only be updated to
> follow kafka/librdkafka updates.
>
> Jon
>
> On Thu, Nov 16, 2017 at 10:10 AM Otto Fowler 
> wrote:
>
> > How often to we expect to change this?  If it is effectively pinned
> then a
> > release process is not that bad.
> >
> >
> > On November 16, 2017 at 10:06:53, Nick Allen (n...@nickallen.org)
> wrote:
> >
> > >
> > > I would suggest that we institute a release procedure for the
> package
> > > itself, but I don't think it necessarily has to line up with metron
> > > releases (happy to be persuaded otherwise). Then we can just link
> metron
> > > to metron-bro-plugin-kafka by pointing to specific
> > > metron-bro-plugin-kafka releases (git tags
> > > <
> >
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> > > versioning>
> > > ).
> > > Right now, full-dev spins up against the
> > > apache/metron-bro-plugin-kafka master branch, which is not a good
> idea to
> > > have in place for an upcoming release. That is the crux of why I
> think we
> > > need to finalize the move to bro 2.5.2 and the plugin packaging
> before
> > our
> > > next release (working on it as we speak).
> > > Jon
> >
> >
> > ​I replayed Jon's comments from the other thread above.​
> >
> > My initial thought, is that I would not want to manage two separate
> release
> > processes. I don't want to have a roll call, cut release candidates
> and
> > test both.
> >
> > I was thinking we would just need to change some of the
> behind-the-scenes
> > processes handled by the release manager. This is o

Re: [DISCUSS] NPM / Node Problems

[zeo...@gmail.com]

That was also required for bro 2.5.2, so I did that here
.
Feel free to reuse the approach elsewhere

Jon

On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler 
wrote:

> First issue is that we need c++ 11 on centos 6.8
>
>
>
> On November 27, 2017 at 09:53:55, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Well, that’s good news on that issue. Reproducing the problem is half way
> to solving it, right?
>
> I would still say there are some systemic things going on that have
> manifested in a variety of ways on both the users and dev list, so it’s
> worth us having a good look at a more robust approach to node dependencies
> (both npm ones, and the native ones)
>
> Simon
>
> On 27 Nov 2017, at 13:30, Otto Fowler  wrote:
>
> I can reproduce the failure in out ansible docker build container, which is
> also centos.
> The issue is building our node on centos in all these cases.
>
>
>
> On November 27, 2017 at 07:02:51, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Thinking about this, doesn’t our build plugin explicitly install it’s own
> node? So actually all the node version things may be a red herring, since
> this is under our control through the pom. Not sure if we actually
> exercising this control. It seems that some of the errors people report are
> more to do with compilation failures for native node modules, which is
> doesn’t pin (i.e. things like system library dependencies). I’m not sure
> what we have in the dependency tree that requires complex native
> dependencies, but this might just be one of those node things we could doc
> around.
>
> This scenario would be fixed by standardising the build container.
>
> Yarn’s big thing is that it enables faster dependency resolution and local
> caching, right? It does not seem to address any of the problems we see, but
> sure, it’s the shiny new dependency system for node modules, which might
> make npm less horrible to deal with, so worth looking into.
>
> The other issue that I’ve seen people run into a lot is flat out download
> errors. This could be helped by finding our versions, maybe with yarn, but
> let’s face it, package-lock.json could also do that with npm, albeit with a
> slightly slower algorithm. However, short of bundling and hosting deps
> ourselves, I suspect the download errors are beyond our control, and
> certainly beyond the scope of this project (fix maven, fix npm, fix all the
> node hosting servers…)
>
> Simon
>
>
> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda <
> raghumitra@gmail.com>
> wrote:
> >
> > Looking at some of the build failure emails and past experience i
> > would suggest having a node & npm version check in our build scripts
> > and moving dependency management to yarn.
> >
> > We need not restrict the build to a specific version of node & npm but
> > we can surely suggest a min version required to build UI successfully.
> >
> > -Raghu
> >
> >
> >
> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
> >  wrote:
> >> Agreeing with Nick, it seems like the main reason people are building
> themselves, and hitting all these environmental issues, is that we do not
> as a project produce binary release artefacts (the rpms which users could
> just install) and instead leave that for the commercial distributors to do.
> >>
> >> Yarn may help with some of the dependency version issues we’re having,
> but not afaik with the core missing library headers / build tools / node
> and npm version issue, those would seem to fit a documentation fix and
> improvements to platform-info to flag the problems, so this can then be a
> pre-flight check tool as well as a diagnostic tool.
> >>
> >> Another option I would put on the table is to standardise our build
> environment, so that the non-java bits are run in a standard docker image
> or something fo the sort, that way we can take control of all the
> environmental and OS dependent pieces, much as we do right now with the rpm
> build sections of the mpack build.
> >>
> >> The challenge here will be adding the relevant maven support. At the
> moment we’re relying on the maven npm and node build plugins, this would
> likely need replacing with something custom and a challenge to support to
> go dow this route.
> >>
> >> Perhaps the real answer here is to push people who are just kicking the
> tyres towards a binary distribution, or at least rpm artefacts as part of
> the Apache release to give them a head start for a happy path on a known
> good OS environment.
> >>
> >> Simon
> >>
> >>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
> >>>
> >>> Yes, it is a problem. I think you've identified a couple important
> things
> >>> that we could address in parallel. I see these as challenges we need to
> >>> solve for the dev community.
> >>>
> >>> (1) NPM is causing us some major headaches. Which version do we
> require?
>
> >>> How do I install that version (on Mac, Windows

Re: [DISCUSS] NPM / Node Problems

[zeo...@gmail.com]

Note that I cleaned up the ansible scripts that install C++ 11 in my latest
PR <https://github.com/apache/metron/pull/847/files>, but it's not super
relevant to this conversation.

Jon

On Mon, Nov 27, 2017 at 10:42 AM zeo...@gmail.com  wrote:

> That was also required for bro 2.5.2, so I did that here
> <https://github.com/apache/metron/commit/59fe1b453279bf5c7df627ea656c762b3a98e777>.
> Feel free to reuse the approach elsewhere
>
> Jon
>
> On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler 
> wrote:
>
>> First issue is that we need c++ 11 on centos 6.8
>>
>>
>>
>> On November 27, 2017 at 09:53:55, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Well, that’s good news on that issue. Reproducing the problem is half way
>> to solving it, right?
>>
>> I would still say there are some systemic things going on that have
>> manifested in a variety of ways on both the users and dev list, so it’s
>> worth us having a good look at a more robust approach to node dependencies
>> (both npm ones, and the native ones)
>>
>> Simon
>>
>> On 27 Nov 2017, at 13:30, Otto Fowler  wrote:
>>
>> I can reproduce the failure in out ansible docker build container, which
>> is
>> also centos.
>> The issue is building our node on centos in all these cases.
>>
>>
>>
>> On November 27, 2017 at 07:02:51, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Thinking about this, doesn’t our build plugin explicitly install it’s own
>> node? So actually all the node version things may be a red herring, since
>> this is under our control through the pom. Not sure if we actually
>> exercising this control. It seems that some of the errors people report
>> are
>> more to do with compilation failures for native node modules, which is
>> doesn’t pin (i.e. things like system library dependencies). I’m not sure
>> what we have in the dependency tree that requires complex native
>> dependencies, but this might just be one of those node things we could doc
>> around.
>>
>> This scenario would be fixed by standardising the build container.
>>
>> Yarn’s big thing is that it enables faster dependency resolution and local
>> caching, right? It does not seem to address any of the problems we see,
>> but
>> sure, it’s the shiny new dependency system for node modules, which might
>> make npm less horrible to deal with, so worth looking into.
>>
>> The other issue that I’ve seen people run into a lot is flat out download
>> errors. This could be helped by finding our versions, maybe with yarn, but
>> let’s face it, package-lock.json could also do that with npm, albeit with
>> a
>> slightly slower algorithm. However, short of bundling and hosting deps
>> ourselves, I suspect the download errors are beyond our control, and
>> certainly beyond the scope of this project (fix maven, fix npm, fix all
>> the
>> node hosting servers…)
>>
>> Simon
>>
>>
>> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda <
>> raghumitra@gmail.com>
>> wrote:
>> >
>> > Looking at some of the build failure emails and past experience i
>> > would suggest having a node & npm version check in our build scripts
>> > and moving dependency management to yarn.
>> >
>> > We need not restrict the build to a specific version of node & npm but
>> > we can surely suggest a min version required to build UI successfully.
>> >
>> > -Raghu
>> >
>> >
>> >
>> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>> >  wrote:
>> >> Agreeing with Nick, it seems like the main reason people are building
>> themselves, and hitting all these environmental issues, is that we do not
>> as a project produce binary release artefacts (the rpms which users could
>> just install) and instead leave that for the commercial distributors to
>> do.
>> >>
>> >> Yarn may help with some of the dependency version issues we’re having,
>> but not afaik with the core missing library headers / build tools / node
>> and npm version issue, those would seem to fit a documentation fix and
>> improvements to platform-info to flag the problems, so this can then be a
>> pre-flight check tool as well as a diagnostic tool.
>> >>
>> >> Another option I would put on the table is to standardise our build
>> environment, so that the non-java bits are run in a standard docker image
>> or something fo the sort, that way we can tak

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

The reason we decided to do that was because it is the best way for it to
be used (and thus improved on and quality tested) by the broader bro
community.  If it's any indication of it's popularity, there was just an
email on the bro mailing list about the plugin a few days ago, and I've
already received a PR against my temporary holding place for this plugin to
improve it (GitHub.com/jonzeolla/jzeolla-metron-bro-plugin-kafka), which
would never have happened if it wasn't a standalone repo.  Previously Nick
pushed the code into a bro/plugins repo, which is now deprecated in favor
of bro packages, but while it was there the community was able to identify
issues which helped improve its quality.

By putting it in its own repo we were also able to turn it into a bro
package, similar to debs or rpms but slightly different.  This has numerous
benefits, including turning setup into single command that installs and
loads the plugin, in addition to handling version management and performing
on installation unit tests to identify issues before use.

I would advocate to treat this code as a dependancy, but not necessarily a
core part of Metron, like we currently do.  This also allows us to maintain
separate versioning and release cycles (i.e. not necessarily lined up with
Metron releases), which I expect will be very simple, infrequent, and
easily tested.  Happy to discuss this further - thanks,

Jon

On Mon, Nov 27, 2017, 17:58 James Sirota  wrote:

> I agree with Nick.  Since the plugin is tightly coupled with Metron why
> not just pull it into the main repo and version it with the rest of the
> code?  Do we really need the second repo for the plug-in?
>
> Thanks,
> James
>
>
>
> 16.11.2017, 08:06, "Nick Allen" :
> >>  I would suggest that we institute a release procedure for the package
> >>  itself, but I don't think it necessarily has to line up with metron
> >>  releases (happy to be persuaded otherwise). Then we can just link
> metron
> >>  to metron-bro-plugin-kafka by pointing to specific
> >>  metron-bro-plugin-kafka releases (git tags
> >>  <
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> >>  versioning>
> >>  ).
> >>  Right now, full-dev spins up against the
> >>  apache/metron-bro-plugin-kafka master branch, which is not a good idea
> to
> >>  have in place for an upcoming release. That is the crux of why I think
> we
> >>  need to finalize the move to bro 2.5.2 and the plugin packaging before
> our
> >>  next release (working on it as we speak).
> >>  Jon
> >
> > ​I replayed Jon's comments from the other thread above.​
> >
> > My initial thought, is that I would not want to manage two separate
> release
> > processes. I don't want to have a roll call, cut release candidates and
> > test both.
> >
> > I was thinking we would just need to change some of the behind-the-scenes
> > processes handled by the release manager. This is one area where I had
> > thought using a submodule in Git would help.
> >
> > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen  wrote:
> >
> >>  + Restarting the thread to include mentors.
> >>
> >>  The code of the 'Kafka Plugin for Bro' is now maintained in the
> external
> >>  repository that we set up a while back.
> >>
> >> - Metron Core: git://git.apache.org/metron.git
> >> - Kafka Plugin for Bro: git://git.apache.org/
> >> metron-bro-plugin-kafka.git
> >>
> >>  (Q) Do we need to change anything in the release procedure to account
> for
> >>  this?
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
-- 

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

In an attempt to keep this from becoming unbearably long, I will try
to keep my responses short, but I would be happy to elaborate.  That's a
fairly good timeline and summary, but here are some clarifications in
corresponding order:

- The plugin history is quite short and you can probably get a good bit of
context just by looking at the commits
.
- The plugin is only useful to the bro community, but it is rather popular.
- The Bro team created the idea of bro packages, which can include bro
plugins, bro scripts, or BroControl plugins.  So, instead of having a
'plugins' repo, they moved to have a 'packages' repo which is by default
referenced by a bro-pkg tool they wrote for package management.
- I believe I kicked this off (or at least I did in my head) when I started
complaining 
about the plugin divergence that occurred due to the move to bro/plugins
(the right move at the time), but Metron's use of a local directory that
hadn't been kept up to date.  My current efforts are an attempt to make
sure this doesn't happen again, and to take advantage of the bro-pkg
benefits.
- The gap between ~3/31 and actual progress on 11/12 is completely on me -
I had intentions of doing this work sooner but failed to do so.
- You can most definitely still install/use the bro plugin without using
bro-pkg.  In fact, the README in my PR

still has the instructions on how to do so.

Q1:  The simple explanation is that the only thing that makes a plugin a
bro package is the inclusion of a bro-pkg.meta file
,
and it includes a build_command which could easily be manually performed to
install by hand (assuming dependencies are met).

I've worked with other projects that use submodules and while I'm fine
discussing it, I suggest that we don't implement it.  I put together a
quick example of why here[1], using the bro project as an example since
it's top of mind.

Q2:   I think the answer to Q1 answers this.  There is absolutely nothing
stopping a git clone && cd $dir && configure && make && make install, but
using bro-pkg to install/load takes into account dependencies

and
unit tests

when it is loaded (and thus fails early and more intuitively).  It only
must be a separate repo (or, more technically correct, a git branch that
includes only the package) because of how bro-pkg works.  If you'd like to
get an idea of how this would work in application for Bro users, you can
see my test instructions here
 (specifically
step #3).  If a 0.1 tag gets pushed to apache/metron-bro-plugin-kafka, the
command could be `bro-pkg install metron-bro-plugin-kafka --version 0.1` or
`bro-pkg install apache/metron-bro-plugin-kafka --version 0.1` due to this
 (the
--force is just to remove user interaction, for an ansible spin-up).


1:  To clone the Bro git repo, you must run `git clone --recursive
https://github.com/bro/bro` (note the --recursive).  Not too big of a deal,
but requires that you remember it and existing instructions/blog posts may
give users inaccurate steps.  Let's make this worse and try to checkout
their latest release, v2.5.2, and automatically update the submodules
appropriately via `git checkout v2.5.2 --recurse-submodules`.  This fails
because aux/plugins (https://github.com/bro/plugins) was removed since
their latest release.  Okay, we can work around this using `git checkout
v2.5.2` and then remember to `git submodule update` every time you checkout
a release or branch.  But because they have nested submodules, we actually
need to run `git submodule update --recursive`.  I can't imagine opting
*into* a workflow anything like this.  There are other options as well,
such as git subtrees, but those I am less familiar with.

Jon

On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler  wrote:

> I am not sure that our use of the plugin necessarily equates to it being
> implicitly coupled to Metron.  It seems like the Right Thing To Do[image:
> ™], esp.
>  for an Apache project would be to make this available for use by the
> greater bro community.
> Unless we expect to do extensive iterative work on the plugin, which would
> then make the decision to spin it out now premature.
>
> Then again, I might be wrong ;)
>
>
> On November 27, 2017 at 19:58:11, Matt Foley (ma...@apache.org) wrote:
>
> [Please pardon me that the below is a little labored. I’m trying to
> understand the implications for both release and use, which requires some
> explanation as w

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

rberization
> > and cause exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta
> alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject
> > for stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting
> > permissions on files when writing (ottobackwards) closes
> apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration
> > Values in the REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar
> > function closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen"  wrote:
> >
> > I just wanted to send an update on where we are at.  We've
> > gotten a lot
> > done here recently as you can see below.
> >
> >   ✓ DONE (1) First, METRON-1289 needs to go in.  This one was
> > a fairly big
> > effort and I am hearing that we are pretty close.
> >
> >   ✓ DONE (2) METRON-1294 fixes an issue in how field types
> are
> > looked-up.
> >
> >   ✓ DONE (3) METRON-1290 is next.  While this may have been
> > fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> >   ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > logic.
> >
> >   ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> >
> >   (6) That leads us to Raghu's UI work in METRON-1252.  This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> >   (7) At this point, we should have our best effort at
> running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> >   (8) After we cut the release, we can introduce the work for
> > ES 5.x in
> > METRON-939.  I know we will need lots of help testing and
> > reviewing this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved
> > BEFORE we
> > release.  We need to come to a consensus on how to release
> > having moved our
> > Bro Plugin to a separate repo.  I don't think we've heard
> from
> > everyone on
> > this.  I'd urge everyone to chime in so we can choose a path
> > forward.
> >
> > If anyone is totally confused in regards to that discussion,
> I
> > can try and
> > send an options summary again as a separate discuss thread.
> > The original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/
> > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > 3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > n...@nickallen.org> wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion.  It sounds like
> most
> > people are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch.  I
> > > have checked-in with those doing the heavy lifting and have
> > compiled a more
> > > detailed plan based on where we are at now.  To the best of
> > my knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > >   (1) First, METRON-1289 needs to go in.  This one was a
> > fairly big effort
> > > and I am hearing that we are pretty close.
> > >
> > >   (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > >
> > >   (3) METRON-1290 is next.  While this may have been fixed
> > in M-1289,
> > > there may be some test cases we want from this PR.
> > >
> > >   (4) METRON-1301 addres

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

I would prefer a, but I was initially thinking of doing the plugin first
and then get in the two PRs out to use this new tag, which are already +1'd
and just waiting on this conversation.  For reference,
https://github.com/apache/metron/pull/847 and
https://github.com/apache/metron-bro-plugin-kafka/pull/4

Jon

On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:

> It seems to me, as I believe I have stated before that a) feels like the
> proper way to handle this.  It is how I have seen other projects like NiFi
> handle things as well.
>
>
>
> On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
>
> Okay, looking at this from the perspective of making a release:
>
>
>
> We have two choices:
>
> a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> metron-bro-plugin-kafka, at the same time and using the same process
> (modulo the necessary) as Metron.  This is dirt simple.
>
> b) I or someone needs to:
>
> - open a jira,
>
> - add the submodule to the Metron code tree,
>
> - possibly (optionally) add build mechanism to the maven poms, and
>
> - document as much as we think appropriate regarding what it is, how to
> build it, and how to update it,
>
> and commit that before the 0.4.2 release.
>
>
>
> What is the will of the community?
>
> Thanks,
>
> --Matt
>
>
>
> From: Nick Allen 
> Reply-To: "dev@metron.apache.org" 
> Date: Tuesday, November 28, 2017 at 9:09 AM
> To: "dev@metron.apache.org" 
> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>
>
>
> I'll add a bit to Jon's technical comments.
>
>
>
> * We only created a separate repo because it was a technical requirement to
> leverage the bro-pkg mechanism.
>
> * Leveraging the new bro-pkg mechanism has many advantages as outlined by
> Jon.
>
> * Enabling the bro-pkg mechanism is backwards compatible. I can install the
> plugin exactly how we use to.
>
>
>
> While I agree with Jon's technical comments, I disagree with the
> non-technical ones.
>
>
>
> (1) I do not want to change our release management process. While we needed
> to make a new repo (a technical change), I did not want that to change how
> we operate as a community (our procedures, policies, versioning and release
> cycles).
>
>
>
> (2) I see no value in adopting a separate release management process for
> the Bro plugin alone. Having a separate release process does not make the
> plugin *more* available to the Bro community.
>
>
>
> (3) I also see no value in positioning the plugin to be spun-out of the
> Metron project. It is a part of Metron and I want to see it benefit and
> evolve "the Apache-way".
>
>
>
> In my mind, the best way to accommodate the additional repo, while
> minimizing changes to our release management process, is to treat the new
> repo as a submodule. I fail to see significant downsides to this approach.
> A few extract command-line options do not seem overly onerous to me.
>
>
>
> Many thanks go to Jon for all the hard work he has put in enhancing the
> plugin.
>
>
>
>
>
> On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com 
> wrote:
>
> In an attempt to keep this from becoming unbearably long, I will try to
> keep my responses short, but I would be happy to elaborate. That's a fairly
> good timeline and summary, but here are some clarifications in
> corresponding order:
>
>
>
> - The plugin history is quite short and you can probably get a good bit of
> context just by looking at the commits.
>
> - The plugin is only useful to the bro community, but it is rather popular.
>
> - The Bro team created the idea of bro packages, which can include bro
> plugins, bro scripts, or BroControl plugins. So, instead of having a
> 'plugins' repo, they moved to have a 'packages' repo which is by default
> referenced by a bro-pkg tool they wrote for package management.
>
> - I believe I kicked this off (or at least I did in my head) when I started
> complaining about the plugin divergence that occurred due to the move to
> bro/plugins (the right move at the time), but Metron's use of a local
> directory that hadn't been kept up to date. My current efforts are an
> attempt to make sure this doesn't happen again, and to take advantage of
> the bro-pkg benefits.
>
> - The gap between ~3/31 and actual progress on 11/12 is completely on me -
> I had intentions of doing this work sooner but failed to do so.
>
> - You can most definitely still install/use the bro plugin without using
> bro-pkg. In fact, the README in my PR still has the i

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone

Jon

On Thu, Dec 7, 2017 at 2:32 PM Matt Foley  wrote:

> I can start the release process tonight.
>
>
>
> Jon, you mentioned you want to commit
>
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>
> before the release.  Is it convenient for you to do so today?
>
>
>
> Thanks,
>
> --Matt
>
>
>
> From: Nick Allen 
> Date: Thursday, December 7, 2017 at 10:13 AM
> To: "dev@metron.apache.org" 
> Cc: Matt Foley 
> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>
>
>
> I am more interested in getting a release cut.  If me moving to the (a)
> camp gets us to consensus and cuts a release faster, then I'll do it.
> Let's get this release train moving.
>
>
>
> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet 
> wrote:
>
> Do we have any further discussion on this?  Pardon me if I misstate
> anyone's position, but it seems like we have a couple people (Otto and Jon
> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
> section of people like myself without a particular horse in the race.
>
> It seems like we need to come to some sort of consensus so that we can get
> the release bus moving again, and right now it seems like (a) is gathering
> more explicit support.  Do we have a compelling reason to not do (a)? To be
> honest, my main worry is more "If we do (a) are we going to be miserable if
> we need to iterate or adjust?" I'm not seeing anything that suggests
> anything too terrible, so unless we see some more discussion, I suggest we
> move forward with (a).
>
>
> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com  wrote:
>
> > I would prefer a, but I was initially thinking of doing the plugin first
> > and then get in the two PRs out to use this new tag, which are already
> +1'd
> > and just waiting on this conversation.  For reference,
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:
> >
> > > It seems to me, as I believe I have stated before that a) feels like
> the
> > > proper way to handle this.  It is how I have seen other projects like
> > NiFi
> > > handle things as well.
> > >
> > >
> > >
> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
> > >
> > > Okay, looking at this from the perspective of making a release:
> > >
> > >
> > >
> > > We have two choices:
> > >
> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > > metron-bro-plugin-kafka, at the same time and using the same process
> > > (modulo the necessary) as Metron.  This is dirt simple.
> > >
> > > b) I or someone needs to:
> > >
> > > - open a jira,
> > >
> > > - add the submodule to the Metron code tree,
> > >
> > > - possibly (optionally) add build mechanism to the maven poms, and
> > >
> > > - document as much as we think appropriate regarding what it is,
> how
> > to
> > > build it, and how to update it,
> > >
> > > and commit that before the 0.4.2 release.
> > >
> > >
> > >
> > > What is the will of the community?
> > >
> > > Thanks,
> > >
> > > --Matt
> > >
> > >
> > >
> > > From: Nick Allen 
> > > Reply-To: "dev@metron.apache.org" 
> > > Date: Tuesday, November 28, 2017 at 9:09 AM
> > > To: "dev@metron.apache.org" 
> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> > Bro'
> > >
> > >
> > >
> > > I'll add a bit to Jon's technical comments.
> > >
> > >
> > >
> > > * We only created a separate repo because it was a technical
> requirement
> > to
> > > leverage the bro-pkg mechanism.
> > >
> > > * Leveraging the new bro-pkg mechanism has many advantages as outlined
> by
> > > Jon.
> > >
> > > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> > the
> > > plugin exactly how we use to.
> > >
> > >
> > >
> > > While I agree with Jon's technical comments, I disagree with the
> > > non-technica

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

FYI to be uber clear about the effects of what I'm doing, spinning up
full-dev only when including the sensors will be broken on the bro plugin
install step between when I push the changes, and when mattf pushes the 0.1
tag to apache/metron-bro-plugin-kafka.

Jon

On Thu, Dec 7, 2017 at 3:05 PM zeo...@gmail.com  wrote:

> Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone
>
> Jon
>
> On Thu, Dec 7, 2017 at 2:32 PM Matt Foley  wrote:
>
>> I can start the release process tonight.
>>
>>
>>
>> Jon, you mentioned you want to commit
>>
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>>
>> before the release.  Is it convenient for you to do so today?
>>
>>
>>
>> Thanks,
>>
>> --Matt
>>
>>
>>
>> From: Nick Allen 
>> Date: Thursday, December 7, 2017 at 10:13 AM
>> To: "dev@metron.apache.org" 
>> Cc: Matt Foley 
>> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>>
>>
>>
>> I am more interested in getting a release cut.  If me moving to the (a)
>> camp gets us to consensus and cuts a release faster, then I'll do it.
>> Let's get this release train moving.
>>
>>
>>
>> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet 
>> wrote:
>>
>> Do we have any further discussion on this?  Pardon me if I misstate
>> anyone's position, but it seems like we have a couple people (Otto and Jon
>> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably
>> a
>> section of people like myself without a particular horse in the race.
>>
>> It seems like we need to come to some sort of consensus so that we can get
>> the release bus moving again, and right now it seems like (a) is gathering
>> more explicit support.  Do we have a compelling reason to not do (a)? To
>> be
>> honest, my main worry is more "If we do (a) are we going to be miserable
>> if
>> we need to iterate or adjust?" I'm not seeing anything that suggests
>> anything too terrible, so unless we see some more discussion, I suggest we
>> move forward with (a).
>>
>>
>> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com 
>> wrote:
>>
>> > I would prefer a, but I was initially thinking of doing the plugin first
>> > and then get in the two PRs out to use this new tag, which are already
>> +1'd
>> > and just waiting on this conversation.  For reference,
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>> >
>> > Jon
>> >
>> > On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:
>> >
>> > > It seems to me, as I believe I have stated before that a) feels like
>> the
>> > > proper way to handle this.  It is how I have seen other projects like
>> > NiFi
>> > > handle things as well.
>> > >
>> > >
>> > >
>> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
>> > >
>> > > Okay, looking at this from the perspective of making a release:
>> > >
>> > >
>> > >
>> > > We have two choices:
>> > >
>> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
>> > > metron-bro-plugin-kafka, at the same time and using the same process
>> > > (modulo the necessary) as Metron.  This is dirt simple.
>> > >
>> > > b) I or someone needs to:
>> > >
>> > > - open a jira,
>> > >
>> > > - add the submodule to the Metron code tree,
>> > >
>> > > - possibly (optionally) add build mechanism to the maven poms, and
>> > >
>> > > - document as much as we think appropriate regarding what it is,
>> how
>> > to
>> > > build it, and how to update it,
>> > >
>> > > and commit that before the 0.4.2 release.
>> > >
>> > >
>> > >
>> > > What is the will of the community?
>> > >
>> > > Thanks,
>> > >
>> > > --Matt
>> > >
>> > >
>> > >
>> > > From: Nick Allen 
>> > > Reply-To: "dev@metron.apache.org" 
>> > > Date: Tuesday, November 28, 2017 at 9:09 AM
>> > > To: "dev@metron.apache.org" 
>> > > Subject: Re: [

Re: New PMC members

[zeo...@gmail.com]

Congratulations, guys!  Well deserved by all 3.

Jon

On Thu, Dec 7, 2017 at 10:48 AM Kyle Richardson 
wrote:

> Congratulations guys! Well deserved.
>
> -Kyle
>
> On Thu, Dec 7, 2017 at 10:18 AM, Nick Allen  wrote:
>
> > Congrats to all 3 for joining the PMC!
> >
> > On Thu, Dec 7, 2017 at 10:12 AM, Casey Stella 
> wrote:
> >
> > > Well, obviously, I meant Metron instead of Impala.  To this point, we
> > > should have a wiki page around templates for this, similar to the
> impala
> > > project. :)
> > >
> > > On Thu, Dec 7, 2017 at 10:06 AM, Casey Stella 
> > wrote:
> > >
> > > > The Project Management Committee (PMC) for Apache Impala has invited
> > Otto
> > > > Fowler, Michael Miklavcic and Justin Leet  to become a PMC member and
> > we
> > > > are pleased to announce that they have accepted.
> > > >
> > > > Congratulations and welcome!
> > > >
> > > >
> > >
> >
>
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

I poked around the RC briefly and spun up full dev with and without
sensors, no issues so far.

Jon

On Fri, Dec 8, 2017 at 4:34 AM Matt Foley  wrote:

> Colleagues,
> I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
>
> Given the complexity of this RC, I’d appreciate if a couple people would
> be willing to kick the tires before we put it up for a vote.
>
> I will myself be going thru the Verify Build process this weekend, as I
> won’t be able to do it Friday.
>
> Thanks,
> --Matt
>
>
> On 12/4/17, 2:05 PM, "zeo...@gmail.com"  wrote:
>
> Can we resolve the conversation regarding the second repo?  I was
> waiting
> to get more input/preferences from people  There's also a documentation
> update that fixes a few broken Stellar docs that already has aa +1, I
> just
> need to merge it.
>
> Jon
>
> On Mon, Dec 4, 2017, 17:01 Casey Stella  wrote:
>
> > I would be in favor of a release at this point.
> >
> > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley  wrote:
> >
> > > Hey all,
> > > I see METRON-1252 was resolved over the weekend.  Shall I go ahead
> and
> > > start the process with 0.4.2 release?
> > > Does anyone have any commits they feel strongly should go in
> before 0.4.2
> > > is done, or are we ready to call it good?
> > >
> > > I believe there is consensus the 0.4.2 release should include a
> release
> > of
> > > the current state of the metron-bro-plugin-kafka.  I will continue
> the
> > > discussion in that thread as to the process for accomplishing
> that, but
> > > plan on it happening.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/26/17, 6:26 PM, "Matt Foley"  wrote:
> > >
> > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > holiday.
> > > Regarding status of the release effort, still pending
> METRON-1252, so
> > > not making the release branch yet.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/17/17, 1:32 PM, "Matt Foley"  wrote:
> > >
> > > (With release manager hat on)
> > >
> > > The community has proposed a release of Metron in the near
> > future,
> > > focusing on Meta-alerts running in Elasticsearch.
> > > Congrats on getting so many of the below already done.  At
> this
> > > point, only METRON-1252, and the discussion of how to handle joint
> > release
> > > of the Metron bro plugin, remain as gating items for the release.
> I
> > > project these will be resolved next week, so let’s propose the
> following:
> > >
> > > Sometime next week, after the last bits are done, I’ll
> start the
> > > release process and create the release branch.
> > >
> > > The proposed new version will be 0.4.2, unless there are
> backward
> > > incompatible changes that support making it 0.5.0.
> > > Currently there are NO included Jiras labeled
> > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > ***If anyone knows that some of the commits included since
> 0.4.1
> > > introduce backward incompatibility, please say so now on this
> thread, and
> > > mark the Jira as such.***
> > >
> > > The 90 or so jiras/commits already in master branch since
> 0.4.1
> > > are listed below.
> > > Thanks,
> > > --Matt
> > >
> > > METRON-1301 Alerts UI - Sorting on Triage Score
> Unexpectedly
> > > Filters Some Records (nickwallen) closes apache/metron#832
> > > METRON-1294 IP addresses are not formatted correctly
> in facet
> > > and group results (merrimanr) closes apache/metron#827
> > > METRON-1291 Kafka produce REST endpoint does not work
> in a
> > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > METRON-1290 Only first 10 alerts are update when a
> MetaAlert
> > > status is changed to inactive (justinleet) closes apache/metron#842
> > > METRON-1311 Service Check Should Check Elasticsearch
> Index
>   

Re: [DISCUSS] Community Meetings

[zeo...@gmail.com]

I think this is a great idea.  Hangouts works well but last I checked has a
user # limitation.  I don't have any other good suggestions, sorry, but I'm
in to attend.

Jon

On Mon, Dec 11, 2017, 16:42 Otto Fowler  wrote:

> I think that we all want to have regular community meetings.  We may be
> better able to keep to a regular schedule with these meetings if we spread
> out the responsibility for them from James and Casey, both of whom have a
> lot on their plate already.
>
> I would be willing to coordinate and run the meetings, and would welcome
> anyone else who wants to help when they can.
>
> The only issue for me is I do not have a web-ex account that I can use to
> hold the meeting.  So I’ll need some recommendations for a suitable
> alternative.  I have not been able to find an Apache Friendly alternative,
> in the same way that Atlassian is apache friendly.
>
>
> So - from what I can see we need to:
>
> - Talk through who is going to do it
> - How are we going to host it
> - When are we going to do it
>
> Anything else?
>
> ottO
>
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

t;
> /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
> >
> > *
> >
> >
> >
> > On December 8, 2017 at 04:34:24, Matt Foley (ma...@apache.org)
> wrote:
> >
> > Colleagues,
> > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> >
> > Given the complexity of this RC, I’d appreciate if a couple people
> would be
> > willing to kick the tires before we put it up for a vote.
> >
> > I will myself be going thru the Verify Build process this weekend,
> as I
> > won’t be able to do it Friday.
> >
> > Thanks,
> > --Matt
> >
> >
> > On 12/4/17, 2:05 PM, "zeo...@gmail.com"  wrote:
> >
> > Can we resolve the conversation regarding the second repo? I was
> waiting
> > to get more input/preferences from people There's also a
> documentation
> > update that fixes a few broken Stellar docs that already has aa +1,
> I just
> > need to merge it.
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 17:01 Casey Stella  wrote:
> >
> > > I would be in favor of a release at this point.
> > >
> > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley 
> wrote:
> > >
> > > > Hey all,
> > > > I see METRON-1252 was resolved over the weekend. Shall I go
> ahead and
> > > > start the process with 0.4.2 release?
> > > > Does anyone have any commits they feel strongly should go in
> before
> > 0.4.2
> > > > is done, or are we ready to call it good?
> > > >
> > > > I believe there is consensus the 0.4.2 release should include a
> release
> > > of
> > > > the current state of the metron-bro-plugin-kafka. I will
> continue the
> > > > discussion in that thread as to the process for accomplishing
> that, but
> > > > plan on it happening.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/26/17, 6:26 PM, "Matt Foley"  wrote:
> > > >
> > > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > > holiday.
> > > > Regarding status of the release effort, still pending
> METRON-1252, so
> > > > not making the release branch yet.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/17/17, 1:32 PM, "Matt Foley"  wrote:
> > > >
> > > > (With release manager hat on)
> > > >
> > > > The community has proposed a release of Metron in the near
> > > future,
> > > > focusing on Meta-alerts running in Elasticsearch.
> > > > Congrats on getting so many of the below already done. At this
> > > > point, only METRON-1252, and the discussion of how to handle
> joint
> > > release
> > > > of the Metron bro plugin, remain as gating items for the
> release. I
> > > > project these will be resolved next week, so let’s propose the
> > following:
> > > >
> > > > Sometime next week, after the last bits are done, I’ll start the
> > > > release process and create the release branch.
> > > >
> > > > The proposed new version will be 0.4.2, unless there are backward
> > > > incompatible changes that support making it 0.5.0.
> > > > Currently there are NO included Jiras labeled
> > > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > > ***If anyone knows that some of the commits included since 0.4.1
> > > > introduce backward incompatibility, please say so now on this
> thread,
> > and
> > > > mark the Jira as such.***
> > > >
> > > > The 90 or so jiras/commits already in master branch since 0.4.1
> > > > are listed below.
> > > > Thanks,
> > > > --Matt
> > > >
> > > > METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> > > > Filters Some Records (nickwallen) closes apa

Re: [DISCUSS] Community Meetings

[zeo...@gmail.com]

This sounds great Otto, thanks.

Jon

On Thu, Dec 14, 2017 at 11:24 AM Laurens Vets  wrote:

>
> Sounds good to me :)
>
> On 2017-12-14 05:59, Otto Fowler wrote:
> > Ok,
> >
> > So we will be concerned with two types of meetings.  I’ll take
> > responsibility for calling the meetings and ‘moderation’.
> >
> > Dev meetings
> >  - feedback on how things are going overall
> >  - discussions on specific technical problems
> >  - discussion of possible improvements
> >
> > User meetings
> >  - demos
> >  - user content ( how I’m using metron )
> >  - some unavoidable discussion on problems
> >  - some requirements gathering triage
> >
> > ALL
> >  - I will call
> >  - I will gather input for agenda
> >  - I will distribute the agenda
> >  - I will distribute the notes to the list and on confluence
> >  - No decisions will be made, only discussed and then put to list
> >  - besides general nodes, breakout messages for topical discussion or
> > decisions
> >
> >
> >
> > How does that sound?
> >
> >
> > On December 13, 2017 at 16:41:29, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > +1
> >
> >
> > On December 13, 2017 at 16:39:52, James Sirota (jsir...@apache.org)
> > wrote:
> >
> > I can set up a dedicated Zoom room with a recurrent meeting and give
> > PMC
> > members rights to the room. I think hosting these meetings should not
> > be a
> > problem. I would vote not to record them, but rather provide the notes
> > after the meeting. It's a lot easier to skim through the notes than
> > jump
> > around in a recording. As Simon mentioned, I would also make it
> > explicitly
> > clear that the meetings are dev meetings. These are not user Q&A and
> > are
> > not meant to be overviews of how different features of Metron work. If
> > we
> > want to do feature demos or provide user content I would want that to
> > be in
> > its own separate meeting.
> >
> > Thanks,
> > James
> >
> > 13.12.2017, 05:00, "Otto Fowler" :
> >> I am ok with just notes and no recording.
> >>
> >> On December 13, 2017 at 04:37:20, Simon Elliston Ball (
> >> si...@simonellistonball.com) wrote:
> >>
> >> Good points Larry, we would need to get consent from everyone on the
> >> call
> >> to record to properly comply with regulations in some countries. We
> >> would
> >> definitely need someone to step up as note taker.
> >>
> >> Something else to think about is intended audience. Previously we’ve
> >> had
> >> meeting like this which have been very detailed Dev@ focussed (which
> >> is a
> >> great thing) but have rather alienated participants in User@ land. We
> >> need
> >> to make it clear what level we’re talking about to be inclusive.
> >>
> >> Simon
> >>
> >>>  On 13 Dec 2017, at 00:44, larry mccay  wrote:
> >>>
> >>>  Not sure about posting the recordings - you will need to check and
> >>> make
> >>>  sure that doesn't violate anything.
> >>>
> >>>  Just a friendly reminder...
> >>>  It is important that meetings have notes and a summary that is sent
> >>> out
> >>>  describing topics to be decided on the mailing list.
> >>>  No decisions can be made in the community meeting itself - this
> >>> gives
> >>>  others in other timezones and commitments review and voice in the
> >>
> >> decisions.
> >>>  If it didn't happen on the mailing lists then it didn't happen. :)
> >>>
> >>>  On Tue, Dec 12, 2017 at 1:39 PM, Simon Elliston Ball <
> >>>  si...@simonellistonball.com> wrote:
> >>>
>   Yes, I do.
> 
>   I suspect the best bet will be to post recordings somewhere on the
>   apache.org  metron site.
> 
>   Simon
> 
> >  On 12 Dec 2017, at 18:36, Otto Fowler 
> > wrote:
> >
> >  Excellent, do you have the > 40 min + record option?
> >
> >  On December 12, 2017 at 13:19:55, Simon Elliston Ball (
> >  si...@simonellistonball.com) wrote:
> >
> >  Happy to volunteer a zoom room. That seems to have worked for most
> > in
> >>
> >> the
> >  past.
> >
> >  Simon
> >
> >>  On 12 Dec 2017, at 18:09, Otto Fowler 
> > wrote:
> >>
> >>  Thanks! I think I’d like something hosted though.
> >>
> >>  On December 12, 2017 at 11:18:52, Ahmed Shah (
>   ahmeds...@cmail.carleton.ca)
> >>  wrote:
> >>
> >>  Hello,
> >>
> >>  wrt "- How are we going to host it"...
> >>
> >>  I've used BigBlueButton as an end user at our University.
> >>
> >>  It is LGPL open source.
> >>
> >>  https://bigbluebutton.org/
> >>  https://bigbluebutton.org/developers/
> >>
> >>  -Ahmed
> >>
> >>  ___
> >>  Ahmed Shah (PMP, M. Eng.)
> >>  Cybersecurity Analyst & Developer
> >>  GCR - Cybersecurity Operations Center
> >>  Carleton University -
> >> cugcr.com
> >>
> >>  
> >>  From: Otto Fowler 
> >>  Sent: December 11, 2017 4:4

Re: [DISCUSS] Stellar Documentation Autogeneration

[zeo...@gmail.com]

A huge +1 from me.  This would be great

On Thu, Dec 14, 2017 at 3:39 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 from me, great idea Justin. I did a bit of digging around also and the
> Doclet approach you're already using seems the way to go. I didn't come
> across any libraries that would make this easier or better. Not sure if
> Swagger has anything along these lines?
>
> On Thu, Dec 14, 2017 at 1:00 PM, Otto Fowler 
> wrote:
>
> > I think this is a great idea, and I looked at the POC and it isn’t as bad
> > as you make it out to be;)
> >
> > What I would like to see is documentation for Stellar functions, by
> > namespace generated. I would also
> > like the capability to document at the namespace level.
> >
> > Often we have namespace level concepts that don’t fit into any given
> > function’s documentation.
> > Setting aside the how of the namespace documentation for a moment, based
> on
> > the POC I would
> > suggest that we
> >
> > * find all namespaces
> > * create a page per namespace
> > * document each function in it’s namespace’s page
> > * include the namespace doc in that page
> >
> > Each module that exports stellar function’s should have it’s own
> > documentation.  As part of breaking stellar out to it’s own module
> > we should remove stellar documentation from stellar common that applies
> to
> > functions outside that module.
> >
> >
> >
> > On December 14, 2017 at 14:32:56, Justin Leet (justinjl...@gmail.com)
> > wrote:
> >
> > I think it would be valuable to have the documentation around Stellar
> being
> > autogenerated. We have most of the info we'd want in the @Stellar
> > annotation, and ideally, we could just pull this info out and produce
> some
> > docs similar to what we already manually maintain. This came up a bit in
> > the context of https://issues.apache.org/jira/browse/METRON-1361
> >
> > I put together a super, super (super!) rough POC of using the approach of
> > Javadoc-style doclet processing that reads the annotations and kicks out
> > something pretty close to the current docs (without any fancy stuff like
> > the table of contents and so on).
> >
> > Right now, there'd be a good deal more to do that to make it usable. Off
> > the top of my head, the main things I wanted to look at before really
> even
> > taking an actual stab at it are
> >
> > 1) abstracting out the markdown formatting from the annotation parsing
> > 2) Making sure we can integrate this approach without breaking current
> > Javadocs
> > 3) Managing things across projects (since we put in Stellar functions all
> > over).
> > 4) Slightly more though about how we'd manage it.
> >
> > Otto's alluded to having a couple thoughts, and I'm more than happy to
> get
> > a better idea of what we want the end state to look like (either this or
> > something else, e.g. an annotation processor during compile phase or if
> > someone knows a tool that takes care of this sort of thing.)
> >
> > Any thoughts?
> >
>
-- 

Jon

Re: [DEV COMMUNITY MEETING] Call for Ideas and Schedule

[zeo...@gmail.com]

I like your list of potential topics.  I'm also in to attend - that time
works well for me.

I would be interested in talking about our release process, as I would like
to suggest that we formalize upgrade and installation instructions to be
included as a part of a release, and talk through any concerns/questions
with the secondary repo.  I know I have at least one discussion point on
each of those suggestions.

Also, maybe we can chat about adding security scanning to our code base,
including on PRs.

Jon

On Thu, Dec 14, 2017, 15:43 Otto Fowler  wrote:

> Dev Community Meeting Call
>
> I would like to propose a developer community meeting.
>
> I propose that we set the meeting early next week, and will throw out
> Monday, December 18th at 09:30AM PST, 12:30 on the East Coast and 5:30 in
> London Towne.
>
> This meeting will be held over a web-ex, the details of which will be
> included in the actual meeting notice.
>
> Please reply to this with scheduling concerns and topic suggestions.
> Potential Topics
>
>- Call for reviewers, ideas how to get more involvement, what people can
>do to help
>- Feature branches : we have two now, what are they and how are we going
>to work on them
>- Extension Repository: Default deployment and installation of parsers
>as it relates to ‘777’
>- General ‘777’ discussion
>
> Developer Community Meeting Disclaimers
>
>- Developer Community meetings are a means for realtime discussion of
>development issues
>- These meetings are not specifically aimed at demonstrations, unless
>one is required or requested as part of such discussion
>- These meetings are geared towards Metron development issues, not user
>issues with deployment or shipped functionality
>- There are *NO* decisions made in these meetings. The mailing list is
>the official communication record of the Apache Metron Project, and as
> such
>all public decisions are to be made on the list, as to give the greatest
>opportunity for community involvement.
>- There *ARE* proposals that can be made and discussed in these
>meetings, that will then be discussed on list for decision.
>- Notes will be taken of these meetings, and they will be posted to the
>list
>- There may also be breakout posts to the list per proposal or topic,
>for more detailed discussion
>
-- 

Jon

Re: [VOTE] Metron Release Candidate 0.4.2-RC2

[zeo...@gmail.com]

+1 (non-binding), also validated using Otto's script (super good work).

Downloaded, validated checksums/sigs, bulit, ran tests, spun up full-dev,
did some basic poking around.

Jon

On Tue, Dec 19, 2017 at 2:45 PM Nick Allen  wrote:

> +1  I validated using Otto's great script.
>
> * Validated the list of changes
> * Checksums
> * Sigs
> * Build
> * Tests
> * Full Dev
>
> On Tue, Dec 19, 2017 at 6:23 AM, Matt Foley  wrote:
>
> > Colleagues,
> > This is a call to vote on releasing Apache Metron 0.4.2 and its
> associated
> > metron-bro-plugin-kafka 0.1.0.
> > The release candidate is available at https://dist.apache.org/repos/
> > dist/dev/metron/0.4.2-RC2/
> >
> > Full list of changes in this release:
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/CHANGES and
> >
> https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/CHANGES.bro-plugin
> >
> > The github tags to be voted upon are:
> > (apache/metron) apache-metron-0.4.2-rc2 and
> (apache/metron-bro-plugin-kafka)
> > 0.1
> >
> > The source archives being voted upon can be found here:
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/
> > apache-metron-0.4.2-rc2.tar.gz
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/
> > apache-metron-bro-plugin-kafka_0.1.0.tar.gz
> >
> > The site-book is at:
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/
> > site-book/index.html
> >
> > Other release files, signatures and digests can be found here:
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/
> >
> > The release artifacts are signed with the following key:
> > 4169 AA27 ECB3 1663 in https://dist.apache.org/repos/
> > dist/dev/metron/0.4.2-RC2/KEYS
> >
> > Please vote on releasing this package as Apache Metron 0.4.2 and Apache
> > Metron-bro-plugin-kafka 0.1.0
> >
> > When voting, please list the actions taken to verify the release.
> >
> > Recommended build validation and verification instructions are posted
> here:
> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > or you are encouraged to try the new release verification script that
> Otto
> > published via email on 11 Dec, available at
> > https://github.com/ottobackwards/Metron-and-Nifi-
> > Scripts/blob/master/metron/metron-rc-check
> >
> > This vote will be open until 9am PST on Friday 22 Dec 2017.
> >
> > Thank you,
> > --Matt
> >
> >
> >
> >
> >
>
-- 

Jon

Re: [DISCUSS] Lowering the barrier to entry to for new users

[zeo...@gmail.com]

I agree we should streamline #2 and lower the bar, and we can readdress if
we are getting PRs that don't follow the contributing guidelines.  We
should also make a contributing.md as not everybody knows about the wiki.

For #3, I think the scripts that Nick, Otto, and others have written for
looking at PRs, testing RCs, etc. should get moved in and referenced more
frequently to users.  I believe progress on this has already started, but
we should continue to contribute little utils that simplify baseline tasks.

Finally, I think the migration of docs from wiki to git, and hosted in the
site-book is very important.  I know this was on my list to do and I will
still happily do it when I can get to it, but it may unfortunately be a
while.  Happy to take volunteers or crowd source this a bit.

Jon

On Tue, Dec 19, 2017, 11:56 Laurens Vets  wrote:

> On 2017-12-19 06:19, Justin Leet wrote:
> > One of the topics that came up in recent community meeting was about
> > lowering the barrier to entry for new users.
> >
> > This is a fairly broad topic that I think covers a few different
> > subtopics.
> >
> > 1) Addressing (or making it easier to address) some of the things we've
> > seen on the user group from people getting started.
> > 2) Making contributing easier and the ways to do so more obvious.  This
> > includes things like making it easier to find on our site (compare our
> > page
> > to Storm's, for example).  It also includes things like reassessing our
> > PR
> > template (For example, is everything still useful enough to keep it?).
> > 3) Anything else that would make help users adopt Metron and become
> > actively involved in reviewing, fixes, docs, and all the other sorts of
> > things that make our stuff better.
> >
> > I'm mostly going to open this up to a general discussion and
> > brainstorming,
> > and presumably we come out with some tickets at the end of this.
>
> I use a github gist that Otto once created for me, maybe I can try and
> turn that into something more official?
>
-- 

Jon

Re: [DISCUSS] Stellar in a Zeppelin Notebook

[zeo...@gmail.com]

This is some awesome work, I'm looking forward to being able to play with
it.

Jon

On Tue, Dec 19, 2017 at 1:12 PM Nick Allen  wrote:

> Yes, I definitely want auto-complete also.
>
> I am factoring out some of the logic you did for auto-complete in the REPL
> in hopes of being able to apply that in Zeppelin.  I believe I saw
> references in the other Zeppelin interpreters for handling auto-complete,
> but I'll have to research it a bit more.
>
> On Tue, Dec 19, 2017 at 1:07 PM, Casey Stella  wrote:
>
> > I love it!  I wonder if we could get more of the REPL-like experience
> > (i.e. I crave autocomplete ;) if we integrated it with jquery shell like
> > they did with nanook (https://github.com/aeshell/nanook).  I know
> > zeppelin lets you integrate with more complex javascript.   Regardless,
> > this is awesome, great job!
> >
> > On Tue, Dec 19, 2017 at 12:44 PM, Otto Fowler 
> > wrote:
> >
> >> That looks great!
> >>
> >>
> >> On December 19, 2017 at 12:34:47, Nick Allen (n...@nickallen.org)
> wrote:
> >>
> >> Ah, dang.  Hopefully this works...
> >>
> >> https://www.dropbox.com/s/44qz3518dn4jtzq/Stellar%20in%20a%
> >> 20Zeppelin%20Notebook.png?dl=0
> >>
> >> On Tue, Dec 19, 2017 at 10:23 AM, Otto Fowler 
> >> wrote:
> >>
> >> > The image is stripped for me, can you post it as a link?
> >> >
> >> > This seems like it would look awesome ;)
> >> >
> >> >
> >> > On December 19, 2017 at 10:03:26, Nick Allen (n...@nickallen.org)
> >> wrote:
> >> >
> >> > (1) I love the REPL, but I hate how inaccessible it is.
> >> >
> >> > (2) I love our use cases
> >> >  >> raphic_login_outliers> and
> >> > examples
> >> >  >> cs/metron-profiler#creating-profiles>,
> >> > but I hate how difficult it is for a new user to run them.
> >> >
> >> > (3) I love the extensibility of Metron, but I hate looking at JSON.
> >> >
> >> > (4) I love the Profiler, but I hate not being able to *see* my
> profiles
> >> as
> >> > plots.
> >> >
> >> > ...
> >> >
> >> > Let me introduce, Stellar running in a Zeppelin Notebook.
> >> >
> >> > (1) Access the REPL from any web browser.
> >> >
> >> > (2) Create executable use cases that can be easily shared between
> users.
> >> >
> >> > (3) Use the simpler management functions to interact with Metron (less
> >> > JSON).
> >> >
> >> > (4) Extract your profiles and create a time series plot.
> >> >
> >> >
> >> >
> >> > [image: Inline image 1]
> >> > The screenshot above is a very lightweight MVP showing that we can run
> >> > Stellar from Zeppelin.  I have a lot more work ahead in refactoring
> the
> >> > existing Stellar Shell/REPL functionality so that we get the same
> >> > experience in Zeppelin as we get on the command line.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
>
-- 

Jon

Re: [DISCUSS] Resources for how to contribute to Apache Metron

[zeo...@gmail.com]

For nearly everybody I've talked to about this project that had complaints,
I've heard something about the significant barrier to entry, divided into
two general categories.  Category 1 is that a lot of security teams lack
substantial experience with Hadoop and would like to get a better
understanding of how the involved components fit together - not
just kafka goes to storm goes to kafka, or a link to the kafka docs for
details about kafka, but a little bit more detail as to _why_ those
components are in use in metron, what properties those components possess
at a high level _which makes them appealing to us_, and how they're
_currently used_ in the metron environment.  Category 2 is that it is
generally more difficult than it should be to get a testing/poc environment
running - running it on a laptop (especially non-macOS) can be a pain to
get running, some laptops simply cannot run it, etc.  I've heard a few
times that a company uses Azure (not AWS) and they would like to quickly
spin it up there.

Just my $0.02

Jon

On Tue, Dec 19, 2017 at 9:02 AM Otto Fowler  wrote:

> Like any project, Apache Metron needs to maintain and grow it’s contributor
> community. We think that we could be doing a better job of this, and would
> like to discuss issues and possible improvements. Issues
>
> What are some of the issues that may inhibit people contributing?
>
>- Barrier of entry (issues getting Metron running in vagrant or local)
>- Documentation : finding current
>- Documentation : content and quality
>- Source Code navigation/documentation/guides
>- Testing guides
>- Use Case Guides
>- Don’t know how they *can* contribute
>- Others that I’m missing?
>
> Remediation Barrier of entry
>
> How can we make the local deployment workflow easier ( other discuss thread
> touches on this)?
> Documentation : Finding Current
>
> When people look for Metron info, where are they looking? What comes up in
> search? - Hortonworks Community forums ( preview release stuff ? ), old
> blog posts? - Mailing list archives? - wiki? (not current) - site-book?
>
> How can we reduce the out of data information, and make the relevant
> information more prominent?
> Documentation : Content and Quality
>
> ( this is a little bit of a chicken and egg issue, since documentation is a
> wonderful way to contribute…. ) - Up to data architecture documentation -
> Non-developer focused ‘feature’ documentation - Developer focused
> documentation ( how to add a XX guides )
> Source Code Guides
>
>- Structure of the code tree
>- What is where, how it is logically setup
>- How to maintain concistancy when working in the code
>- Javadoc
>
> Testing Guides
>
>- Tests that we have are buried in PR’s
>- No regression tests
>
> Use case guides
>
>- more how-to guides
>
> Contributing guide
>
>- right now, have dev env guide
>- review and submit doc changes
>- review PR guide
>- pr testing guide ( better pr testing steps?)
>
> These are things I can think of, anyone have any comment, additions,
> priorities?
>
-- 

Jon

Re: Secure code analysis

[zeo...@gmail.com]

Just following up on this conversation again -

I have discussed this ad-hoc with a few PMC members recently and wanted to
bring it up on the list.  Veracode has provided us with a 100% free portal
to scan the Metron code with, but in order to integrate, the safest option
is probably to use the ASF's jenkins server (as I'm not aware of a safe way
to automatically pass API creds to Veracode from GitHub).  My long-term
interest here would be to scan and clean up the code base generally, and
then to try and scan PRs for concerns (non-blocking).  Perhaps at some
point, if we identify that these scans are actually useful and not
false-positive prone/onerous, we could turn this into a blocking
requirement for contributions.  Being a security project, I feel that we
should be doing as much as we can to ensure that what we're providing is
safe.

I looked briefly at the Veracode Jenkins integrations, and the ASF Jenkins
setup.  It looks like Veracode has a Jenkins plugin
<https://help.veracode.com/reader/PgbNZUD7j8aY7iG~hQZWxQ/_4G8gT1rhWMgVVtCI1C57A>,
Jenkins has a plugin for Veracode in its plugin repo
<https://plugins.jenkins.io/veracode-scanner> (not supported by Veracode),
the ASF supports adding plugins
<https://wiki.apache.org/general/Jenkins#How_do_I_install_a_new_Jenkins_plugin.3F>
to their Jenkins servers (although I think
<http://What_do_Administrators_do.3F> the admins are supposed to do this),
and Metron is not yet set up <https://builds.apache.org/view/M-R/> on the
ASF Jenkins server.  The ASF seems to support giving non-PMC committers
access <https://wiki.apache.org/general/Jenkins#How_do_I_get_an_account> to
Jenkins, but it requires that the PMC chair do some work, and generally it
looks like they want admins
<https://wiki.apache.org/general/Jenkins#FAQ_For_Administrators>/PMC
<https://wiki.apache.org/general/Jenkins#FAQ_For_PMCs> members to be
involved (I also don't have access to the builds JIRA project
<https://issues.apache.org/jira/projects/BUILDS>, if it really exists).

I'm happy to play around with this and see how it could be useful, but in
order to do so I need to get some additional authorization.  Does anybody
have any concerns with delegating this access to me, or with this general
approach?

Jon

On Fri, Dec 16, 2016 at 11:39 AM James Sirota  wrote:

> That would be great. I can work with them
>
> 15.12.2016, 18:38, "zeo...@gmail.com" :
> > I recently discussed this topic with Veracode regarding the metron
> project
> > and they mentioned there may be interest in providing free services,
> > however they would need to work with an official project rep. If there's
> > interest in pursuing this please let me know.
> >
> > On Thu, Jun 2, 2016, 21:17 zeo...@gmail.com  wrote:
> >
> >>  Per the other discussion it is possible that this conflicts with the
> >>  Apache stance for vulnerability disclosure/management. I'm going to
> hold
> >>  off on any additional effort until I know more.
> >>
> >>  Jon
> >>
> >>  On Tue, May 31, 2016, 16:07 James Sirota  wrote:
> >>
> >>  Jon, would it be possible for you to scan Metron from your own branch?
> >>  I'd like to know if this is useful at all. If we get value out of it
> I'll
> >>  run this down and see how we can get it hooked up.
> >>
> >>  31.05.2016, 10:08, "Nick Allen" :
> >>  > I connect Travis to my own personal fork of Metron so that the CI
> builds
> >>  > run on my own branches before I submit PRs. Thinking you could do the
> >>  same
> >>  > with this. Maybe I'm wrong.
> >>  >
> >>  > On Tue, May 31, 2016 at 1:06 PM, zeo...@gmail.com 
> >>  wrote:
> >>  >
> >>  >> To register project on Coverity Scan, you must be contributor or
> >>  maintainer
> >>  >> of the project.
> >>  >>
> >>  >> It may also be worth mentioning that there are a ton of Apache
> projects
> >>  >> already registered, including Ambari, Drill, Flume, Hadoop, HBase,
> >>  NiFi,
> >>  >> Oozie, Ranger, Sqoop, Spark, Storm, Tez, etc. See
> >>  >> https://scan.coverity.com/projects?page=2
> >>  >>
> >>  >> Jon
> >>  >>
> >>  >> On Tue, May 31, 2016 at 12:52 PM Nick Allen 
> >>  wrote:
> >>  >>
> >>  >> > You could set it up on your own fork of Metron in Github. Then you
> >>  can
> >>  >> > tell us if it is useful at all.
> >>  >> >
> >>  >> > On Sat, May 28, 2016 at 2:36 PM, zeo...@gmail.c

Re: Secure code analysis

[zeo...@gmail.com]

Sure, not a problem.

(1) I went to an event where a presenter from Veracode was calling out some
bugs in open source projects, and that Veracode wanted to be a part of the
solution.  As such, they offered to give free analysis to open source
projects that reach out.  At this point the account that I have access to
is just for the Apache Metron project, but it is possible that the
relationship could grow if it makes sense for other projects.  For
instance, this <https://twitter.com/PeteChestna/status/943845893597483008>.

(2) No specific reason - in the past I looked at Coverity (see below in
this thread) but was deterred from personally setting it up due to some of
their policies about who can register new scans (i.e. I was not a committer
at the time I believe, and that level of involvement was requested).  I
have used Veracode in the past, along with others (AppScan, Fortify, etc.),
and had a good experience albeit in a very different setting than this.  I
would be more than happy to play around with any of these kinds of services
and no affinity to one or the other, but right now the only thing I
actually have access to is Veracode and free options like Coverity.

Veracode is a proprietary cloud-hosted platform that has dynamic and static
scan offerings, and they have various integrations
<https://community.veracode.com/s/integrations> with build systems (maven,
Jenkins, Bamboo, etc.) and IDEs (IntelliJ, Eclipse, etc.).  They also
appear to have opened up their training materials
<https://community.veracode.com/s/education-and-training>, which are handy
to point to from time to time.  I've worked with it in the past and things
largely seem to work as you would expect, although it has been 5 years
since I really used their products regularly.

(3) I have been manually making submissions dating back to 2017-02-13, but
because the file transfer is uploaded from my home Internet (upload speeds
of ~6Mbps), it takes quite a while and so I don't do it very frequently.
Usually just around releases.

Jon

On Sat, Dec 23, 2017 at 11:13 AM Nick Allen  wrote:

> > Veracode has provided us with a 100% free portal to scan the Metron code
> with, but in order to integrate, the safest option is probably to use the
> ASF's jenkins server
>
> (1) Can you describe this more?   How has this been provided?  Is this for
> all Apache projects; just Metron?  Was this based on a relationship you
> have within CA?
>
>
> (2) Why Veracode?  Can you describe this platform more?  Is it open source
> or proprietary?  Why is this better than alternatives?
>
>
> (3) I have no objection to experimenting with the service to see if it
> provides actionable results, but is there no simpler way to do this?  It
> doesn't seem like we should have to mess with a bunch of Apache
> infrastructure to see if the service works at a basic level.  Can't we
> manually submit master and/or previous releases to Veracode to see if we
> get actionable results?
>
>
>
>
>
> On Thu, Dec 21, 2017 at 10:48 AM, zeo...@gmail.com 
> wrote:
>
> > Just following up on this conversation again -
> >
> > I have discussed this ad-hoc with a few PMC members recently and wanted
> to
> > bring it up on the list.  Veracode has provided us with a 100% free
> portal
> > to scan the Metron code with, but in order to integrate, the safest
> option
> > is probably to use the ASF's jenkins server (as I'm not aware of a safe
> way
> > to automatically pass API creds to Veracode from GitHub).  My long-term
> > interest here would be to scan and clean up the code base generally, and
> > then to try and scan PRs for concerns (non-blocking).  Perhaps at some
> > point, if we identify that these scans are actually useful and not
> > false-positive prone/onerous, we could turn this into a blocking
> > requirement for contributions.  Being a security project, I feel that we
> > should be doing as much as we can to ensure that what we're providing is
> > safe.
> >
> > I looked briefly at the Veracode Jenkins integrations, and the ASF
> Jenkins
> > setup.  It looks like Veracode has a Jenkins plugin
> > <https://help.veracode.com/reader/PgbNZUD7j8aY7iG~hQZWxQ/
> > _4G8gT1rhWMgVVtCI1C57A>,
> > Jenkins has a plugin for Veracode in its plugin repo
> > <https://plugins.jenkins.io/veracode-scanner> (not supported by
> Veracode),
> > the ASF supports adding plugins
> > <https://wiki.apache.org/general/Jenkins#How_do_I_
> > install_a_new_Jenkins_plugin.3F>
> > to their Jenkins servers (although I think
> > <http://What_do_Administrators_do.3F> the admins are supposed to do
> this),
> > and Metron is not yet set up <https://bui

Re: Anand is a new Committer!

[zeo...@gmail.com]

Welcome aboard, Anand!  Congrats

Jon

On Thu, Jan 11, 2018 at 10:41 AM Otto Fowler 
wrote:

> Congratulations and welcome Anand!
>
>
> On January 11, 2018 at 09:29:24, Casey Stella (ceste...@gmail.com) wrote:
>
> The Project Management Committee (PMC) for Apache Metron has invited Anand
> Subramanian to become a committer and we are pleased to announce that they
> have accepted.
>
> Congratulations and welcome, Anand!
>


-- 

Jon

Re: [DISCUSS] Time to remove github updates from dev?

[zeo...@gmail.com]

I would give that  +1 as well.

Jon

On Fri, Jan 19, 2018 at 3:32 PM Casey Stella  wrote:

> I could get behind that.
>
> On Fri, Jan 19, 2018 at 3:31 PM, Andre  wrote:
>
> > Folks,
> >
> > May I suggest Metron follows the NiFi mailing list strategy (we got
> > inspired by another project but I don't recall the name) and remove the
> > github comments from the dev list?
> >
> > Within NiFi we have both the dev and the issues lists. dev is for humans,
> > issues is for JIRA and github commits.[1]
> >
> > This allows the list thread list to be cleaner and is particularly
> helpful
> > for those reading the list from a list aggregation service.
> >
> > Cheers
> >
> >
> > [1] https://lists.apache.org/list.html?iss...@nifi.apache.org
> >
>


-- 

Jon

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

[zeo...@gmail.com]

I agree with having a metron_ prefix for ES indexes, and the timing.

Jon

On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> With the completion of https://github.com/apache/metron/pull/840
> (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings for a
> major release rev of Metron in the upcoming release (currently slotted to
> 0.4.3, I believe). Since there are non-backwards compatible changes
> pertaining to ES indexing, it seems like a good opportunity to revisit our
> index naming standards.
>
> I propose we add a simple prefix "metron_" to all Metron indexes. There are
> numerous reasons for doing so
>
>- removes the likelihood of index name collisions when we perform
>operations on index wildcard names, e.g. "enrichment_*, indexing_*,
> etc.".
>- ie, this allows us to be more friendly in a multi-tenant ES
>environment for relatively low engineering cost.
>- simplifies the Kibana dashboard a bit. We currently needed to create a
>special index pattern in order to accommodate multi-index pattern
> matching
>across all metron-specific indexes. Using metron_* would be much simpler
>and less prone to error.
>- easier for customers to debug and identify Metron-specific indexes and
>associated data
>
>
> The reason for making these changes now is that we already have breaking
> changes with ES. Leveraging existing indexed data rather than deleting
> indexes and starting from scractch already requires a re-indexing/migration
> step, so there is no additional effort on the part of users if they choose
> to attempt a migration. It further makes sense with our current work
> towards upgrading Solr.
>
> We already have a battery of integration and manual tests after the ES
> upgrade work that can be leveraged to validate the changes.
>
> Mike Miklavcic
>


-- 

Jon

Re: Metron User Community Meeting Call

[zeo...@gmail.com]

Thanks Otto, I'm in to attend at that time/place.

Jon

On Thu, Jan 25, 2018, 14:45 Otto Fowler  wrote:

> I would like to propose a Metron user community meeting. I propose that we
> set the meeting next week, and will throw out Wednesday, January 31st at
> 09:30AM PST, 12:30 on the East Coast and 5:30 in London Towne. This meeting
> will be held over a web-ex, the details of which will be included in the
> actual meeting notice.
> Topics
>
> We have a volunteer for a community member presentation:
>
> Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR -
> Cybersecurity Operations Center Carleton University - cugcr.com
>
> Ahmed would like to talk to the community about
>
>-
>
>Who the GCR group is
>-
>
>How they use Metron 0.4.1
>-
>
>Walk through their dashboards, UI management screen, nifi
>-
>
>Challenges we faced up until now
>
> I would like to thank Ahmed for stepping forward for this meeting.
>
> If you have something you would like to present or talk about please reply
> here! Maybe we can have people ask for “A better explanation of feature
> X” type things?
> Metron User Community Meetings
>
> User Community Meetings are a means for realtime discussion of experiences
> with Apache Metron, or demonstration of how the community is using or will
> be using Apache Metron.
>
> These meetings are geared towards:
>
>-
>
>Demonstrations and knowledge sharing as opposed to technical
>discussion or implementation details from members of the Apache Metron
>Community
>-
>
>Existing Feature demonstrations
>-
>
>Proposed Feature demonstrations
>-
>
>Community feedback
>
> These meetings are *not* for :
>
>-
>
>Support discussions. Those are best left to the mailing lists.
>-
>
>Development discussions. There is another type of meeting for that.
>
>
>
>

-- 

Jon

[REQUEST] Add Ian as an Assignee in JIRA

[zeo...@gmail.com]

Can someone add Ian Abreu as a potential assignee on JIRA?  He has a PR
 open against his
ticket  in the bro
plugin repo.  Thanks,

Jon
-- 

Jon

Re: [DISCUSS] Profiler Enhancement

[zeo...@gmail.com]

Scenario 2 is one that I'm specifically interested in, I have that exact
use case right now.  I can see Scenario 1 being useful in the future as
well.

I'm also interested in a conversation along the lines of what Otto brought
up (i.e. I would like to re-ingest data to redo parsing, enrichments, etc.)
but happy to keep that conversation separate or for the future.

Really just wanted to comment that this work effort has a huge +1 from me
and is something I've been following.

This work should interact nicely with METRON-1397
, because users may need to
ingest bulk data from time to time that is a result of a system export.

Jon

On Mon, Feb 5, 2018 at 9:38 AM Otto Fowler  wrote:

> I think that is fine,  we can use that and work out the UX to manage new or
> replace.  Maybe we can do Profile Compare down the line?
>
> On February 5, 2018 at 09:28:16, Nick Allen (n...@nickallen.org) wrote:
>
> > If we replay a set of data with a new version of a profile I think it
> will always have to be a new profile and not ‘replace’ the old one.
> Series1, Seriers2  etc?
>
> As part of this effort (unless there is a compelling reason) I wouldn't
> change that behavior.  The profile data is stored based on profile name +
> entity + timestamp (I'm glossing over some of the details, but that's
> effectively what happens).  If you change the definition of a profile, but
> the name does not change, then you would replace the existing profile
> data.  If you do not want to replace, then you should change the name of
> the profile.
>
> Now is this the best way to store the data?  I am not sure.  It is a
> complex discussion all by itself, but is something that I would rather
> handle as a separate effort.
>
>
>
> On Fri, Feb 2, 2018 at 5:42 PM, Otto Fowler 
> wrote:
>
> > You know, I am going to back this up.
> > I usually thing of replay as replay, profiler or not, but that is not
> true.
> > Replay of data through the full pipeline (parsers/enrichement) has more
> > consequences or concerns, so we can drop this.
> > I don’t want to expand the scope of your idea.  We can reuse/refactor to
> > the other case (parser + enrichment) later.
> > Sorry.
> >
> >
> > ——
> >
> > So, about re-writing.
> > If we replay a set of data with a new version of a profile I think it
> will
> > always have to be a new profile and not ‘replace’
> > the old one.   Series1, Seriers2  etc?
> >
> >
> >
> >
> > On February 2, 2018 at 17:24:46, Nick Allen (n...@nickallen.org) wrote:
> >
> > I think that is definitely a reasonable extension.
> >
> > In this case would we need any additional actions to indicate that data
> > will be overwritten?
> >
> > I am trying to think of other additional needs that this use case has
> over
> > the others.
> >
> > On Feb 2, 2018 12:38 PM, "Otto Fowler"  wrote:
> >
> >> Scenario 3:
> >> As a Security ?  I have modified a profile or parser configuration (
> >> replay is replay ), and I want to run the new version
> >> against my old data.
> >>
> >>
> >>
> >> On February 2, 2018 at 12:19:54, Nick Allen (n...@nickallen.org) wrote:
> >>
> >> I have been thinking about an enhancement to the Profiler for quite some
> >> time. Actually, my first pass at defining this was called "Replay
> >> Telemetry through Profiler" back in METRON-594 [1].
> >>
> >> I'd like to first discuss the use case to make sure we start out on the
> >> right foot. Here is how I would define the use cases for this
> >> functionality.
> >>
> >> *> Scenario 1: Model Development*
> >>
> >> As a Security Data Scientist, I want to understand the historical
> >> behaviors
> >> and trends of a profile that I have created so that I can understand if
> it
> >> is valuable for model building.
> >>
> >> There are two possible negative outcomes that the Security Data
> Scientist
> >> must be aware of when creating profiles.
> >>
> >>
> >> - The profile might have been defined incorrectly resulting in a feature
> >> set that does not match reality (a bug in the profile definition).
> >>
> >>
> >> - The profile might have been defined correctly, but the feature set
> >> itself has no predictive value.
> >>
> >> Analyzing the profile over archived, historical telemetry allows the
> >> Security Data Scientist to better to mitigate both of these negative
> >> outcomes.
> >>
> >>
> >> *> Scenario 2: Model Deployment*
> >>
> >> As a Security Platform Engineer, I want to generate a profile using
> >> archived telemetry when I deploy a new model to production so that
> models
> >> depending on that profile can begin to function on day 1.
> >>
> >>
> >>
> >> (Q) Do these make sense? Am I missing anything? Too broad or too narrow?
> >>
> >> Once we nail down the use case(s), I'll delete the old JIRA and create a
> >> new JIRA with the use cases. That would give us a place to start on the
> >> technical details of the implementation.
> >>
> >> [1] https://issues.apache.org/jira/browse/METRON-594
> >>
> >>
>
-- 

Jon

DataWorks Summit San Jose

[zeo...@gmail.com]

Hi All,

Just a heads up that *the San Jose DataWorks Summit's call for papers is
coming to a close soon *(February 9th, in 2 days!).  If you are doing
anything cool with open source big data and security that you want to talk
about, please submit to the Cyber Security track.  I'm hoping to attend and
I'm sure there will be others from the community in attendance as well.

Feel free to reply here (or to me directly) if you have any questions about
submitting or even if you just want to meet up at the event.  For more
information check out the website here
.

Jon
-- 

Jon

Re: metron-bro-plugin kafka

[zeo...@gmail.com]

Try

redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG,
FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG,
Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG,
Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);

Note that you usually wouldn't want to send reporter.log, as that's where
errors get sent and it could become an infinite loop.

Jon

On Tue, Feb 13, 2018, 05:26 bharath phatak  wrote:

> Hi Team,
>
> Can some one help me out on the list of
> redef Kafka::logs_to_send values?
>
> I want to push all logs generated by bro to Kafka.
>
> I tried adding log file name but getting bro is crashing
>
> Ex weird::LOG, Files::LOG
>
> Thanks,
> Bharath
>


-- 

Jon

Re: metron-bro-plugin kafka

[zeo...@gmail.com]

Okay, great.  It's possible that you need to do something like the
following to get known devices:

 echo "redef Software::asset_tracking = ALL_HOSTS;" >>
/usr/local/bro/share/bro/site/local.bro

These snippets are from my testing instructions related to adding support
for bro 2.5.2 logs (link <https://github.com/apache/metron/pull/844>).
They should find their way into the plugin README eventually.

Jon

On Tue, Feb 13, 2018 at 6:35 AM bharath phatak 
wrote:

> Hi Jon,
>
> Other than Known::DEVICES_LOG rest all worked.
>
> Thanks,
> Bharath
> On Tue, Feb 13, 2018, 4:15 PM zeo...@gmail.com  wrote:
>
> > Try
> >
> > redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG,
> > FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG,
> > Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG,
> > Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);
> >
> > Note that you usually wouldn't want to send reporter.log, as that's where
> > errors get sent and it could become an infinite loop.
> >
> > Jon
> >
> > On Tue, Feb 13, 2018, 05:26 bharath phatak 
> > wrote:
> >
> > > Hi Team,
> > >
> > > Can some one help me out on the list of
> > > redef Kafka::logs_to_send values?
> > >
> > > I want to push all logs generated by bro to Kafka.
> > >
> > > I tried adding log file name but getting bro is crashing
> > >
> > > Ex weird::LOG, Files::LOG
> > >
> > > Thanks,
> > > Bharath
> > >
> >
> >
> > --
> >
> > Jon
> >
>
-- 

Jon

Re: [DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron

[zeo...@gmail.com]

I agree, the first approach makes the most sense to me.

Jon

On Wed, Feb 21, 2018 at 11:45 AM Nick Allen  wrote:

> +1 to the first approach, as you've laid it out.  That makes the most sense
> to me.  We need a way to rev the version of the ES Mpack independent of the
> ES version.
>
> On Wed, Feb 21, 2018 at 11:30 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Anyone have any opinions on how we should version the ES/Kibana MPack?
> >
> > We currently rev the Metron one based on current Metron version and apply
> > it to both the overall MPack as well as the individual service versions,
> > e.g. metron_mpack-0.4.3.0/common-services/METRON/0.4.3. For ES we have
> > been
> > keeping the service version matched to the current ES version because
> it's
> > independent of Metron, ie 5.6.2. The upshot is that we can handle this
> one
> > of two ways.
> >
> >1. Keep the same approach after splitting off ES and Kibana - ES MPack
> >version is set to the current Metron version (0.4.3) and the service
> > itself
> >is set to the ES version (5.6.2).
> >2. Use ES version for both the MPack version and service version
> > (5.6.2).
> >
> >
> > I personally recommend and prefer the first approach because it allows us
> > to make changes to the mpack itself without necessarily changing the
> > version of ES or Kibana, which is something that is likely to happen.
> It's
> > also consistent and seamless with our current versioning approach.
> Lastly,
> > I don't believe the ES versions provide much contextual sense in the
> Metron
> > world for an MPack version - setting the service version definitely makes
> > sense to indicate what exact ES version we're using, but the MPack is
> > really our way of providing custom functionality that wraps a specific
> > version of ES/Kibana. Hope this makes sense.
> >
> > Mike
> >
> >
> > On Sat, Feb 17, 2018 at 11:13 AM, Nick Allen  wrote:
> >
> > > +1 I agree with Otto's idea.  It makes a lot of sense for Elasticsearch
> > and
> > > Kibana to live in a separate Mpack.
> > >
> > > This provides us a path forward to support additional indexers like
> Solr.
> > >
> > > We should also not force our users to install an external component
> (like
> > > Elasticsearch) using the Mpack.  There are just too many installation
> > > configurations for us to reasonably support; especially on larger
> > > installations.  Supporting the Elasticsearch MPack is a project unto
> > > itself.  That being said, the functionality will still be there for
> those
> > > that want to use it.
> > >
> > >
> > >
> > > On Fri, Feb 16, 2018 at 4:10 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > This came up earlier when discussing work around the ES upgrade:
> > > > https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd
> > > > 74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E
> > > > https://lists.apache.org/thread.html/8ec83b6a3ef39057c9466ff72a2f63
> > > > c9308452f1ebc1804e67cb495b@%3Cdev.metron.apache.org%3E
> > > >
> > > > Looks like Otto made this suggestion and Kyle is on board. I was
> > > originally
> > > > opposed to this because it did not seem worth the effort to support 2
> > > > separate MPacks. However, now that we are working on the Solr
> upgrade,
> > it
> > > > seems like an appropriate solution for enabling us to make the
> indexing
> > > > piece pluggable. I propose that we commence with this solution.
> > > >
> > > > Cheers,
> > > > Mike
> > > >
> > >
> >
>
-- 

Jon

Re: [DISCUSS] Generic Syslog Parsing capability for parsers

[zeo...@gmail.com]

So I've kept my ear to the ground regarding this topic for a while now, and
had some conversations a year or so ago about the idea as well.  At the
very least, I think having the concept of a pre-parser is a good one, if
not chaining an arbitrary number of parsers together.  I see this as an
important way to reduce the complexity of implementing new parsers and
getting more community involvement/contributions.

Syslog headers are a solid use case to start with because a lot of
implementations fail to properly implement it on the sending side, at least
in the real world scenarios that I've seen.  Having a way to extend the
parser to easily handle incorrect implementations of syslog would be great,
but anything that can pre-parse or trim the syslog headers to make parsing
further along in the pipeline more simple would help.

Another idea that would be attractive would be the ability to do
opportunistic parsing given an ordered list of parsers and some criteria
for successful parsing (which I admittedly am not sure how to solve) which
(at least in my mind) would require similar logic to parser chaining.  In
some highly decentralized organizations this would be helpful as it takes
the configuration effort off of the team sending the logs (and thus makes
them more willing to send logs _at all_) and pushes it onto the team
parsing and/or storing them.

I'm not suggesting we attempt to crack that second nut here, I would love
to see that use case in mind during discussions.

TL;DR:  +1

Jon

On Tue, Mar 20, 2018 at 6:14 PM Otto Fowler  wrote:

> I think the chaining of parsers, or ability to compose parsers is a good
> idea, but with reference to the pr mentioned, I would have some number of
> StellarChainLinks as opposed re-implementing stellar in chainlinks.
> Although it is NiFi-y.  But since I write Processors too, that is fine.
>
>
> On March 20, 2018 at 18:05:12, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> It seems like parser chaining is becomes a hot topic on the repo too with
> https://github.com/apache/metron/pull/969#partial-pull-merging <
> https://github.com/apache/metron/pull/969#partial-pull-merging>
>
> I would like to discuss the option, and how we might architect, of
> configuring parsers to operate on the output of parsers. This may also give
> us the opportunity to be more efficient in scenarios where people have
> large numbers of sources, and so use up a lot of slots for lower volume
> parsers for example.
>
> I have a bunch of ideas around this, but am more keen to hear what everyone
> else thinks at this stage. How should we go about fixing parser config so
> that it’s clearer (removing the need for people to reinvent the parser
> wheel as we’ve seen in a few places) and also more concise and powerful
> (consolidating the parsing of transports such as syslog and content such as
> application logs, or types of device logs).
>
> If this can lead to a more efficient way of handling both the syslog
> problem, and the kind of problem that leads to switching between grok
> statements in something like our ASA parser then all the better. I suspect
> that there might also be a case for multi-level chaining here too, since
> some things are embedded in multiple transports, or might have complex
> fields that want ‘sub-parsing’.
>
> Of course one of the key values of Metron is its speed, so maybe
> formalising some of the microbenchmarking approaches a few of us have been
> working on might help here too. I’ve got a few bits of micro-benching
> infrastructure around CEF and ASA, and I believe there’s also been some
> work to load and perf test things like enrichment that might be leveraged.
>
> Thoughts on a dev board?
>
> Simon
>
> > On 20 Mar 2018, at 21:47, Otto Fowler  wrote:
> >
> > I entered METRON–1453  >
> a
> > little while ago while working on the PR#579
> > .
> >
> > "We have several parsers now, with many imaginable that are based on
> > syslog, where the format is SYSLOG HEADER MESSAGE.
> >
> > With message being in a different format. It would be great is we had a
> way
> > to generically handle syslog headers, such that ANY parser data could
> come
> > over syslog.
> >
> > Either you could have a custom parser, or configure CSV or JSON such that
> > they could be the payload, such that you can handle JSON over syslog by
> > configuration only."
> >
> > The idea would be that the parser bolt would use the configuration to
> > trigger parsing the incoming message as syslog formatted, and pass the
> > message part to the parser, and put the syslog parts in the message(s)
> > after parsing.
> >
> > As part of this I did some work on parsing syslog, using both grok and a
> > DSL that I did from the spec :
> https://github.com/ottobackwards/grok-v-antlr
> >
> > The DSL is slower, but grok cannot handle multiple structured data
> entries,
> > and the DSL can. I’m not good enoug

Re: Secure code analysis

[zeo...@gmail.com]

I would like to volunteer some effort to see how we might be able to
integrate Veracode scans with the ASF Jenkins instance to see how it could
be useful, but in order to do so I need to get some additional
authorization.  *Would a PMC member mind getting me access* so I can take a
look, given that nobody seems to have had an issue with this?  For
reference, from my prior email:

The ASF seems to support giving non-PMC committers access
<https://wiki.apache.org/general/Jenkins#How_do_I_get_an_account> to
Jenkins, but it requires that the PMC chair do some work, and generally it
looks like they want admins
<https://wiki.apache.org/general/Jenkins#FAQ_For_Administrators>/PMC
<https://wiki.apache.org/general/Jenkins#FAQ_For_PMCs> members to be
involved (I also don't have access to the builds JIRA project
<https://issues.apache.org/jira/projects/BUILDS>, if it really exists).

Jon

On Sun, Jan 7, 2018 at 8:16 AM Nadir Hajiyani 
wrote:

> Here is the documentation for various Veracode integrations -
> https://help.veracode.com/reader/QJgoLlv~uqsO6Zvu9jG9pw/
> h2NG_xyaRqXJtAUioBS2SA
>
> A few options can be explored here, like:
>
>- Sending the scans directly via the IDE (Eclipse, IntelliJ, Visual
>Studio)
>- Utilizing the API Wrapper
>- Using the Upload API (Easier said than done)
>
>
> On Sun, Dec 24, 2017 at 9:58 AM, Nick Allen  wrote:
>
> > > 3) I have been manually making submissions dating back to 2017-02-13,
> but
> >
> > Oh, great.
> > ​So your general impression based on those submissions is that this would
> > be useful for us?
> >
> > I didn't realize that you had already been reviewing the output of the
> tool
> > over a period of time.
> >
> > Thanks, Jon
> >
> >
> > On Dec 23, 2017 8:32 PM, "zeo...@gmail.com"  wrote:
> >
> > Sure, not a problem.
> >
> > (1) I went to an event where a presenter from Veracode was calling out
> some
> > bugs in open source projects, and that Veracode wanted to be a part of
> the
> > solution.  As such, they offered to give free analysis to open source
> > projects that reach out.  At this point the account that I have access to
> > is just for the Apache Metron project, but it is possible that the
> > relationship could grow if it makes sense for other projects.  For
> > instance, this <
> https://twitter.com/PeteChestna/status/943845893597483008
> > >.
> >
> > (2) No specific reason - in the past I looked at Coverity (see below in
> > this thread) but was deterred from personally setting it up due to some
> of
> > their policies about who can register new scans (i.e. I was not a
> committer
> > at the time I believe, and that level of involvement was requested).  I
> > have used Veracode in the past, along with others (AppScan, Fortify,
> etc.),
> > and had a good experience albeit in a very different setting than this.
> I
> > would be more than happy to play around with any of these kinds of
> services
> > and no affinity to one or the other, but right now the only thing I
> > actually have access to is Veracode and free options like Coverity.
> >
> > Veracode is a proprietary cloud-hosted platform that has dynamic and
> static
> > scan offerings, and they have various integrations
> > <https://community.veracode.com/s/integrations> with build systems
> (maven,
> > Jenkins, Bamboo, etc.) and IDEs (IntelliJ, Eclipse, etc.).  They also
> > appear to have opened up their training materials
> > <https://community.veracode.com/s/education-and-training>, which are
> handy
> > to point to from time to time.  I've worked with it in the past and
> things
> > largely seem to work as you would expect, although it has been 5 years
> > since I really used their products regularly.
> >
> > (3) I have been manually making submissions dating back to 2017-02-13,
> but
> > because the file transfer is uploaded from my home Internet (upload
> speeds
> > of ~6Mbps), it takes quite a while and so I don't do it very frequently.
> > Usually just around releases.
> >
> > Jon
> >
> > On Sat, Dec 23, 2017 at 11:13 AM Nick Allen  wrote:
> >
> > > > Veracode has provided us with a 100% free portal to scan the Metron
> > code
> > > with, but in order to integrate, the safest option is probably to use
> the
> > > ASF's jenkins server
> > >
> > > (1) Can you describe this more?   How has this been provided?  Is this
> > for
> > > all Apache projects; just Metron?  Was this based on a relationship you
> > > have 

GeoLite deprecating legacy DBs

[zeo...@gmail.com]

Looks like we will need to update the Geo DBs that we use for enrichment.


Updated versions of the GeoLite Legacy databases are now only available to
redistribution license customers, although anyone can continue to download
the March 2018 GeoLite Legacy builds. Starting January 2, 2019, the last
build will be removed from our website. GeoLite Legacy database users will
need to switch to the GeoLite2 or commercial GeoIP databases and update
their integrations by January 2, 2019.

New Database Format Available: ... For our latest database format, please
see our GeoLite2 Databases.

https://dev.maxmind.com/geoip/legacy/geolite/

Jon
-- 

Jon

Re: GeoLite deprecating legacy DBs

[zeo...@gmail.com]

Okay great, sorry I missed that conversation.  I did a quick search earlier
on my phone of where we pull the DBs from and if you strip out the file
GeoLite2-City.mmdb.gz it redirects to the legacy page, but I guess that's
just a strange redirection and not an indication that that download area
only holds the legacy DBs.

Jon

On Fri, Apr 13, 2018 at 7:03 AM Nick Allen  wrote:

> See also https://issues.apache.org/jira/browse/METRON-1517
>
> On Fri, Apr 13, 2018, 5:40 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > Don’t we already use the GeoLite2 database? Mine are all
> > /apps/metron/geo/default/GeoLite2-City.mmdb.gz downloaded from
> > http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
> > which seems to match the new format page.
> >
> > Am I missing something Jon, or are you referring to the old geo
> enrichment?
> >
> > Simon
> >
> >
> > > On 13 Apr 2018, at 10:27, zeo...@gmail.com  wrote:
> > >
> > > Looks like we will need to update the Geo DBs that we use for
> enrichment.
> > >
> > >
> > > Updated versions of the GeoLite Legacy databases are now only available
> > to
> > > redistribution license customers, although anyone can continue to
> > download
> > > the March 2018 GeoLite Legacy builds. Starting January 2, 2019, the
> last
> > > build will be removed from our website. GeoLite Legacy database users
> > will
> > > need to switch to the GeoLite2 or commercial GeoIP databases and update
> > > their integrations by January 2, 2019.
> > >
> > > New Database Format Available: ... For our latest database format,
> please
> > > see our GeoLite2 Databases.
> > >
> > > https://dev.maxmind.com/geoip/legacy/geolite/
> > >
> > > Jon
> > > --
> > >
> > > Jon
> >
> >
>
-- 

Jon

Re: [VOTE] Development Guidelines Addendum on Inactive Pull Requests

[zeo...@gmail.com]

+1 (non-binding)

On Fri, Apr 20, 2018 at 9:42 AM Michel Sumbul 
wrote:

> +1
>
> 2018-04-20 14:40 GMT+01:00 Otto Fowler :
>
> > +1
> >
> >
> > On April 20, 2018 at 09:30:30, Nick Allen (n...@nickallen.org) wrote:
> >
> > I am proposing the following addition to the project's development
> > guidelines [1]. Based on these guidelines, an abandoned pull request can
> > be closed in roughly 6 weeks time (4 weeks of inactivity plus 2 weeks to
> > respond to a committer's request.)
> >
> > Please vote +1, 0, or -1 and also indicate if your vote is binding or
> > non-binding. More information on voting can be found in the Apache Metron
> > By-Laws [2].
> >
> > This vote will remain open for at least 72 hours, excluding this weekend.
> > I plan to close the vote no sooner than Wednesday, April 25, 2018 at 8:00
> > AM EST.
> >
> > The discuss thread that preceeded this vote can be found here [3].
> >
> > --
> >
> > 2.6.1 Inactive Pull Requests
> >
> >
> > Contributions can often take a significant amount of time to complete the
> > code review process. This process requires active participation from the
> > contributor. If the contributor is unable to actively participate, the
> > pull request is unlikely to successfully complete this process.
> >
> > Pull Requests that have failed to receive active participation from the
> > contributor for an extended period of time risk being abandoned. Any
> > committer can submit a request for Apache Infra to close a pull request
> > that has been abandoned according to the following guidelines.
> >
> >
> > - A pull request is 'inactive' if no comments or updates have been made
> > by the contributor in the previous 4 weeks.
> >
> >
> > - For any 'inactive' pull request, a committer can request from the
> > contributor justification for keeping the pull request open.
> >
> >
> > - The committer's request should be made as a public comment on the pull
> > request. The committer should refer the contributor to these development
> > guidelines for inactive pull requests.
> >
> >
> > - If the contributor publically responds to the request, the pull
> > request is no longer consider 'inactive'.
> >
> >
> > - If the contributor does not respond to the request within 2 weeks, the
> > pull request is considered 'abandoned'.
> >
> >
> > - A committer can cast a -1 vote on any 'abandoned' pull request using
> > these development guidelines as justification.
> >
> >
> > - A committer can submit a request to Apache Infra to close the
> > 'abandoned' pull request based on this -1 vote.
> >
> > --
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> >
> > [2] https://cwiki.apache.org/confluence/display/METRON/
> > Apache+Metron+Bylaws
> >
> > [3]
> > https://lists.apache.org/thread.html/a4e72af67994c8e818f843a9ea8cc2
> > 86d81b5c72002fd011d66111f6@%3Cdev.metron.apache.org%3E
> >
>
-- 

Jon

Re: [DISCUSS] Pcap UI user requirements

[zeo...@gmail.com]

>From my perspective PCAP is primarily used as a follow-on to an alert or
meta-alert - people very rarely use PCAP for initial hunting.  I know this
has been brought up by Otto, Mike, and Ryan across the two related threads
and I think it's all spot on.  Going from an alert or meta-alert to pulling
PCAP would by far the primary use case for this in every SOC I've ever
worked in (not necessarily a representative sample).

I also have some additional thoughts on the feature side after doing some
brainstorming and talking to two of the SOCs I work most with:
 - Limit the size of the PCAP, not just the date range, and maybe even have
a configurable cluster-wide admin max for PCAP retrieval, set to 0/infinite
by default.
 - Set priority of PCAP queries.  Perhaps there's an automated
pcap retrieval 'just in case', which should have a lower priority than an
interactive request via the UI.
 - Ability to pause/resume (not just cancel) jobs.
 - Configurable cluster-wide admin max # of current PCAP queries, set to
0/infinite by default.
 - Ability to pull PCAP live off the wire and stream it into a file.
 - Ability to filter PCAP by providing a BPF filter to apply in server-side
post-processing (less efficient, but very versatile).
 - Request what PCAP data exists in the cluster (answering "how far back
can I go?")
 - This is obvious and is probably assumed, but queries based on any set of
the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.

Jon

On Fri, May 4, 2018 at 9:44 AM Otto Fowler  wrote:

> That is the ‘views’ part.
>
> We can have options on the data output, if you have output full data, then
> we can have different views and interactions for inspection and level of
> detail.
>
>
>
> On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com) wrote:
>
> It can be like a report but also to investigate some case where the user
> want to see the whole packet (all the bits and bytes). Like in wireshark,
> something interactive no?
>
> 2018-05-04 14:33 GMT+01:00 Otto Fowler :
>
> > The PCAP Query seems more like PCAP Report to me. You are generating a
> > report based on parameters.
> > That report is something that takes some time and external process to
> > generate… ie you have to wait for it.
> >
> > I can almost imagine a flow where you:
> >
> > * Are in the AlertUI
> > * Ask to generate a PCAP report based on some selected alerts/meta-alert,
> > possibly picking from on or more report ‘templates’
> > that have query options etc
> > * The report request is ‘queued’, that is dispatched to be be
> > executed/generated
> > * You as a user have a ‘queue’ of your report results, and when the
> report
> > is done it is queued there
> > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > info/meta has the yarn details )
> > * You can select the report from your queue and view it either in a new
> UI
> > or custom component
> > * You can then apply a different ‘view’ to the report or work with the
> > report data
> > * You can print / save etc
> > * You can associate the report with the alerts ( again in the report info
> )
> > with…. a ‘case’ or ‘ticket’ or investigation something or other
> >
> >
> > We can introduce extensibility into the report templates, report views (
> > thinks that work with the json data of the report )
> >
> > Something like that.
> >
> >
> > On May 4, 2018 at 09:19:15, Ryan Merriman (merrim...@gmail.com) wrote:
> >
> > Continuing a discussion that started in a discuss thread about exposing
> > Pcap query capabilities in the back end. How should we expose this
> feature
> > to users? Should it be integrated into the Alerts UI or be separate
> > standalone UI?
> >
> > To summarize the general points made in the other thread:
> >
> > - Adding this capability to the Alerts UI will make it more of a
> > composite app. Is that really what we want since we have separate UIs for
> > Alerts and management?
> > - Would it be better to bring it in on it's own so it can be released
> > with qualifiers and tested with the right expectations without affecting
> > the Alerts UI?
> > - There are some use cases that begin with an infosec analyst doing a
> > search on alerts
> > followed by them going to query pcap data corresponding to the
> > threats they're investigating. Would having these features in the same
> > UI streamline this process?
> >
> > There was also mention of some features we should consider:
> >
> > - Pcap queries should be made asynchronous via the UI
> > - Take care that a user doesn't hit refresh or POST multiple times and
> kick
> > off 50 mapreduce jobs
> > - Options for managing the YARN queue that is used
> > - Provide a "cancel" option that kills the MR job, or tell the user to
> > go to the CLI to kill their job
> > - Managing data if multiple users run queries
> > - Strategy for cleaning up jobs and implementing a TTL (I think this one
> > will be tricky and definitely needs discussion)
> > - Date range or other query limits
> >
>

Re: [DISCUSS] Release?

[zeo...@gmail.com]

We should also mention the Upgrade of ElasticSearch and Kibana

Jon

On Wed, May 9, 2018 at 12:49 PM Nick Allen  wrote:

> Oh, and also the Solr work that is currently in a feature branch.  We would
> have to get the work finished up and merged though.  Sounds like we are
> real close on that.
>
> On Wed, May 9, 2018 at 12:47 PM, Nick Allen  wrote:
>
> > The next part of the conversation would be, what should the version
> number
> > be?  To help with that, I have tried to summarize the changes in the
> > release.  Of course, this is going to be heavily biased towards my own
> > interests, so please feel free to chime in if I have missed anything.
> >
> >
> >- Significant performance improvements in Parsing, Enrichments, and
> >Indexing
> >
> >
> >- Support for Ubuntu installations
> >
> >
> >- Support for Storm 1.1 and HDP 2.6
> >
> >
> >- Event time processing in the Profiler
> >
> >
> >- Complex document support in the JSONMapParser
> >
> >
> >- Support for the Elasticsearch X-Pack
> >
> >
> >- Run Stellar in a Zeppelin Notebook
> >
> >
> >- Oodles of bug fixes and usability improvements
> >
> >
> >
> >
> > On Wed, May 9, 2018 at 12:14 PM, Nick Allen  wrote:
> >
> >> Something like this might be more digestible for these purposes.
> >>
> >> $git log --pretty="%cr %s" tags/apache-metron-0.4.2-release..HEAD
> >>
> >> 88 minutes ago METRON-1530 Default proxy config settings in
> >> metron-contrib need to be updated (sardell via merrimanr) closes
> >> apache/metron#998
> >> 5 days ago METRON-1545 Upgrade Spring and Spring Boot (merrimanr) closes
> >> apache/metron#1008
> >> 7 days ago METRON-1543 Unable to Set Parser Output Topic in Sensor
> Config
> >> (nickwallen) closes apache/metron#1007
> >> 2 weeks ago METRON-1539: Specialized RENAME field transformer closes
> >> apache/incubator-metron#1002
> >> 2 weeks ago METRON-1520: Add caching for stellar field transformations
> >> closes apache/incubator-metron#990
> >> 2 weeks ago METRON-1529 CONFIG_GET Fails to Retrieve Latest Config When
> >> Run in Zeppelin REPL (nickwallen) closes apache/metron#997
> >> 2 weeks ago METRON-1511 Unable to Serialize Profiler Configuration
> >> (nickwallen) closes apache/metron#982
> >> 3 weeks ago METRON-1528: Fix missing file in metron.spec (mmiklavc via
> >> mmiklavc) closes apache/metron#996
> >> 3 weeks ago METRON-1445: Update performance tuning guide with more
> >> explicit parameter instructions (mmiklavc via mmiklavc) closes
> >> apache/metron#988
> >> 3 weeks ago METRON-1502 Upgrade Doxia plugin to 1.8 (justinleet) closes
> >> apache/metron#974
> >> 3 weeks ago METRON-1527: Remove dead test file sitting in source folder
> >> (mmiklavc via mmiklavc) closes apache/metron#994
> >> 3 weeks ago METRON-1499 Enable Configuration of Unified Enrichment
> >> Topology via Ambari (nickwallen) closes apache/metron#984
> >> 3 weeks ago METRON-1515: Errors loading stellar functions currently bomb
> >> the entire topology, they should be recoverable closes
> >> apache/incubator-metron#985
> >> 4 weeks ago METRON-1522 Fix the typo errors at profile debugger readme
> >> (MohanDV via nickwallen) closes apache/metron#992
> >> 4 weeks ago METRON-1519 Indexing Error Topic Property Not Displayed in
> >> MPack (nickwallen) closes apache/metron#987
> >> 4 weeks ago METRON-1347: Indexing Topology should fail tuples without a
> >> source.type (cstella via mmiklavc) closes apache/metron#863
> >> 4 weeks ago Add support for Vagrant Cachier plugin if present
> >> (simonellistonball via mmiklavc) closes apache/metron#993
> >> 4 weeks ago METRON-1521: JSONMapParser is no longer serializable closes
> >> apache/incubator-metron#991
> >> 4 weeks ago METRON-1516 Support for Ansible 2.5.0 (ottobackwards) closes
> >> apache/metron#989
> >> 4 weeks ago METRON-1494 Profiler Emits Messages to Kafka When Not Needed
> >> (nickwallen) closes apache/metron#967
> >> 4 weeks ago METRON-1510 Update Metron website to include info about
> >> github update subscription (anandsubbu) closes apache/metron#981
> >> 4 weeks ago METRON-1518 Build Failure When Using Profile HDP-2.5.0.0
> >> (nickwallen) closes apache/metron#986
> >> 4 weeks ago METRON-1465:Support for Elasticsearch X-pack (wardbekker via
> >> mmiklavc) closes apache/metron#946
> >> 4 weeks ago METRON-1504: Enriching missing values does not match the
> >> semantics between the new enrichment topology and old closes
> >> apache/incubator-metron#976
> >> 5 weeks ago METRON-1505 Intermittent Profiler Integration Test Failure
> >> (nickwallen) closes apache/metron#977
> >> 5 weeks ago METRON-1449 Set Zookeeper URL for Stellar Running in
> Zeppelin
> >> Notebook (nickwallen) closes apache/metron#931
> >> 5 weeks ago METRON-1462: Separate ES and Kibana from Metron Mpack
> >> (mmiklavc via mmiklavc) closes apache/metron#943
> >> 5 weeks ago METRON-1501 Parser messages that fail to validate are
> dropped
> >> silently (cestella via justinleet) closes apache/metron#972
> >> 5 weeks 

Re: [DISCUSS] Pcap panel architecture

[zeo...@gmail.com]

This looks really great and gets me excited to maybe revisit some old
conversations about PCAP capture in Metron.  The only thing that I think
it's missing is the ability to filter using bpf.  I think the same thing
can technically be accomplished by using packet_filter and I wouldn't throw
a fit if that's considered a follow-on, but bpf is the standard language
that people who do packet munging for a living know.

Jon

On Wed, May 9, 2018 at 1:00 PM Ryan Merriman  wrote:

> Now that we are confident we can run submit a MR job from our current REST
> application, is this the desired approach?  Just want to confirm.
>
> Next I think we should map out what the REST interface will look like.
> Here are the endpoints I'm thinking about:
>
> GET /api/v1/pcap/metadata?basePath
>
> This endpoint will return metadata of pcap data stored in HDFS.  This would
> include pcap size, date ranges (how far back can I go), etc.  It would
> accept an optional HDFS basePath parameter for cases where pcap data is
> stored in multiple places and/or different from the default location.
>
> POST /api/v1/pcap/query
>
> This endpoint would accept a pcap request, submit a pcap query job, and
> return a job id.  The request would be an object containing the parameters
> documented here:  https://github.com/apache/metron/tree/master/
> metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
> would be associated with a user that submits it.  An exception will be
> returned for violating constraints like too many queries submitted, query
> parameters out of limits, etc.
>
> GET /api/v1/pcap/status/
>
> This endpoint will return the status of a running job.  I imagine this is
> just a proxy to the YARN REST api.  We can discuss the implementation
> behind these endpoints later.
>
> GET /api/v1/pcap/stop/
>
> This endpoint would kill a running pcap job.  If the job has already
> completed this is a noop.
>
> GET /api/v1/pcap/list
>
> This endpoint will list a user's submitted pcap queries.  Items in the list
> would contain job id, status (is it finished?), start/end time, and number
> of pages.  Maybe there is some overlap with the status endpoint above and
> the status endpoint is not needed?
>
> GET /api/v1/pcap/pdml//
>
> This endpoint will return pcap results for the given page in pdml format (
> https://wiki.wireshark.org/PDML).  Are there other formats we want to
> support?
>
> GET /api/v1/pcap/raw//
>
> This endpoint will allow a user to download raw pcap results for the given
> page.
>
> DELETE /api/v1/pcap/
>
> This endpoint will delete pcap query results.  Not sure yet how this fits
> in with our broader cleanup strategy.
>
> This should get us started.  What did I miss and what would you change
> about these?  I did not include much detail related to security, cleanup
> strategy, or underlying implementation details but these are items we
> should discuss at some point.
>
> On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Sweet! That's great news. The pom changes are a lot simpler than I
> > expected. Very nice.
> >
> > On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman 
> wrote:
> >
> > > Finally figured it out.  Commit is here:
> > > https://github.com/merrimanr/incubator-metron/commit/
> > > 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
> > >
> > > It came down to figuring out the right combination of maven
> dependencies
> > > and passing in the HDP version to REST as a Java system property.  I
> also
> > > included some HDFS setup tasks.  I tested this in full dev and can now
> > > successfully run a pcap query and get results.  All you should have to
> do
> > > is generate some pcap data first.
> > >
> > > On Tue, May 8, 2018 at 4:17 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > @Ryan - pulled your branch and experimented with a few things. In
> doing
> > > so,
> > > > it dawned on me that by adding the yarn and hadoop classpath, you
> > > probably
> > > > didn't introduce a new classpath issue, rather you probably just
> moved
> > > onto
> > > > the next classpath issue, ie hbase per your exception about hbase
> jaxb.
> > > > Anyhow, I put up a branch with some pom changes worth trying in
> > > conjunction
> > > > with invoking the rest app startup via "/usr/bin/yarn jar"
> > > >
> > > > https://github.com/mmiklavc/metron/tree/ryan-rest-test
> > > >
> > > > https://github.com/mmiklavc/metron/commit/
> > 5ca23580fc6e043fafae2327c80b65
> > > > b20ca1c0c9
> > > >
> > > > Mike
> > > >
> > > >
> > > > On Tue, May 8, 2018 at 7:44 AM, Simon Elliston Ball <
> > > > si...@simonellistonball.com> wrote:
> > > >
> > > > > That would be a step closer to something more like a micro-service
> > > > > architecture. However, I would want to make sure we think about the
> > > > > operational complexity, and mpack implications of having another
> > server
> > > > > installed and running somewhere on the cluster (also, ssl,
> kerberos,
> > > etc
> > > >

Re: [DISCUSS] Release?

[zeo...@gmail.com]

I agree that it's probably time (more likely, overdue) for a release.
Based off of looking at all of those changes I would also suggest going to
at least 0.5.x.

It probably makes sense to take a look at Upgrading.md
<https://github.com/apache/metron/blob/master/Upgrading.md> (and related
docs) to make sure it's accurate as well.

Jon

On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Good call - I thought that made our last release, but this would be the 2nd
> follow-on from when Nick originally posed a breakdown
> 4 months ago METRON-939: Upgrade ElasticSearch and Kibana (mmiklavc via
> mmiklavc) closes apache/metron#840
>
>
> https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
>
> On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com 
> wrote:
>
> > We should also mention the Upgrade of ElasticSearch and Kibana
> >
> > Jon
> >
> > On Wed, May 9, 2018 at 12:49 PM Nick Allen  wrote:
> >
> > > Oh, and also the Solr work that is currently in a feature branch.  We
> > would
> > > have to get the work finished up and merged though.  Sounds like we are
> > > real close on that.
> > >
> > > On Wed, May 9, 2018 at 12:47 PM, Nick Allen 
> wrote:
> > >
> > > > The next part of the conversation would be, what should the version
> > > number
> > > > be?  To help with that, I have tried to summarize the changes in the
> > > > release.  Of course, this is going to be heavily biased towards my
> own
> > > > interests, so please feel free to chime in if I have missed anything.
> > > >
> > > >
> > > >- Significant performance improvements in Parsing, Enrichments,
> and
> > > >Indexing
> > > >
> > > >
> > > >- Support for Ubuntu installations
> > > >
> > > >
> > > >- Support for Storm 1.1 and HDP 2.6
> > > >
> > > >
> > > >- Event time processing in the Profiler
> > > >
> > > >
> > > >- Complex document support in the JSONMapParser
> > > >
> > > >
> > > >- Support for the Elasticsearch X-Pack
> > > >
> > > >
> > > >- Run Stellar in a Zeppelin Notebook
> > > >
> > > >
> > > >- Oodles of bug fixes and usability improvements
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 9, 2018 at 12:14 PM, Nick Allen 
> > wrote:
> > > >
> > > >> Something like this might be more digestible for these purposes.
> > > >>
> > > >> $git log --pretty="%cr %s" tags/apache-metron-0.4.2-release..HEAD
> > > >>
> > > >> 88 minutes ago METRON-1530 Default proxy config settings in
> > > >> metron-contrib need to be updated (sardell via merrimanr) closes
> > > >> apache/metron#998
> > > >> 5 days ago METRON-1545 Upgrade Spring and Spring Boot (merrimanr)
> > closes
> > > >> apache/metron#1008
> > > >> 7 days ago METRON-1543 Unable to Set Parser Output Topic in Sensor
> > > Config
> > > >> (nickwallen) closes apache/metron#1007
> > > >> 2 weeks ago METRON-1539: Specialized RENAME field transformer closes
> > > >> apache/incubator-metron#1002
> > > >> 2 weeks ago METRON-1520: Add caching for stellar field
> transformations
> > > >> closes apache/incubator-metron#990
> > > >> 2 weeks ago METRON-1529 CONFIG_GET Fails to Retrieve Latest Config
> > When
> > > >> Run in Zeppelin REPL (nickwallen) closes apache/metron#997
> > > >> 2 weeks ago METRON-1511 Unable to Serialize Profiler Configuration
> > > >> (nickwallen) closes apache/metron#982
> > > >> 3 weeks ago METRON-1528: Fix missing file in metron.spec (mmiklavc
> via
> > > >> mmiklavc) closes apache/metron#996
> > > >> 3 weeks ago METRON-1445: Update performance tuning guide with more
> > > >> explicit parameter instructions (mmiklavc via mmiklavc) closes
> > > >> apache/metron#988
> > > >> 3 weeks ago METRON-1502 Upgrade Doxia plugin to 1.8 (justinleet)
> > closes
> > > >> apache/metron#974
> > > >> 3 weeks ago METRON-1527: Remove dead test file sitting in source
> > folder
> > > >> (mmiklavc via mmiklavc) clo

Re: [DISCUSS] Release?

[zeo...@gmail.com]

Nick - that was this
<https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9>
 and this
<https://github.com/apache/metron-bro-plugin-kafka/commit/4db999e82cbb91e989eaf00a88e94ffd2459f3a3>,
which just barely snuck it into the same release cycle as 0.4.2.  The
plugin has two new commits since then (1 bugfix 1 feature) - if we want to
include version 0.2 of the plugin with a metron 0.5.0 release we would need
to make a 0.2 tag against HEAD of the plugin repo, then increment the
version in the ansible playbooks here
<https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30>
.

Jon

On Wed, May 9, 2018 at 2:26 PM Otto Fowler  wrote:

> I think we should have that.
>
>
> On May 9, 2018 at 14:16:23, Michael Miklavcic (michael.miklav...@gmail.com
> )
> wrote:
>
> One item we haven't gotten around to was redoing the index names to use a
> metron_ prefix. I'm the one that pushed the original DISCUSS thread on
> this, but haven't had a chance to advance it. Does anyone have any strong
> opinions on it? I originally thought it made sense to include alongside the
> other major ES and Solr changes.
>
> On Wed, May 9, 2018 at 12:13 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I'm also a +1 on 0.5.0. This is a fairly big release.
> >
> > On Wed, May 9, 2018 at 12:05 PM, Nick Allen  wrote:
> >
> >> +1 to 0.5.0
> >>
> >> On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com 
> >> wrote:
> >>
> >> > I agree that it's probably time (more likely, overdue) for a release.
> >> > Based off of looking at all of those changes I would also suggest
> going
> >> to
> >> > at least 0.5.x.
> >> >
> >> > It probably makes sense to take a look at Upgrading.md
> >> > <https://github.com/apache/metron/blob/master/Upgrading.md> (and
> >> related
> >> > docs) to make sure it's accurate as well.
> >> >
> >> > Jon
> >> >
> >> > On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
> >> > michael.miklav...@gmail.com> wrote:
> >> >
> >> > > Good call - I thought that made our last release, but this would be
> >> the
> >> > 2nd
> >> > > follow-on from when Nick originally posed a breakdown
> >> > > 4 months ago METRON-939: Upgrade ElasticSearch and Kibana (mmiklavc
> >> via
> >> > > mmiklavc) closes apache/metron#840
> >> > >
> >> > >
> >> > > https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f
> >> > 36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
> >> > >
> >> > > On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com  >
> >> > > wrote:
> >> > >
> >> > > > We should also mention the Upgrade of ElasticSearch and Kibana
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > On Wed, May 9, 2018 at 12:49 PM Nick Allen 
> >> wrote:
> >> > > >
> >> > > > > Oh, and also the Solr work that is currently in a feature
> >> branch. We
> >> > > > would
> >> > > > > have to get the work finished up and merged though. Sounds like
> >> we
> >> > are
> >> > > > > real close on that.
> >> > > > >
> >> > > > > On Wed, May 9, 2018 at 12:47 PM, Nick Allen  >
> >> > > wrote:
> >> > > > >
> >> > > > > > The next part of the conversation would be, what should the
> >> version
> >> > > > > number
> >> > > > > > be? To help with that, I have tried to summarize the changes
> in
> >> > the
> >> > > > > > release. Of course, this is going to be heavily biased towards
> >> my
> >> > > own
> >> > > > > > interests, so please feel free to chime in if I have missed
> >> > anything.
> >> > > > > >
> >> > > > > >
> >> > > > > > - Significant performance improvements in Parsing,
> >> Enrichments,
> >> > > and
> >> > > > > > Indexing
> >> > > > > >
> >> > > > > >
> >> > > > > > - Support for Ubuntu installations
> >> > > > > >
> >

Re: [DISCUSS] Pcap UI user requirements

[zeo...@gmail.com]

I had a feeling it may be that way.  Unless anyone else knows of a better
approach, it's probably most reasonable to push that into a follow-on JIRA
and not over-complicate the current activities.

Jon

On Wed, May 9, 2018 at 2:33 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> We are limited by Yarn and MapReduce applications in the case of
> pause/resume - I could be wrong, but I don't think that's something that's
> supported unless you're talking about multiple MR jobs strung together.
>
>
> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application
>
> I don't see anything suggesting "SUSPENDED" or "PAUSED" as we have
> available in workflow engines like Oozie.
>
> "The valid application state can be one of the following:  ALL, NEW,
> NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED"
>
> Same goes for MR job commands:
>
> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html#job
>
> Mike
>
> On Mon, May 7, 2018 at 2:04 PM, zeo...@gmail.com  wrote:
>
> > From my perspective PCAP is primarily used as a follow-on to an alert or
> > meta-alert - people very rarely use PCAP for initial hunting.  I know
> this
> > has been brought up by Otto, Mike, and Ryan across the two related
> threads
> > and I think it's all spot on.  Going from an alert or meta-alert to
> pulling
> > PCAP would by far the primary use case for this in every SOC I've ever
> > worked in (not necessarily a representative sample).
> >
> > I also have some additional thoughts on the feature side after doing some
> > brainstorming and talking to two of the SOCs I work most with:
> >  - Limit the size of the PCAP, not just the date range, and maybe even
> have
> > a configurable cluster-wide admin max for PCAP retrieval, set to
> 0/infinite
> > by default.
> >  - Set priority of PCAP queries.  Perhaps there's an automated
> > pcap retrieval 'just in case', which should have a lower priority than an
> > interactive request via the UI.
> >  - Ability to pause/resume (not just cancel) jobs.
> >  - Configurable cluster-wide admin max # of current PCAP queries, set to
> > 0/infinite by default.
> >  - Ability to pull PCAP live off the wire and stream it into a file.
> >  - Ability to filter PCAP by providing a BPF filter to apply in
> server-side
> > post-processing (less efficient, but very versatile).
> >  - Request what PCAP data exists in the cluster (answering "how far back
> > can I go?")
> >  - This is obvious and is probably assumed, but queries based on any set
> of
> > the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.
> >
> > Jon
> >
> > On Fri, May 4, 2018 at 9:44 AM Otto Fowler 
> > wrote:
> >
> > > That is the ‘views’ part.
> > >
> > > We can have options on the data output, if you have output full data,
> > then
> > > we can have different views and interactions for inspection and level
> of
> > > detail.
> > >
> > >
> > >
> > > On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com)
> > wrote:
> > >
> > > It can be like a report but also to investigate some case where the
> user
> > > want to see the whole packet (all the bits and bytes). Like in
> wireshark,
> > > something interactive no?
> > >
> > > 2018-05-04 14:33 GMT+01:00 Otto Fowler :
> > >
> > > > The PCAP Query seems more like PCAP Report to me. You are generating
> a
> > > > report based on parameters.
> > > > That report is something that takes some time and external process to
> > > > generate… ie you have to wait for it.
> > > >
> > > > I can almost imagine a flow where you:
> > > >
> > > > * Are in the AlertUI
> > > > * Ask to generate a PCAP report based on some selected
> > alerts/meta-alert,
> > > > possibly picking from on or more report ‘templates’
> > > > that have query options etc
> > > > * The report request is ‘queued’, that is dispatched to be be
> > > > executed/generated
> > > > * You as a user have a ‘queue’ of your report results, and when the
> > > report
> > > > is done it is queued there
> > > > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > > > info/meta has the yarn details )
> > > > * You can select t

Re: [DISCUSS] Pcap UI user requirements

[zeo...@gmail.com]

Regarding the prioritization, that is what I was thinking as well, I just
wasn't as prescriptive with my suggestion.

I did look for a java implementation and failed to find one (the closest I
found was the Apache-licened bcc project <https://github.com/iovisor/bcc>).
Perhaps someone else's google-fu is better than mine though.

Not sure that BPF has the ability to depend on state (I feel it would be
very unlikely), but if it does I've never used it.  My 10 minute read of
the original paper (admittedly > 20 years old) and some other supplementary
info seems to support my prior thoughts.

Jon

On Wed, May 9, 2018 at 2:59 PM Casey Stella  wrote:

> A couple of thoughts on cluster overuse:
> * Definitely can't pause/resume MR jobs, unfortunately
> * The traditional approach to managing overuse of cluster resources and
> prioritization in Yarn is via the scheduler.  I'd suggest rather than
> building this ourselves, we allow users to be associated with yarn queues
> so their jobs are submitted to the correct queue and don't clobber the
> cluster.
>
> Beyond that, I like the BPF idea as a filter.  Any idea if there's a java
> BPF implementation?  Also, keep in mind that our query mechanism is a map
> and a reduce job, so any filtering system which depends on state (e.g.
> previous packets by time) is going to trigger another architecture.
>
> On Mon, May 7, 2018 at 4:05 PM zeo...@gmail.com  wrote:
>
> > From my perspective PCAP is primarily used as a follow-on to an alert or
> > meta-alert - people very rarely use PCAP for initial hunting.  I know
> this
> > has been brought up by Otto, Mike, and Ryan across the two related
> threads
> > and I think it's all spot on.  Going from an alert or meta-alert to
> pulling
> > PCAP would by far the primary use case for this in every SOC I've ever
> > worked in (not necessarily a representative sample).
> >
> > I also have some additional thoughts on the feature side after doing some
> > brainstorming and talking to two of the SOCs I work most with:
> >  - Limit the size of the PCAP, not just the date range, and maybe even
> have
> > a configurable cluster-wide admin max for PCAP retrieval, set to
> 0/infinite
> > by default.
> >  - Set priority of PCAP queries.  Perhaps there's an automated
> > pcap retrieval 'just in case', which should have a lower priority than an
> > interactive request via the UI.
> >  - Ability to pause/resume (not just cancel) jobs.
> >  - Configurable cluster-wide admin max # of current PCAP queries, set to
> > 0/infinite by default.
> >  - Ability to pull PCAP live off the wire and stream it into a file.
> >  - Ability to filter PCAP by providing a BPF filter to apply in
> server-side
> > post-processing (less efficient, but very versatile).
> >  - Request what PCAP data exists in the cluster (answering "how far back
> > can I go?")
> >  - This is obvious and is probably assumed, but queries based on any set
> of
> > the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.
> >
> > Jon
> >
> > On Fri, May 4, 2018 at 9:44 AM Otto Fowler 
> > wrote:
> >
> > > That is the ‘views’ part.
> > >
> > > We can have options on the data output, if you have output full data,
> > then
> > > we can have different views and interactions for inspection and level
> of
> > > detail.
> > >
> > >
> > >
> > > On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com)
> > wrote:
> > >
> > > It can be like a report but also to investigate some case where the
> user
> > > want to see the whole packet (all the bits and bytes). Like in
> wireshark,
> > > something interactive no?
> > >
> > > 2018-05-04 14:33 GMT+01:00 Otto Fowler :
> > >
> > > > The PCAP Query seems more like PCAP Report to me. You are generating
> a
> > > > report based on parameters.
> > > > That report is something that takes some time and external process to
> > > > generate… ie you have to wait for it.
> > > >
> > > > I can almost imagine a flow where you:
> > > >
> > > > * Are in the AlertUI
> > > > * Ask to generate a PCAP report based on some selected
> > alerts/meta-alert,
> > > > possibly picking from on or more report ‘templates’
> > > > that have query options etc
> > > > * The report request is ‘queued’, that is dispatched to be be
> > > > executed/generated
> > > > * You as a user have a ‘queue’ of your report

Re: [DISCUSS] Release?

[zeo...@gmail.com]

I tend to like grouping the es changes into one release (i.e. include the
index name change) and solr into another (next release).

I think we go too long between releases myself and wouldn't be against
doing two releases just a couple of months apart.

Jon

On Wed, May 9, 2018, 14:47 Michael Miklavcic 
wrote:

> I don't have a strong opinion. The ES upgrade alone is a massive feature.
> It could make it easier to include the index change I mentioned along with
> Solr as a follow-up. I think if we did split, we could arguably start on
> the next release with Solr almost immediately.
>
> On Wed, May 9, 2018, 12:40 PM Nick Allen  wrote:
>
> > Simon brought up the idea of including the Solr enhancements (currently
> in
> > a feature branch) for the release.  What are people's opinions on this?
> Is
> > this something that is a blocker for the release?
> >
> > IMO, there is so much already in master waiting to be released that I
> don't
> > see a need to include it in the next release.  I'd be happy to see that
> > Solr work drive a follow-on release.
> >
> >
> >
> >
> > On Wed, May 9, 2018 at 2:16 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > One item we haven't gotten around to was redoing the index names to
> use a
> > > metron_ prefix. I'm the one that pushed the original DISCUSS thread on
> > > this, but haven't had a chance to advance it. Does anyone have any
> strong
> > > opinions on it? I originally thought it made sense to include alongside
> > the
> > > other major ES and Solr changes.
> > >
> > > On Wed, May 9, 2018 at 12:13 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > I'm also a +1 on 0.5.0. This is a fairly big release.
> > > >
> > > > On Wed, May 9, 2018 at 12:05 PM, Nick Allen 
> > wrote:
> > > >
> > > >> +1 to 0.5.0
> > > >>
> > > >> On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com 
> > > >> wrote:
> > > >>
> > > >> > I agree that it's probably time (more likely, overdue) for a
> > release.
> > > >> > Based off of looking at all of those changes I would also suggest
> > > going
> > > >> to
> > > >> > at least 0.5.x.
> > > >> >
> > > >> > It probably makes sense to take a look at Upgrading.md
> > > >> > <https://github.com/apache/metron/blob/master/Upgrading.md> (and
> > > >> related
> > > >> > docs) to make sure it's accurate as well.
> > > >> >
> > > >> > Jon
> > > >> >
> > > >> > On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
> > > >> > michael.miklav...@gmail.com> wrote:
> > > >> >
> > > >> > > Good call - I thought that made our last release, but this would
> > be
> > > >> the
> > > >> > 2nd
> > > >> > > follow-on from when Nick originally posed a breakdown
> > > >> > > 4 months ago METRON-939: Upgrade ElasticSearch and Kibana
> > (mmiklavc
> > > >> via
> > > >> > > mmiklavc) closes apache/metron#840
> > > >> > >
> > > >> > >
> > > >> > >
> > https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f
> > > >> > 36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
> > > >> > >
> > > >> > > On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com <
> > zeo...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > We should also mention the Upgrade of ElasticSearch and Kibana
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > > On Wed, May 9, 2018 at 12:49 PM Nick Allen <
> n...@nickallen.org>
> > > >> wrote:
> > > >> > > >
> > > >> > > > > Oh, and also the Solr work that is currently in a feature
> > > >> branch.  We
> > > >> > > > would
> > > >> > > > > have to get the work finished up and merged though.  Sounds
> > like
> > > >> we
> > > &g

Re: [DISCUSS] Pcap panel architecture

[zeo...@gmail.com]

At the very least there needs to be the ability to share downloaded PCAPs
with other users and/or have roles that can see all pcaps.  A platform
engineer may want to clean up old pcaps after x time, or a manger may ask
an analyst to find all of the traffic that exhibits xyz behavior, dump a
pcap, and then point him to it so the manager can review.  Since the
pcap may be huge, we wouldn't want to try to push people to sending it via
email, uploading to a file server, finding an external hard drive, etc.

Jon

On Thu, May 10, 2018 at 10:16 AM Ryan Merriman  wrote:

> Mike, I believe the /pcapGetter/getPcapsByIdentifiers endpoint exposes the
> fixed query option which we have covered.  I agree with you that
> deprecating the metron-api module should be a goal of this feature.
>
> On Wed, May 9, 2018 at 1:36 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > This looks like a pretty good start Ryan. Does the metadata endpoint
> cover
> > this https://github.com/apache/metron/tree/master/
> > metron-platform/metron-api#the-pcapgettergetpcapsbyidentifiers-endpoint
> > from the original metron-api? If so, then we would be able to deprecate
> the
> > existing metron-api project. If we later go to micro-services, a pcap
> > module would spin back into the fold, but it would probably look
> different
> > from metron-api.
> >
> > I commented on the UI thread, but to reiterate for the purpose of backend
> > functionality here I don't believe there is a way to "PAUSE" or "SUSPEND"
> > jobs. That said, I think GET /api/v1/pcap/stop/ is sufficient for
> > the job management operations.
> >
> > On Wed, May 9, 2018 at 11:00 AM, Ryan Merriman 
> > wrote:
> >
> > > Now that we are confident we can run submit a MR job from our current
> > REST
> > > application, is this the desired approach?  Just want to confirm.
> > >
> > > Next I think we should map out what the REST interface will look like.
> > > Here are the endpoints I'm thinking about:
> > >
> > > GET /api/v1/pcap/metadata?basePath
> > >
> > > This endpoint will return metadata of pcap data stored in HDFS.  This
> > would
> > > include pcap size, date ranges (how far back can I go), etc.  It would
> > > accept an optional HDFS basePath parameter for cases where pcap data is
> > > stored in multiple places and/or different from the default location.
> > >
> > > POST /api/v1/pcap/query
> > >
> > > This endpoint would accept a pcap request, submit a pcap query job, and
> > > return a job id.  The request would be an object containing the
> > parameters
> > > documented here:  https://github.com/apache/metron/tree/master/
> > > metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
> > > would be associated with a user that submits it.  An exception will be
> > > returned for violating constraints like too many queries submitted,
> query
> > > parameters out of limits, etc.
> > >
> > > GET /api/v1/pcap/status/
> > >
> > > This endpoint will return the status of a running job.  I imagine this
> is
> > > just a proxy to the YARN REST api.  We can discuss the implementation
> > > behind these endpoints later.
> > >
> > > GET /api/v1/pcap/stop/
> > >
> > > This endpoint would kill a running pcap job.  If the job has already
> > > completed this is a noop.
> > >
> > > GET /api/v1/pcap/list
> > >
> > > This endpoint will list a user's submitted pcap queries.  Items in the
> > list
> > > would contain job id, status (is it finished?), start/end time, and
> > number
> > > of pages.  Maybe there is some overlap with the status endpoint above
> and
> > > the status endpoint is not needed?
> > >
> > > GET /api/v1/pcap/pdml//
> > >
> > > This endpoint will return pcap results for the given page in pdml
> format
> > (
> > > https://wiki.wireshark.org/PDML).  Are there other formats we want to
> > > support?
> > >
> > > GET /api/v1/pcap/raw//
> > >
> > > This endpoint will allow a user to download raw pcap results for the
> > given
> > > page.
> > >
> > > DELETE /api/v1/pcap/
> > >
> > > This endpoint will delete pcap query results.  Not sure yet how this
> fits
> > > in with our broader cleanup strategy.
> > >
> > > This should get us started.  What did I miss and what would you change
> > > about these?  I did not include much detail related to security,
> cleanup
> > > strategy, or underlying implementation details but these are items we
> > > should discuss at some point.
> > >
> > > On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Sweet! That's great news. The pom changes are a lot simpler than I
> > > > expected. Very nice.
> > > >
> > > > On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman 
> > > wrote:
> > > >
> > > > > Finally figured it out.  Commit is here:
> > > > > https://github.com/merrimanr/incubator-metron/commit/
> > > > > 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
> > > > >
> > > > > It came down to figuring out the right combination of maven
> > > dependencies
> > > > > and passing i

Re: [DISCUSS] Pcap panel architecture

[zeo...@gmail.com]

I think baby steps are fine - admin gets access to all, otherwise you only
see your own pcaps, but we file a jira for a future add of API security,
which more mature SOCs that align with the Metron personas will need.

Jon

On Fri, May 11, 2018, 09:27 Ryan Merriman  wrote:

> That's a good point Jon.  There are different levels of effort associated
> with different options.  If we want to allow pcaps to be shared with
> specific users, we will need to introduce ACL security in our REST
> application using something like the ACL capability that comes with Spring
> Security or Ranger.  This would be more complex to design and implement.
> If we want something more broad like admin roles that can see all or
> allowing pcap files to become public, this would be less work.  Do you
> think ACL security is required or would the other options be acceptable?
>
> On Thu, May 10, 2018 at 2:47 PM, zeo...@gmail.com 
> wrote:
>
> > At the very least there needs to be the ability to share downloaded PCAPs
> > with other users and/or have roles that can see all pcaps.  A platform
> > engineer may want to clean up old pcaps after x time, or a manger may ask
> > an analyst to find all of the traffic that exhibits xyz behavior, dump a
> > pcap, and then point him to it so the manager can review.  Since the
> > pcap may be huge, we wouldn't want to try to push people to sending it
> via
> > email, uploading to a file server, finding an external hard drive, etc.
> >
> > Jon
> >
> > On Thu, May 10, 2018 at 10:16 AM Ryan Merriman 
> > wrote:
> >
> > > Mike, I believe the /pcapGetter/getPcapsByIdentifiers endpoint exposes
> > the
> > > fixed query option which we have covered.  I agree with you that
> > > deprecating the metron-api module should be a goal of this feature.
> > >
> > > On Wed, May 9, 2018 at 1:36 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > This looks like a pretty good start Ryan. Does the metadata endpoint
> > > cover
> > > > this https://github.com/apache/metron/tree/master/
> > > > metron-platform/metron-api#the-pcapgettergetpcapsbyidentifier
> > s-endpoint
> > > > from the original metron-api? If so, then we would be able to
> deprecate
> > > the
> > > > existing metron-api project. If we later go to micro-services, a pcap
> > > > module would spin back into the fold, but it would probably look
> > > different
> > > > from metron-api.
> > > >
> > > > I commented on the UI thread, but to reiterate for the purpose of
> > backend
> > > > functionality here I don't believe there is a way to "PAUSE" or
> > "SUSPEND"
> > > > jobs. That said, I think GET /api/v1/pcap/stop/ is sufficient
> > for
> > > > the job management operations.
> > > >
> > > > On Wed, May 9, 2018 at 11:00 AM, Ryan Merriman 
> > > > wrote:
> > > >
> > > > > Now that we are confident we can run submit a MR job from our
> current
> > > > REST
> > > > > application, is this the desired approach?  Just want to confirm.
> > > > >
> > > > > Next I think we should map out what the REST interface will look
> > like.
> > > > > Here are the endpoints I'm thinking about:
> > > > >
> > > > > GET /api/v1/pcap/metadata?basePath
> > > > >
> > > > > This endpoint will return metadata of pcap data stored in HDFS.
> This
> > > > would
> > > > > include pcap size, date ranges (how far back can I go), etc.  It
> > would
> > > > > accept an optional HDFS basePath parameter for cases where pcap
> data
> > is
> > > > > stored in multiple places and/or different from the default
> location.
> > > > >
> > > > > POST /api/v1/pcap/query
> > > > >
> > > > > This endpoint would accept a pcap request, submit a pcap query job,
> > and
> > > > > return a job id.  The request would be an object containing the
> > > > parameters
> > > > > documented here:  https://github.com/apache/metron/tree/master/
> > > > > metron-platform/metron-pcap-backend#query-filter-utility.  A
> > query/job
> > > > > would be associated with a user that submits it.  An exception will
> > be
> > > > > returned for violating constraints like too many queries submitted,
> > > query
> > > &g

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

[zeo...@gmail.com]

We did discuss doing a release since there were two new commits, but I
don't think it was included in this round.

Jon

On Sat, May 26, 2018, 10:22 Otto Fowler  wrote:

> Is there a BRO RC # for this?
>
>
> On May 25, 2018 at 14:53:25, Nick Allen (n...@nickallen.org) wrote:
>
> +1 Release this package as Apache Metron 0.5.0-RC1
>
> Ran through all validation steps using the `metron-rc-check` script, which
> included running all the tests, license checks, and spun-up the CentOS dev
> environment successfully.
>
>
>
> On Fri, May 25, 2018 at 1:40 PM, Nick Allen  wrote:
>
> > This release does not contain an updated Bro plugin so the RC check
> script
> > does not currently work. Try using the patch at
> > https://github.com/apache/metron/pull/1034.
> >
> >
> > On Thu, May 24, 2018 at 3:23 PM, Justin Leet  wrote:
> >
> >> Hi all,
> >>
> >> This is a call to vote on releasing Apache Metron 0.5.0
> >>
> >> Full list of changes in this release:
> >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/CHANGES
> >>
> >> The tag/commit to be voted upon is:
> >>
> >> (apache/metron) apache-metron-0.5.0-rc1
> >>
> >> The source archive being voted upon can be found here:
> >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
> >> apache-metron-0.5.0-rc1.tar.gz
> >>
> >> Other release files, signatures and digests can be found here:
> >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
> >>
> >> The release artifacts are signed with the following key:
> >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/KEYS
> >>
> >> Please vote on releasing this package as Apache Metron 0.5.0-RC1
> >>
> >> When voting, please list the actions taken to verify the release.
> >>
> >> Recommended build validation and verification instructions are posted
> >> here:
> >> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >>
> >> This vote will be open until 4pm EDT on Tuesday May 29 2018, to account
> >> for
> >> the weekend.
> >>
> >> [ ] +1 Release this package as Apache Metron 0.3.0-RC1
> >>
> >> [ ] 0 No opinion
> >>
> >> [ ] -1 Do not release this package because...
> >>
> >
> >
>
-- 

Jon

Re: Bro plugin release process docs?

[zeo...@gmail.com]

I did a bit of poking around and I don't believe we ever formally wrote
that down.  The last release happened as a combination of actions from
mattf and myself (mostly mattf).

The plugin has two new commits since the last release (1 bugfix 1 feature)
- if we want to couple version 0.2 of the plugin with a metron 0.5.0 release we
would need to make a 0.2 tag against HEAD of the plugin repo, then
increment the version in the ansible playbooks here

.

Or, if we just want to make a plugin release without changing
apache/metron, we could just make a 0.2 tag in the plugin repo, and then
release it in a more disjointed way.  I know that's not super helpful since
I don't have documentation for doing an apache release of the plugin other
than hacking something together based on what's out there for apache/metron.

Reference conversations:
 -
https://lists.apache.org/thread.html/2606166bd5e864f1b56db302099c9a8042cdadec8fa2692fef49493f@%3Cdev.metron.apache.org%3E
 -
https://lists.apache.org/thread.html/3aecbadbf3353e98c03ca4b680fcd998d0cd2bf5a4319238dd85ae75@%3Cdev.metron.apache.org%3E

Jon

On Fri, May 25, 2018 at 2:00 PM Justin Leet  wrote:

> At the risk of exposing my ignorance, do we have the bro plugin release
> process documented anywhere?  We have a doc for the main release (
> https://cwiki.apache.org/confluence/display/METRON/Release+Process), but I
> haven't noticed one for the bro plugin.
>
> For the current RC, it's not included and it wasn't pushed for (it has less
> changes for obvious reasons).  However, we should be making sure to
> validate if its necessary to release and having the process documented.
>
> Justin
>
-- 

Jon

Re: Knox SSO feature branch PRs: a quick demo

[zeo...@gmail.com]

Nice run through Simon that was very helpful for me to catch up on the work
you've been doing.  Appreciate the focus on this too, when talking to
others about Metron I have heard a few times that they were interested in
features that it seems we will soon have.  Hopefully I'll have a chance to
take a swing through your FB/PRs next week as well.

Jon

On Thu, Aug 2, 2018, 15:10 Michael Miklavcic 
wrote:

> Nice! Thanks for the walk-through, Simon. Agreed that this should assist
> reviewers in understanding the feature branch and PR break-down.
>
> Cheers,
> Mike
>
> On Thu, Aug 2, 2018 at 8:51 AM larry mccay  wrote:
>
> > Hi Simon -
> >
> > I like how you walk through those various PRs and describe what is done
> at
> > each step.
> > Please feel free sure to bring any suggestions that you may have for
> > improvements in Knox and KnoxSSO to the community.
> > The public cert issue is a pain for sure - we do have a knoxcli command
> for
> > exporting it but you need to be on the Knox machine to do that.
> > I've been considering add an API to the admin API for retrieving it as
> well
> > - but of course it will need to be up and running for that.
> >
> > thanks,
> >
> > --larry
> >
> >
> > On Wed, Aug 1, 2018 at 11:34 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> > > I've recently put in a number of PRs on the Knox feature branch, and
> > > thought it might be useful to post a quick 'sprint demo' style
> > explanation
> > > of what the various PRs and functionality entails:
> > > https://youtu.be/9OJz6hg0N1I
> > >
> > > Hope this helps with review process. There are a couple of areas where
> > that
> > > need a little follow on improvement (Ambari mpack cosmetic oddness
> > mainly).
> > > Any thoughts and assistance on that would be very greatly appreciated.
> > >
> > > Simon
> > >
> >
>
-- 

Jon

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

I agree - I would love to see a release not long after the PCAP FB gets
into master, and 0.6.0 makes sense to me.

I'd also like to see a 0.2 release of metron-bro-plugin-kafka.  There is
one new commit, and I have a PR open which is waiting on some tests before
it's ready to be evaluated/merged.  I will try to get that work done asap.
As of right now metron's dev ansible scripts pin to a specific release of
metron-bro-plugin-kafka (0.1
<0.1https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml>),
and I'm fine leaving that as is until after the coming release, but we
could also do a metron-bro-plugin-kafka release first and then update
metron to point the dev environment to the new package prior to the
upcoming RC.

I would also like to discuss what the roadmap looks like for a 1.0 release
and perhaps a more regular release schedule.  I have some thoughts but
don't want to hijack this thread.

Jon

On Wed, Aug 15, 2018 at 9:11 AM Justin Leet  wrote:

> Hi all,
>
> It's been a little while since the last release, and a couple major items
> have gone in since then (or are hopefully close to going in!).  In
> particular, I'd personally like to see a release with our Solr work
>  and the
> close-to-completion PCAP Query Panel
> .  There is a thread
> <
> https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E
> >
> around what's left before merging the PCAP feature branch, I encourage you
> to take a look. There are also some nice-to-haves as well as some Apache
> cleanup around the RAT tool and typescript files
> .
>
> Version Number
> I'm proposing bumping to 0.6.0, in particular because of the Solr and PCAP
> efforts. We can adjust that as necessary.
>
> I'm proposing we release this from the Metron master branch, plus any
> commits the community considers necessary.  Note that I'm proposing that
> this release occur after the PCAP feature branch is merged into master.
>
> Proposed Timeframe
> I would tentatively like to start work on the RC Wednesday, September 5th.
> It's a little further out than usual, but I wanted to kick off the
> discussion before Labor Day and to give ongoing  time to settle. And also
> because I'll be unavailable around Labor Day.
>
> JIRA Status
> There are 31 open PRs at https://github.com/apache/metron/pulls. We should
> work on getting anything we feel merits inclusion closed out. Please
> respond with any tickets we'd like included.
>
> A couple of these are for the PCAP feature branch, and there will be at
> least one more for documentation.
>
> There will be updates necessary to get our Jira up to date.  I'll follow up
> on that, and ask that everyone double check their tickets.
>
> There have been 106 commits since the 0.5.0 release (listed at the end of
> message). There will be a few more when we pull in the PCAP feature branch.
>
> Completed PRs as of Aug 15 as generated by git log --pretty="%cr %s"
> tags/apache-metron-0.5.0-release..HEAD.
>
> 5 days ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
> via mmiklavc) closes apache/metron#1152
> 13 days ago METRON-1701 Update General notes on the installation of Pycapa
> on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
> 3 weeks ago METRON-1650 Packaging docker containers are too large
> (jameslamb via merrimanr) closes apache/metron#1091
> 3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
> management pack repo info closes apache/incubator-metron#1052
> 3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
> apache/incubator-metron#1126
> 3 weeks ago METRON-1694: Clean up Metron REST docs closes
> apache/incubator-metron#1131
> 4 weeks ago METRON-1606 Add a 'wrap' to incoming messages in the
> metron json parser (ottobackwards) closes apache/metron#1054
> 4 weeks ago METRON-1672 Add metron-alerts's UI unit tests to travis
> build process (justinleet) closes apache/metron#1106
> 4 weeks ago METRON-1684 Fix Markdown problems in 3rdPartyParser.md
> (justinleet) closes apache/metron#1110
> 4 weeks ago METRON-1657 Parser aggregation in storm (justinleet) closes
> apache/metron#1099
> 4 weeks ago METRON-1651 Fixing failing protractor e2e test (tiborm via
> merrimanr) closes apache/metron#1095
> 4 weeks ago METRON-1673 Fix Javadoc errors (justinleet) closes
> apache/metron#1107
> 4 weeks ago METRON-1620: Fixes for forensic clustering use case example
> (mmiklavc via mmiklavc) closes apache/metron#1065
> 4 weeks ago METRON-1659: The platform-info.sh should check for the vagrant
> hostmanager plugin closes apache/incubator-metron#1100
> 4 weeks ago METRON-1658: Upgrade bro to 2.5.4 closes
> apache/incubator-metron#1101
> 4 weeks ago METRON-1236 Add start/stop/restart commands that execute
> successfully, w

Re: [DISCUSS] Release cadence

[zeo...@gmail.com]

I'm a fan of a hybrid time/feature-based cadence.  Something like "When 3
months has passed since our last release, or a sufficiently complicated
change has been introduced to master (like merging a FB), a discuss thread
is started".  I'm primarily thinking of what the upgrade path looks like
(more on that in a "how do we get to 1.0" discuss).

Jon

On Wed, Aug 15, 2018 at 11:02 AM Justin Leet  wrote:

> Hi all,
>
> In concert with the discuss thread on a potential 0.6.0 release, I'd also
> like start a discussion about our release cadence.  We've generally been
> pretty relaxed around doing releases, and I'm curious what people's
> thoughts are on adopting a somewhat more regular schedule.
>
> Couple questions I think are relevant
> 1. Is this something we should work towards and, if we do, how do we want
> to go about it?
>
>- "Whenever someone feels like pushing out a discuss thread"?
>- "Let's just start a discuss thread every X and if we want to release
>we release"?
>- "let's try to get a release out every X and what's on the bus is on
>the bus"?
>- Something else?
>
> 2. Assuming we do want to do more regular releases, what's the timeframe
> we'd like to shoot for?
>
> Personally, I'd like to just start a discuss thread regularly, with the
> built-in expectation that not every thread should necessarily lead to a
> release. I don't want to be forcing release overhead when there's not
> enough to merit a release, but releasing more often than we often do now
> would provide a lot of values to users.
>
> In terms of timeframe, I tend to think a 2-3 month cadence for the threads
> is reasonable. It's long enough to potentially accrue enough features to
> merit a release, but short enough that when we pass on a release we're
> probably fine just waiting for another cycle to come around.  The last
> release was ~2 months ago and we have a good amount of stuff here, but I
> also don't expect two feature branches going in to be the norm.
>
> I'd expect whatever comes out of this thread to also be relatively
> informal. At least right now, I don't feel like we need a rigid schedule,
> and I'd still like people to feel encouraged to propose a release,
> particularly when there are a couple major features or critical fixes.
> Alternatively, I would expect some of these discuss threads to conclude,
> "We should do a release, but let's wait a couple waits for these tickets to
> finish up" (e.g. like the Pcap query panel).
>
> Justin
>
-- 

Jon

[DISCUSS] Getting to a 1.0 release

[zeo...@gmail.com]

So, as has been discussed in a few

other

recent dev list threads, I would like to discuss what a Metron 1.0 release
looks like.

In order to kick off the conversation, I would like to make a few
suggestions regarding "what 1.0 means to me," but I'm very interested to
hear everybody else's opinions.

In order to go 1.0 I believe we should have:
1. A clear, supported method of upgrading from one version of Metron to the
next.  We have attempted
 to make this
easier in the past, but it is currently not

supported

.
2. Authentication for all of the UIs and APIs should be secure and support
SSO.  I believe this is in progress via METRON-1663
.
3. Each of our personas

should
be well documented, understood, and supported.
 - The current state of documentation is, in my opinion, inadequate and I
admit I am partially to blame for this.  I suggest we define a strict
approach for documentation, align to it (such as perhaps migrating all
useful wiki documentation to git), and enforce it.
 - I would consider METRON-1699
 as a critical item for
a Security Data Scientist, but it is currently not clear to me where the
line exists between some of the other personas, or that each persona has
been sufficiently implemented.
4. A performance tuning guide should be available for all of the main
components, whether as an independent document or as a part of a larger
document.
5. Simple data ingest.
 - Similar to the ongoing conversation for NiFi integration
,
we should be able to say that we have broken down the barriers to getting
data into a Metron cluster in easy and efficient ways.  In addition to
NiFi, having support for other popular tools such as beats
, fluentd ,
etc.
 - Parsers should be pluggable, with independent tests and the ability to
make versioned modifications with roll-backs.

What else?  Are any of these items not necessary for a 1.0?

Jon
-- 

Jon

Re: Need a slack invite

[zeo...@gmail.com]

Invite sent.

Jon

On Mon, Aug 27, 2018, 03:36 Karthik D B  wrote:

> Hi Team,
> I’m a non-ASF committers, I Would like to join the Metron Slack Channel.
> pls. Send an invite.
> Thanks,
> Karthik DB

-- 

Jon

Re: [ANNOUNCE] - Apache Metron Slack channel

[zeo...@gmail.com]

Invite sent.

Jon

On Mon, Aug 27, 2018, 02:45 Ali Nazemian  wrote:

> Can I be invited as well?
>
> On Thu, Aug 16, 2018 at 4:37 AM Otto Fowler 
> wrote:
>
> > Done
> >
> >
> > On August 15, 2018 at 14:22:45, Vets, Laurens (laur...@daemon.be) wrote:
> >
> > Could I be invited?
> >
> > On 15-Aug-18 09:48, Michael Miklavcic wrote:
> > > + Metron user list
> > >
> > > On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > >> Turns out we are able to invite folks on an ad-hoc basis. See
> > instructions
> > >> here -
> > >>
> https://cwiki.apache.org/confluence/display/METRON/Community+Resources
> > >>
> > >>
> > >> On Wed, Aug 15, 2018 at 9:23 AM Michael Miklavcic <
> > >> michael.miklav...@gmail.com> wrote:
> > >>
> > >>> It's another option with different features. I imagine many people
> will
> > >>> use both.
> > >>>
> > >>> On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball <
> > >>> si...@simonellistonball.com> wrote:
> > >>>
> >  Since this is committers only, would it make more sense to stick to
> > IRC?
> >  Or
> >  is exclusivity the idea?
> > 
> >  On 15 August 2018 at 16:09, Nick Allen  wrote:
> > 
> > > Thanks for the instructions!
> > >
> > > On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > >> The Metron community has a Slack channel available for
> communication
> > >> (similar to the existing IRC channel, only on Slack).
> > >>
> > >> To join:
> > >>
> > >> 1. Go to slack.com.
> > >> 2. For organization/group, you'll enter "the-asf"
> > >> 3. Use your Apache email for your login
> > >> 4. Click "Channels" and look for #metron (Created by ottO June 15,
> > > 2018)
> > >> Best
> > >> Mike Miklavcic
> > >>
> > 
> > 
> >  --
> >  --
> >  simon elliston ball
> >  @sireb
> > 
> >
>
>
> --
> A.Nazemian
>
-- 

Jon

Re: [DISCUSS] Getting to a 1.0 release

[zeo...@gmail.com]

..@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 to separating developer docs and user docs. How should we
> approach
> > > > that.
> > > > > Have a separate doc book? I haven’t had a ton of time to contribute
> > to
> > > > code
> > > > > lately but I’d be happy to help write some of these.
> > > > >
> > > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen 
> > wrote:
> > > > >
> > > > > > Personally, I think the state of our docs and web presence is an
> > > > > inhibitor
> > > > > > to growing the Metron community.  Unless we can offer concise,
> > > > compelling
> > > > > > answers to the basic questions (What can I do with Metron?  Who
> > does
> > > it
> > > > > > help? How do I do that?), potential users and contributors are
> > unable
> > > > to
> > > > > > see the value of Metron.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen 
> > > > wrote:
> > > > > >
> > > > > > > I'd like to see us focus on improving our docs before a version
> > > 1.0.
> > > > > > > Right now we just stitch together a bunch of READMEs, which is
> a
> > > > great
> > > > > > > stride from where we started, but is not ideal.
> > > > > > >
> > > > > > > Our docs should focused on the user and use cases; What can I
> do
> > > with
> > > > > > > Metron?  Who does it help? How do I do that?
> > > > > > >
> > > > > > > The docs should be separate from the code base to allow for an
> > > > > > > organization that is focused on the user rather than the
> > > > > implementation.
> > > > > > > This allows the READMEs to focus on the developer and the
> > > > > implementation,
> > > > > > > which should make them more digestible too.  The docs should be
> > > > version
> > > > > > > controlled and maintained through PRs, just like the code.  We
> > > should
> > > > > > take
> > > > > > > just as much pride in our docs as we do in our code.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > > > > > > si...@simonellistonball.com> wrote:
> > > > > > >
> > > > > > >> Agreed, should we add TDE by default, and get the ranger
> > policies
> > > on
> > > > > by
> > > > > > >> default? That leaves secured in Kafka, which would have to be
> > > built
> > > > > into
> > > > > > >> the consumers and producers to encrypt into the on disk Kafka
> > > > topics.
> > > > > > Does
> > > > > > >> that seem necessary to people? It would have performance
> > > > implications
> > > > > > for
> > > > > > >> sure.
> > > > > > >>
> > > > > > >> Simon
> > > > > > >>
> > > > > > >> > On 15 Aug 2018, at 21:26, Otto Fowler <
> > ottobackwa...@gmail.com>
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > Well, I look at it like this.
> > > > > > >> >
> > > > > > >> > The Secure Vault was part of the original metron pitch, and
> > many
> > > > may
> > > > > > >> have used that as part of their evaluations.
> > > > > > >> > “Look, it is going to have a security vault type thing, it
> is
> > on
> > > > the
> > > > > > >> roadmap”.
> > > > > > >> >
> > > > > > >> > Regardless of the implementation, conceptually, security of
> > data
> > > > at
> > > > > > >> rest is important, and is a major outstanding item or the core
> > > > metron
> > > > > > >> proposition.
> > > > > > >> >
> > > > > > >> &

Re: IRC Channel -> OPS?

[zeo...@gmail.com]

Isn't it Casey?

Jon

On Wed, Aug 29, 2018, 08:41 Otto Fowler  wrote:

> Who has ops in the irc channel?
> Can you pop in and set the topic to something like:
> “There is an ASF slack with an active metron channel, please email
> dev@metron.apache.org and request an invite”
>
-- 

Jon

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

 ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago METRON-1617: Make threat triage score function with
> >> dots as
> >> > > > well as colons closes apache/incubator-metron#1062
> >> > > > 9 weeks ago METRON-1613 Metaalerts status update broken in Alerts
> UI
> >> > > > (merrimanr) closes apache/metron#1059
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago METRON-1588 Migrate storm-kafka-client to 1.2.1 closes
> >> > > > apache/incubator-metron#1039
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago Merge branch 'feature/METRON-1416-upgrade-solr' of
> >> > > > https://git-wip-us.apache.org/repos/asf/metron into
> >> > > > feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago METRON-1587 Make collection utility work for HDP
> search
> >> > > > (merrimanr) closes apache/metron#1043
> >> > > > 9 weeks ago METRON-1612 Fix website download links (justinleet)
> >> closes
> >> > > > apache/metron#1058
> >> > > > 9 weeks ago METRON-1608 Add configuration for threat.triage.field
> >> name
> >> > > > (merrimanr) closes apache/metron#1055
> >> > > > 10 weeks ago METRON-1585 SolrRetrieveLatestDao does not use the
> >> > > collection
> >> > > > lookup (justinleet via merrimanr) closes apache/metron#1050
> >> > > > 10 weeks ago METRON-1533 Create KAFKA_FIND Stellar Function
> >> > (nickwallen)
> >> > > > closes apache/metron#1025
> >> > > > 10 weeks ago METRON-1601: Rename metaalert alert nested field to
> >> > > > metron_alert to avoid collision closes
> apache/incubator-metron#1049
> >> > > > 10 weeks ago METRON-1572 Enhance KAFKA_PUT function (nickwallen)
> >> closes
> >> > > > apache/metron#1024
> >> > > > 10 weeks ago METRON-1607 update public web site to point at 0.5.0
> >> new
> >> > > > release (justinleet) closes apache/metron#1053
> >> > > > 10 weeks ago METRON-1568: Stellar should have a _ special variable
> >> > which
> >> > > > returns the message in map form closes
> apache/incubator-metron#1021
> >> > > > 2 months ago METRON-1594: KafkaWriter is asynchronous and may lose
> >> data
> >> > > on
> >> > > > node failure (mmiklavc via mmiklavc) closes apache/metron#1045
> >> > > > 2 months ago METRON-1603: Fix multivalue field errors in Bro Solr
> >> > schema
> >> > > > (mmiklavc via mmiklavc) closes apache/metron#1051
> >> > > > 2 months ago METRON-1584 Indexing Topology Crashes with Invalid
> >> Message
> >> > > > (nickwallen) closes apache/metron#1036
> >> > > > 2 months ago METRON-1547 Solr Comment Fields (justinleet) closes
> >> > > > apache/metron#1037
> >> > > > 2 months ago METRON-1553 Validate JIRA Script Error (nickwallen)
> >> closes
> >> > > > apache/metron#1013
> >> > > > 2 months ago METRON-1592 Unable to use third party parser with
> Storm
> >> > > > versions >= 1.1.0 (nickwallen) closes apache/metron#1042
> >> > > > 2 months ago METRON-1598 NoClassDefFoundError when running with
> >> > > > Elasticsearch X-Pack (nickwallen) closes apache/metron#1048
> >> > > > 2 months ago METRON-1589 '/api/v1/search/search' fails when 'Solr
> >> > > Zookeeper
> >> > > > Urls' has comma separated multiple zookeeper urls (justinleet)
> >> closes
> >> > > > apache/metron#1040
> >> > > > 2 months ago METRON-1593 Setting Metron rest additional classpath
> >> > removes
> >> > > > HBase and Hadoop configs from classpath (merrimanr) closes
> >> > > > apache/m

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

I lied, we didn't need to update our btests because it's limited to a major
and minor version.

https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34

Jon

On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com  wrote:

> I looked into x.y.z back when we released 0.1 and it was not possible in
> bro-pkg at the time but now it is
> <https://github.com/bro/package-manager/issues/32>.  In order to do this,
> we'll also need to configure bro-pkg.meta to require the proper version of
> bro-pkg, as well as update the btests for the new version string.  I will
> throw together a JIRA and PR to do all this in case we decide to align with
> x.y.z; we can trash it if we decide to stay with x.y.
>
> Jon
>
> On Wed, Sep 5, 2018 at 7:35 PM Justin Leet  wrote:
>
>> Long story short, while preparing the release candidate, I discovered our
>> metron-bro-plugin-kafka is inconsistently versioned.
>>
>> In the repo, it's x.y (e.g. 0.2)
>> See:
>> https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18
>>
>> In our released artifact, it's x.y.z (e.g. 0.1.0)
>> See http://archive.apache.org/dist/metron/0.4.2/
>>
>> Going forward from this release, I'd like the artifact to be consistent
>> with the repo.  I'd personally prefer x.y.z to be entirely consistent
>> throughout Metron, but if there's a particular reason why it was x.y I'm
>> happy to entertain it.
>>
>> If we choose to move to x.y.z, I can provide a PR to update the version to
>> 0.2.0 unless someone else wants to volunteer. Otherwise, I'd like to
>> release the artifact as apache-metron-bro-plugin-kafka_0.2.tar.gz
>>
>> Justin
>>
>> On Tue, Sep 4, 2018 at 12:15 PM Justin Leet 
>> wrote:
>>
>> > As an update, I'll be working on starting the release process rolling
>> > today. The PCAP Query panel feature branch is in master and I haven't
>> heard
>> > of any other potential blockers. I'd appreciate everyone updating their
>> > JIRAs with their state (complete, etc.) and version (0.6.0) as needed.
>> >
>> > Do we have anything that needs to go into the UPGRADING.md file?
>> >
>> > A PR is out for updating the version from 0.5.1 to 0.6.0. Please see
>> > https://github.com/apache/metron/pull/1183. I still need to spin up
>> full
>> > dev etc. before this is ready to be merged.
>> >
>> > Updated list of PRs that have made it into master as of Sept 4, 2018
>> >
>> > 6 days ago METRON-1751 Storm Profiler dies when consuming null message
>> > (nickwallen) closes apache/metron#1176
>> > 6 days ago METRON-1757 Storm Profiler Serialization Exception
>> (nickwallen)
>> > closes apache/metron#1178
>> > 6 days ago METRON-1743 CEF testPaloAltoCEF test using a confusing
>> variable
>> > name (JonZeolla via justinleet) closes apache/metron#1173
>> > 8 days ago METRON-1752 Prevent package.lock from changing during build
>> > (sardell via merrimanr) closes apache/metron#1177
>> > 8 days ago METRON-1724 Date/time validation missing in PCAP query
>> (tiborm
>> > via nickwallen) closes apache/metron#1172
>> > 3 weeks ago METRON-1554 Pcap Query Panel (merrimanr) closes
>> > apache/metron#1169
>> > 3 weeks ago METRON-1739 UDP packets are not handled (merrimanr) closes
>> > apache/metron#1168
>> > 3 weeks ago METRON-1727: Alerts are not populated on the alerts UI after
>> > enabling X-pack for Elastic search (MohanDV via mmiklavc) closes
>> > apache/metron#1141
>> > 3 weeks ago METRON-1738: Pcap directories should have correct
>> permissions
>> > (merrimanr via mmiklavc) closes apache/metron#1166
>> > 3 weeks ago METRON-1737: Document Job cleanup (merrimanr via mmiklavc)
>> > closes apache/metron#1164
>> > 3 weeks ago METRON-1732: Fix job status liveness bug and parallelize
>> > finalizer file writing (mmiklavc via mmiklavc) closes apache/metron#1157
>> > 3 weeks ago METRON-1735 Empty print status option causes NPE (merrimanr)
>> > closes apache/metron#1160
>> > 3 weeks ago METRON-1733 PCAP UI - PCAP queries don't work on Safari
>> > (sardell via merrimanr) closes apache/metron#1158
>> > 3 weeks ago METRON-1734 Src and Dst port filters are incorrect after
>> > changing to empty (merrimanr) closes apache/metron#1159
>> > 4 weeks ago METRON-1725 Add ability to specify YARN queue for pcap jobs
>> > (merrimanr) closes apache/metron#1153
>> > 4 weeks ago METRON-1731: 

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

I think mattf-horton just did that as a part of convention.  He handled
that part, and I did the 0.1 tagging (as a prereq to this
<https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30>)
last time the package was released.

Jon

On Wed, Sep 5, 2018 at 8:28 PM Justin Leet  wrote:

> Any idea why we released it as 0.1.0 in the artifacts version?  I'm fine
> with doing x.y if we need to, but I would like the artifact versioning to
> be consistent if possible.
>
> On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com  wrote:
>
> > I lied, we didn't need to update our btests because it's limited to a
> major
> > and minor version.
> >
> >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34
> >
> > Jon
> >
> > On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com 
> wrote:
> >
> > > I looked into x.y.z back when we released 0.1 and it was not possible
> in
> > > bro-pkg at the time but now it is
> > > <https://github.com/bro/package-manager/issues/32>.  In order to do
> > this,
> > > we'll also need to configure bro-pkg.meta to require the proper version
> > of
> > > bro-pkg, as well as update the btests for the new version string.  I
> will
> > > throw together a JIRA and PR to do all this in case we decide to align
> > with
> > > x.y.z; we can trash it if we decide to stay with x.y.
> > >
> > > Jon
> > >
> > > On Wed, Sep 5, 2018 at 7:35 PM Justin Leet 
> > wrote:
> > >
> > >> Long story short, while preparing the release candidate, I discovered
> > our
> > >> metron-bro-plugin-kafka is inconsistently versioned.
> > >>
> > >> In the repo, it's x.y (e.g. 0.2)
> > >> See:
> > >>
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18
> > >>
> > >> In our released artifact, it's x.y.z (e.g. 0.1.0)
> > >> See http://archive.apache.org/dist/metron/0.4.2/
> > >>
> > >> Going forward from this release, I'd like the artifact to be
> consistent
> > >> with the repo.  I'd personally prefer x.y.z to be entirely consistent
> > >> throughout Metron, but if there's a particular reason why it was x.y
> I'm
> > >> happy to entertain it.
> > >>
> > >> If we choose to move to x.y.z, I can provide a PR to update the
> version
> > to
> > >> 0.2.0 unless someone else wants to volunteer. Otherwise, I'd like to
> > >> release the artifact as apache-metron-bro-plugin-kafka_0.2.tar.gz
> > >>
> > >> Justin
> > >>
> > >> On Tue, Sep 4, 2018 at 12:15 PM Justin Leet 
> > >> wrote:
> > >>
> > >> > As an update, I'll be working on starting the release process
> rolling
> > >> > today. The PCAP Query panel feature branch is in master and I
> haven't
> > >> heard
> > >> > of any other potential blockers. I'd appreciate everyone updating
> > their
> > >> > JIRAs with their state (complete, etc.) and version (0.6.0) as
> needed.
> > >> >
> > >> > Do we have anything that needs to go into the UPGRADING.md file?
> > >> >
> > >> > A PR is out for updating the version from 0.5.1 to 0.6.0. Please see
> > >> > https://github.com/apache/metron/pull/1183. I still need to spin up
> > >> full
> > >> > dev etc. before this is ready to be merged.
> > >> >
> > >> > Updated list of PRs that have made it into master as of Sept 4, 2018
> > >> >
> > >> > 6 days ago METRON-1751 Storm Profiler dies when consuming null
> message
> > >> > (nickwallen) closes apache/metron#1176
> > >> > 6 days ago METRON-1757 Storm Profiler Serialization Exception
> > >> (nickwallen)
> > >> > closes apache/metron#1178
> > >> > 6 days ago METRON-1743 CEF testPaloAltoCEF test using a confusing
> > >> variable
> > >> > name (JonZeolla via justinleet) closes apache/metron#1173
> > >> > 8 days ago METRON-1752 Prevent package.lock from changing during
> build
> > >> > (sardell via merrimanr) closes apache/metron#1177
> > >> > 8 days ago METRON-1724 Date/time validation missing in PCAP query
> > >> (tiborm
> > >> > via nickwallen) cl

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

Either is fine with me.  If it's x.y in some parts of the app I prefer to
keep it consistent throughout, but I'm also fine with lining up with
Apache/Metron where we can.

I also refreshed myself on why we avoided x.y.z initially and it was
actually for this exact reason, we wanted consistent versioning throughout
a repo.  This issue is with the bro plugins themselves, not bro-pkg, so I
submit a JIRA <https://bro-tracker.atlassian.net/browse/BIT-1985>.

Jon


On Wed, Sep 5, 2018, 21:51 Justin Leet  wrote:

> Makes sense. Do we have any objection to just going to the artifact being
> 0.2?  Or do we want to keep the mixed versioning and just live with it, at
> least for now?
>
> On Wed, Sep 5, 2018 at 8:58 PM zeo...@gmail.com  wrote:
>
> > I think mattf-horton just did that as a part of convention.  He handled
> > that part, and I did the 0.1 tagging (as a prereq to this
> > <
> >
> https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30
> > >)
> > last time the package was released.
> >
> > Jon
> >
> > On Wed, Sep 5, 2018 at 8:28 PM Justin Leet 
> wrote:
> >
> > > Any idea why we released it as 0.1.0 in the artifacts version?  I'm
> fine
> > > with doing x.y if we need to, but I would like the artifact versioning
> to
> > > be consistent if possible.
> > >
> > > On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > I lied, we didn't need to update our btests because it's limited to a
> > > major
> > > > and minor version.
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34
> > > >
> > > > Jon
> > > >
> > > > On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com 
> > > wrote:
> > > >
> > > > > I looked into x.y.z back when we released 0.1 and it was not
> possible
> > > in
> > > > > bro-pkg at the time but now it is
> > > > > <https://github.com/bro/package-manager/issues/32>.  In order to
> do
> > > > this,
> > > > > we'll also need to configure bro-pkg.meta to require the proper
> > version
> > > > of
> > > > > bro-pkg, as well as update the btests for the new version string.
> I
> > > will
> > > > > throw together a JIRA and PR to do all this in case we decide to
> > align
> > > > with
> > > > > x.y.z; we can trash it if we decide to stay with x.y.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Wed, Sep 5, 2018 at 7:35 PM Justin Leet 
> > > > wrote:
> > > > >
> > > > >> Long story short, while preparing the release candidate, I
> > discovered
> > > > our
> > > > >> metron-bro-plugin-kafka is inconsistently versioned.
> > > > >>
> > > > >> In the repo, it's x.y (e.g. 0.2)
> > > > >> See:
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18
> > > > >>
> > > > >> In our released artifact, it's x.y.z (e.g. 0.1.0)
> > > > >> See http://archive.apache.org/dist/metron/0.4.2/
> > > > >>
> > > > >> Going forward from this release, I'd like the artifact to be
> > > consistent
> > > > >> with the repo.  I'd personally prefer x.y.z to be entirely
> > consistent
> > > > >> throughout Metron, but if there's a particular reason why it was
> x.y
> > > I'm
> > > > >> happy to entertain it.
> > > > >>
> > > > >> If we choose to move to x.y.z, I can provide a PR to update the
> > > version
> > > > to
> > > > >> 0.2.0 unless someone else wants to volunteer. Otherwise, I'd like
> to
> > > > >> release the artifact as apache-metron-bro-plugin-kafka_0.2.tar.gz
> > > > >>
> > > > >> Justin
> > > > >>
> > > > >> On Tue, Sep 4, 2018 at 12:15 PM Justin Leet <
> justinjl...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > As an update, I'll be working on starting the release process
> > > rolling
> > &

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

I'm not aware of the bro plugin artifacts being used in any way.

Jon

On Thu, Sep 6, 2018, 10:59 Justin Leet  wrote:

> Do we use the artifacts directly at all? Or is it through bro-pkg only?
>
> Also, It's very possible I'm making a mountain out of a molehill here, and
> if it's something that's not particularly important, it might be worthwhile
> to just stick with what we did last time, and just table this discussion
> until post release.  It feels pretty nitpicky at this point, and if the
> practical implications are pretty minor, I'd rather just get an RC out.
>
> On Thu, Sep 6, 2018 at 10:02 AM zeo...@gmail.com  wrote:
>
> > Either is fine with me.  If it's x.y in some parts of the app I prefer to
> > keep it consistent throughout, but I'm also fine with lining up with
> > Apache/Metron where we can.
> >
> > I also refreshed myself on why we avoided x.y.z initially and it was
> > actually for this exact reason, we wanted consistent versioning
> throughout
> > a repo.  This issue is with the bro plugins themselves, not bro-pkg, so I
> > submit a JIRA <https://bro-tracker.atlassian.net/browse/BIT-1985>.
> >
> > Jon
> >
> >
> > On Wed, Sep 5, 2018, 21:51 Justin Leet  wrote:
> >
> > > Makes sense. Do we have any objection to just going to the artifact
> being
> > > 0.2?  Or do we want to keep the mixed versioning and just live with it,
> > at
> > > least for now?
> > >
> > > On Wed, Sep 5, 2018 at 8:58 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > I think mattf-horton just did that as a part of convention.  He
> handled
> > > > that part, and I did the 0.1 tagging (as a prereq to this
> > > > <
> > > >
> > >
> >
> https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30
> > > > >)
> > > > last time the package was released.
> > > >
> > > > Jon
> > > >
> > > > On Wed, Sep 5, 2018 at 8:28 PM Justin Leet 
> > > wrote:
> > > >
> > > > > Any idea why we released it as 0.1.0 in the artifacts version?  I'm
> > > fine
> > > > > with doing x.y if we need to, but I would like the artifact
> > versioning
> > > to
> > > > > be consistent if possible.
> > > > >
> > > > > On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com 
> > > > wrote:
> > > > >
> > > > > > I lied, we didn't need to update our btests because it's limited
> > to a
> > > > > major
> > > > > > and minor version.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com <
> zeo...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I looked into x.y.z back when we released 0.1 and it was not
> > > possible
> > > > > in
> > > > > > > bro-pkg at the time but now it is
> > > > > > > <https://github.com/bro/package-manager/issues/32>.  In order
> to
> > > do
> > > > > > this,
> > > > > > > we'll also need to configure bro-pkg.meta to require the proper
> > > > version
> > > > > > of
> > > > > > > bro-pkg, as well as update the btests for the new version
> string.
> > > I
> > > > > will
> > > > > > > throw together a JIRA and PR to do all this in case we decide
> to
> > > > align
> > > > > > with
> > > > > > > x.y.z; we can trash it if we decide to stay with x.y.
> > > > > > >
> > > > > > > Jon
> > > > > > >
> > > > > > > On Wed, Sep 5, 2018 at 7:35 PM Justin Leet <
> > justinjl...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Long story short, while preparing the release candidate, I
> > > > discovered
> > > > > > our
> > > > > > >> metron-bro-plugin-kafka is inconsistently versioned.
> 

Re: [DISCUSS] Feature branches post-merge

[zeo...@gmail.com]

Yeah I don't have a good reason to suggest we keep 'em. so +1 to deleting
old FBs.

Jon

On Fri, Sep 7, 2018 at 12:14 PM Nick Allen  wrote:

> +1 delete old feature branches.
>
> BTW, there is a branch out there called METRON-113 that we probably need to
> clean-up.  I'm not sure where that came from or why its still around.
> Probably a fat-finger from long ago.
>
> On Fri, Sep 7, 2018 at 12:00 PM Otto Fowler 
> wrote:
>
> > I would drop them.
> > I’ve already clean up FB’s around dead things.
> >
> >
> >
> > On September 6, 2018 at 13:42:55, Michael Miklavcic (
> > michael.miklav...@gmail.com) wrote:
> >
> > What are we doing with feature branches once they're complete and merged
> > into master? Is our expectation that we'll keep feature branches in
> > perpetuity, or should we plan to do some house cleaning once they've been
> > merged? I did a quick check of NiFi and Kafka and don't see much by way
> of
> > feature branches in their repos. I see plenty of RC's in both the
> branches
> > and tags listings, but nothing FB related. In previous discussions, we
> > talked quite a bit about us "trailblazing here," so it may be that this
> is
> > simply without much precedent and entirely for us to decide. I can
> > definitely see value in maintaining them for future reference, as it does
> > offer a nice bucket in which to collect the commits and discussion
> nicely,
> > but I wanted to get others' thoughts.
> >
> > Best,
> > Mike
> >
>
-- 

Jon

Re: [DISCUSS] Split apart releases for core Metron and the Bro plugin

[zeo...@gmail.com]

+1 to defer for this release and +1 to Justin's suggested release/dist
directory breakout and complete separation.

Jon

On Fri, Sep 7, 2018 at 1:43 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 to deferring for this release and having the separation like NiFi. Since
> we're bootstrapping from their process, what are they doing? I would assume
> we'd want some sort of vote for the plugin version change as well.
>
> On Fri, Sep 7, 2018 at 10:15 AM Nick Allen  wrote:
>
> > +1 for complete separation as you've described.
> >
> > On Fri, Sep 7, 2018 at 11:31 AM Justin Leet 
> wrote:
> >
> > > I would like this to be a complete separation.  Complete with separate
> > RCs,
> > > separate call to vote, etc. There's a bit more overhead, but plugin
> > > releases should be rarer and as the release infra gets improved and
> > > scripted out more, I don't think it'll end up being much more than
> > bundling
> > > it together.
> > >
> > > On Fri, Sep 7, 2018 at 11:27 AM Nick Allen  wrote:
> > >
> > > > > Other projects, e.g. NiFi , split
> > apart
> > > > these releases within their dist directories.
> > > >
> > > > I prefer the way Nifi organizes it.  Definitely seems more logically
> > > > organized.
> > > >
> > > >
> > > > > If we split them apart, we can make the releases independently.
> This
> > > > fixes the problem of aligning the versions (simply release the plugin
> > > > first, update full-dev, release core Metron).
> > > >
> > > > Does this entail a complete separation; including a separate
> > > call-to-vote?
> > > > One vote for core Metron and a separate vote for plugin?
> > > >
> > > >
> > > > > Do we want try to get this separation done after the current
> release
> > > > cycle is over?
> > > >
> > > > +1 Let's wait for the next release to hash this out.
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Sep 7, 2018 at 10:27 AM Justin Leet 
> > > wrote:
> > > >
> > > > > Right now, we tie together our main release and the Bro plugin, as
> > seen
> > > > in
> > > > > our 0.4.2 release
> > > > > https://archive.apache.org/dist/metron/0.4.2/ and the current RC.
> > > > >
> > > > > Other projects, e.g. NiFi , split
> > apart
> > > > > these
> > > > > releases within their dist directories.
> > > > >
> > > > > In our case this might look something like
> > > > > 0.5.0/
> > > > > metron-bro-plugin-kafka
> > > > > - 0.2.0/
> > > > >
> > > > > Right now, with the releases tied together, we aren't upgrading
> > > full-dev
> > > > > with the version of the plugin (because we're releasing
> > simultaneously
> > > > and
> > > > > can't update the version number).
> > > > >
> > > > > If we split them apart, we can make the releases independently.
> This
> > > > fixes
> > > > > the problem of aligning the versions (simply release the plugin
> > first,
> > > > > update full-dev, release core Metron).  The plugin also updates
> > > > > substantially less often and we can just do those releases at a
> > cadence
> > > > we
> > > > > choose.
> > > > >
> > > > > Any thoughts on doing this?
> > > > > Do we want try to get this separation done after the current
> release
> > > > cycle
> > > > > is over?
> > > > > If we do, do we have a preferred layout? I didn't see anything
> Apache
> > > > > preferred in a quick search, but I definitely could have missed
> > > something
> > > > > (and https://checker.apache.org/projs/nifi.html looks clean for
> > NiFi,
> > > > so I
> > > > > assume it's fine.)
> > > > >
> > > >
> > >
> >
>
-- 

Jon

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

Yup, your first paragraph is exactly right.

Jon

On Fri, Sep 7, 2018 at 1:23 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I'm a little rusty on my C++, but to summarize what I've gathered from this
> thread and linked code: The Bro plugin infrastructure does not support a
> patch version currently, only major.minor (x.y). But bro-pkg, the bro
> package installation mechanism, does support it now. Which is unfortunately
> moot bc unlike other 3rd party scripts that can support vX.Y.Z, this is a
> plugin and the API does not support the 'Y'.
>
> I think I'm in favor of keeping X.Y (0.2) for now. I would urge this
> approach by comparison to calculating significant figures - if my tooling
> is only able to calculate 2 decimal places, e.g. 1.23, I shouldn't be
> recording 1.230 or 1.23 as this isn't reflective of the tool's actual
> fidelity. Likewise, the Bro plugin API doesn't allow X.Y.Z right now, only
> X.Y per Jon's comments. Not strictly speaking an apples to apples
> comparison, but it's a justifiable approach imho. Once the plugin
> infrastructure supports it, we can add the extra patch version to reflect
> this additional available state info.
>
> Best,
> Mike
>
>
> On Thu, Sep 6, 2018 at 7:34 PM zeo...@gmail.com  wrote:
>
> > I'm not aware of the bro plugin artifacts being used in any way.
> >
> > Jon
> >
> > On Thu, Sep 6, 2018, 10:59 Justin Leet  wrote:
> >
> > > Do we use the artifacts directly at all? Or is it through bro-pkg only?
> > >
> > > Also, It's very possible I'm making a mountain out of a molehill here,
> > and
> > > if it's something that's not particularly important, it might be
> > worthwhile
> > > to just stick with what we did last time, and just table this
> discussion
> > > until post release.  It feels pretty nitpicky at this point, and if the
> > > practical implications are pretty minor, I'd rather just get an RC out.
> > >
> > > On Thu, Sep 6, 2018 at 10:02 AM zeo...@gmail.com 
> > wrote:
> > >
> > > > Either is fine with me.  If it's x.y in some parts of the app I
> prefer
> > to
> > > > keep it consistent throughout, but I'm also fine with lining up with
> > > > Apache/Metron where we can.
> > > >
> > > > I also refreshed myself on why we avoided x.y.z initially and it was
> > > > actually for this exact reason, we wanted consistent versioning
> > > throughout
> > > > a repo.  This issue is with the bro plugins themselves, not bro-pkg,
> > so I
> > > > submit a JIRA <https://bro-tracker.atlassian.net/browse/BIT-1985>.
> > > >
> > > > Jon
> > > >
> > > >
> > > > On Wed, Sep 5, 2018, 21:51 Justin Leet 
> wrote:
> > > >
> > > > > Makes sense. Do we have any objection to just going to the artifact
> > > being
> > > > > 0.2?  Or do we want to keep the mixed versioning and just live with
> > it,
> > > > at
> > > > > least for now?
> > > > >
> > > > > On Wed, Sep 5, 2018 at 8:58 PM zeo...@gmail.com 
> > > > wrote:
> > > > >
> > > > > > I think mattf-horton just did that as a part of convention.  He
> > > handled
> > > > > > that part, and I did the 0.1 tagging (as a prereq to this
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30
> > > > > > >)
> > > > > > last time the package was released.
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Wed, Sep 5, 2018 at 8:28 PM Justin Leet <
> justinjl...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Any idea why we released it as 0.1.0 in the artifacts version?
> > I'm
> > > > > fine
> > > > > > > with doing x.y if we need to, but I would like the artifact
> > > > versioning
> > > > > to
> > > > > > > be consistent if possible.
> > > > > > >
> > > > > > > On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com <
> > zeo...@gmail.com>
> > > > > > wrote:
> > > > > > &g

Metron dev environments moving to require Ansible 2.4+

[zeo...@gmail.com]

Hi All,

As it currently sits, once METRON-1758
 is merged into the code base,
Ansible 2.4 or later will be required to use any of the Metron ansible
playbooks.  This is in contrast to the prior version requirements outlined
in Metron documentation which specifically point to 2.0.0.2 and 2.2.0.0 as
supported/recommended Ansible versions.  If you install Ansible 2.5.0
exactly you should not experience any issues spinning up pre- and post-
merge versions of Metron.

I am broadcasting this to both the user and dev communities in advance of
any changes to provide an opportunity to voice any concerns.  Thanks,

Jon
-- 

Jon

Re: Metron dev environments moving to require Ansible 2.4+

[zeo...@gmail.com]

Do you mean this
<https://cwiki.apache.org/confluence/display/METRON/Downgrade+Ansible>?  It
was the only reference I could find on the wiki.  All of the READMEs should
be updated as a part of the PR, but feel free to provide your input if I
missed anything.

Jon

On Fri, Sep 28, 2018 at 10:15 AM Otto Fowler 
wrote:

> We should make sure the non-source documentation is updated
>
>
> On September 28, 2018 at 09:32:52, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> Hi All,
>
> As it currently sits, once METRON-1758
> <https://github.com/apache/metron/pull/1179> is merged into the code
> base, Ansible 2.4 or later will be required to use any of the Metron
> ansible playbooks.  This is in contrast to the prior version requirements
> outlined in Metron documentation which specifically point to 2.0.0.2 and
> 2.2.0.0 as supported/recommended Ansible versions.  If you install Ansible
> 2.5.0 exactly you should not experience any issues spinning up pre- and post-
> merge versions of Metron.
>
> I am broadcasting this to both the user and dev communities in advance of
> any changes to provide an opportunity to voice any concerns.  Thanks,
>
> Jon
> --
>
> Jon
>
> --

Jon

Re: Metron dev environments moving to require Ansible 2.4+

[zeo...@gmail.com]

Hi All,

This has been pushed to master.  Please updated your Ansible
appropriately.  Thanks!

Jon

On Fri, Sep 28, 2018 at 12:18 PM Otto Fowler 
wrote:

> Yeah,  I thought we had more but maybe they where removed.
> Many places in *.md files referencing Ansible and versions too
>
>
> On September 28, 2018 at 11:45:14, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> Do you mean this
> <https://cwiki.apache.org/confluence/display/METRON/Downgrade+Ansible>?
> It was the only reference I could find on the wiki.  All of the READMEs
> should be updated as a part of the PR, but feel free to provide your input
> if I missed anything.
>
> Jon
>
> On Fri, Sep 28, 2018 at 10:15 AM Otto Fowler 
> wrote:
>
>> We should make sure the non-source documentation is updated
>>
>>
>> On September 28, 2018 at 09:32:52, zeo...@gmail.com (zeo...@gmail.com)
>> wrote:
>>
>> Hi All,
>>
>> As it currently sits, once METRON-1758
>> <https://github.com/apache/metron/pull/1179> is merged into the code
>> base, Ansible 2.4 or later will be required to use any of the Metron
>> ansible playbooks.  This is in contrast to the prior version requirements
>> outlined in Metron documentation which specifically point to 2.0.0.2 and
>> 2.2.0.0 as supported/recommended Ansible versions.  If you install Ansible
>> 2.5.0 exactly you should not experience any issues spinning up pre- and post-
>> merge versions of Metron.
>>
>> I am broadcasting this to both the user and dev communities in advance of
>> any changes to provide an opportunity to voice any concerns.  Thanks,
>>
>> Jon
>> --
>>
>> Jon
>>
>> --
>
> Jon
>
> --

Jon

Re: [DISCUSS] Split apart releases for core Metron and the Bro plugin

[zeo...@gmail.com]

Good catch.  I'm not sure what the right approach is there, I wonder what
other projects do for the same situation

Jon

On Mon, Oct 8, 2018 at 3:24 PM Justin Leet  wrote:

> https://github.com/apache/metron/pull/1188 scripts the creation of RC's.
> It does this split, and allows for independent releases. The main catch is
> a slight hangup with the KEYS file.
>
> The metron-bro-kafka-plugin repo doesn't contain a KEYS file, it currently
> has just been aligned with the main repo's KEYS file.The KEYS file doesn't
> need to live in each submodule's directory, just in the main folder (
> https://dist.apache.org/repos/dist/release/metron/).  This means it's not
> a
> problem for the plugin repo to be missing it.
>
> The main catch is that right now, we only publish the KEYS file when we
> release the main repo. In the (admittedly rare) situation where we have a
> new release manager who has to update the KEYS file, we'd currently need to
> do a metron release before we could do a plugin release.
>
> Should we split out and document a KEYS update from the main repo?
> Essentially, just run through the normal PR process and then have a post
> merge step of kicking up the updated KEYS file.  Then Release Process
> <https://cwiki.apache.org/confluence/display/METRON/Release+Process> would
> just get updated to reflect this process.
>
>
> On Fri, Sep 7, 2018 at 3:15 PM Casey Stella  wrote:
>
> > +1 to defer for this release and complete separation.  Good fences make
> > good submodules. ;)
> >
> > On Fri, Sep 7, 2018 at 2:33 PM zeo...@gmail.com 
> wrote:
> >
> > > +1 to defer for this release and +1 to Justin's suggested release/dist
> > > directory breakout and complete separation.
> > >
> > > Jon
> > >
> > > On Fri, Sep 7, 2018 at 1:43 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > +1 to deferring for this release and having the separation like NiFi.
> > > Since
> > > > we're bootstrapping from their process, what are they doing? I would
> > > assume
> > > > we'd want some sort of vote for the plugin version change as well.
> > > >
> > > > On Fri, Sep 7, 2018 at 10:15 AM Nick Allen 
> wrote:
> > > >
> > > > > +1 for complete separation as you've described.
> > > > >
> > > > > On Fri, Sep 7, 2018 at 11:31 AM Justin Leet  >
> > > > wrote:
> > > > >
> > > > > > I would like this to be a complete separation.  Complete with
> > > separate
> > > > > RCs,
> > > > > > separate call to vote, etc. There's a bit more overhead, but
> plugin
> > > > > > releases should be rarer and as the release infra gets improved
> and
> > > > > > scripted out more, I don't think it'll end up being much more
> than
> > > > > bundling
> > > > > > it together.
> > > > > >
> > > > > > On Fri, Sep 7, 2018 at 11:27 AM Nick Allen 
> > > wrote:
> > > > > >
> > > > > > > > Other projects, e.g. NiFi <http://apache.org/dist/nifi/>,
> > split
> > > > > apart
> > > > > > > these releases within their dist directories.
> > > > > > >
> > > > > > > I prefer the way Nifi organizes it.  Definitely seems more
> > > logically
> > > > > > > organized.
> > > > > > >
> > > > > > >
> > > > > > > > If we split them apart, we can make the releases
> independently.
> > > > This
> > > > > > > fixes the problem of aligning the versions (simply release the
> > > plugin
> > > > > > > first, update full-dev, release core Metron).
> > > > > > >
> > > > > > > Does this entail a complete separation; including a separate
> > > > > > call-to-vote?
> > > > > > > One vote for core Metron and a separate vote for plugin?
> > > > > > >
> > > > > > >
> > > > > > > > Do we want try to get this separation done after the current
> > > > release
> > > > > > > cycle is over?
> > > > > > >
> > > > > > > +1 Let's wait for the next release to hash this out.
> > > > > > >
> > > > > > &

Re: Bro plugin release process docs?

[zeo...@gmail.com]

So I was poking around on the plugin today and noticed that we have
a apache-metron-bro-plugin-kafka_0.2.0-release and
apache-metron-bro-plugin-kafka_0.2.0-rc1
tag, but no 0.2 (which is what bro-pkg would point to).  Anybody have any
concerns if I push the 0.2 tag as discussed above?  Then we could update
the bro package manager, and finally update what the apache/metron full-dev
environment(s) point to (0.2 as opposed to 0.1).  Thanks,

Jon

On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com  wrote:

> I did a bit of poking around and I don't believe we ever formally wrote
> that down.  The last release happened as a combination of actions from
> mattf and myself (mostly mattf).
>
> The plugin has two new commits since the last release (1 bugfix 1 feature)
> - if we want to couple version 0.2 of the plugin with a metron 0.5.0
> release we would need to make a 0.2 tag against HEAD of the plugin repo,
> then increment the version in the ansible playbooks here
> <https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30>
> .
>
> Or, if we just want to make a plugin release without changing
> apache/metron, we could just make a 0.2 tag in the plugin repo, and then
> release it in a more disjointed way.  I know that's not super helpful since
> I don't have documentation for doing an apache release of the plugin other
> than hacking something together based on what's out there for apache/metron.
>
> Reference conversations:
>  -
> https://lists.apache.org/thread.html/2606166bd5e864f1b56db302099c9a8042cdadec8fa2692fef49493f@%3Cdev.metron.apache.org%3E
>  -
> https://lists.apache.org/thread.html/3aecbadbf3353e98c03ca4b680fcd998d0cd2bf5a4319238dd85ae75@%3Cdev.metron.apache.org%3E
>
> Jon
>
> On Fri, May 25, 2018 at 2:00 PM Justin Leet  wrote:
>
>> At the risk of exposing my ignorance, do we have the bro plugin release
>> process documented anywhere?  We have a doc for the main release (
>> https://cwiki.apache.org/confluence/display/METRON/Release+Process), but
>> I
>> haven't noticed one for the bro plugin.
>>
>> For the current RC, it's not included and it wasn't pushed for (it has
>> less
>> changes for obvious reasons).  However, we should be making sure to
>> validate if its necessary to release and having the process documented.
>>
>> Justin
>>
> --
>
> Jon
>
-- 

Jon

Re: Bro plugin release process docs?

[zeo...@gmail.com]

I wonder if we should also update the
https://cwiki.apache.org/confluence/display/METRON/Release+Process instructions
to include tagging for the bro plugin, or if we were going to split out the
release processes?  I'd be happy to update the instructions (Step 8) if
that's the right place for now and I didn't miss a new place for the plugin
release instructions.

Jon

On Wed, Oct 10, 2018 at 4:31 PM zeo...@gmail.com  wrote:

> So I was poking around on the plugin today and noticed that we have
> a apache-metron-bro-plugin-kafka_0.2.0-release and 
> apache-metron-bro-plugin-kafka_0.2.0-rc1
> tag, but no 0.2 (which is what bro-pkg would point to).  Anybody have any
> concerns if I push the 0.2 tag as discussed above?  Then we could update
> the bro package manager, and finally update what the apache/metron full-dev
> environment(s) point to (0.2 as opposed to 0.1).  Thanks,
>
> Jon
>
> On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com  wrote:
>
>> I did a bit of poking around and I don't believe we ever formally wrote
>> that down.  The last release happened as a combination of actions from
>> mattf and myself (mostly mattf).
>>
>> The plugin has two new commits since the last release (1 bugfix 1
>> feature) - if we want to couple version 0.2 of the plugin with a metron
>> 0.5.0 release we would need to make a 0.2 tag against HEAD of the plugin
>> repo, then increment the version in the ansible playbooks here
>> <https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30>
>> .
>>
>> Or, if we just want to make a plugin release without changing
>> apache/metron, we could just make a 0.2 tag in the plugin repo, and then
>> release it in a more disjointed way.  I know that's not super helpful since
>> I don't have documentation for doing an apache release of the plugin other
>> than hacking something together based on what's out there for apache/metron.
>>
>> Reference conversations:
>>  -
>> https://lists.apache.org/thread.html/2606166bd5e864f1b56db302099c9a8042cdadec8fa2692fef49493f@%3Cdev.metron.apache.org%3E
>>  -
>> https://lists.apache.org/thread.html/3aecbadbf3353e98c03ca4b680fcd998d0cd2bf5a4319238dd85ae75@%3Cdev.metron.apache.org%3E
>>
>> Jon
>>
>> On Fri, May 25, 2018 at 2:00 PM Justin Leet 
>> wrote:
>>
>>> At the risk of exposing my ignorance, do we have the bro plugin release
>>> process documented anywhere?  We have a doc for the main release (
>>> https://cwiki.apache.org/confluence/display/METRON/Release+Process),
>>> but I
>>> haven't noticed one for the bro plugin.
>>>
>>> For the current RC, it's not included and it wasn't pushed for (it has
>>> less
>>> changes for obvious reasons).  However, we should be making sure to
>>> validate if its necessary to release and having the process documented.
>>>
>>> Justin
>>>
>> --
>>
>> Jon
>>
> --
>
> Jon
>
-- 

Jon

Re: Bro plugin release process docs?

[zeo...@gmail.com]

Yeah you're right when I looked closer to make the change it was step 10.
I pushed a manual 0.2 tag to metron-bro-plugin-kafka and did a quick update
to the cwiki, but it could use some additional love, especially if we want
to split out the release processes generally.

Jon

On Wed, Oct 10, 2018 at 5:32 PM Justin Leet  wrote:

> Yeah, we need to update the release process instructions. It should
> actually be step 10 shouldn't it?  We want to keep the rc tag as-is because
> it may get rejected/cancelled, and update the final tag as the release. I
> already need to overhaul that page to reflect the addition of the RC
> creation script, so I can also update it.
>
> I didn't remember there was a specific reason for the git tagging, so I
> just followed the convention we used for the main repo, which is why we
> have this problem.
>
> The release candidate script also needs to be updated (because it needs to
> know the proper prior tag to pull). It's not hard to do, but we'll need to
> open a Jira and update that portion of the script. It's pretty easy to do,
> it's just the portion here
> <
> https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245
> >
> .
>
> On Wed, Oct 10, 2018 at 5:09 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > +1 to all of that from me, Jon. Thanks for taking care of this.
> >
> > On Wed, Oct 10, 2018 at 2:34 PM zeo...@gmail.com 
> wrote:
> >
> > > I wonder if we should also update the
> > > https://cwiki.apache.org/confluence/display/METRON/Release+Process
> > > instructions
> > > to include tagging for the bro plugin, or if we were going to split out
> > the
> > > release processes?  I'd be happy to update the instructions (Step 8) if
> > > that's the right place for now and I didn't miss a new place for the
> > plugin
> > > release instructions.
> > >
> > > Jon
> > >
> > > On Wed, Oct 10, 2018 at 4:31 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > So I was poking around on the plugin today and noticed that we have
> > > > a apache-metron-bro-plugin-kafka_0.2.0-release and
> > > apache-metron-bro-plugin-kafka_0.2.0-rc1
> > > > tag, but no 0.2 (which is what bro-pkg would point to).  Anybody have
> > any
> > > > concerns if I push the 0.2 tag as discussed above?  Then we could
> > update
> > > > the bro package manager, and finally update what the apache/metron
> > > full-dev
> > > > environment(s) point to (0.2 as opposed to 0.1).  Thanks,
> > > >
> > > > Jon
> > > >
> > > > On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com 
> > > wrote:
> > > >
> > > >> I did a bit of poking around and I don't believe we ever formally
> > wrote
> > > >> that down.  The last release happened as a combination of actions
> from
> > > >> mattf and myself (mostly mattf).
> > > >>
> > > >> The plugin has two new commits since the last release (1 bugfix 1
> > > >> feature) - if we want to couple version 0.2 of the plugin with a
> > metron
> > > >> 0.5.0 release we would need to make a 0.2 tag against HEAD of the
> > plugin
> > > >> repo, then increment the version in the ansible playbooks here
> > > >> <
> > >
> >
> https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30
> > > >
> > > >> .
> > > >>
> > > >> Or, if we just want to make a plugin release without changing
> > > >> apache/metron, we could just make a 0.2 tag in the plugin repo, and
> > then
> > > >> release it in a more disjointed way.  I know that's not super
> helpful
> > > since
> > > >> I don't have documentation for doing an apache release of the plugin
> > > other
> > > >> than hacking something together based on what's out there for
> > > apache/metron.
> > > >>
> > > >> Reference conversations:
> > > >>  -
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/2606166bd5e864f1b56db302099c9a8042cdadec8fa2692fef49493f@%3Cdev.metron.apache.org%3E
> > > >>  -
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/3aecbadbf3353e98c03ca4b680fcd998d0cd2bf5a4319238dd85ae75@%3Cdev.metron.apache.org%3E
> > > >>
> > > >> Jon
> > > >>
> > > >> On Fri, May 25, 2018 at 2:00 PM Justin Leet 
> > > >> wrote:
> > > >>
> > > >>> At the risk of exposing my ignorance, do we have the bro plugin
> > release
> > > >>> process documented anywhere?  We have a doc for the main release (
> > > >>> https://cwiki.apache.org/confluence/display/METRON/Release+Process
> ),
> > > >>> but I
> > > >>> haven't noticed one for the bro plugin.
> > > >>>
> > > >>> For the current RC, it's not included and it wasn't pushed for (it
> > has
> > > >>> less
> > > >>> changes for obvious reasons).  However, we should be making sure to
> > > >>> validate if its necessary to release and having the process
> > documented.
> > > >>>
> > > >>> Justin
> > > >>>
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > > --
> > > >
> > > > Jon
> > > >
> > > --
> > >
> > > Jon
> > >
> >
>
-- 

Jon

Re: Bro plugin release process docs?

[zeo...@gmail.com]

Is there a reason why the prefix for apache/metron ends with a -, whereas
the plugin ends with a _ separator?  I would like to see it more uniform
using the _

Jon

On Thu, Oct 11, 2018 at 9:04 AM Justin Leet  wrote:

> That first half of that page will be rewritten in the next couple days to
> reflect the script to prepare release candidate; I didn't want to change
> that page until the PR actually went in.
>
> The release process, barring the KEYS issue (and a patch to the script to
> handle the bro plugin tag), is pretty much set to be managed separately at
> this point.  The docs just need an overhaul, so someone who's not me knows
> what to do.
>
> On Wed, Oct 10, 2018 at 7:01 PM zeo...@gmail.com  wrote:
>
> > Yeah you're right when I looked closer to make the change it was step 10.
> > I pushed a manual 0.2 tag to metron-bro-plugin-kafka and did a quick
> update
> > to the cwiki, but it could use some additional love, especially if we
> want
> > to split out the release processes generally.
> >
> > Jon
> >
> > On Wed, Oct 10, 2018 at 5:32 PM Justin Leet 
> wrote:
> >
> > > Yeah, we need to update the release process instructions. It should
> > > actually be step 10 shouldn't it?  We want to keep the rc tag as-is
> > because
> > > it may get rejected/cancelled, and update the final tag as the
> release. I
> > > already need to overhaul that page to reflect the addition of the RC
> > > creation script, so I can also update it.
> > >
> > > I didn't remember there was a specific reason for the git tagging, so I
> > > just followed the convention we used for the main repo, which is why we
> > > have this problem.
> > >
> > > The release candidate script also needs to be updated (because it needs
> > to
> > > know the proper prior tag to pull). It's not hard to do, but we'll need
> > to
> > > open a Jira and update that portion of the script. It's pretty easy to
> > do,
> > > it's just the portion here
> > > <
> > >
> >
> https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245
> > > >
> > > .
> > >
> > > On Wed, Oct 10, 2018 at 5:09 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > +1 to all of that from me, Jon. Thanks for taking care of this.
> > > >
> > > > On Wed, Oct 10, 2018 at 2:34 PM zeo...@gmail.com 
> > > wrote:
> > > >
> > > > > I wonder if we should also update the
> > > > > https://cwiki.apache.org/confluence/display/METRON/Release+Process
> > > > > instructions
> > > > > to include tagging for the bro plugin, or if we were going to split
> > out
> > > > the
> > > > > release processes?  I'd be happy to update the instructions (Step
> 8)
> > if
> > > > > that's the right place for now and I didn't miss a new place for
> the
> > > > plugin
> > > > > release instructions.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Wed, Oct 10, 2018 at 4:31 PM zeo...@gmail.com  >
> > > > wrote:
> > > > >
> > > > > > So I was poking around on the plugin today and noticed that we
> have
> > > > > > a apache-metron-bro-plugin-kafka_0.2.0-release and
> > > > > apache-metron-bro-plugin-kafka_0.2.0-rc1
> > > > > > tag, but no 0.2 (which is what bro-pkg would point to).  Anybody
> > have
> > > > any
> > > > > > concerns if I push the 0.2 tag as discussed above?  Then we could
> > > > update
> > > > > > the bro package manager, and finally update what the
> apache/metron
> > > > > full-dev
> > > > > > environment(s) point to (0.2 as opposed to 0.1).  Thanks,
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com <
> zeo...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> I did a bit of poking around and I don't believe we ever
> formally
> > > > wrote
> > > > > >> that down.  The last release happened as a combination of
> actions
> > > from
> > > > > >> mattf and myself (mo

Re: Bro plugin release process docs?

[zeo...@gmail.com]

Okay, I'll PR something since I'm looking at it

Jon

On Thu, Oct 11, 2018 at 9:45 AM Justin Leet  wrote:

> If I recall correctly, they were already different, presumably from the
> more manual process, and I wanted keep things close to what they already
> were before trying to clean that sort of thing up. We can definitely unify
> them, and if we need to code in any exceptions to deal with pre-unified
> versions, we can.  It just wasn't high on my priority list.
>
> In the script, the change to the separator itself is easy, but I'd have to
> double check if there's anything else that needs to happen to make sure
> tags and such line up.
>
> On Thu, Oct 11, 2018 at 9:18 AM zeo...@gmail.com  wrote:
>
> > Is there a reason why the prefix for apache/metron ends with a -, whereas
> > the plugin ends with a _ separator?  I would like to see it more uniform
> > using the _
> >
> > Jon
> >
> > On Thu, Oct 11, 2018 at 9:04 AM Justin Leet 
> wrote:
> >
> > > That first half of that page will be rewritten in the next couple days
> to
> > > reflect the script to prepare release candidate; I didn't want to
> change
> > > that page until the PR actually went in.
> > >
> > > The release process, barring the KEYS issue (and a patch to the script
> to
> > > handle the bro plugin tag), is pretty much set to be managed separately
> > at
> > > this point.  The docs just need an overhaul, so someone who's not me
> > knows
> > > what to do.
> > >
> > > On Wed, Oct 10, 2018 at 7:01 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > Yeah you're right when I looked closer to make the change it was step
> > 10.
> > > > I pushed a manual 0.2 tag to metron-bro-plugin-kafka and did a quick
> > > update
> > > > to the cwiki, but it could use some additional love, especially if we
> > > want
> > > > to split out the release processes generally.
> > > >
> > > > Jon
> > > >
> > > > On Wed, Oct 10, 2018 at 5:32 PM Justin Leet 
> > > wrote:
> > > >
> > > > > Yeah, we need to update the release process instructions. It should
> > > > > actually be step 10 shouldn't it?  We want to keep the rc tag as-is
> > > > because
> > > > > it may get rejected/cancelled, and update the final tag as the
> > > release. I
> > > > > already need to overhaul that page to reflect the addition of the
> RC
> > > > > creation script, so I can also update it.
> > > > >
> > > > > I didn't remember there was a specific reason for the git tagging,
> > so I
> > > > > just followed the convention we used for the main repo, which is
> why
> > we
> > > > > have this problem.
> > > > >
> > > > > The release candidate script also needs to be updated (because it
> > needs
> > > > to
> > > > > know the proper prior tag to pull). It's not hard to do, but we'll
> > need
> > > > to
> > > > > open a Jira and update that portion of the script. It's pretty easy
> > to
> > > > do,
> > > > > it's just the portion here
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245
> > > > > >
> > > > > .
> > > > >
> > > > > On Wed, Oct 10, 2018 at 5:09 PM Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > > > +1 to all of that from me, Jon. Thanks for taking care of this.
> > > > > >
> > > > > > On Wed, Oct 10, 2018 at 2:34 PM zeo...@gmail.com <
> zeo...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > I wonder if we should also update the
> > > > > > >
> > https://cwiki.apache.org/confluence/display/METRON/Release+Process
> > > > > > > instructions
> > > > > > > to include tagging for the bro plugin, or if we were going to
> > split
> > > > out
> > > > > > the
> > > > > > > release processes?  I'd be happy to update the instructions
> (Step
> > > 8)
> > > > if
> > > > > > > that's 

Re: Bro plugin release process docs?

[zeo...@gmail.com]

I've updated the cwiki
<https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=66854770&selectedPageVersions=43&selectedPageVersions=40>
and opened a PR <https://github.com/apache/metron/pull/1236> for this, open
to feedback on any of the changes I made/suggested.  Since I assume we
wouldn't want to make any changes to releases retroactively, I added a note
to the cwiki to note the history (see "Historical Note" under section 5).
Thanks,

Jon

On Thu, Oct 11, 2018 at 11:05 AM zeo...@gmail.com  wrote:

> Okay, I'll PR something since I'm looking at it
>
> Jon
>
> On Thu, Oct 11, 2018 at 9:45 AM Justin Leet  wrote:
>
>> If I recall correctly, they were already different, presumably from the
>> more manual process, and I wanted keep things close to what they already
>> were before trying to clean that sort of thing up. We can definitely unify
>> them, and if we need to code in any exceptions to deal with pre-unified
>> versions, we can.  It just wasn't high on my priority list.
>>
>> In the script, the change to the separator itself is easy, but I'd have to
>> double check if there's anything else that needs to happen to make sure
>> tags and such line up.
>>
>> On Thu, Oct 11, 2018 at 9:18 AM zeo...@gmail.com 
>> wrote:
>>
>> > Is there a reason why the prefix for apache/metron ends with a -,
>> whereas
>> > the plugin ends with a _ separator?  I would like to see it more uniform
>> > using the _
>> >
>> > Jon
>> >
>> > On Thu, Oct 11, 2018 at 9:04 AM Justin Leet 
>> wrote:
>> >
>> > > That first half of that page will be rewritten in the next couple
>> days to
>> > > reflect the script to prepare release candidate; I didn't want to
>> change
>> > > that page until the PR actually went in.
>> > >
>> > > The release process, barring the KEYS issue (and a patch to the
>> script to
>> > > handle the bro plugin tag), is pretty much set to be managed
>> separately
>> > at
>> > > this point.  The docs just need an overhaul, so someone who's not me
>> > knows
>> > > what to do.
>> > >
>> > > On Wed, Oct 10, 2018 at 7:01 PM zeo...@gmail.com 
>> > wrote:
>> > >
>> > > > Yeah you're right when I looked closer to make the change it was
>> step
>> > 10.
>> > > > I pushed a manual 0.2 tag to metron-bro-plugin-kafka and did a quick
>> > > update
>> > > > to the cwiki, but it could use some additional love, especially if
>> we
>> > > want
>> > > > to split out the release processes generally.
>> > > >
>> > > > Jon
>> > > >
>> > > > On Wed, Oct 10, 2018 at 5:32 PM Justin Leet 
>> > > wrote:
>> > > >
>> > > > > Yeah, we need to update the release process instructions. It
>> should
>> > > > > actually be step 10 shouldn't it?  We want to keep the rc tag
>> as-is
>> > > > because
>> > > > > it may get rejected/cancelled, and update the final tag as the
>> > > release. I
>> > > > > already need to overhaul that page to reflect the addition of the
>> RC
>> > > > > creation script, so I can also update it.
>> > > > >
>> > > > > I didn't remember there was a specific reason for the git tagging,
>> > so I
>> > > > > just followed the convention we used for the main repo, which is
>> why
>> > we
>> > > > > have this problem.
>> > > > >
>> > > > > The release candidate script also needs to be updated (because it
>> > needs
>> > > > to
>> > > > > know the proper prior tag to pull). It's not hard to do, but we'll
>> > need
>> > > > to
>> > > > > open a Jira and update that portion of the script. It's pretty
>> easy
>> > to
>> > > > do,
>> > > > > it's just the portion here
>> > > > > <
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245
>> > > > > >
>> > > > > .
>> > > > >
>> > > > > On Wed, Oct 10, 2018 at 5:09 PM Michael Mikl

Bro plugin unit tests failing

[zeo...@gmail.com]

So it seems that the last commit before the 0.2 release of
metron-bro-plugin-kafka broke the one basic unit test that we had.  Since
metron 0.6.0 pins to 0.1

this wouldn't cause an obvious issue if you spun up the sensors in
full-dev, and even if we were to point metron to 0.2 of the plugin we
wouldn't have necessarily seen the issue because we do a --force

to remove interaction.

After we get a chance to review/merge a recent PR
, we should have
a few more 'actual' unit tests, and I would like to get travis setup for
the repo  to actually
check those (PR incoming) so it's less likely this sort of thing would
happen again.  However, it doesn't appear I have the right permissions to
be able to get travis moving on my own.  Does anybody else have the right
access, or is this an asf infrastructure ticket?

I'd also like us to consider doing a 0.3 release for the plugin so we
aren't distributing something that has broken tests.  Thoughts on that?

Jon
-- 

Jon

Re: Bro plugin unit tests failing

[zeo...@gmail.com]

All good suggestions.  I don't think I'll personally have time in the short
term to draft a release discussion email, but I do think it's a good idea.

I can take a stab at the README link to contributing and a PR template for
the plugin repo.

Jon

On Mon, Oct 15, 2018 at 8:09 AM Justin Leet  wrote:

> Should we just put out a discuss for releasing in general, with this
> discussion noted?  The outcome of the release cadence discussion was to put
> out a thread around each board report. I believe that's a few days from
> now, so it's good timing. As an aside, part of the post release activities
> for the plugin should probably be upping the plugin version in the main
> repo to use the new version.
>
> While we're on the topic, is there anything else we should do to improve
> the plugin repo?  Off the top of my head, setting up a version of the PR
> template would be helpful.  Maybe adding a section to the README.md linking
> to the CONTRIBUTING.md of the main repo?
>
> On Sun, Oct 14, 2018 at 11:14 AM Otto Fowler 
> wrote:
>
>> It is INFRA, see INFRA-17091 for example.
>>
>>
>> On October 12, 2018 at 20:47:24, zeo...@gmail.com (zeo...@gmail.com)
>> wrote:
>>
>> So it seems that the last commit before the 0.2 release of
>> metron-bro-plugin-kafka broke the one basic unit test that we had. Since
>> metron 0.6.0 pins to 0.1
>> <
>>
>> https://github.com/apache/metron/blob/Metron_0.6.0/metron-deployment/ansible/roles/bro/vars/main.yml#L30
>> >
>>
>> this wouldn't cause an obvious issue if you spun up the sensors in
>> full-dev, and even if we were to point metron to 0.2 of the plugin we
>> wouldn't have necessarily seen the issue because we do a --force
>> <
>>
>> https://github.com/apache/metron/blob/Metron_0.6.0/metron-deployment/ansible/roles/bro/tasks/metron-bro-plugin-kafka.yml#L35
>> >
>>
>> to remove interaction.
>>
>> After we get a chance to review/merge a recent PR
>> <https://github.com/apache/metron-bro-plugin-kafka/pull/2>, we should
>> have
>> a few more 'actual' unit tests, and I would like to get travis setup for
>> the repo <https://travis-ci.org/apache/metron-bro-plugin-kafka> to
>> actually
>> check those (PR incoming) so it's less likely this sort of thing would
>> happen again. However, it doesn't appear I have the right permissions to
>> be able to get travis moving on my own. Does anybody else have the right
>> access, or is this an asf infrastructure ticket?
>>
>> I'd also like us to consider doing a 0.3 release for the plugin so we
>> aren't distributing something that has broken tests. Thoughts on that?
>>
>> Jon
>> --
>>
>> Jon
>>
> --

Jon

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

I agree with a metron-bro-plugin-kafka release of 0.3.0 (0.3 in bro-pkg),
assuming we can get apache/metron-bro-plugin-kafka#2 in.  I'm working on
adding travis to the metron-bro-plugin-kafka repo, but I'm not sure when I
will have enough time to finish my work there and wouldn't want to hold up
a release just for that.

Jon

On Tue, Oct 16, 2018 at 12:26 PM Justin Leet  wrote:

> Hi all,
>
> As you might recall from a prior discussion about release cadence, we were
> interested in initiating release threads near our board reports to see if
> we wanted to do releases or not. Additionally, the work is done to do two
> separate releases, so our options are releasing both, a single one, or
> neither.
>
> Having said that, a metron-bro-kafka-plugin 0.3.0 release came up on this
> thread
> <
> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> >.
> In particular, the prospect of a release came up in the context of having a
> version with better (and working) testing.
>
> Version Number
> If we choose to do a core Metron release, I propose 0.6.1. For
> metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the versioning for
> the plugin is a bit different in that we need x.y for bro-pkg, instead of
> x.y.z, so we wouldn't release 0.2.1.
>
> I would personally be in favor of doing a plugin release, but I'm less
> inclined to do a core Metron release.
>
> For Metron, I believe the main feature is the batch profiler work, but
> there's a fair amount of fixes and improvements.
>
> For the plugin, I believe the main improvements would be around the fixing
> our testing and more maintenance type work.
>
> JIRA status
> *metron*
> There are 24 opens PRs at https://github.com/apache/metron/pulls.  If we
> do
> decide to release, we should work on getting anything meriting inclusion
> closed out.
>
> There have been 49 commits since the 0.6.0 release (listed at the end of
> the message). This assumes a release from master.
>
> *metron-bro-kafka-plugin*
> There are 6 open PRs at
> https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we do decide
> to release, we should get anything we need closed out.
>
> There have been 2 commits since the 0.2.0 release (listed at the end of the
> message).  This assumes a release from master.
>
> *Metron changelog*
> METRON-1801 Allow Customization of Elasticsearch Document ID
> (nickwallen) closes apache/metron#1218
> METRON-1799 Remove outdated bylaws from site. (justinleet) closes
> apache/metron#1216
> METRON-1769 Script creation of a release candidate (justinleet) closes
> apache/metron#1188
> METRON-1761 Allow a grok statement to be applied to each line in a
> file. (ottobackwards) closes apache/metron#1184
> METRON-1813 Stellar REPL Not Initialized with Client JAAS (nickwallen)
> closes apache/metron#1232
> METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc)
> closes apache/metron#1230
> METRON-1811 Alert Search Fails When Sorting by Alert Status (merrimanr)
> closes apache/metron#1231
> METRON-1809 Support Column Oriented Input with Batch Profiler
> (nickwallen) closes apache/metron#1229
> METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via mmiklavc)
> closes apache/metron#1224
> METRON-1792 Simplify Profile Definitions in Integration Tests
> (nickwallen) closes apache/metron#1211
> METRON-1807 Auto populate the recommended values to some of the metron
> config parameters  (MohanDV via merrimanr) closes apache/metron#1227
> METRON-1808 Add Ansible created pyc to gitignore (justinleet) closes
> apache/metron#1228
> METRON-1695 Expose pcap properties through Ambari (anandsubbu) closes
> apache/metron#1207
> METRON-1771 Update REST endpoints to support eventually consistent UI
> updates (merrimanr) closes apache/metron#1190
> METRON-1791 Add GUID to Messages Produced by Profiler (nickwallen)
> closes apache/metron#1210
> METRON-1804 Update version to 0.6.1 (justinleet) closes
> apache/metron#1220
> METRON-1798 Add mpack support for parser aggregation (anandsubbu)
> closes apache/metron#1215
> METRON-1750 Create Parser for Syslog RFC 5424 Messages (ottobackwards)
> closes apache/metron#1175
> METRON-1794 Include User Details When Escalating Alerts (nickwallen)
> closes apache/metron#1212
> METRON-1782 Add Kafka Partition and Offset to Profiler Debug Logs
> (nickwallen) closes apache/metron#1202
> METRON-1758 Add support for Ansible 2.6 in dev (JonZeolla via
> jonzeolla) closes apache/metron#1179
> METRON-1784: Re-allow remote ssh and scp in Centos full dev (mmiklavc
> via mmiklavc) closes apache/metron#1204
> METRON-1787 Input Time Constraints for Batch Profiler (nickwallen)
> closes apache/metron#1209
> METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
> (nickwallen) closes apache/metron#1185
> METRON-1786 Pcap Topology Status Incorrect (MohanDV via nickwall

Re: Invite to Slack Channel

[zeo...@gmail.com]

Invite sent

On Mon, Oct 22, 2018 at 9:26 AM Muhammed Irshad  wrote:

> Some one get me also the slack channel link ?
> Thanks,
> Muhammed Irshad
> Q*Burst*
> www.qburst.com
>
>
> On Wed, Oct 17, 2018 at 7:33 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Sent
> >
> > On Wed, Oct 17, 2018 at 7:23 AM Tibor Meller 
> > wrote:
> >
> > > Hi Guys,
> > > Can you add me to the apache metron slack chanel?
> > >
> > > Thanks,
> > >
> > > On Thu, Oct 4, 2018 at 1:14 PM Otto Fowler 
> > > wrote:
> > >
> > > > Done
> > > >
> > > >
> > > > On October 4, 2018 at 05:35:06, Tamás Fodor (ftamas.m...@gmail.com)
> > > wrote:
> > > >
> > > > Hello,
> > > >
> > > > Michael, can you add me as well?
> > > >
> > > > Thank you in advance!
> > > >
> > > > Tamas
> > > >
> > > > On Wed, Oct 3, 2018 at 4:27 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Sent
> > > > >
> > > > > On Wed, Oct 3, 2018 at 8:17 AM Shane Ardell <
> > shane.m.ard...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > Is it possible for someone to send me an invite to the Metron
> Slack
> > > > > > channel?
> > > > > >
> > > > > > Regards,
> > > > > > Shane
> > > > > >
> > > > >
> > > >
> > >
> >
>
-- 

Jon Zeolla

Re: [DISCUSS] Day 1 User Experience - Getting Metron Running

[zeo...@gmail.com]

Yeah I would +1 katakoda.  I also think that it would help to start
distributing RPMs, DEBs, and the mpacks with the releases, as well as
consider a service like opensuse's build service for nightlies, etc.

Jon

On Fri, Oct 26, 2018 at 6:25 AM Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> Great idea! This will be a HUGE improvement in the user experience for
> first timers to Metron. Katakoda seems very interesting - simple and
> straight-forward. I loved the way you can provide instructions, commands
> (that can be directly clicked!), links, explanation and so on.
>
> Regards,
> Anand
>
> On 10/25/18, 7:49 PM, "Nick Allen"  wrote:
>
> We all know spinning up the development environment is a pain.
> Unfortunately, it is the only way for a new user to get a feel for
> Metron.
> We need a better way to introduce new users to Metron.
>
> I am hoping we can brainstorm ways to improve that experience.  Here
> are a
> few thoughts that might help start a discussion.
>
> (1) Create a *KataKoda* [1] based demo.  I ran across this after
> finding
> Apache Ozone's demo [2], which I think is great.
>
>
>- A user does not need to download or install anything.  It is a
>   completely hosted offering.
>   - Provides a step-by-step demo experience that could guide users
>   through creating an enrichment, defining a profile, managing
> alerts.
>   - Would require a Metron on Docker solution.
>
> (2) Create a *Vagrant Cloud* [3] hosted image of "Full Dev" with
> everything
> installed and ready to rock.  A user would just need to install
> Vagrant and
> run:
>
> vagrant init metron/0.6.0
>
> vagrant up
>
>
>- Reduces the number of dependencies needed to get Metron
> up-and-running.
>   - Significantly increases the success rate of new users getting
>   Metron running.
>   - Still results in "Full Dev" Metron which requires too many
>   resources for the average computer.
>
> Are these good options? What other approaches could we take?  Hopefully
> some JIRAs might fall out of this discussion.
>
> - Nick
>
>
> --
> [1] https://www.katacoda.com
> [2] https://www.katacoda.com/elek/scenarios/ozone101
> [3] https://app.vagrantup.com/boxes/search
>
>
> --

Jon Zeolla

Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

[zeo...@gmail.com]

+1 totally agree.

Jon

On Fri, Nov 2, 2018, 1:31 AM Anand Subramanian 
wrote:

> Piling on my +1 (non-binding) as well.
>
> On 11/2/18, 4:41 AM, "Ryan Merriman"  wrote:
>
> +1
>
> On Thu, Nov 1, 2018 at 5:38 PM Casey Stella 
> wrote:
>
> > +1
> > On Thu, Nov 1, 2018 at 18:34 Nick Allen  wrote:
> >
> > > +1
> > >
> > > On Thu, Nov 1, 2018, 6:27 PM Justin Leet 
> wrote:
> > >
> > > > +1, I haven't seen any case where the split-join topology isn't
> made
> > > > obsolete by the unified topology.
> > > >
> > > > On Thu, Nov 1, 2018 at 6:17 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Fellow Metronians,
> > > > >
> > > > > We've had the unified enrichment topology around for a number
> of
> > months
> > > > > now, it has proved itself stable, and there is yet to be a
> time that
> > I
> > > > have
> > > > > seen the split-join topology outperform the unified one. Here
> are
> > some
> > > > > simple reasons to deprecate the split-join topology.
> > > > >
> > > > >1. Unified topology performs better.
> > > > >2. The configuration, especially for performance tuning is
> much,
> > > much
> > > > >simpler in the unified model.
> > > > >3. The footprint within the cluster is smaller.
> > > > >4. One of the first activities for any install is that we
> spend
> > time
> > > > >instructing users to switch to the unified topology.
> > > > >5. One less moving part to maintain.
> > > > >
> > > > > I'd like to recommend that we deprecate the split-join
> topology and
> > > make
> > > > > the unified enrichment topology the new default.
> > > > >
> > > > > Best,
> > > > > Mike
> > > > >
> > > >
> > >
> >
>
>
> --

Jon Zeolla

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

So, about this release, anybody have time to review
apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13?

Jon

On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> And I do think we will be ready to roll another Metron release in the near
> future as well.
>
> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet  wrote:
>
> > I tend to agree with Mike. I think we could do a release of the main
> > project and get benefit, but I think skipping this cycle (especially
> since
> > we had a release last month) let's us get things like the ES client
> > migration in and settled and just generally make recommending a newer
> > version easier.
> >
> > The main reason for doing a release for the Bro plugin is to address
> > shortcomings that we discovered and are addressing.
> >
> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Migrating the ES client, refactoring parser bolt. There are some
> > interface
> > > changes in flight right now that I think would be beneficial to see in
> > the
> > > next release.
> > >
> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen  wrote:
> > >
> > > > I am in favor of a release for both.
> > > >
> > > > There are a lot of really useful bug fixes, management of pcap
> through
> > > > Ambari, more flexibility for configuring JAAS in Ambari, increased
> > > > Elasticsearch performance, the Syslog parser, and the Batch Profiler,
> > > among
> > > > others. I would be happy with calling it a 0.6.1 point release.
> > > >
> > > > Mike - What is outstanding that you would like to see in the release?
> > > >
> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin release.
> It
> > > > seems
> > > > > like it's ready to go, and I think there are a few more things I'd
> > like
> > > > to
> > > > > see get into our next Metron release so I'm good with holding off
> > > there.
> > > > >
> > > > > Mike
> > > > >
> > > > > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet <
> justinjl...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > As you might recall from a prior discussion about release
> cadence,
> > we
> > > > > were
> > > > > > interested in initiating release threads near our board reports
> to
> > > see
> > > > if
> > > > > > we wanted to do releases or not. Additionally, the work is done
> to
> > do
> > > > two
> > > > > > separate releases, so our options are releasing both, a single
> one,
> > > or
> > > > > > neither.
> > > > > >
> > > > > > Having said that, a metron-bro-kafka-plugin 0.3.0 release came up
> > on
> > > > this
> > > > > > thread
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> > > > > > >.
> > > > > > In particular, the prospect of a release came up in the context
> of
> > > > > having a
> > > > > > version with better (and working) testing.
> > > > > >
> > > > > > Version Number
> > > > > > If we choose to do a core Metron release, I propose 0.6.1. For
> > > > > > metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the
> > > versioning
> > > > > for
> > > > > > the plugin is a bit different in that we need x.y for bro-pkg,
> > > instead
> > > > of
> > > > > > x.y.z, so we wouldn't release 0.2.1.
> > > > > >
> > > > > > I would personally be in favor of doing a plugin release, but I'm
> > > less
> > > > > > inclined to do a core Metron release.
> > > > > >
> > > > > > For Metron, I believe the main feature is the batch profiler
> work,
> > > but
> > > > > > there's a fair amount of fixes and improvements.
> > > > > >
> > > > > > For the plugin, I believe the main improvements would be around
> the
> > > > > fixing
> > > > > > our testing and more maintenance type work.
> > > > > >
> > > > > > JIRA status
> > > > > > *metron*
> > > > > > There are 24 opens PRs at https://github.com/apache/metron/pulls
> .
> > > If
> > > > we
> > > > > > do
> > > > > > decide to release, we should work on getting anything meriting
> > > > inclusion
> > > > > > closed out.
> > > > > >
> > > > > > There have been 49 commits since the 0.6.0 release (listed at the
> > end
> > > > of
> > > > > > the message). This assumes a release from master.
> > > > > >
> > > > > > *metron-bro-kafka-plugin*
> > > > > > There are 6 open PRs at
> > > > > > https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we
> do
> > > > > decide
> > > > > > to release, we should get anything we need closed out.
> > > > > >
> > > > > > There have been 2 commits since the 0.2.0 release (listed at the
> > end
> > > of
> > > > > the
> > > > > > message).  This assumes a release from master.
> > > > > >
> > > > > > *Metron changelog*
> > > > > > METRON-1801 Allow Customization of Elasticsearch Document ID
> > > > > > (nickwallen

Re: [DISCUSS] Knox SSO feature branch review and features

[zeo...@gmail.com]

Phew, that was quite the thread to catch up on.

I agree that this should be optional/pluggable to start, and I'm interested
to hear the issues as they relate to upgrading an existing cluster (given
the suggested approach) and exposing both legacy and knox URLs at the same
time.

Jon

On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic 
wrote:

> A couple more things, and I think this goes without saying - whatever we do
> with Knox should NOT
>
>1. Require unit and integration tests to use Knox
>2. Break fulldev
>
> Also, I don't know that I saw you mention this, but I'm unsure how we
> should leverage Knox as a core piece of the platform. i.e. should we make
> this required or optional? I'm open to hearing opinions on this, but I'm
> inclined to keep this a pluggable option.
>
> Mike
>
>
> On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Thanks for the update Ryan. Per my earlier comments, I thought it might
> be
> > the case that we could dramatically simplify this by leveraging Knox's
> > proxy capabilities, and per your research that appears to be the case.
> This
> > is a dramatic simplification and improvement of this feature imo, +1. I'm
> > also +1 on a couple distinct steps that you've laid out: fix the UI
> issues
> > in master, then add Knox for SSO. That should help mitigate issues with
> > merge conflicts with ongoing development.
> >
> > > I think it will be a challenge exposing the UIs through both the Knox
> > url and legacy urls at the same time.
> > I'm not sure I understand the issue here. Are you referring to this
> > comment? "Added a ng build option to build the UI with base href set to
> > Knox base path." Isn't it just a matter of URL rewriting/forwarding? I
> > thought we'd be exposing the URL's directly in one context, and through
> > Knox in the other. Either way, it seems like we should be able to
> provide a
> > dynamic base path through configuration in our web applications. I'd
> expect
> > to modify that property based on whether Knox is configured or not.
> >
> > > I'm also not clear on how one would use Knox with REST set to legacy
> > JDBC-based authentication. As far as I know Knox does not support JDBC so
> > there would be a mismatch between Knox and REST.
> > I'm OK with not having Knox work with JDBC. That's a feature of Knox and
> > probably not something we care much about.
> >
> > >We could initially make Knox an optional feature that requires setup
> with
> > the help of some documentation (like Kerberos) while keeping the system
> the
> > way it is now by default.
> > Sounds good to me.
> >
> > > I imagine we'll deprecate JDBC-based authentication at some point so
> > that may be a good time to switch.
> > I would like to announce deprecation in our next release and move to
> > remove it in a following release.
> >
> > Thanks for taking this on and great job laying things out.
> >
> > Thanks,
> > Mike
> >
> > On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman 
> wrote:
> >
> >> I have spent some time recently reviewing this discussion and the
> feature
> >> branch that Simon put out.  I think this is an important feature and
> want
> >> to move it forward.  I started another discussion on adding Knox to our
> >> stack but this discussion has a lot of good context so I will continue
> it
> >> here.
> >>
> >> I think the main point of contention was that this feature branch
> included
> >> several different architectural changes and it was unclear if they were
> >> needed and if so, could be done separately.  Fortunately LDAP
> >> authentication has been accepted into master so we can cross it off the
> >> list.  From my understanding of the points people have made, that leaves
> >> Knox related SSO changes and migrating expressjs to a different,
> JVM-based
> >> web server that includes proxying capabilities (Zuul).
> >>
> >> I think everyone agrees that if we can limit the scope to just Knox
> >> related
> >> SSO changes that would be ideal.  I believe I have found a way to do
> that
> >> while working on a small POC this week.  The key to this (Simon alluded
> to
> >> it earlier) is to put both REST and our UIs behind Knox.  I initially
> was
> >> focused on just adding REST as a service in Knox and decided to
> experiment
> >> with also adding our UIs.  After I did this it became clear that this
> >> simplifies things considerably:
> >>
> >>- The REST app and the UIs are now served from the same host so CORS
> >>concerns go away.
> >>- We no longer need to worry about proxying REST requests from the
> UIs
> >>with express or Zuul because Knox handles that for us.  This will
> make
> >> our
> >>express configuration even simpler.  In fact, all we need is a simple
> >> way
> >>to serve static UI assets.
> >>- We no longer need to check for SSO tokens and redirect in the UI
> >>web/app servers (or the REST app for that matter) because Knox
> handles
> >> that
> >>for us.
> >>- The

Re: [DISCUSS] Slack Channel Use

[zeo...@gmail.com]

Spot on Justin, I totally agree.  My only nit is that often it's much
easier troubleshooting in Slack as opposed to the mailing lists, so I'm
game to allow some troubleshooting in Slack as long as the issue and
resolution makes it back to the lists.  Given that slack message history is
being kept (although to what degree I'm not sure), we could fairly easily
link to the start of a discussion in slack in the wrap-up mailing list
email for future reference.

Jon

On Mon, Nov 12, 2018 at 2:08 PM Casey Stella  wrote:

> Piling on, +1 to what Justin said.
>
> On Mon, Nov 12, 2018 at 12:42 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I'm also +1 to Justin's points.
> >
> > On Mon, Nov 12, 2018 at 10:38 AM Nick Allen  wrote:
> >
> > > +1 to all your points Justin.
> > >
> > > On Mon, Nov 12, 2018 at 10:08 AM Justin Leet 
> > > wrote:
> > >
> > > > I wanted to add back onto this thread after putting some more thought
> > > into
> > > > it.
> > > >
> > > > I like Slack for the type of small developer "what's going on here?"
> > type
> > > > discussions.  That's the kind of thing I like being real-time ("Hey,
> > full
> > > > dev is acting weird", "What's the basic layout of this stuff?",
> > "Anybody
> > > > else seen this test failure?", etc.).  I think we've been pretty good
> > > about
> > > > keeping our decision type dev discussions to the list (e.g. this
> exact
> > > > conversation).
> > > >
> > > > We've been doing this more, but I would like to see more of the user
> > and
> > > > troubleshooting move to the list.  I think we've gotten a bit better
> > > about
> > > > it as we've settled into Slack, but having that sort of helpful stuff
> > > > exposed and searchable for users who come in afterwards is a big
> > selling
> > > > point of the lists, imo.
> > > >
> > > > To add onto this, I'd probably like to see
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/METRON/Community+Resources#CommunityResources-ApacheMetronCommunityResources
> > > > (and any other relevant links) updated to emphasize a Slack focus on
> > > > developing Metron itself, and the user lists for configuration,
> > > > troubleshooting, etc.
> > > >
> > > > Essentially, I'm proposing:
> > > > Dev list / Jira / PRs as usual for any actual decisions + concrete
> > > feature
> > > > discussion/review.
> > > > Slack for Metron development "Hey, anyone seen this or have insight
> or
> > a
> > > > starting point?" and "I'm seeing something weird in our tests" type
> > stuff
> > > > User list for usage and troubleshooting questions.  Generally,
> > > discussions
> > > > like this in Slack should be redirected to the user list.
> > > >
> > > > Is this a reasonable way separate our concerns here?
> > > >
> > > > On Wed, Oct 24, 2018 at 11:37 AM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Yeah, I'm also surprised by that comment about the mailing list
> > > activity.
> > > > > Our dev/user list discussions are by far more active than they've
> > ever
> > > > > been. Just have a look at the list of DISCUSS threads that have
> come
> > up
> > > > in
> > > > > the past few months and it's clear that not only participation has
> > > > > increased, but diversity of topic and participant.
> > > > >
> > > > > On Wed, Oct 24, 2018 at 8:08 AM Casey Stella 
> > > wrote:
> > > > >
> > > > > > Not for nothing, but at least according to the last board report
> > > that I
> > > > > > submitted, the user@ traffic is up 100% and the dev list traffic
> > is
> > > > flat
> > > > > > as
> > > > > > compared to last quarter.  That's not to say that we couldn't
> stand
> > > > more
> > > > > > discussion on the lists, but a lot of the dev discussion happens
> on
> > > > > github
> > > > > > and JIRA and I'm happy to see an uptick in user traffic.
> > > > > >
> > > > > > On Wed, Oct 24, 2018 at 10:05 AM Otto Fowler <
> > > ottobackwa...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I wouldn’t be so quick to related the slack discussion with
> > > perceived
> > > > > > > activity on the list.
> > > > > > > That is more do to the other things that are bigger issues.
> > > > > > >
> > > > > > >
> > > > > > > On October 24, 2018 at 07:15:30, Nick Allen (
> n...@nickallen.org)
> > > > > wrote:
> > > > > > >
> > > > > > > > I have heard recently people thought Metron is sort of dead
> > just
> > > > > > because
> > > > > > > the mailing list is not so active anymore!
> > > > > > >
> > > > > > > That is exactly my concern.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Oct 24, 2018, 2:49 AM Ali Nazemian <
> > alinazem...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > I kind of expect to have Slack for more dev related
> discussions
> > > > > rather
> > > > > > > than
> > > > > > > > user QA. I guess it is quite common to expect mailing list to
> > be
> > > > used
> > > > > > for
> > > > > > > > the purpose of knowledge sharing to make sure it will be
> > > accessible
> 

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

In my opinion metron-bro-plugin-kafka is ready for a release.  Anything
else people would want to see?  Once it gets released, I would like to
update full dev to use the newest version prior to any future metron
release (0.6.1 or whatever we choose).

Jon

On Wed, Nov 7, 2018 at 8:07 PM zeo...@gmail.com  wrote:

> So, about this release, anybody have time to review
> apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13?
>
> Jon
>
> On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> And I do think we will be ready to roll another Metron release in the near
>> future as well.
>>
>> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet 
>> wrote:
>>
>> > I tend to agree with Mike. I think we could do a release of the main
>> > project and get benefit, but I think skipping this cycle (especially
>> since
>> > we had a release last month) let's us get things like the ES client
>> > migration in and settled and just generally make recommending a newer
>> > version easier.
>> >
>> > The main reason for doing a release for the Bro plugin is to address
>> > shortcomings that we discovered and are addressing.
>> >
>> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
>> > michael.miklav...@gmail.com> wrote:
>> >
>> > > Migrating the ES client, refactoring parser bolt. There are some
>> > interface
>> > > changes in flight right now that I think would be beneficial to see in
>> > the
>> > > next release.
>> > >
>> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen 
>> wrote:
>> > >
>> > > > I am in favor of a release for both.
>> > > >
>> > > > There are a lot of really useful bug fixes, management of pcap
>> through
>> > > > Ambari, more flexibility for configuring JAAS in Ambari, increased
>> > > > Elasticsearch performance, the Syslog parser, and the Batch
>> Profiler,
>> > > among
>> > > > others. I would be happy with calling it a 0.6.1 point release.
>> > > >
>> > > > Mike - What is outstanding that you would like to see in the
>> release?
>> > > >
>> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
>> > > > michael.miklav...@gmail.com> wrote:
>> > > >
>> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin release.
>> It
>> > > > seems
>> > > > > like it's ready to go, and I think there are a few more things I'd
>> > like
>> > > > to
>> > > > > see get into our next Metron release so I'm good with holding off
>> > > there.
>> > > > >
>> > > > > Mike
>> > > > >
>> > > > > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet <
>> justinjl...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > As you might recall from a prior discussion about release
>> cadence,
>> > we
>> > > > > were
>> > > > > > interested in initiating release threads near our board reports
>> to
>> > > see
>> > > > if
>> > > > > > we wanted to do releases or not. Additionally, the work is done
>> to
>> > do
>> > > > two
>> > > > > > separate releases, so our options are releasing both, a single
>> one,
>> > > or
>> > > > > > neither.
>> > > > > >
>> > > > > > Having said that, a metron-bro-kafka-plugin 0.3.0 release came
>> up
>> > on
>> > > > this
>> > > > > > thread
>> > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
>> > > > > > >.
>> > > > > > In particular, the prospect of a release came up in the context
>> of
>> > > > > having a
>> > > > > > version with better (and working) testing.
>> > > > > >
>> > > > > > Version Number
>> > > > > > If we choose to do a co

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

I'm good with that release schedule, and using version 0.7.0 for the
apache/metron release.

I opened up METRON-1881 and have a branch ready to PR for the 0.3
plugin upgrade.

Jon

On Fri, Nov 16, 2018 at 10:08 AM Otto Fowler 
wrote:

> Can you generate the jiras that would be included in the release?
>
>
> On November 16, 2018 at 10:05:50, Justin Leet (justinjl...@gmail.com)
> wrote:
>
> Given that we've had a couple major PRs (the ES client migration along with
> the Angular upgrade stuff, and I'm sure others), I'd be in favor of
> releasing both the plugin and the main repo.
>
> I'd be in favor of doing something like:
> metron-bro-plugin-kafka release 0.3.0
> PR to update full dev
> metron release 0.7.0 (given that things changed a good amount)
>
> I would also expect this process to most likely be kicked off post
> Thanksgiving break (Thursday 22nd and Friday 23rd for anybody not in the
> US).
>
> I can kick out another summarization thread if we want, but basically:
> * Nov 26 - Start the Bro plugin release process
> * Once that finishes, PR to update full dev with new plugin version
> * Once PR is in, start metron release process (hopefully) sometime the week
> of the 3rd?
>
> Are there any objections to staggering the releases like that? They could
> also be done together, but it means that we have to update full dev to
> match the plugin version post release.
>
> On Wed, Nov 14, 2018 at 10:29 AM zeo...@gmail.com 
> wrote:
>
> > In my opinion metron-bro-plugin-kafka is ready for a release. Anything
> > else people would want to see? Once it gets released, I would like to
> > update full dev to use the newest version prior to any future metron
> > release (0.6.1 or whatever we choose).
> >
> > Jon
> >
> > On Wed, Nov 7, 2018 at 8:07 PM zeo...@gmail.com 
> wrote:
> >
> > > So, about this release, anybody have time to review
> > > apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13?
> > >
> > > Jon
> > >
> > > On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > >> And I do think we will be ready to roll another Metron release in the
> > near
> > >> future as well.
> > >>
> > >> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet 
> > >> wrote:
> > >>
> > >> > I tend to agree with Mike. I think we could do a release of the main
> > >> > project and get benefit, but I think skipping this cycle (especially
> > >> since
> > >> > we had a release last month) let's us get things like the ES client
> > >> > migration in and settled and just generally make recommending a
> newer
> > >> > version easier.
> > >> >
> > >> > The main reason for doing a release for the Bro plugin is to address
> > >> > shortcomings that we discovered and are addressing.
> > >> >
> > >> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
> > >> > michael.miklav...@gmail.com> wrote:
> > >> >
> > >> > > Migrating the ES client, refactoring parser bolt. There are some
> > >> > interface
> > >> > > changes in flight right now that I think would be beneficial to
> see
> > in
> > >> > the
> > >> > > next release.
> > >> > >
> > >> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen 
> > >> wrote:
> > >> > >
> > >> > > > I am in favor of a release for both.
> > >> > > >
> > >> > > > There are a lot of really useful bug fixes, management of pcap
> > >> through
> > >> > > > Ambari, more flexibility for configuring JAAS in Ambari,
> increased
> > >> > > > Elasticsearch performance, the Syslog parser, and the Batch
> > >> Profiler,
> > >> > > among
> > >> > > > others. I would be happy with calling it a 0.6.1 point release.
> > >> > > >
> > >> > > > Mike - What is outstanding that you would like to see in the
> > >> release?
> > >> > > >
> > >> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
> > >> > > > michael.miklav...@gmail.com> wrote:
> > >> > > >
> > >> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin
> > release.
>

Re: [ANNOUNCE] Shane Ardell is a committer

[zeo...@gmail.com]

Congrats Shane!

Jon

On Mon, Nov 19, 2018 at 10:43 AM Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> Many congratulations, Shane!
>
> Cheers,
> Anand
>
> On 11/19/18, 8:36 PM, "James Sirota"  wrote:
>
>
> The Project Management Committee (PMC) for Apache Metron has invited
> Shane Ardell to become a committer and we are pleased to announce that he
> has accepted.  I wanted to congratulate Shane on this achievement.
>
>
> Being a committer enables easier contribution to the project since
> there is no need to go via the patch submission process. This should enable
> better productivity. Being a PMC member enables assistance with the
> management and to guide the direction of the project.
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>
>
> --

Jon Zeolla

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

he/metron#1201
> > > METRON-1759 PCAP UI: Removing wrong Input annotations from pcap
> panel
> > > component (tiborm via nickwallen) closes apache/metron#1180
> > > METRON-1772 Support alternative input formats in the Batch Profiler
> > > (nickwallen) closes apache/metron#1191
> > > METRON-1770 Add Docs for Running the Profiler with Spark on YARN
> > > (nickwallen) closes apache/metron#1189
> > > METRON-1774 Allow user to configure JAAS client in Ambari
> > (nickwallen)
> > > closes apache/metron#1192
> > > METRON-1760 Kill PCAP job should prompt for confirmation (ruffle
> via
> > > nickwallen) closes apache/metron#1199
> > > METRON-1777: Fix Elasticsearch X-Pack sample pom in documentation
> > > (mmiklavc via mmiklavc) closes apache/metron#1196
> > > METRON-1781 Fix RPM Spec File (nickwallen) closes
> apache/metron#1200
> > > METRON-1780 Fix broken website images (justinleet) closes
> > > apache/metron#1198
> > > METRON-1476 Update to Angular 6.1.3 (sardell via nickwallen) closes
> > > apache/metron#1096
> > > METRON-1776 Update public web site to point at 0.6.0 new release
> > > (justinleet) closes apache/metron#1195
> > > METRON-1775 Transient exception could prevent expired profiles from
> > > being flushed (nickwallen) closes apache/metron#1194
> > > METRON-1717 Relocate Storm Profiler Code (nickwallen) closes
> > > apache/metron#1187
> > > METRON-1748 Improve Storm Profiler Integration Test (nickwallen)
> > closes
> > > apache/metron#1174
> > > METRON-1741 Move REPL Port of Profiler to Separate Project
> > (nickwallen)
> > > closes apache/metron#1170
> > > METRON-1715 Create DEB Packaging for Batch Profiler (nickwallen)
> > closes
> > > apache/metron#1167
> > > METRON-1736 Enhance Batch Profiler Integration Test (nickwallen)
> > closes
> > > apache/metron#1162
> > > METRON-1714 Create RPM Packaging for the Batch Profiler
> (nickwallen)
> > > closes apache/metron#1163
> > > METRON-1708 Run the Batch Profiler in Spark (nickwallen) closes
> > > apache/metron#1161
> > > METRON-1707 Port Profiler to Spark (nickwallen) closes
> > > apache/metron#1150
> > >     METRON-1705 Create ProfilePeriod Using Period ID (nickwallen)
> closes
> > > apache/metron#1148
> > > METRON-1706 HbaseClient.mutate should return the number of
> mutations
> > > (nickwallen) closes apache/metron#1147
> > > METRON-1704 Message Timestamp Logic Should be Shared (nickwallen)
> > > closes apache/metron#1146
> > > METRON-1703 Make Core Profiler Components Serializable (nickwallen)
> > > closes apache/metron#1145
> > >
> > >
> > > *apache/metron-bro-plugin-kafka*
> > > git log "master" "^tags/apache-metron-bro-plugin-kafka_0.2.0-release"
> > > --no-merges | grep -E "^[[:blank:]]+METRON" | sed 's/\[//g' | sed
> > 's/\]//g'
> > > | awk '!x[$1]++'
> > > METRON-1827 Update librdkafka in metron-bro-plugin-kafka (JonZeolla
> > via
> > > jonzeolla) closes apache/metron-bro-plugin-kafka#13
> > > METRON-1866 Improve metron-bro-plugin-kafka documentation
> (JonZeolla
> > > via jonzeolla) closes apache/metron-bro-plugin-kafka#17
> > > METRON-1304 Allow metron-bro-plugin-kafka to include or exclude
> logs
> > > (JonZeolla via nickwallen) closes apache/metron-bro-plugin-kafka#2
> > > METRON-1865 Fix metron-bro-plugin-kafka tests (JonZeolla via
> > jonzeolla)
> > > closes apache/metron-bro-plugin-kafka#16
> > > METRON-1828 Improve bro plugin contributing documentation
> (JonZeolla)
> > > closes apache/metron-bro-plugin-kafka#14
> > > METRON-1818 Remove config_files from bro-pkg.meta (JonZeolla)
> closes
> > > apache/metron-bro-plugin-kafka#11
> > > METRON-1800 Increment metron-bro-plugin-kafka version (JonZeolla
> via
> > > jonzeolla) closes apache/metron-bro-plugin-kafka#10
> > > METRON-1773 Bro plugin docs should refer to Apache Metron project
> > > (nickwallen) closes apache/metron-bro-plugin-kafka#9
> > >
> > > On Sun, Nov 18, 2018 at 7:52 AM zeo...@gmail.com 
> > wrote:
> > >
> > > > I'm good with that release schedule, and using version 0.7.0 for the
> > > > apache/metron release.
> > > >
> > > > I opened up METRON

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

(nickwallen) closes apache/metron#1202
> > > > METRON-1758 Add support for Ansible 2.6 in dev (JonZeolla via
> > > > jonzeolla) closes apache/metron#1179
> > > > METRON-1784: Re-allow remote ssh and scp in Centos full dev
> > (mmiklavc
> > > > via mmiklavc) closes apache/metron#1204
> > > > METRON-1787 Input Time Constraints for Batch Profiler
> (nickwallen)
> > > > closes apache/metron#1209
> > > > METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to
> > Elasticsearch
> > > > (nickwallen) closes apache/metron#1185
> > > > METRON-1786 Pcap Topology Status Incorrect (MohanDV via
> nickwallen)
> > > > closes apache/metron#1206
> > > > METRON-1709 Add controls to start / stop the PCAP topology from
> > > Ambari.
> > > > (MohanDV via nickwallen) closes apache/metron#1201
> > > > METRON-1759 PCAP UI: Removing wrong Input annotations from pcap
> > panel
> > > > component (tiborm via nickwallen) closes apache/metron#1180
> > > > METRON-1772 Support alternative input formats in the Batch
> Profiler
> > > > (nickwallen) closes apache/metron#1191
> > > > METRON-1770 Add Docs for Running the Profiler with Spark on YARN
> > > > (nickwallen) closes apache/metron#1189
> > > > METRON-1774 Allow user to configure JAAS client in Ambari
> > > (nickwallen)
> > > > closes apache/metron#1192
> > > > METRON-1760 Kill PCAP job should prompt for confirmation (ruffle
> > via
> > > > nickwallen) closes apache/metron#1199
> > > > METRON-1777: Fix Elasticsearch X-Pack sample pom in documentation
> > > > (mmiklavc via mmiklavc) closes apache/metron#1196
> > > > METRON-1781 Fix RPM Spec File (nickwallen) closes
> > apache/metron#1200
> > > > METRON-1780 Fix broken website images (justinleet) closes
> > > > apache/metron#1198
> > > > METRON-1476 Update to Angular 6.1.3 (sardell via nickwallen)
> closes
> > > > apache/metron#1096
> > > > METRON-1776 Update public web site to point at 0.6.0 new release
> > > > (justinleet) closes apache/metron#1195
> > > > METRON-1775 Transient exception could prevent expired profiles
> from
> > > > being flushed (nickwallen) closes apache/metron#1194
> > > > METRON-1717 Relocate Storm Profiler Code (nickwallen) closes
> > > > apache/metron#1187
> > > > METRON-1748 Improve Storm Profiler Integration Test (nickwallen)
> > > closes
> > > > apache/metron#1174
> > > > METRON-1741 Move REPL Port of Profiler to Separate Project
> > > (nickwallen)
> > > > closes apache/metron#1170
> > > > METRON-1715 Create DEB Packaging for Batch Profiler (nickwallen)
> > > closes
> > > > apache/metron#1167
> > > > METRON-1736 Enhance Batch Profiler Integration Test (nickwallen)
> > > closes
> > > > apache/metron#1162
> > > > METRON-1714 Create RPM Packaging for the Batch Profiler
> > (nickwallen)
> > > > closes apache/metron#1163
> > > > METRON-1708 Run the Batch Profiler in Spark (nickwallen) closes
> > > > apache/metron#1161
> > > > METRON-1707 Port Profiler to Spark (nickwallen) closes
> > > > apache/metron#1150
> > > > METRON-1705 Create ProfilePeriod Using Period ID (nickwallen)
> > closes
> > > > apache/metron#1148
> > > > METRON-1706 HbaseClient.mutate should return the number of
> > mutations
> > > > (nickwallen) closes apache/metron#1147
> > > > METRON-1704 Message Timestamp Logic Should be Shared (nickwallen)
> > > > closes apache/metron#1146
> > > > METRON-1703 Make Core Profiler Components Serializable
> (nickwallen)
> > > > closes apache/metron#1145
> > > >
> > > >
> > > > *apache/metron-bro-plugin-kafka*
> > > > git log "master" "^tags/apache-metron-bro-plugin-kafka_0.2.0-release"
> > > > --no-merges | grep -E "^[[:blank:]]+METRON" | sed 's/\[//g' | sed
> > > 's/\]//g'
> > > > | awk '!x[$1]++'
> > > > METRON-1827 Update librdkafka in metron-bro-plugin-kafka
> (JonZeolla
> > > via
> > > > jonzeolla) closes apache/metron-bro-plugin-kafka#13
> > > > METRON-1866 Improve metron-bro-plugin-kafka documentation

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[zeo...@gmail.com]

-1

In my testing it appears that an issue was introduced in 0.2 which is
causing a segfault on the destructor (
https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57).
I've opened METRON-1910 and am testing a fix now.

On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:

> This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
> The release candidate is available at:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
>
> Full list of changes in this release:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES
>
> The tag to be voted upon is:
> apache-metron-bro-plugin-kafka_0.3.0-rc1
> <
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc1
> >
>
> The source archives being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/apache-metron-bro-plugin-kafka_0.3.0-rc1.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
>
> The release artifacts are signed with the following key:
> *https://dist.apache.org/repos/dist/release/metron/KEYS*
> 
> Please vote on releasing this package as Apache Metron-bro-plugin-kafka
> 0.3.0
>
> When voting, please list the actions taken to verify the release.
>
> Recommended build validation and verification instructions are posted here:
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
> This is only for the plugin, so the VM validation is not required.
>
> This vote will be open for until 3pm EDT on Friday November 27 2018.
>
> [ ] +1 Release this package as Apache Metron 0.3.0-RC1
>
> [ ] 0 No opinion
>
> [ ] -1 Do not release this package because...
>
-- 

Jon Zeolla