Re: [VOTE] Move Apache Metron to the Apache Attic and Dissolve PMC

[zeo...@gmail.com]

+1

--
Jon Zeolla
@jonzeolla

PittSec | BSidesPGH | SteelCityInfoSec

On Mon, Nov 16, 2020, 11:33 AM Casey Stella  wrote:

> +1
>
> On Mon, Nov 16, 2020 at 09:01 Justin Leet  wrote:
>
> > Hi all,
> >
> > This is a vote thread to retire Metron to the Attic, and dissolve the
> PMC.
> > This follows a discussion thread on the dev list ([DISCUSS] Retire Metron
> > to the Attic
> > <
> >
> https://lists.apache.org/thread.html/reb31f643fac20d3ad09521fd702b19922412b7a4e8e08062968268c5%40%3Cdev.metron.apache.org%3E
> > >).
> > More details can be found in that discussion, but the most relevant link
> is
> > the specific process at Moving a project to the Attic
> > .
> >
> > As noted in the process page, this is a PMC vote. As usual, feel
> encouraged
> > to contribute non-binding votes.
> >
> > The vote will run 72 hours, until Nov 19th at 9:00 am EST.
> >
> > Thank you,
> > Justin
> >
>

Re: [DISCUSS] Retire Metron to the Attic

[zeo...@gmail.com]

I also agree with a move to the attic.  +1 to Otto's comment about forking
the kafka plugin.

--
Jon Zeolla
@jonzeolla

PittSec | BsidesPGH | SteelCityInfoSec


On Mon, Nov 9, 2020 at 1:30 PM Otto Fowler  wrote:

>  I am in support of this as well,
>
> We have substantial work to do to get metron off of the HDP version it is
> on, which I think is the minimal work to keep it viable.  We do not however
> have the people to finish the work that was started.
>
> Jon Zeolla and myself will be forking the metron-bro-kafka-plugin out, as
> it _has_ continued to be active.
>
>
>
> From: Casey Stella  
> Reply: dev@metron.apache.org  <
> dev@metron.apache.org>
> Date: November 9, 2020 at 10:48:22
> To: dev@metron.apache.org  
> Subject:  Re: [DISCUSS] Retire Metron to the Attic
>
> Hi all,
> I'm in complete support of this. Given the current level of interest, I
> believe that we should move this project to the attic.
>
> Best,
>
> Casey Stella
>
> On Mon, Nov 2, 2020 at 7:28 PM Justin Leet  wrote:
>
> > Hi all,
> >
> > I want to start a discussion in the community to consider retiring Metron
> > to the Apache Attic and dissolve the project management committee. For
> > anyone unaware, an overview is available at Apache Attic
> > . In short, it's a way to provide a wind-down
> > process for projects reaching their end of life. This process, and
> > potential next steps, can be found here:
> > http://attic.apache.org/process.html.
> >
> > We've seen a substantial decrease in contributions (both code and mailing
> > lists) over the past year or so (see:
> > https://github.com/apache/metron/commits/master). There's been some
> > interest in the project, but in my personal opinion, not enough to build
> or
> > sustain real momentum. Additionally, there's been an inability to close
> the
> > loop on paring down the project's scope and building a solid foundation
> for
> > future work (see: Development Activity has dropped to effectively 0, what
> > should we do?
> > <
> >
>
> https://lists.apache.org/thread.html/rf7ea1c1afb347e352efff50f58fbd58779f71e6c814d2a5563e381d7%40%3Cdev.metron.apache.org%3E
> > >).
> > Finally, the last release was over a year ago and I don't see a release
> > naturally building anytime soon. We haven't seen much active development
> or
> > participation for a substantial period of time, which to me implies we've
> > reached a natural end of life.
> >
> > The most obvious practical effects would be no new releases, setting
> source
> > read-only, mailing lists closed down, automated processes ended, etc. In
> > addition, the PMC would also be dissolved as part of this process.
> > Community members may still fork all or part of Metron, but should keep
> in
> > mind that the Metron trademark remains with the ASF.
> >
> > What opinions are there either in favor of or against moving the project
> to
> > the Attic? Any questions about the process (or comments/corrections
> anyone
> > would like to add)?
> >
> > Thanks,
> > Justin
> >
>

Re: Any relation to Spot?

[zeo...@gmail.com]

Nope, different projects with similar goals.  Metron came from Cisco
OpenSOC and Spot came from ONI.

Jon Zeolla

On Thu, Apr 9, 2020, 5:57 PM Yerex, Tom  wrote:

> Good afternoon,
>
> I hope everyone is safe and healthy. I tripped across the Apache Spot
> project while working through some documentation and it made me wonder if
> there is any relation between Apache Metron and Apache Spot?
>
> hxxp://spot.incubator.apache.org/
>
> Cheers,
>
> Tom.
>

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC3

[zeo...@gmail.com]

+1 ran the RC script, spun up end to end successfully, manual validation,
etc.

- Jon Zeolla
zeo...@gmail.com


On Thu, Oct 10, 2019 at 3:10 PM Otto Fowler  wrote:

> +1 binding Ran RC script including the docker end to end testing
>
>
>
>
> On October 10, 2019 at 14:38:45, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I have written a new script for validation of bro-kafka RC’s
>
>
> https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/bro-kafka/metron-bro-kafka-rc-check
>
> Please give it a shot. If there are no issues with it, it will go into the
> plugin project after the rc is done ( can’t test the rc without the rc )
>
>
>
> On October 10, 2019 at 14:31:44, Justin Leet (l...@apache.org) wrote:
>
> This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
> The release candidate is available at:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/
> Full list of changes in this release:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/CHANGES
>
> The tag to be voted upon is:
> apache-metron-bro-plugin-kafka_0.3.0-rc2
> <
>
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc3
> >
>
> The source archives being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/apache-metron-bro-plugin-kafka_0.3.0-rc3.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/
>
> The release artifacts are signed with the following key:
> *https://dist.apache.org/repos/dist/release/metron/KEYS*
> <https://dist.apache.org/repos/dist/release/metron/KEYS>
> Please vote on releasing this package as Apache Metron-bro-plugin-kafka
> 0.3.0
>
> When voting, please list the actions taken to verify the release.
>
> Recommended build validation and verification instructions are posted here:
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
> This is only for the plugin, so the VM validation is not required.
>
> This vote will be open until 3pm EDT on Tuesday October 15 2019, to account
> for the weekend.
>
> [ ] +1 Release this package as Apache Metron 0.3.0-RC3
>
> [ ] 0 No opinion
>
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC2

[zeo...@gmail.com]

>From my perspective, yes

Jon Zeolla

On Thu, Oct 10, 2019, 7:24 AM Justin Leet  wrote:

> Are we at the point where we're comfortable spinning up another RC?
>
> On Thu, Oct 3, 2019 at 5:04 PM Justin Leet  wrote:
>
> > The vote has failed. The voting was:
> > 2 -1’s (binding)
> >
> > A new RC will be created once we're satisfied the latest fix has resolved
> > issues.
> >
> > On Tue, Oct 1, 2019, 2:47 PM zeo...@gmail.com  wrote:
> >
> >> -1 as well, validated the issue that Otto was seeing.
> >>
> >> I'm also testing to ensure that the fix properly addressed the issue and
> >> will respond if I see any issues that would block a fast follow RC3.
> >>
> >> - Jon Zeolla
> >> zeo...@gmail.com
> >>
> >>
> >> On Tue, Oct 1, 2019 at 3:27 PM Otto Fowler 
> >> wrote:
> >>
> >> > The fix has landed.
> >> >
> >> > Also, here is a quick script to try for validation
> >> >
> >> >
> >> >
> >>
> https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/bro-kafka/metron-bro-kafka-rc-check
> >> >
> >> >
> >> >
> >> >
> >> > On October 1, 2019 at 07:41:02, Otto Fowler (ottobackwa...@gmail.com)
> >> > wrote:
> >> >
> >> > https://github.com/apache/metron-bro-plugin-kafka/pull/37
> >> >
> >> >
> >> >
> >> > On September 30, 2019 at 18:44:29, Otto Fowler (
> ottobackwa...@gmail.com
> >> )
> >> > wrote:
> >> >
> >> > –1 binding.
> >> >
> >> > A new feature to our docker testing to add support for the git
> version /
> >> > branch of the plugin to test was added during development.
> >> >
> >> > This feature however makes it impossible to run the docker testing
> suite
> >> > against the signed src tarball for verification.
> >> >
> >> > I have created METRON–2269 for this issue.
> >> >
> >> >
> >> >
> >> > On September 30, 2019 at 09:56:54, Justin Leet (l...@apache.org)
> wrote:
> >> >
> >> > This is a call to vote on releasing Apache Metron-bro-plugin-kafka
> 0.3.0
> >> > The release candidate is available at:
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
> >> > Full list of changes in this release:
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/CHANGES
> >> >
> >> > The tag to be voted upon is:
> >> > apache-metron-bro-plugin-kafka_0.3.0-rc2
> >> > <
> >> >
> >> >
> >>
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc2
> >> > >
> >> >
> >> > The source archives being voted upon can be found here:
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/apache-metron-bro-plugin-kafka_0.3.0-rc2.tar.gz
> >> >
> >> > Other release files, signatures and digests can be found here:
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
> >> >
> >> > The release artifacts are signed with the following key:
> >> > *https://dist.apache.org/repos/dist/release/metron/KEYS*
> >> > <https://dist.apache.org/repos/dist/release/metron/KEYS>
> >> > Please vote on releasing this package as Apache
> Metron-bro-plugin-kafka
> >> > 0.3.0
> >> >
> >> > When voting, please list the actions taken to verify the release.
> >> >
> >> > Recommended build validation and verification instructions are posted
> >> here:
> >> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >> >
> >> > This is only for the plugin, so the VM validation is not required.
> >> >
> >> > This vote will be open for until 10am EDT on Thursday September 3
> 2019.
> >> >
> >> > [ ] +1 Release this package as Apache Metron 0.3.0-RC2
> >> >
> >> > [ ] 0 No opinion
> >> >
> >> > [ ] -1 Do not release this package because...
> >> >
> >>
> >
>

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC2

[zeo...@gmail.com]

-1 as well, validated the issue that Otto was seeing.

I'm also testing to ensure that the fix properly addressed the issue and
will respond if I see any issues that would block a fast follow RC3.

- Jon Zeolla
zeo...@gmail.com


On Tue, Oct 1, 2019 at 3:27 PM Otto Fowler  wrote:

> The fix has landed.
>
> Also, here is a quick script to try for validation
>
>
> https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/bro-kafka/metron-bro-kafka-rc-check
>
>
>
>
> On October 1, 2019 at 07:41:02, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> https://github.com/apache/metron-bro-plugin-kafka/pull/37
>
>
>
> On September 30, 2019 at 18:44:29, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> –1 binding.
>
> A new feature to our docker testing to add support for the git version /
> branch of the plugin to test was added during development.
>
> This feature however makes it impossible to run the docker testing suite
> against the signed src tarball for verification.
>
> I have created METRON–2269 for this issue.
>
>
>
> On September 30, 2019 at 09:56:54, Justin Leet (l...@apache.org) wrote:
>
> This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
> The release candidate is available at:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
> Full list of changes in this release:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/CHANGES
>
> The tag to be voted upon is:
> apache-metron-bro-plugin-kafka_0.3.0-rc2
> <
>
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc2
> >
>
> The source archives being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/apache-metron-bro-plugin-kafka_0.3.0-rc2.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
>
> The release artifacts are signed with the following key:
> *https://dist.apache.org/repos/dist/release/metron/KEYS*
> <https://dist.apache.org/repos/dist/release/metron/KEYS>
> Please vote on releasing this package as Apache Metron-bro-plugin-kafka
> 0.3.0
>
> When voting, please list the actions taken to verify the release.
>
> Recommended build validation and verification instructions are posted here:
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
> This is only for the plugin, so the VM validation is not required.
>
> This vote will be open for until 10am EDT on Thursday September 3 2019.
>
> [ ] +1 Release this package as Apache Metron 0.3.0-RC2
>
> [ ] 0 No opinion
>
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[zeo...@gmail.com]

If it's possible I would prefer to get the process moving sooner rather
than later so perhaps I could mention it to people at ZeekWeek and improve
our adoption rates.

Jon Zeolla

On Sun, Sep 29, 2019, 2:16 PM Justin Leet  wrote:

> Jon actually wouldn't be able to cut a release candidate immediately
> anyway; he doesn't have an entry in the KEYS file.
>
> As backgound noted in:
> https://cwiki.apache.org/confluence/display/METRON/Release+Process:
>
> "Currently, the KEYS file is only pushed with a main Metron release.  This
> > means that, when there is a new release manager, they will be unable to
> > release apache/metron-bro-kafka-plugin without a release of
> apache/metron."
>
>
> Nobody's particularly cared about fixing this, given that the plugin
> releases more rarely than the main repo.
>
>
> I should be able to cut an RC tomorrow, but when is a bit more
> questionable. The other catch is that I'm doing some travel late this week,
> so my ability to actually push the release post vote (assuming success) or
> cut a new RC (assuming failure) is pretty limited. I'll leave it up to the
> community if everyone would rather live with the possibility that there's a
> delay post vote or if we'd rather start next week.
>
> On Sun, Sep 29, 2019 at 12:47 PM zeo...@gmail.com 
> wrote:
>
> > Justin Leet was running this release previously
> >
> > Jon Zeolla
> >
> > On Sun, Sep 29, 2019, 12:07 PM Otto Fowler 
> > wrote:
> >
> > > If you are doing the RM duties, just go a head and cut the RC.
> > >
> > >
> > >
> > >
> > > On September 29, 2019 at 11:35:10, zeo...@gmail.com (zeo...@gmail.com)
> > > wrote:
> > >
> > > Hi everyone,
> > >
> > > This took much longer than expected, but we recently cleaned up the
> last
> > > blocker to getting a 0.3 release out. As a reminder, currently the
> latest
> > > apache/metron-bro-plugin-kafka release has a bug causing the tests to
> > > fail. Master no longer has this, and in fact has a fair amount of
> > > improvements both in features and automated testing which went in since
> > > this rc. I would like to see them all included in the new 0.3 rc (i.e.
> > > align the release with master).
> > >
> > > We may also want to consider a fast following release that includes the
> > two
> > > current open PRs (after review) that were waiting on the bro/zeek 3.0
> > > release (which recently happened). Because these cause breaking
> changes I
> > > do not think they should be in the 0.3 release, but rather are good
> > > candidates for a 1.0.0 release (moving from x.y to x.y.z versioning as
> > > well). At that time we may also want to consider a project rename due
> to
> > > the bro project rename to zeek, but I'll leave that for a separate
> thread
> > > for after we get 0.3 out.
> > >
> > > Jon Zeolla
> > >
> > > On Fri, Nov 30, 2018, 6:14 PM Justin Leet 
> wrote:
> > >
> > > > The vote has failed. The voting was:
> > > > 1 -1’s (binding)
> > > >
> > > > A new RC will be created once this issue is resolved.
> > > >
> > > > On Wed, Nov 28, 2018 at 10:49 AM zeo...@gmail.com 
> > > > wrote:
> > > >
> > > > > -1
> > > > >
> > > > > In my testing it appears that an issue was introduced in 0.2 which
> is
> > > > > causing a segfault on the destructor (
> > > > >
> > > > >
> > > >
> > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57
> > > > > ).
> > > > > I've opened METRON-1910 and am testing a fix now.
> > > > >
> > > > > On Tue, Nov 27, 2018 at 2:36 PM Justin Leet 
> wrote:
> > > > >
> > > > > > This is a call to vote on releasing Apache
> Metron-bro-plugin-kafka
> > > > 0.3.0
> > > > > > The release candidate is available at:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> > > > > >
> > > > > > Full list of changes in this release:
> > > > > >
> > > > > >
> > > > >
> > >

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[zeo...@gmail.com]

Justin Leet was running this release previously

Jon Zeolla

On Sun, Sep 29, 2019, 12:07 PM Otto Fowler  wrote:

> If you are doing the RM duties, just go a head and cut the RC.
>
>
>
>
> On September 29, 2019 at 11:35:10, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> Hi everyone,
>
> This took much longer than expected, but we recently cleaned up the last
> blocker to getting a 0.3 release out. As a reminder, currently the latest
> apache/metron-bro-plugin-kafka release has a bug causing the tests to
> fail. Master no longer has this, and in fact has a fair amount of
> improvements both in features and automated testing which went in since
> this rc. I would like to see them all included in the new 0.3 rc (i.e.
> align the release with master).
>
> We may also want to consider a fast following release that includes the two
> current open PRs (after review) that were waiting on the bro/zeek 3.0
> release (which recently happened). Because these cause breaking changes I
> do not think they should be in the 0.3 release, but rather are good
> candidates for a 1.0.0 release (moving from x.y to x.y.z versioning as
> well). At that time we may also want to consider a project rename due to
> the bro project rename to zeek, but I'll leave that for a separate thread
> for after we get 0.3 out.
>
> Jon Zeolla
>
> On Fri, Nov 30, 2018, 6:14 PM Justin Leet  wrote:
>
> > The vote has failed. The voting was:
> > 1 -1’s (binding)
> >
> > A new RC will be created once this issue is resolved.
> >
> > On Wed, Nov 28, 2018 at 10:49 AM zeo...@gmail.com 
> > wrote:
> >
> > > -1
> > >
> > > In my testing it appears that an issue was introduced in 0.2 which is
> > > causing a segfault on the destructor (
> > >
> > >
> >
>
> https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57
> > > ).
> > > I've opened METRON-1910 and am testing a fix now.
> > >
> > > On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:
> > >
> > > > This is a call to vote on releasing Apache Metron-bro-plugin-kafka
> > 0.3.0
> > > > The release candidate is available at:
> > > >
> > > >
> > >
> >
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> > > >
> > > > Full list of changes in this release:
> > > >
> > > >
> > >
> >
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES
> > > >
> > > > The tag to be voted upon is:
> > > > apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > > <
> > > >
> > >
> >
>
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > > >
> > > >
> > > > The source archives being voted upon can be found here:
> > > >
> > > >
> > >
> >
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/apache-metron-bro-plugin-kafka_0.3.0-rc1.tar.gz
> > > >
> > > > Other release files, signatures and digests can be found here:
> > > >
> > > >
> > >
> >
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> > > >
> > > > The release artifacts are signed with the following key:
> > > > *https://dist.apache.org/repos/dist/release/metron/KEYS*
> > > > <https://dist.apache.org/repos/dist/release/metron/KEYS>
> > > > Please vote on releasing this package as Apache
> Metron-bro-plugin-kafka
> > > > 0.3.0
> > > >
> > > > When voting, please list the actions taken to verify the release.
> > > >
> > > > Recommended build validation and verification instructions are posted
> > > here:
> > > > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > > >
> > > > This is only for the plugin, so the VM validation is not required.
> > > >
> > > > This vote will be open for until 3pm EDT on Friday November 27 2018.
> > > >
> > > > [ ] +1 Release this package as Apache Metron 0.3.0-RC1
> > > >
> > > > [ ] 0 No opinion
> > > >
> > > > [ ] -1 Do not release this package because...
> > > >
> > > --
> > >
> > > Jon Zeolla
> > >
> >
>
> On

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[zeo...@gmail.com]

Hi everyone,

This took much longer than expected, but we recently cleaned up the last
blocker to getting a 0.3 release out.  As a reminder, currently the latest
apache/metron-bro-plugin-kafka release has a bug causing the tests to
fail.  Master no longer has this, and in fact has a fair amount of
improvements both in features and automated testing which went in since
this rc.  I would like to see them all included in the new 0.3 rc (i.e.
align the release with master).

We may also want to consider a fast following release that includes the two
current open PRs (after review) that were waiting on the bro/zeek 3.0
release (which recently happened).  Because these cause breaking changes I
do not think they should be in the 0.3 release, but rather are good
candidates for a 1.0.0 release (moving from x.y to x.y.z versioning as
well).  At that time we may also want to consider a project rename due to
the bro project rename to zeek, but I'll leave that for a separate thread
for after we get 0.3 out.

Jon Zeolla

On Fri, Nov 30, 2018, 6:14 PM Justin Leet  wrote:

> The vote has failed. The voting was:
> 1 -1’s (binding)
>
> A new RC will be created once this issue is resolved.
>
> On Wed, Nov 28, 2018 at 10:49 AM zeo...@gmail.com 
> wrote:
>
> > -1
> >
> > In my testing it appears that an issue was introduced in 0.2 which is
> > causing a segfault on the destructor (
> >
> >
> https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57
> > ).
> > I've opened METRON-1910 and am testing a fix now.
> >
> > On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:
> >
> > > This is a call to vote on releasing Apache Metron-bro-plugin-kafka
> 0.3.0
> > > The release candidate is available at:
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> > >
> > > Full list of changes in this release:
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES
> > >
> > > The tag to be voted upon is:
> > > apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > <
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > >
> > >
> > > The source archives being voted upon can be found here:
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/apache-metron-bro-plugin-kafka_0.3.0-rc1.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> > >
> > > The release artifacts are signed with the following key:
> > > *https://dist.apache.org/repos/dist/release/metron/KEYS*
> > > <https://dist.apache.org/repos/dist/release/metron/KEYS>
> > > Please vote on releasing this package as Apache Metron-bro-plugin-kafka
> > > 0.3.0
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > Recommended build validation and verification instructions are posted
> > here:
> > > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > >
> > > This is only for the plugin, so the VM validation is not required.
> > >
> > > This vote will be open for until 3pm EDT on Friday November 27 2018.
> > >
> > > [ ] +1 Release this package as Apache Metron 0.3.0-RC1
> > >
> > > [ ] 0 No opinion
> > >
> > > [ ] -1 Do not release this package because...
> > >
> > --
> >
> > Jon Zeolla
> >
>

On Fri, Nov 30, 2018, 6:14 PM Justin Leet  wrote:

> The vote has failed. The voting was:
> 1 -1’s (binding)
>
> A new RC will be created once this issue is resolved.
>
> On Wed, Nov 28, 2018 at 10:49 AM zeo...@gmail.com 
> wrote:
>
> > -1
> >
> > In my testing it appears that an issue was introduced in 0.2 which is
> > causing a segfault on the destructor (
> >
> >
> https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57
> > ).
> > I've opened METRON-1910 and am testing a fix now.
> >
> > On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:
> >
> > > This is a call to vote on releasing Apache Metron-bro-plugin-kafka
> 0.3.0
> > > The release candidate is available at:
&

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[zeo...@gmail.com]

I agree that having a scripted approach for backup and restore of Metron
configs should be necessary for such a large change/upgrade process.
Having been through this many times in the past I can tell you that the
difficulty of upgrading (whether perceived or actual) holds back adoption
of the platform.

Jon Zeolla

On Tue, Aug 27, 2019, 5:25 PM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Not sure it’s in the scope of the project to handle the HDP upgrade as
> well, I would scope it to metron config only, and point at the extensive
> upgrade capability of Ambari, rather than us trying to recreate the way the
> distribution works.
>
> Simon
>
> On Tue, 27 Aug 2019 at 22:23, Otto Fowler  wrote:
>
> > If anyone can think of the things that need to be backed up, please
> > comment the jira.
> >
> >
> >
> >
> > On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Good idea METRON–2239 [blocker].
> >
> >
> >
> > On August 27, 2019 at 16:30:13, Simon Elliston Ball (
> > si...@simonellistonball.com) wrote:
> >
> > You could always submit a Jira :)
> >
> > On Tue, 27 Aug 2019 at 21:27, Otto Fowler 
> wrote:
> >
> >> You are right, that is much better than backup_metron_configs.sh.
> >>
> >>
> >>
> >> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> >> si...@simonellistonball.com) wrote:
> >>
> >> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> >> kinda already do.
> >>
> >> Simon
> >>
> >> On Tue, 27 Aug 2019 at 20:19, Otto Fowler 
> >> wrote:
> >>
> >>> Maybe we need some automated method to backup configurations and
> restore
> >>> them.
> >>>
> >>>
> >>>
> >>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> >>> michael.miklav...@gmail.com) wrote:
> >>>
> >>> > Once you back up your metron configs, the same configs that worked on
> >>> the previous version will continue to work on the version running on
> HDP
> >>> 3.x.  If there is any discrepancy between the two or additional
> settings
> >>> will be required, those will be documented in the release notes.  From
> the
> >>> Metron perspective, this upgrade would be no different than simply
> >>> upgrading to the new Metron version.
> >>>
> >>> This upgrade cannot be performed the same way we've done it in the
> past.
> >>> A number of platform upgrades, including OS, are required:
> >>>
> >>>1. Requires the OS to be updated on all nodes because there are no
> >>>Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
> >>>2. The final new HBase code will not run on HDP 2.6
> >>>3. The MPack changes for new Ambari are not backwards compatible
> >>>4. YARN and HDFS/MR are also at risk to be backwards incompatible
> >>>
> >>>
> >>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> >>> michael.miklav...@gmail.com> wrote:
> >>>
>  Adding the dev list back into the thread (a reply-all was missed).
> 
>  On Tue, Aug 27, 2019 at 10:49 AM James Sirota 
>  wrote:
> 
> > I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
> > everyone will likely need to migrate by end of year.  Doing this
> platform
> > upgrade sooner will give everyone visibility into what Metron on HDP
> 3.x
> > looks like so they have time to plan and upgrade at their own pace.
> > Feature-wise, the Metron application itself will be unchanged.  It is
> > merely the platform underneath that is changing.  HDP itself can be
> > upgraded per instructions here:
> >
> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
> >
> > Once you back up your metron configs, the same configs that worked on
> > the previous version will continue to work on the version running on
> HDP
> > 3.x.  If there is any discrepancy between the two or additional
> settings
> > will be required, those will be documented in the release notes.
> From the
> > Metron perspective, this upgrade would be no different than simply
> > upgrading to the new Metron version.
> >
> > James
> >
> >
> > 27.08.2019, 07:09, "Simon Elliston Ball" <
> si...@simonellistonball.com
> > >:
> >
> > Something worth noting here is that HDP 2.6.5 is quite old and
> > approaching EoL rapidly, so the issue of upgrade is urgent. I am
> aware of a
> > large number of users who require this upgrade ASAP, and in fact an
> aware
> > of zero users who wish to remain on HDP 2.
> >
> > Perhaps those users who want to stay on the old platform can stick
> > their hands up and raise concerns, but this move will likely have to
> happen
> > very soon.
> >
> > Simon
> >
> > On Tue, 27 Aug 2019 at 15:04, Otto Fowler 
> > wrote:
> >
> > Although we had the discussion, and some great ideas where passed
> > around, I do not believe we came to some kind of consensus on what
> 1.0
> > should look like. So that discussion would h

Re: What's the status of Metron

[zeo...@gmail.com]

I just sent an invite for the ASF slack.  Check out #Metron once you're in
there.

There are some various network diagrams but nothing that I would consider
holistic.  Here are some pointers (in order)

https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture

https://github.com/apache/metron/blob/master/metron-platform/metron-parsing/README.md#parser-architecture

https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-storm/README.md#enrichment-architecture

https://github.com/apache/metron/blob/master/metron-platform/metron-indexing/metron-indexing-storm/README.md#indexing-architecture

Note that the wiki is generally very outdated but the logical diagram on
that page isn't bad.

Jon Zeolla

On Sat, Jun 8, 2019, 5:25 AM Phillip Henry  wrote:

> ... also, is there a Gitter channel where we can ask questions and discuss
> things in a more real-time manner? A lot of people find email somewhat
> cumbersome.
>
> Regards,
>
> Phillip
>
>
> On Sat, Jun 8, 2019 at 9:31 AM Phillip Henry 
> wrote:
>
> > Thanks, guys. I'm munching through the documentation now but would really
> > appreciate a steer towards perhaps a diagram of the architecture. Does it
> > exist?
> >
> > Mike, I'm reading the link you sent and slowly digesting it. I hope I can
> > help to contribute to the use cases in the near future.
> >
> > Regards,
> >
> > Phillip
> >
> >
> > On Fri, Jun 7, 2019 at 6:56 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> Welcome Phillip!
> >>
> >> +1 to all of what Otto said.
> >>
> >> One area where we would like to expand, and your DS skills could be
> >> useful,
> >> is in some of the analytics use cases. For example, I'm currently
> working
> >> on finishing up a PR for a data sketch function that can estimate top-k
> >> results in an infinite streaming data set. We've got functions around
> >> things like median absolute deviation, and I wrote a wrapper for a hyper
> >> log log plus implementation. It would be awesome to get some
> contributions
> >> to our use cases that leverage some of our existing analytics functions
> as
> >> well as new implementations. Here's a forensic clustering use case, for
> >> example -
> >>
> https://github.com/apache/metron/tree/master/use-cases/forensic_clustering
> >> .
> >> This is a bit more advanced for someone new to the project, but I'm just
> >> sharing opportunities and a roadmap of things that would be extremely
> >> beneficial.
> >>
> >> Best,
> >> Mike Miklavcic
> >>
> >> On Fri, Jun 7, 2019 at 11:14 AM Otto Fowler 
> >> wrote:
> >>
> >> > Hi Phillip and welcome!
> >> >
> >> > You certainly can help.
> >> >
> >> > As far as the status, there are several ways to look at it aren’t
> there?
> >> >
> >> > - we have active committers and reviewers, but not as many as we would
> >> like
> >> > - we have an active users like, but not insanely so
> >> > - metron _is_ part of a large bistro package HCP / cloudera ? and
> >> deployed,
> >> > supported and used there
> >> >
> >> >
> >> > A lot of work is going on right now in the following areas
> >> >
> >> > - refactoring the UI technologies and testability
> >> > - making the build better, speed, shading etc
> >> > - de-coupling storm from our core technologies so we can introduce or
> >> > support alternative processing ( spark for example )
> >> > - work on our deployment / dev experience ( vagrant/docker )
> >> >
> >> > and other things I’ve not thought of, and have PR’s etc.
> >> >
> >> > we can always use help with documentation, reviews, testing, how-to’s
> or
> >> > development :)
> >> >
> >> >
> >> >
> >> > On June 7, 2019 at 12:40:38, Phillip Henry (londonjava...@gmail.com)
> >> > wrote:
> >> >
> >> > Hello, all.
> >> >
> >> > The Metron docs mentioned something about introducing oneself upon
> >> joining
> >> > the mailing list so here goes. I'm a Scala/Java/Data
> Engineer/Scientist
> >> > working in a SOC at a leading telco.
> >> >
> >> > I've only recently come across Metron and was wondering what the
> >> community
> >> > was like, how healthy the project is (I see commits in the last few
> >> days so
> >> > that's good) and wondering if I can help.
> >> >
> >> > Regards,
> >> >
> >> > Phillip
> >> >
> >>
> >
>

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

[zeo...@gmail.com]

+1 non-binding

I would only prefer that we change "Appropriate architecture diagrams
should be created in" to "Appropriate architecture diagrams must be created
in" but I'm good either way.

- Jon Zeolla
zeo...@gmail.com


On Fri, May 3, 2019 at 10:18 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Yes, it is free James. We made sure of that in the original discussion.
>
> On Thu, May 2, 2019 at 9:33 PM James Sirota  wrote:
>
> > i am ok with it as long as we are not forcing people to buy stuff
> >
> > 02.05.2019, 18:18, "Michael Miklavcic" :
> > > Here's the latest discussion on the subject:
> > >
> >
> https://lists.apache.org/thread.html/0aa2b0b9ed4a0f0b0d8bb018c618e62de196565f9af71f347e504076@%3Cdev.metron.apache.org%3E
> > >
> > > I'd like to propose a vote to change our dev guidelines which will
> > clarify
> > > the tooling we use to produce diagrams and share the source files for
> > those
> > > diagrams. I propose the dev guidelines
> > >
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> > and
> > > PR checklist
> > >
> >
> https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
> > > be
> > > changed in the following ways:
> > >
> > >1. Under "1.1 Contributing A Code Change"
> > >   1. Change <<"New features and significant bug fixes should be
> > >   documented in the JIRA and appropriate architecture diagrams
> > should be
> > >   attached. Major features may require a vote.">> to <<"New
> features
> > >   and significant bug fixes should be documented in the JIRA.
> > Appropriate
> > >   architecture diagrams should be created in https://www.draw.io/
> > > and committed
> > >   to source control as per section 2.4. Diagrams may be requested
> of
> > PR
> > >   submitters during review either as documentation or as an aid to
> > the
> > >   reviewer. Major features may also require a vote.">>
> > >2. Under "2.4 Documentation"
> > >   1. New line item <<"Diagrams - We save architecture diagram
> source
> > >   files in an xml format rendered by draw.io (instructions below).
> > This
> > >   is the free tool of choice that we've agreed to use for
> exchanging
> > >   diagrams and their source files in Metron.">>
> > >   2. New line item < > >   "/images-source" and rendered diagrams and images
> > belong in
> > >   "/images."
> > >   3. New subsection <<"Creating and Modifying Diagrams">>. This
> > section
> > >   would provide basic instructions for downloading source files
> from
> > >   draw.io.
> > >3. Add a new checkbox item under PR checklist heading "For
> > documentation
> > >related changes" with the following text
> > >   1. Have you ensured that any documentation diagrams have been
> > >   updated, along with their source files, using draw.io? See
> > >
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> > > for
> > >   instructions.
> > >4. Here is the Jira for migrating/redoing existing diagrams
> > >   1. https://issues.apache.org/jira/browse/METRON-2099
> > >
> > > We require a minimum of 72 hours for a vote, not typically including
> > > weekend days. I'd like to leave this vote open until Wednesday 5/8,
> 12PM
> > > EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your
> vote
> > is
> > > binding or non-binding.
> >
> > ---
> > Thank you,
> >
> > James Sirota
> > PMC- Apache Metron
> > jsirota AT apache DOT org
> >
> >
>

Re: [DISCUSS] Next Release

[zeo...@gmail.com]

Any chance we could do a quick cleanup of METRON-1795?  Seems like it
should be assigned to Jagdeep Singh (jagdeep.sing...@team.telstra.com), but
I couldn't find that account in the assignee field.

- Jon Zeolla
zeo...@gmail.com


On Thu, Apr 25, 2019 at 12:33 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Just a heads up, we'll also want this PR in for the release because without
> it fulldev won't build. https://github.com/apache/metron/pull/1391. It's
> got a +1 from me and should be merged shortly.
>
> On Thu, Apr 25, 2019 at 6:53 AM Justin Leet  wrote:
>
> > Great, thanks Nick!
> >
> > I'll work on cutting the release candidate later today. That'll probably
> > happen in the afternoon (EDT), and I'll let everyone know here and in the
> > Slack room when it's available.
> >
> > On Thu, Apr 25, 2019 at 8:41 AM Nick Allen  wrote:
> >
> > > Justin - I cleaned up the last bit of stragglers in JIRA.  JIRA should
> be
> > > ready for the release.
> > >
> > > $ /dev-utilities/release-utils/validate-jira-for-release
> --version=0.7.1
> > > --start=tags/apache-metron_0.7.0-release
> > > ...
> > >JIRA  STATUS FIX VERSION
> > >  ASSIGNEEFIX
> > > METRON-2091Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2078Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2065Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2067Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2074Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2082Done   0.7.1
> > > Mohan
> > > METRON-2006Done   0.7.1
> Justin
> > > Leet
> > > METRON-2071Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2014Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2026Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2062Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2050Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2060Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2064Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2066Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-1654Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2053Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2022Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2056Done   0.7.1
>  Nick
> > > Allen
> > > METRON-2023Done   0.7.1   Tibor
> > > Meller
> > > METRON-2039Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2052Done   0.7.1   Tibor
> > > Meller
> > > METRON-2051Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2023Done   0.7.1   Tibor
> > > Meller
> > > METRON-2046Done   0.7.1  Anand
> > > Subramanian
> > > METRON-2029Done   0.7.1   Shane
> > > Ardell
> > > METRON-2032Done   0.7.1  Anand
> > > Subramanian
> > > METRON-2038Done   0.7.1
>  Nick
> > > Allen
> > > METRON-2035Done   0.7.1
>  Nick
> > > Allen
> > > METRON-2041Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2036Done   0.7.1  Michael
> > > Miklavcic
> > > METRON-2030Done   0.7.1  Ryan
> > > Merriman
> > > METRON-2031Done   0.7.1   Tibor
> > > Meller
> > > METRON-2012Done

Re: [DISCUSS] Next Release

[zeo...@gmail.com]

On a related note, I would love to get 0.2 and 0.3 versions for
metron-bro-plugin-kafka so we can clean up the JIRAs.

Currently the metron-bro-plugin-kafka plugin uses major.minor versioning
but once zeek 3.0 is released we plan to align with the metron project and
do major.minor.patch versioning.  When that happens, are we okay with
sharing version numbers?

- Jon Zeolla
zeo...@gmail.com


On Tue, Apr 23, 2019 at 1:42 PM Justin Leet  wrote:

> Absolutely. It'll probably be tomorrow before that gets into full swing.
>
> I don't believe we have a "0.7.1" release in Jira, and I (oddly enough)
> don't believe I have permissions to create it.  We'll need someone to get
> that created (James, maybe?).
>
> Until then, I'd like to ask everyone to make sure Jira is up to date
> (particularly with assignee, release version (or at least next+1 which is
> easy to find), along with status). Typically, I've taken care of getting
> Jira up to date in the last couple releases, but it would be helpful if
> everyone took a few minutes to update their own tickets, so I'm just doing
> touch-ups (and updating my own tickets).  If I can grab some time, I'll try
> to get a list of tickets to update to the appropriate contributors.
>
> Thanks,
> Justin
>
>
>
> On Tue, Apr 23, 2019, 09:54 Nick Allen  wrote:
>
> > Justin - Can you kick-off the release process?
> >
> > On Wed, Apr 17, 2019 at 11:51 AM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > I don't think it should hold up the release. It's *extremely* rare,
> even
> > > though it's come up twice. I wanted to be preemptive about bringing up
> > the
> > > intermittent failure list because we agreed as a community on this
> being
> > a
> > > regular release cycle review step. I think all those voting on the
> > release
> > > should have a full awareness of issues seen and vote accordingly as to
> > > whether we should move forward or address a problem first. Having
> > reviewed
> > > the test-failures list, I'm in favor of moving forward with the
> release.
> > >
> > > On Wed, Apr 17, 2019 at 9:29 AM Nick Allen  wrote:
> > >
> > > > Oh, and that failure from 21 days ago was caused by a downstream
> issue
> > > > brought in by PR #1364, that we then later reverted [1].  So at least
> > > that
> > > > particular issue has been identified and addressed and should no
> longer
> > > > impacts builds.
> > > >
> > > > ---
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/metron/commit/cad679a5948ce0ee9866e61bbf75b1f5f255682c
> > > >
> > > > On Wed, Apr 17, 2019 at 11:25 AM Nick Allen 
> > wrote:
> > > >
> > > > > Are you suggesting the release should wait on this?  Personally, I
> > > don't
> > > > > feel that we have any *persistently intermittent* test failures
> that
> > > > would
> > > > > block a release.  The last build failure on master I see was from
> 21
> > > days
> > > > > ago.
> > > > >
> > > > > Otherwise, I'd really like to kick off the next release.
> > > > >
> > > > > On Tue, Apr 16, 2019 at 11:25 AM Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > >> FYI, I just saw one of our reported intermittent test failures pop
> > up
> > > > >> again
> > > > >> today - https://issues.apache.org/jira/browse/METRON-1814
> > > > >>
> > > > >> On Mon, Apr 15, 2019 at 2:22 PM Michael Miklavcic <
> > > > >> michael.miklav...@gmail.com> wrote:
> > > > >>
> > > > >> > I want to thread the needle on this. I just reviewed a PR from
> > Ryan
> > > > that
> > > > >> > addresses this and gave it a +1.
> > > > >> > https://github.com/apache/metron/pull/1381
> > > > >> >
> > > > >> > I think Jon Zeolla and Justin Leet should have an opportunity to
> > > chime
> > > > >> in
> > > > >> > as they also had commented about this request in the original
> PR.
> > > > >> >
> > > > >> > One other thing to note - we had agreed last time around to do a
> > > scan
> > > > >> ove

Re: [DISCUSS] Format for sharing architecture source files and rendered images

[zeo...@gmail.com]

I'm also partial to draw.io.

Jon Zeolla

On Wed, Apr 17, 2019, 9:48 PM Otto Fowler  wrote:

> Also, the section should either have a blurb and like for draw.io or a
> reference footnote etc.
>
>
> On April 17, 2019 at 21:36:03, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I think we should try draw.io, and should add ( with vote i think ) a new
> section to the developer documention like:
> Architecture Diagrams
>
> The project supports the XXX format along with accompanying image
> version in  format for architectural diagrams.
>
> Architectural diagrams are important because reasons, and while they may
> not be required for a PR with architectural implications, they may be
> requested as part of the review process when deemed prudent.
>
> —
>
> Along with that, we can discuss how to store them in git and have them
> integrated with the documentation *and* the website.
>
>
>
>
> On April 16, 2019 at 11:57:46, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hi devs,
>
> We've talked about this fairly ad hoc on PRs before, and I thought it worth
> bringing up more formally. Architecture diagrams are a strategically useful
> addition to the core developer documentation. Examples include:
>
> 1.
>
> https://github.com/apache/metron/tree/master/metron-platform/metron-parsing#parser-architecture
> 2.
>
> https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment/metron-enrichment-storm#enrichment-architecture
> 3.
>
> https://github.com/apache/metron/tree/master/metron-platform/metron-job#job-state-statechart
> 4.
>
> https://github.com/apache/metron/tree/master/metron-interface#knox-requestresponse-flow
>
> We currently have a couple formats for maintaining the source diagram files
> across the project as well as a few diagrams with no source files. I've
> personally used draw.io in the past because it's publicly available and
> free to use. I also see Powerpoint.
>
> 1. No source file -
>
> https://github.com/apache/metron/blob/master/metron-analytics/metron-maas-service/maas_arch.png
> 2. draw.io source -
>
> https://github.com/apache/metron/blob/master/metron-platform/metron-job/metron-job_state_statechart_diagram.xml
> 3. powerpoint -
>
> https://github.com/apache/metron/blob/master/metron-interface/flow_diagrams.pptx
>
> I'm interested in what the developer community has access to and would
> prefer. As I mentioned before, I'm partial to draw.io if only because it
> does not require a fee to use. PowerPoint is probably easier to use, but it
> does limit the contributors who can edit the images. Are there other
> options that people have used they would prefer over any of the above? Are
> we still interested in maintaining the source files (my personal vote is
> yes)?
>
> Best,
> Mike
>

Re: Problems with Dev deployment.

[zeo...@gmail.com]

Wow, I didn't realize that never got finalized/merged.  Looks like there is
a failure in travis on that PR, if you get that wrapped up I would think we
should take another look at that and maybe get it merged.  It has been a
while but I recall I was pretty happy with it after my review cycle with
you.

- Jon Zeolla
zeo...@gmail.com


On Wed, Apr 10, 2019 at 9:17 AM Otto Fowler  wrote:

> These issues are the reason https://github.com/apache/metron/pull/1261 was
> done.  It would be nice if we could get by them.
>
>
> On April 10, 2019 at 08:13:04, Dale Richardson (tigerqu...@outlook.com)
> wrote:
>
> Older pre-req versions are mentioned at:
>
>
> https://metron.apache.org/current-book/metron-deployment/vagrant/codelab-platform/index.html
> Metron – Developer Image for Apache Metron on Virtualbox<
>
> https://metron.apache.org/current-book/metron-deployment/vagrant/codelab-platform/index.html
> >
>
> Developer Image for Apache Metron on Virtualbox. This image is a fully
> functional Metron installation that has been pre-loaded with Ambari, HDP
> and Metron.
> metron.apache.org
>
>
> https://metron.apache.org/current-book/metron-deployment/vagrant/full-dev-platform/index.html
> Metron – Full Development Platform<
>
> https://metron.apache.org/current-book/metron-deployment/vagrant/full-dev-platform/index.html
> >
>
> Full Development Platform. This project fully automates the provisioning
> and deployment of Apache Metron and all necessary prerequisites on a
> single, virtualized host running on Virtualbox.
> metron.apache.org
>
>
>
> https://metron.apache.org/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
> Metron – Quick Development Platform<
>
> https://metron.apache.org/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
> >
>
> Quick Development Platform. This project fully automates the provisioning
> and deployment of Apache Metron and all necessary prerequisites on a
> single, virtualized host running on Virtualbox.
> metron.apache.org
>
>
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
>
> Metron Install on Ubuntu/Debian single-node VM with Vagrant and Ambari -
> Metron - Apache Software Foundation - cwiki.apache.org<
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548>
> Contributed by Umesh Kaushik < umesh.kaus...@bhujang.net > Introduction.
> These instructions are for an Ubuntu host, with occasional comments about
> how to do similar tasks in CentOS.
> cwiki.apache.org
>
> Which links to
> https://gist.github.com/dpalomar/96b826dac5c2e8b62cbf4c86dbd1c9df
>
>
>
> 
> From: Michael Miklavcic 
> Sent: Wednesday, 10 April 2019 4:15 AM
> To: dev@metron.apache.org
> Subject: Re: Problems with Dev deployment.
>
> Where are you seeing 2.0.0.2 for Ansible? Should be 2.4.0+ now.
>
> https://github.com/apache/metron/blob/master/metron-deployment/development/centos6/README.md
>
>
>
> On Tue, Apr 9, 2019, 6:59 PM Michael Miklavcic <
> michael.miklav...@gmail.com>
>
> wrote:
>
> > That would be awesome man! Yes, Jira tickets for every change. I don't
> > recall off the top of my head the version - I'll have to look when I get
> > back in front of a computer.
> >
> > On Tue, Apr 9, 2019, 6:56 PM Dale Richardson 
> > wrote:
> >
> >> Hi Michael,
> >> Yep that was the issue, looks like "brew install ansible" configures
> >> ansible for python 3. Happy to patch the documentation with updated
> >> install instructions (I will add the MacOS Mojave Drive permission issue
> >> as well). Do you know if the minimum ansible version requirement has
> >> changed at all? (some documentation lists it as 2.0.0.2 or 2.2.2.0). Do
> >> you guys usually put in a Jira to cover small documentation changes at
> all?
> >>
> >> Regards,
> >> Dale.
> >> 
> >> From: Otto Fowler 
> >> Sent: Sunday, 7 April 2019 1:40 PM
> >> To: dev@metron.apache.org
> >> Subject: Re: Problems with Dev deployment.
> >>
> >> Can you pull down
> >>
>
> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fmetron%2Fpull%2F1261&data=02%7C01%7C%7Cc00c98c5300d45adf8c508d6bbab3180%7C84df9e7fe9f640afb435%7C1%7C0%7C636902741276312736&sdata=oylL4njlCkTYaLsyLYl44Zy8%2BoZoB4HVBCb5F2vgxng%3D&reserved=0
> >> and try?
> >> That should eliminate any env. issues.
> >>
> >> I’ll update it to latest master now.
> >&g

Re: [DISCUSS] Next Release

[zeo...@gmail.com]

Isn't the documentation already in progress?

https://github.com/apache/metron/pull/1330#issuecomment-466453372

If not I would still consider it important to complete prior to a release
and I agree with Justin's comments in

https://lists.apache.org/thread.html/50b89b919bd8bef3f7fcdef167cbd7e489fa74a1e2da3e4fddb08b13@


Jon Zeolla

On Thu, Mar 28, 2019, 2:16 PM Michael Miklavcic 
wrote:

> Jon and Ryan - this was a convo/negotiation between you two at the time.
> Any thoughts?
>
> On Thu, Mar 28, 2019 at 9:08 AM Nick Allen  wrote:
>
> > Is anyone volunteering to take this on?  Would be nice to get a release
> > out.
> >
> > On Thu, Mar 14, 2019, 4:53 PM zeo...@gmail.com  wrote:
> >
> > > We should likely get METRON-2014 in, based on
> > >
> > >
> >
> https://lists.apache.org/thread.html/13bd0ed5606ad4f3427f24a8e759d6bcb61ace76d4afcc9f48310a00@%3Cdev.metron.apache.org%3E
> > >
> > > On Thu, Mar 14, 2019 at 4:24 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Ticket is now done and merged. I'm also good on 0.7.1.
> > > >
> > > > On Thu, Mar 14, 2019 at 2:18 PM Justin Leet 
> > > wrote:
> > > >
> > > > > I'm in favor doing a release, pending the ticket Mike pointed out
> > (and
> > > > > anything else someone comes up with).
> > > > >
> > > > > To the best of my knowledge, I think 0.7.1 is sufficient, but if
> > > someone
> > > > > comes up with something, it's not hard to pivot.
> > > > >
> > > > > On Wed, Mar 13, 2019, 13:08 Michael Miklavcic <
> > > > michael.miklav...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I'd like to see this fixed for the next release.
> > > > > > https://issues.apache.org/jira/browse/METRON-2036. Even though
> > it's
> > > a
> > > > > > non-prod issue, this is a core part of our
> > infrastructure/development
> > > > > > lifecycle that is currently broken and fits with our previous
> > > > agreements
> > > > > of
> > > > > > holding a release until all intermittent test failures are
> > addressed.
> > > > > >
> > > > > > On Wed, Mar 13, 2019 at 11:33 AM Nick Allen 
> > > > wrote:
> > > > > >
> > > > > > > I would like to open a discussion in regards to the next
> release.
> > > Our
> > > > > > last
> > > > > > > 0.7.0 release was on Dec 11th.
> > > > > > >
> > > > > > > I believe we have a significant number of bug fixes and
> > performance
> > > > > > > improvements that would make a worthy point release; 0.7.1.
> > > > Although,
> > > > > we
> > > > > > > should review the change log and see if there are any breaking
> > > > changes
> > > > > > that
> > > > > > > would require a bump to the minor version.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > $ git log --format=%B  tags/apache-metron_0.7.0-release..HEAD |
> > > grep
> > > > > > > METRON-
> > > > > > > Merge remote-tracking branch 'apache/master' into METRON-2035
> > > > > > > METRON-2035 Allow User to Configure Role Names for Access
> Control
> > > > > > > METRON-2030 SensorParserGroupControllerIntegrationTest
> > intermittent
> > > > > > errors
> > > > > > > (merrimanr via mmiklavc) closes apache/metron#1352
> > > > > > > METRON-2031 [UI] Turning off initial search request and polling
> > by
> > > > > > default
> > > > > > > on Alerts UI (tiborm via mmiklavc) closes apache/metron#1353
> > > > > > > METRON-2012 Unable to Execute Stellar Functions Against HBase
> in
> > > the
> > > > > REPL
> > > > > > > (nickwallen) closes apache/metron#1345
> > > > > > > METRON-1971 Short timeout value in Cypress may cause build
> > failures
> > > > > > > (sardell) closes apache/metron#1323
> > > > > > > METRON-1940 Check if not and install Elastic search templates /
> > > Solr
> > > 

Re: [DISCUSS] Next Release

[zeo...@gmail.com]

We should likely get METRON-2014 in, based on
https://lists.apache.org/thread.html/13bd0ed5606ad4f3427f24a8e759d6bcb61ace76d4afcc9f48310a00@%3Cdev.metron.apache.org%3E

On Thu, Mar 14, 2019 at 4:24 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Ticket is now done and merged. I'm also good on 0.7.1.
>
> On Thu, Mar 14, 2019 at 2:18 PM Justin Leet  wrote:
>
> > I'm in favor doing a release, pending the ticket Mike pointed out (and
> > anything else someone comes up with).
> >
> > To the best of my knowledge, I think 0.7.1 is sufficient, but if someone
> > comes up with something, it's not hard to pivot.
> >
> > On Wed, Mar 13, 2019, 13:08 Michael Miklavcic <
> michael.miklav...@gmail.com
> > >
> > wrote:
> >
> > > I'd like to see this fixed for the next release.
> > > https://issues.apache.org/jira/browse/METRON-2036. Even though it's a
> > > non-prod issue, this is a core part of our infrastructure/development
> > > lifecycle that is currently broken and fits with our previous
> agreements
> > of
> > > holding a release until all intermittent test failures are addressed.
> > >
> > > On Wed, Mar 13, 2019 at 11:33 AM Nick Allen 
> wrote:
> > >
> > > > I would like to open a discussion in regards to the next release. Our
> > > last
> > > > 0.7.0 release was on Dec 11th.
> > > >
> > > > I believe we have a significant number of bug fixes and performance
> > > > improvements that would make a worthy point release; 0.7.1.
> Although,
> > we
> > > > should review the change log and see if there are any breaking
> changes
> > > that
> > > > would require a bump to the minor version.
> > > >
> > > > Thoughts?
> > > >
> > > > $ git log --format=%B  tags/apache-metron_0.7.0-release..HEAD | grep
> > > > METRON-
> > > > Merge remote-tracking branch 'apache/master' into METRON-2035
> > > > METRON-2035 Allow User to Configure Role Names for Access Control
> > > > METRON-2030 SensorParserGroupControllerIntegrationTest intermittent
> > > errors
> > > > (merrimanr via mmiklavc) closes apache/metron#1352
> > > > METRON-2031 [UI] Turning off initial search request and polling by
> > > default
> > > > on Alerts UI (tiborm via mmiklavc) closes apache/metron#1353
> > > > METRON-2012 Unable to Execute Stellar Functions Against HBase in the
> > REPL
> > > > (nickwallen) closes apache/metron#1345
> > > > METRON-1971 Short timeout value in Cypress may cause build failures
> > > > (sardell) closes apache/metron#1323
> > > > METRON-1940 Check if not and install Elastic search templates / Solr
> > > > collections when indexing server is restarted (MohanDV) closes
> > > > apache/metron#1305
> > > > METRON-2019 Improve Metron REST Logging (merrimanr) closes
> > > > apache/metron#1347
> > > > METRON-2016 Parser aggregate groups should be persisted and available
> > > > through REST (merrimanr) closes apache/metron#1346
> > > > METRON-1987 Upgrade Alert UI to stable Bootstrap 4 (sardell) closes
> > > > apache/metron#1336
> > > > METRON-1968 Messages are lost when a parser produces multiple
> messages
> > > and
> > > > batch size is greater than 1 (merrimanr) closes apache/metron#1330
> > > > METRON-1778 Out-of-order timestamps may delay flush in Storm Profiler
> > > > (nickwallen) closes apache/metron#1197
> > > > METRON-1996 Solr search throws NPE for group search if the group
> > > parameter
> > > > is null or empty (MohanDV via nickwallen) closes apache/metron#1333
> > > > METRON-1944 Unable to Delete a Comment in Alerts UI (ruffle1986 via
> > > > sardell) closes apache/metron#1307
> > > > METRON-2010 Unable to Build Metron Due to Inaccessible Repository
> > > > (nickwallen) closes apache/metron#1343
> > > > METRON-1998 Only one sensor is flushed by tick tuple (merrimanr)
> closes
> > > > apache/metron#1335
> > > > METRON-2009 Address Javadoc checkstyle issues in metron-common
> > > (justinleet)
> > > > closes apache/metron#1342
> > > > METRON-2005 Batch Writer writes 0-byte files to HDFS on rotation
> > > > (justinleet) closes apache/metron#1338
> > > > METRON-2007 Management UI not loading grok statements correctly
> > > (merrimanr)
> > > > closes apache/metron#1340
> > > > METRON-1986 Batch Profiler Fails to Resolve Stats Stellar Functions
> > > > (nickwallen) closes apache/metron#1328
> > > > METRON-1993 Stellar REST_GET should handle responses when content
> > length
> > > is
> > > > less than zero (merrimanr) closes apache/metron#1331
> > > > METRON-1999 Adding validation against special characters to parser
> name
> > > > field (tiborm via sardell) closes apache/metron#1337
> > > > METRON-1985 Improve Error Handling When Cannot Connect to HBase
> > > > (nickwallen) closes apache/metron#1327
> > > > METRON-1974 Batch Profiler Should Handle Errant Profiles Better
> > > > (nickwallen) closes apache/metron#1326
> > > > METRON-1970 Add Metadata to Error Messages Generated During Parsing
> > > > (nickwallen) closes apache/metron#1325
> > > > METRON-1995 Arrow icon in date range selector moved to a wrong
> position
> > > >

Re: [DISCUSS] Central Navigation for Alerts and Management UI

[zeo...@gmail.com]

I use both screens frequently on prod clusters.  I don't know how prevalent
that use case is though.

Jon

On Mon, Mar 11, 2019 at 7:33 AM Shane Ardell 
wrote:

> Good point, Otto. Just posted there now.
>
> On Mon, Mar 11, 2019 at 12:11 PM Otto Fowler 
> wrote:
>
> > Maybe you should post to the users@ list
> >
> >
> > On March 11, 2019 at 06:25:48, Shane Ardell (shane.m.ard...@gmail.com)
> > wrote:
> >
> > Thank you both for explaining the original design choice. That does make
> > sense.
> >
> > However, like both of you point out, I am also curious if anyone has
> direct
> > user feedback indicating a need for either persona to switch between
> > screens.
> >
> > Shane
> >
> > On Tue, Mar 5, 2019 at 7:25 PM Rita McKissick <
> rmckiss...@hortonworks.com>
> > wrote:
> >
> > > That was my thought, too. The Management UI is meant for the Operations
> > > persona.
> > > And the Alerts UI is meant for the SOC analyst persona. If we see a
> need
> > > for either of these
> > > personas to use both of the UIs, then the ability to switch between the
> > > two UIs would
> > > be great. Otherwise, I'm not sure that ability is necessary.
> > >
> > > As an aside, as the tech writer I would love to be able to switch
> between
> > > the two UIs, but I'm not
> > > really one of our supported personas __ Darn!
> > >
> > > Rita
> > >
> > > Rita McKissick ! Sr. Technical Writer
> > > rmckiss...@hortonworks.com
> > > (mobile) 831-234-3676 <(831)%20234-3676>
> > >
> > >
> > > On 3/5/19, 9:50 AM, "Michael Miklavcic" 
> > > wrote:
> > >
> > > The original design was done with the intent to keep the user profiles
> > > (soc
> > > analyst vs ops personnel) separate and enable a microservices-oriented
> > > architecture. I don't have a strong opinion one way or the other, but
> > > I'd
> > > be interested to hear whether others in the community find this wall
> > > useful, or if we should come back to a single pain of glass.
> > >
> > > Mike
> > >
> > > On Tue, Mar 5, 2019 at 9:12 AM Shane Ardell 
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I recently started experimenting with implementing a navigation bar
> > > in both
> > > > the Alerts and Management UI. It would allow us to navigate between
> > > the two
> > > > UIs through links instead of manually entering a url or opening
> > > separate
> > > > tabs from Ambari.
> > > >
> > > > I'm just wondering what everyone's thoughts are. Is this something
> > > we want
> > > > in Metron?
> > > >
> > >
> > >
> > >
> >
>
-- 

Jon Zeolla

Re: [DISCUSS] Upgrading HBase and Kafka support

[zeo...@gmail.com]

So most importantly I want to make sure to give Otto credit for being the
one who cleaned up the rudimentary testing steps we had for testing the
plugin and turned it into the docker end to end.  Right now we manually run
the tests, as there were a few follow-ons we needed to work through before
it's ready for Travis.  In my opinion, once METRON-2003 (PR 26) gets in
it'll be ready to have Travis.  There isn't any current Maven use

Jon

On Fri, Mar 8, 2019 at 12:26 PM Otto Fowler  wrote:

> I believe that the TestContainers allows the ide case
>
>
> On March 8, 2019 at 11:38:24, Michael Miklavcic (
> michael.miklav...@gmail.com)
> wrote:
>
> I'm -1 on #1 unless there's some desperately compelling reason to go that
> route. It would be a regression in our test coverage, and at that point
> it's really just duplicating our unit tests as opposed to checking our
> integration.
>
> I'm good with 3. Gating factors for a successful implementation would be
> that as a developer I can:
>
> 1. Run it in my IDE without having to do anything extra (the beauty of
> the in-mem component is that @BeforeClass spins it up automatically - we
> should keep doing something along those lines)
> 2. Run it via Maven cli
> 3. Run it in Travis as part of our normal build
>
> It's probably worth looking at Kafka's testing infrastructure straight from
> the source - https://github.com/apache/kafka/blob/trunk/tests/README.md.
> They leverage Docker containers now for system tests.
>
> Best,
> Mike
>
>
> On Fri, Mar 8, 2019 at 7:47 AM Ryan Merriman  wrote:
>
> > I have been researching the effort involved to upgrade to HDP 3. Along
> the
> > way I've found a couple challenging issues that we will need to solve,
> both
> > involving our integration testing strategy.
> >
> > The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there
> > have been significant changes to the API. This creates an issue in the
> > KafkaComponent class, which we use as an in-memory Kafka server in
> > integration tests. Most of the classes that were previously used have
> gone
> > away, and to the best of my knowledge, were not supported as public APIs.
> > I also don't see any publicly documented APIs to replace them.
> >
> > The second issue is HBase. We are moving from 1.1.2 to 2.0.2 so another
> > significant change. This creates an issue in the MockHTable class
> > becausethe HTableInterface class has changed to Table, essentially
> > requiring that MockHTable be rewritten to conform to the new interface.
> > It's my opinion that this class is complicated and difficult to maintain
> as
> > it is anyways.
> >
> > These 2 issues have the potential to add a significant amount of work to
> > upgrading Metron to HDP 3. I want to take a step back and review our
> > options before we move forward. Here are some initial thoughts I had on
> > how to approach this. For HBase:
> >
> > 1. Update MockHTable to work with the new HBase API. We would continue
> > using a mock server approach for HBase.
> > 2. Research replacing MockHTable with an in-memory HBase server.
> > 3. Replace MockHTable with a Docker container running HBase.
> >
> > For Kafka:
> >
> > 1. Replace KafkaComponent with a mock server implementation.
> > 2. Update KafkaComponent to work with the new API. We would probably
> > need to leverage some internal Kafka classes. I do not see a testing
> > API
> > documented publicly.
> > 3. Replace KafkaComponent with a Docker container running Kafka.
> >
> > What other options are there? Whatever we choose I think we should follow
> > a similar approach for both (mock servers, in memory servers, Docker,
> other
> > options I'm not thinking of).
> >
> > This will not shock anyone but I would be in favor of Docker containers.
> > They have the advantage of classpath isolation, easy upgrades, and
> accurate
> > integration testing. The downside is we will have to adjusts our tests
> and
> > travis script to incorporate these Docker containers into our build
> > process. We have discussed this at length in the past and it has
> generally
> > stalled for various reasons. Maybe if we move a few services at a time it
> > might be more palatable? As for the other 2 approaches, I think if either
> > worked well we wouldn't be having this discussion. Mock servers are hard
> > to maintain and I don't see in memory testing classes documented in
> > javadocs for either service.
> >
> > Thoughts?
> >
>
-- 

Jon Zeolla

Re: [DISCUSS] Upgrading HBase and Kafka support

[zeo...@gmail.com]

+1 to option 3 on both.  Also strongly in favor of Docker.  We recently
took a similar approach in metron-bro-plugin-kafka as well (link
) to
do end to end testing.

Jon

On Fri, Mar 8, 2019 at 9:53 AM Nick Allen  wrote:

> +1 for option 3.  I am in favor of using Docker for the integration tests
> for all the reasons that you mentioned.
>
> On Fri, Mar 8, 2019 at 9:47 AM Ryan Merriman  wrote:
>
> > I have been researching the effort involved to upgrade to HDP 3.  Along
> the
> > way I've found a couple challenging issues that we will need to solve,
> both
> > involving our integration testing strategy.
> >
> > The first issue is Kafka.  We are moving from 0.10.0 to 2.0.0 and there
> > have been significant changes to the API.  This creates an issue in the
> > KafkaComponent class, which we use as an in-memory Kafka server in
> > integration tests.  Most of the classes that were previously used have
> gone
> > away, and to the best of my knowledge, were not supported as public APIs.
> > I also don't see any publicly documented APIs to replace them.
> >
> > The second issue is HBase.  We are moving from 1.1.2 to 2.0.2 so another
> > significant change.  This creates an issue in the MockHTable class
> > becausethe HTableInterface class has changed to Table, essentially
> > requiring that MockHTable be rewritten to conform to the new interface.
> > It's my opinion that this class is complicated and difficult to maintain
> as
> > it is anyways.
> >
> > These 2 issues have the potential to add a significant amount of work to
> > upgrading Metron to HDP 3.  I want to take a step back and review our
> > options before we move forward.  Here are some initial thoughts I had on
> > how to approach this.  For HBase:
> >
> >1. Update MockHTable to work with the new HBase API.  We would
> continue
> >using a mock server approach for HBase.
> >2. Research replacing MockHTable with an in-memory HBase server.
> >3. Replace MockHTable with a Docker container running HBase.
> >
> > For Kafka:
> >
> >1. Replace KafkaComponent with a mock server implementation.
> >2. Update KafkaComponent to work with the new API.  We would probably
> >need to leverage some internal Kafka classes.  I do not see a testing
> > API
> >documented publicly.
> >3. Replace KafkaComponent with a Docker container running Kafka.
> >
> > What other options are there?  Whatever we choose I think we should
> follow
> > a similar approach for both (mock servers, in memory servers, Docker,
> other
> > options I'm not thinking of).
> >
> > This will not shock anyone but I would be in favor of Docker containers.
> > They have the advantage of classpath isolation, easy upgrades, and
> accurate
> > integration testing.  The downside is we will have to adjusts our tests
> and
> > travis script to incorporate these Docker containers into our build
> > process.  We have discussed this at length in the past and it has
> generally
> > stalled for various reasons.  Maybe if we move a few services at a time
> it
> > might be more palatable?  As for the other 2 approaches, I think if
> either
> > worked well we wouldn't be having this discussion.  Mock servers are hard
> > to maintain and I don't see in memory testing classes documented in
> > javadocs for either service.
> >
> > Thoughts?
> >
>
-- 

Jon Zeolla

Re: [DISCUSS] Architecture documentation

[zeo...@gmail.com]

Sorry for the delay here.  Yup I'm good with where this ended up, thanks!

Jon

On Tue, Feb 26, 2019 at 10:21 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> @Jon - I think this DISCUSS thread is the last gating factor for getting
> this PR in, are you ok with the prescribed approach here? i.e. Ryan has
> already done a lot of clarification and expansion in the README's and code
> comments, and the follow-on will be creating a separate architecture
> document in Metron root that will go into the next release.
>
> Best,
> Mike
>
>
> On Mon, Feb 25, 2019 at 3:25 PM Justin Leet  wrote:
>
> > Re: Labeling in Jira, I'm fine with having a be "Next + 1" from a release
> > management perspective, but I'd still consider at least taking action on
> > followup to be the relevant party's responsibility (implementer or
> whatever
> > the case may be).  We probably should have a more clear way to tag things
> > like this, but I don't believe we do right now. If there's not a harder
> > dependency than my memory, chances are high it gets
> > overlooked/missed/whatever.
> >
> > On Mon, Feb 25, 2019 at 4:32 PM Otto Fowler 
> > wrote:
> >
> > > I really like the idea of architecture.md -> **/architecture.md.
> > >
> > > We overall do not have javadoc in a lot of areas, and could maybe start
> > > working on it as we go and think about asking for it in reviews.
> > > We are also missing the Parser Programmer’s Guide, how to add a parser
> to
> > > the metron system/install etc and other things.
> > >
> > >
> > >
> > > On February 25, 2019 at 15:22:47, Ryan Merriman (merrim...@gmail.com)
> > > wrote:
> > >
> > > I feel like the code itself is pretty well documented. I updated
> existing
> > > javadocs and added javadocs to classes that didn't have them before
> this
> > > PR. In my opinion the level of documentation for these classes has
> > > increased significantly.
> > >
> > > On Mon, Feb 25, 2019 at 1:52 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Tentatively agreed on further clarification of what we consider
> in/out
> > of
> > > > scope for documentation re: document something that wasn't documented
> > > > before. Ryan, can you give a quick summary of what you *have*
> > > added/updated
> > > > in documentation on this PR vs what you want to leave out?
> > > >
> > > > My initial concern in punting on docs right now is that part of what
> > made
> > > > this PR/task more challenging in the first place was not having
> > > > documentation. We risk losing context and detail again if we don't do
> > > this
> > > > immediately. Would it be reasonable to split it up as follows?:
> > > >
> > > > 1. Additional overarching documentation feels out of scope - make it
> a
> > > > follow on (see comments below).
> > > > 2. Adding documentation to our existing README's and java code
> comments
> > > > that describe the new/modified functionality should be in scope
> because
> > > > it's part of the unit of work. I expect that a developer should be
> able
> > > > to
> > > > look at the code, tests, comments, and README's and understand how
> this
> > > > code functions without having to start from scratch.
> > > >
> > > > The way we've handled follow-on work before, at least as far as
> feature
> > > > branches are concerned, was to create Jiras and link them to the
> > > > appropriate discussions for context. Maybe we can take that one step
> > > > further and do the release manager a favor by also labeling the
> > > > required/requested release on the Jira as a gating factor. This
> follows
> > > our
> > > > pattern for intermittent test failure reporting, e.g.
> > > >
> > > >
> > >
> > >
> >
> https://issues.apache.org/jira/browse/METRON-1946?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-failure%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > .
> > > >
> > > > I'm also in favor of continuing to document architecture and
> technical
> > > > details as part of the code base as Ryan and Jon have suggested. I
> > think
> > > we
> > > > should have an "architecture.md" in metron root that replaces this -
> > > >
> > > >
> > >
> > >
> >
> https://github.com/apache/metron/blob/d7d4fd9afb19e2bd2e66babb7e1514a19eae07d0/README.md#navigating-the-architecture
> > > > and covers the broad architecture with links to the appropriate
> modules
> > > for
> > > > detail. Minimally, it would be nice if we had a simple diagram
> showing
> > > the
> > > > basic flow of data in Metron. I think we probably want an updated
> > version
> > > > of this wiki entry from back in the day -
> > > >
> https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture
> > > >
> > > > Best,
> > > > Mike
> > > >
> > > >
> > > > On Mon, Feb 25, 2019 at 7:18 AM Nick Allen 
> wrote:
> > > >
> > > > > I don't think we should hold up this work to document something
> that
> > > > wasn't
> > > > > previously documented. A follow-on is sufficient.
> > > > >
> > 

Re: [DISCUSS] Architecture documentation

[zeo...@gmail.com]

I disagree, I would hold up a code change that has no documentation/no
plans for documentation.  It's also very possible that, if documentation
was to come later on a specific workstream, it could get forgotten.  It
kind of breaks the review process IMO unless there's a compensating
control, such as a JIRA with a specific tag and a reminder in the release
process to check for that.

Jon

On Mon, Feb 25, 2019 at 9:14 AM Nick Allen  wrote:

> > Procedurally, do we have a way to ensure that any follow-on
> documentation happens
> prior to a release (anything in the wiki, etc.)?
>
> If someone thinks the code base needs X before the next release, then they
> can bring up X during the release discussion.  We don't need additional
> procedure around this.
>
> On Mon, Feb 25, 2019 at 9:11 AM zeo...@gmail.com  wrote:
>
> > I agree, I think all docs should be kept in the code base.  I
> > opened METRON-714 ages ago to get the existing cwiki docs over to READMEs
> > as well.
> >
> > I would also like to see us consider a more general/overview
> architecture,
> > or perhaps write each component's architecture in a way that it can be
> > composed into a higher level doc when generating the site-book.  Right
> now
> > the barrier to getting started with Metron is too high, and I think this
> > would make it slightly less imposing.  But this is slightly outside of
> the
> > scope of the current conversation.
> >
> > Procedurally, do we have a way to ensure that any follow-on documentation
> > happens prior to a release (anything in the wiki, etc.)?  I'm fine with
> > splitting the commits generally.
> >
> > Jon
> >
> > On Mon, Feb 25, 2019 at 8:50 AM Ryan Merriman 
> wrote:
> >
> > > Recently I submitted a PR <https://github.com/apache/metron/pull/1330>
> > > that
> > > introduces a large number of changes to a critical part of our code
> base.
> > > Reviewers feel like it is significant enough to document at an
> > > architectural level (and I agree).  There are a couple points I would
> > like
> > > to clarify.
> > >
> > > Generally architectural documentation lives in the README of the
> > > appropriate module.  Do we want to continue documenting architecture
> > here?
> > > I think it makes sense because it will be versioned along with the
> code.
> > > Just wanted to confirm there are no objections to continuing this
> > practice.
> > >
> > > A reviewer suggested we could accept the PR as is and leave the
> > > architectural documentation as a follow on.  I think this makes sense
> > > because it can be tedious to maintain a large PR as other smaller
> commits
> > > are accepted into master.  An important requirement is the
> documentation
> > > follow on must be completed in a timely manner, before the next
> release.
> > > Are there any objections to doing it this way?
> > >
> > --
> >
> > Jon Zeolla
> >
>
-- 

Jon Zeolla

Re: [DISCUSS] Architecture documentation

[zeo...@gmail.com]

I agree, I think all docs should be kept in the code base.  I
opened METRON-714 ages ago to get the existing cwiki docs over to READMEs
as well.

I would also like to see us consider a more general/overview architecture,
or perhaps write each component's architecture in a way that it can be
composed into a higher level doc when generating the site-book.  Right now
the barrier to getting started with Metron is too high, and I think this
would make it slightly less imposing.  But this is slightly outside of the
scope of the current conversation.

Procedurally, do we have a way to ensure that any follow-on documentation
happens prior to a release (anything in the wiki, etc.)?  I'm fine with
splitting the commits generally.

Jon

On Mon, Feb 25, 2019 at 8:50 AM Ryan Merriman  wrote:

> Recently I submitted a PR 
> that
> introduces a large number of changes to a critical part of our code base.
> Reviewers feel like it is significant enough to document at an
> architectural level (and I agree).  There are a couple points I would like
> to clarify.
>
> Generally architectural documentation lives in the README of the
> appropriate module.  Do we want to continue documenting architecture here?
> I think it makes sense because it will be versioned along with the code.
> Just wanted to confirm there are no objections to continuing this practice.
>
> A reviewer suggested we could accept the PR as is and leave the
> architectural documentation as a follow on.  I think this makes sense
> because it can be tedious to maintain a large PR as other smaller commits
> are accepted into master.  An important requirement is the documentation
> follow on must be completed in a timely manner, before the next release.
> Are there any objections to doing it this way?
>
-- 

Jon Zeolla

Metron REST w/o LDAP

[zeo...@gmail.com]

Is it intended that we require METRON_LDAP_PASSWORD when LDAP isn't in use
to start metron-rest?

```
[metroniso@server ~]$ export METRON_LDAP_PASSWORD=anything
[metroniso@server ~]$ /usr/metron/0.7.0/bin/metron-rest.sh
[metroniso@server ~]$ tail -f /var/log/metron/metron-rest.log
# No error
```

```
[metroniso@server ~]$ unset METRON_LDAP_PASSWORD
[metroniso@server ~]$ /usr/metron/0.7.0/bin/metron-rest.sh >/dev/null
[metroniso@server ~]$ tail -f /var/log/metron/metron-rest.log
19/01/26 21:27:03 WARN
context.AnnotationConfigServletWebServerApplicationContext: Exception
encountered during context initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'webSecurityConfig': Injection of autowired dependencies
failed; nested exception is java.lang.IllegalArgumentException: Could not
resolve placeholder 'METRON_LDAP_PASSWORD' in value
"${METRON_LDAP_PASSWORD}"
19/01/26 21:27:03 INFO jpa.LocalContainerEntityManagerFactoryBean: Closing
JPA EntityManagerFactory for persistence unit 'default'
19/01/26 21:27:03 INFO hikari.HikariDataSource: HikariPool-1 - Shutdown
initiated...
19/01/26 21:27:03 INFO hikari.HikariDataSource: HikariPool-1 - Shutdown
completed.
Jan 26, 2019 9:27:03 PM org.apache.catalina.core.StandardService
stopInternal
INFO: Stopping service [Tomcat]
19/01/26 21:27:03 INFO logging.ConditionEvaluationReportLoggingListener:

Error starting ApplicationContext. To display the conditions report re-run
your application with 'debug' enabled.
19/01/26 21:27:03 ERROR boot.SpringApplication: Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'webSecurityConfig': Injection of autowired dependencies
failed; nested exception is java.lang.IllegalArgumentException: Could not
resolve placeholder 'METRON_LDAP_PASSWORD' in value
"${METRON_LDAP_PASSWORD}"
at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:379)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1344)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:578)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
at
org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:760)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at
org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
at
org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
at
org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1255)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1243)
at
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:36)
Caused by: java.lang.IllegalArgumentException: Could not resolve
placeholder 'METRON_LDAP_PASSWORD' in value "${METRON_LDAP_PASSWORD}"
at
org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:172)
at
org.springframework.util.PropertyPlaceholderHelper.replacePlaceholders(PropertyPlaceholderHelper.java:124)
at
org.springframework.core.env.AbstractPropertyResolver.doResolvePlaceholders(AbstractPropertyResolver.java:237)
at
org.springframework.core.env.AbstractPropertyResolver.resolveRequiredPlaceholders(AbstractPropertyResolver.java:211)
at
org.springframework.core.env.AbstractPropertyResolver.resolveNestedPlaceholders(AbstractPropertyResolver.java:228)
at
org.springframework.core.env.PropertySourcesPropertyResolver.getProperty(PropertySourcesPropertyResolver.java:88)
at
org.springframework.core.env.PropertySourcesPropertyResolver.getProperty(PropertySourcesPropertyResolver.java:62)
at
org.springframework.core.env.AbstractEnvironment.getProperty(AbstractEnvironment.java:531)
at
org.springframework.context.su

Re: [DISCUSS] Writer class refactor

[zeo...@gmail.com]

Totally on board with everybody's comments above this point.

Jon

On Fri, Jan 18, 2019, 6:07 PM Michael Miklavcic 
wrote:

> Thanks for the write up, Ryan. I had to touch on some of this when
> refactoring the kafka writer away from the async model so we could
> guarantee delivery. We had potential to drop messages before that change
> because of the async producer calls, which would ack the Storm tuple as
> soon as the writer returned.
>
>- https://github.com/apache/metron/pull/1045
>
> We'll want to talk about these fixes/updates in context of our message
> delivery semantics, both in Storm and Kafka. As it currently stands, we do
> NOT use Storm Trident, which means we have at-least-once message processing
> in Storm. There is an inherent possibility that we will publish duplicate
> messages in some instances. From a Kafka perspective, we have the same
> issue. As of Kafka 0.11.0, they provide a way to get exactly-once
> semantics, but I'm not sure we've done much to explicitly achieve that.
>
>- https://kafka.apache.org/10/documentation.html#semantics
>
> From a Kafka delivery guarantee perspective, it appears we're currently
> setting # required acks to 1 by default. This means we get commit
> confirmation as soon as the leader has written the message to its local
> log. In this case should the leader fail immediately after acknowledging
> the record but before the followers have replicated it then the record will
> be lost. We could investigate settings acks=all or acks=-1, but this would
> be a tradeoff in performance for us.
>
>-
>
> https://github.com/apache/metron/blob/341960b91f8fe742d5cf947633b7edd2275587d5/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java#L87
>- https://kafka.apache.org/10/documentation/#producerconfigs
>
> Per the KafkaProducer documentation, the flush() command will wait until
> all messages are batched and sent, and will return with either success
> (acked) or an error. "A request is considered completed when it is
> successfully acknowledged according to the acks configuration you have
> specified or else it results in an error."
>
>-
>
> https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
>
> With this combination of factors, I believe we can continue to guarantee
> at-least-once semantics in the writer, regardless of batch size. To your
> point about not passing 2 separate lists, I suggest that we modify the API
> by passing in something like Map> so that the
> tuples always get acked with respect to their messages. This way we can
> avoid the tuple-message batch boundary problem by ensuring we only ack a
> tuple when all associated messages are successfully written to Kafka.
>
> Best,
> Mike
>
>
> On Fri, Jan 18, 2019 at 1:31 PM Otto Fowler 
> wrote:
>
> > Agreed
> >
> >
> > On January 18, 2019 at 14:52:32, Ryan Merriman (merrim...@gmail.com)
> > wrote:
> >
> > I am on board with that. In that case, I think it's even more important
> > that we get the Writer interfaces right.
> >
> > On Fri, Jan 18, 2019 at 1:34 PM Otto Fowler 
> > wrote:
> >
> > > I think that the writers should be loaded as, and act as extension
> > points,
> > > such that it is possible to have 3rd party writers, and would structure
> > > them as such.
> > >
> > >
> > >
> > > On January 18, 2019 at 13:55:00, Ryan Merriman (merrim...@gmail.com)
> > > wrote:
> > >
> > > Recently there was a bug reported by a user where a parser that emits
> > > multiple messages from a single tuple doesn't work correctly:
> > > https://issues.apache.org/jira/browse/METRON-1968. This has exposed a
> > > problem with how the writer classes work.
> > >
> > > The fundamental issue is this: the writer classes operate under the
> > > assumption that there is a 1 to 1 mapping between tuples and messages
> to
> > > be
> > > written. A couple of examples:
> > >
> > > KafkaWriter
> > > <
> > >
> >
> >
> https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java#L236
> > >
> >
> > >
> > > -
> > > This class writes messages by iterating through the list of tuples and
> > > fetching the message with the same index. This is the cause of the Jira
> > > above. We could iterate through the message list instead but then we
> > don't
> > > know which tuples have been fully processed. It would be possible for a
> > > batch to be flushed before all messages from a tuple are passed to the
> > > writer.
> > >
> > > BulkWriterComponent
> > > <
> > >
> >
> >
> https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java#L250
> > >
> >
> > >
> > > - The tuple list size is used to determine when a batch should be
> > flushed.
> > > While inherently incorrect in my opinion (should be message list size),
> > > this also causes an issue where only the first message from the last
> > tuple
> > > in

[DISCUSS] Clarify development guidelines

[zeo...@gmail.com]

I was looking at picking up a JIRA which could apply to both apache/metron
and apache/metron-bro-plugin-kafka (upgrade to bro 2.6.1/latest).  It made
me take another look at our dev guidelines to see if they are explicit
about having one JIRA per PR (it doesn't).  Is this something we should
do?  My off the cuff language would be somethign like "JIRA Issues must
explicitly map 1:1 with PRs, even if the same work is being applied across
multiple repositories"?

https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
-- 

Jon Zeolla

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

I agree, I would suggest moving forward with cutting an apache/metron RC.

Jon

On Wed, Dec 5, 2018 at 3:45 PM Nick Allen  wrote:

> I would prefer to just go ahead with the release and not wait on an
> intermittent, integration test related JIRAs.  Just wanting to see if there
> is support for getting a RC out sooner rather than later.
>
> On Tue, Dec 4, 2018 at 4:06 PM zeo...@gmail.com  wrote:
>
> > I agree that we should move forward with the apache/metron 0.7.0 release.
> > If 0.3 gets finalized in time we can include it, but otherwise no big
> deal
> > not including it since the dev environment points to 0.1 which didn't
> have
> > the issue.
> >
> > Jon
> >
> > On Mon, Dec 3, 2018 at 5:09 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > I have one more intermittent failure to add to the list from a timeout
> in
> > > the profiler integration tests.
> > > https://issues.apache.org/jira/browse/METRON-1918
> > >
> > >
> > > On Mon, Dec 3, 2018 at 2:54 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > fwiw, I have not been able to reproduce the integration test failure
> > that
> > > > I logged here - https://issues.apache.org/jira/browse/METRON-1851.
> > > Unless
> > > > anyone else has seen this, either locally or in Travis, I recommend
> we
> > > > close it out as unable to reproduce. If it does ever show up again,
> the
> > > > closed Jira will be out there as a record of it.
> > > >
> > > > On Mon, Dec 3, 2018 at 2:29 PM Justin Leet 
> > > wrote:
> > > >
> > > >> I'm inclined to do move forward with the core repo release. Having
> > said
> > > >> that, there's a few test bugs and such I'd like to see addressed,
> > either
> > > >> "won't fix" or preferably with PRs, before creating an RC (as noted
> > > >> earlier
> > > >> in the thread).  It's probably a good opportunity to ask again if
> > > there's
> > > >> anything outstanding we'd like to see in the release. Is there
> > anything
> > > >> we'd like taken care of that's relatively close to going in?
> > > >>
> > > >> If the plugin gets fixed before we're set to move forward with a
> core
> > > >> release (or we choose not to fix it, given the current version is
> > > >> affected), I'm happy to put out a new RC.
> > > >>
> > > >> On Mon, Dec 3, 2018 at 4:12 PM Michael Miklavcic <
> > > >> michael.miklav...@gmail.com> wrote:
> > > >>
> > > >> > +1 Nick
> > > >> >
> > > >> > On Mon, Dec 3, 2018 at 2:04 PM Nick Allen 
> > wrote:
> > > >> >
> > > >> > > OK, well either way, I see no need to hold up Metron 0.6.1.
> > > >> > >
> > > >> > > On Mon, Dec 3, 2018 at 3:51 PM zeo...@gmail.com <
> zeo...@gmail.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > > > I believe that 0.2.0 is impacted by the bug.
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > > On Mon, Dec 3, 2018 at 3:50 PM Nick Allen  >
> > > >> wrote:
> > > >> > > >
> > > >> > > > > In light of this comment [1], I propose that we move forward
> > > with
> > > >> > > another
> > > >> > > > > Metron release and forgo the Metron Bro Plugin 0.3.0 release
> > > >> until we
> > > >> > > can
> > > >> > > > > resolve METRON-1910 [2].  There is no need to rush the fix
> as
> > > the
> > > >> > > current
> > > >> > > > > 0.2.0 release of the Bro Plugin is not impacted by the bug.
> We
> > > do
> > > >> > have
> > > >> > > a
> > > >> > > > > good amount of other Metron functionality to release though.
> > I
> > > do
> > > >> > not
> > > >> > > > see
> > > >> > > > > a need to hold-up the release.
> > > >> > > > >
> > > >> > > > > ---
> > > >

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

I agree that we should move forward with the apache/metron 0.7.0 release.
If 0.3 gets finalized in time we can include it, but otherwise no big deal
not including it since the dev environment points to 0.1 which didn't have
the issue.

Jon

On Mon, Dec 3, 2018 at 5:09 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I have one more intermittent failure to add to the list from a timeout in
> the profiler integration tests.
> https://issues.apache.org/jira/browse/METRON-1918
>
>
> On Mon, Dec 3, 2018 at 2:54 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > fwiw, I have not been able to reproduce the integration test failure that
> > I logged here - https://issues.apache.org/jira/browse/METRON-1851.
> Unless
> > anyone else has seen this, either locally or in Travis, I recommend we
> > close it out as unable to reproduce. If it does ever show up again, the
> > closed Jira will be out there as a record of it.
> >
> > On Mon, Dec 3, 2018 at 2:29 PM Justin Leet 
> wrote:
> >
> >> I'm inclined to do move forward with the core repo release. Having said
> >> that, there's a few test bugs and such I'd like to see addressed, either
> >> "won't fix" or preferably with PRs, before creating an RC (as noted
> >> earlier
> >> in the thread).  It's probably a good opportunity to ask again if
> there's
> >> anything outstanding we'd like to see in the release. Is there anything
> >> we'd like taken care of that's relatively close to going in?
> >>
> >> If the plugin gets fixed before we're set to move forward with a core
> >> release (or we choose not to fix it, given the current version is
> >> affected), I'm happy to put out a new RC.
> >>
> >> On Mon, Dec 3, 2018 at 4:12 PM Michael Miklavcic <
> >> michael.miklav...@gmail.com> wrote:
> >>
> >> > +1 Nick
> >> >
> >> > On Mon, Dec 3, 2018 at 2:04 PM Nick Allen  wrote:
> >> >
> >> > > OK, well either way, I see no need to hold up Metron 0.6.1.
> >> > >
> >> > > On Mon, Dec 3, 2018 at 3:51 PM zeo...@gmail.com 
> >> > wrote:
> >> > >
> >> > > > I believe that 0.2.0 is impacted by the bug.
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > On Mon, Dec 3, 2018 at 3:50 PM Nick Allen 
> >> wrote:
> >> > > >
> >> > > > > In light of this comment [1], I propose that we move forward
> with
> >> > > another
> >> > > > > Metron release and forgo the Metron Bro Plugin 0.3.0 release
> >> until we
> >> > > can
> >> > > > > resolve METRON-1910 [2].  There is no need to rush the fix as
> the
> >> > > current
> >> > > > > 0.2.0 release of the Bro Plugin is not impacted by the bug. We
> do
> >> > have
> >> > > a
> >> > > > > good amount of other Metron functionality to release though.  I
> do
> >> > not
> >> > > > see
> >> > > > > a need to hold-up the release.
> >> > > > >
> >> > > > > ---
> >> > > > >
> >> > > > > [1]
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/metron-bro-plugin-kafka/pull/20#issuecomment-443481440
> >> > > > > [2] https://github.com/apache/metron-bro-plugin-kafka/pull/20
> >> > > > >
> >> > > > > On Thu, Nov 29, 2018 at 10:29 AM Justin Leet <
> >> justinjl...@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > There's a few issues I would like to see at least triaged and
> >> > > > preferably
> >> > > > > > addressed prior to the release of the main repo. In Jira, we
> >> have a
> >> > > > > > "test-failures" label, that has a few things attached to it.
> If
> >> we
> >> > > know
> >> > > > > of
> >> > > > > > any other Jiras that should have this label attached, please
> do
> >> so
> >> > > and
> >> > > > > I'd
> >> > > > > > appreciate it if you replied to the

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

I believe that 0.2.0 is impacted by the bug.

Jon

On Mon, Dec 3, 2018 at 3:50 PM Nick Allen  wrote:

> In light of this comment [1], I propose that we move forward with another
> Metron release and forgo the Metron Bro Plugin 0.3.0 release until we can
> resolve METRON-1910 [2].  There is no need to rush the fix as the current
> 0.2.0 release of the Bro Plugin is not impacted by the bug. We do have a
> good amount of other Metron functionality to release though.  I do not see
> a need to hold-up the release.
>
> ---
>
> [1]
>
> https://github.com/apache/metron-bro-plugin-kafka/pull/20#issuecomment-443481440
> [2] https://github.com/apache/metron-bro-plugin-kafka/pull/20
>
> On Thu, Nov 29, 2018 at 10:29 AM Justin Leet 
> wrote:
>
> > There's a few issues I would like to see at least triaged and preferably
> > addressed prior to the release of the main repo. In Jira, we have a
> > "test-failures" label, that has a few things attached to it. If we know
> of
> > any other Jiras that should have this label attached, please do so and
> I'd
> > appreciate it if you replied to the thread.  See test-failures
> > <
> >
> https://issues.apache.org/jira/browse/METRON-1851?jql=project%20%3D%20METRON%20AND%20labels%20%3D%20test-failure
> > >
> > .
> >
> > The Jiras are:
> > METRON-1810 <https://issues.apache.org/jira/browse/METRON-1810>
> > METRON-1814 <https://issues.apache.org/jira/browse/METRON-1814>
> > METRON-1851 <https://issues.apache.org/jira/browse/METRON-1851>
> >
> > On Wed, Nov 21, 2018 at 2:20 PM zeo...@gmail.com 
> wrote:
> >
> > > A metron-bro-plugin-kafka 0.3 release is good to go from my side.
> Thanks
> > > for all of the reviews Nick
> > >
> > > On Wed, Nov 21, 2018 at 11:16 AM Nick Allen 
> wrote:
> > >
> > > > Ha.  Yes, that definitely counts and makes a ton of sense.  Thanks!
> > > >
> > > > On Wed, Nov 21, 2018 at 11:00 AM Justin Leet 
> > > > wrote:
> > > >
> > > > > Does "I forgot to pull master fresh before running the command"
> count
> > > as
> > > > a
> > > > > reason?
> > > > >
> > > > > The missing Jiras are:
> > > > >
> > > > > METRON-1890 Metron Vagrant should disable audio (ottobackwards)
> > > > closes
> > > > > apache/metron#1277
> > > > > METRON-1874 Create a Parser Debugger (nickwallen) closes
> > > > > apache/metron#1265
> > > > > METRON-1880 Use Caffeine for Profiler Caching (nickwallen)
> closes
> > > > > apache/metron#1270
> > > > > METRON-1877 Nested IF ELSE statements can cause parse errors in
> > > > Stellar
> > > > > (justinleet) closes apache/metron#1268
> > > > > METRON-1872 Move rat plugin away from snapshot version
> > (justinleet)
> > > > > closes apache/metron#1264
> > > > >
> > > > > On Wed, Nov 21, 2018 at 10:59 AM Nick Allen 
> > > wrote:
> > > > >
> > > > > > Also, I'd like to get this one included in the release.  This is
> > > really
> > > > > > annoying for people just wanting to try out the Profiler.  And
> this
> > > was
> > > > > > 'broken' after the last release, so there currently is no release
> > > with
> > > > > this
> > > > > > problem and I'd like to keep it that way. :)
> > > > > >
> > > > > > https://github.com/apache/metron/pull/1276
> > > > > >
> > > > > > On Wed, Nov 21, 2018 at 10:11 AM Justin Leet <
> > justinjl...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Realized I'd never sent the updated list of Jiras.  I changed
> the
> > > > > command
> > > > > > > slightly (to remove a clause I thought we'd already removed re:
> > > http,
> > > > > and
> > > > > > > added the awk to remove dupes resulting from multiple commits
> > for a
> > > > > > single
> > > > > > > Jira. I'll do a PR for these changes).
> > > > > > >
> > > > > > > *apache/metron*
> > > > > > > git log "master" "^tags/apache-metron-0.6.0-release"
> --no-merges
> > |
&g

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[zeo...@gmail.com]

-1

In my testing it appears that an issue was introduced in 0.2 which is
causing a segfault on the destructor (
https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57).
I've opened METRON-1910 and am testing a fix now.

On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:

> This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
> The release candidate is available at:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
>
> Full list of changes in this release:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES
>
> The tag to be voted upon is:
> apache-metron-bro-plugin-kafka_0.3.0-rc1
> <
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc1
> >
>
> The source archives being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/apache-metron-bro-plugin-kafka_0.3.0-rc1.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
>
> The release artifacts are signed with the following key:
> *https://dist.apache.org/repos/dist/release/metron/KEYS*
> 
> Please vote on releasing this package as Apache Metron-bro-plugin-kafka
> 0.3.0
>
> When voting, please list the actions taken to verify the release.
>
> Recommended build validation and verification instructions are posted here:
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
> This is only for the plugin, so the VM validation is not required.
>
> This vote will be open for until 3pm EDT on Friday November 27 2018.
>
> [ ] +1 Release this package as Apache Metron 0.3.0-RC1
>
> [ ] 0 No opinion
>
> [ ] -1 Do not release this package because...
>
-- 

Jon Zeolla

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

(nickwallen) closes apache/metron#1202
> > > > METRON-1758 Add support for Ansible 2.6 in dev (JonZeolla via
> > > > jonzeolla) closes apache/metron#1179
> > > > METRON-1784: Re-allow remote ssh and scp in Centos full dev
> > (mmiklavc
> > > > via mmiklavc) closes apache/metron#1204
> > > > METRON-1787 Input Time Constraints for Batch Profiler
> (nickwallen)
> > > > closes apache/metron#1209
> > > > METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to
> > Elasticsearch
> > > > (nickwallen) closes apache/metron#1185
> > > > METRON-1786 Pcap Topology Status Incorrect (MohanDV via
> nickwallen)
> > > > closes apache/metron#1206
> > > > METRON-1709 Add controls to start / stop the PCAP topology from
> > > Ambari.
> > > > (MohanDV via nickwallen) closes apache/metron#1201
> > > > METRON-1759 PCAP UI: Removing wrong Input annotations from pcap
> > panel
> > > > component (tiborm via nickwallen) closes apache/metron#1180
> > > > METRON-1772 Support alternative input formats in the Batch
> Profiler
> > > > (nickwallen) closes apache/metron#1191
> > > > METRON-1770 Add Docs for Running the Profiler with Spark on YARN
> > > > (nickwallen) closes apache/metron#1189
> > > > METRON-1774 Allow user to configure JAAS client in Ambari
> > > (nickwallen)
> > > > closes apache/metron#1192
> > > > METRON-1760 Kill PCAP job should prompt for confirmation (ruffle
> > via
> > > > nickwallen) closes apache/metron#1199
> > > > METRON-1777: Fix Elasticsearch X-Pack sample pom in documentation
> > > > (mmiklavc via mmiklavc) closes apache/metron#1196
> > > > METRON-1781 Fix RPM Spec File (nickwallen) closes
> > apache/metron#1200
> > > > METRON-1780 Fix broken website images (justinleet) closes
> > > > apache/metron#1198
> > > > METRON-1476 Update to Angular 6.1.3 (sardell via nickwallen)
> closes
> > > > apache/metron#1096
> > > > METRON-1776 Update public web site to point at 0.6.0 new release
> > > > (justinleet) closes apache/metron#1195
> > > > METRON-1775 Transient exception could prevent expired profiles
> from
> > > > being flushed (nickwallen) closes apache/metron#1194
> > > > METRON-1717 Relocate Storm Profiler Code (nickwallen) closes
> > > > apache/metron#1187
> > > > METRON-1748 Improve Storm Profiler Integration Test (nickwallen)
> > > closes
> > > > apache/metron#1174
> > > > METRON-1741 Move REPL Port of Profiler to Separate Project
> > > (nickwallen)
> > > > closes apache/metron#1170
> > > > METRON-1715 Create DEB Packaging for Batch Profiler (nickwallen)
> > > closes
> > > > apache/metron#1167
> > > > METRON-1736 Enhance Batch Profiler Integration Test (nickwallen)
> > > closes
> > > > apache/metron#1162
> > > > METRON-1714 Create RPM Packaging for the Batch Profiler
> > (nickwallen)
> > > > closes apache/metron#1163
> > > > METRON-1708 Run the Batch Profiler in Spark (nickwallen) closes
> > > > apache/metron#1161
> > > > METRON-1707 Port Profiler to Spark (nickwallen) closes
> > > > apache/metron#1150
> > > > METRON-1705 Create ProfilePeriod Using Period ID (nickwallen)
> > closes
> > > > apache/metron#1148
> > > > METRON-1706 HbaseClient.mutate should return the number of
> > mutations
> > > > (nickwallen) closes apache/metron#1147
> > > > METRON-1704 Message Timestamp Logic Should be Shared (nickwallen)
> > > > closes apache/metron#1146
> > > > METRON-1703 Make Core Profiler Components Serializable
> (nickwallen)
> > > > closes apache/metron#1145
> > > >
> > > >
> > > > *apache/metron-bro-plugin-kafka*
> > > > git log "master" "^tags/apache-metron-bro-plugin-kafka_0.2.0-release"
> > > > --no-merges | grep -E "^[[:blank:]]+METRON" | sed 's/\[//g' | sed
> > > 's/\]//g'
> > > > | awk '!x[$1]++'
> > > > METRON-1827 Update librdkafka in metron-bro-plugin-kafka
> (JonZeolla
> > > via
> > > > jonzeolla) closes apache/metron-bro-plugin-kafka#13
> > > > METRON-1866 Improve metron-bro-plugin-kafka documentation

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

he/metron#1201
> > > METRON-1759 PCAP UI: Removing wrong Input annotations from pcap
> panel
> > > component (tiborm via nickwallen) closes apache/metron#1180
> > > METRON-1772 Support alternative input formats in the Batch Profiler
> > > (nickwallen) closes apache/metron#1191
> > > METRON-1770 Add Docs for Running the Profiler with Spark on YARN
> > > (nickwallen) closes apache/metron#1189
> > > METRON-1774 Allow user to configure JAAS client in Ambari
> > (nickwallen)
> > > closes apache/metron#1192
> > > METRON-1760 Kill PCAP job should prompt for confirmation (ruffle
> via
> > > nickwallen) closes apache/metron#1199
> > > METRON-1777: Fix Elasticsearch X-Pack sample pom in documentation
> > > (mmiklavc via mmiklavc) closes apache/metron#1196
> > > METRON-1781 Fix RPM Spec File (nickwallen) closes
> apache/metron#1200
> > > METRON-1780 Fix broken website images (justinleet) closes
> > > apache/metron#1198
> > > METRON-1476 Update to Angular 6.1.3 (sardell via nickwallen) closes
> > > apache/metron#1096
> > > METRON-1776 Update public web site to point at 0.6.0 new release
> > > (justinleet) closes apache/metron#1195
> > > METRON-1775 Transient exception could prevent expired profiles from
> > > being flushed (nickwallen) closes apache/metron#1194
> > > METRON-1717 Relocate Storm Profiler Code (nickwallen) closes
> > > apache/metron#1187
> > > METRON-1748 Improve Storm Profiler Integration Test (nickwallen)
> > closes
> > > apache/metron#1174
> > > METRON-1741 Move REPL Port of Profiler to Separate Project
> > (nickwallen)
> > > closes apache/metron#1170
> > > METRON-1715 Create DEB Packaging for Batch Profiler (nickwallen)
> > closes
> > > apache/metron#1167
> > > METRON-1736 Enhance Batch Profiler Integration Test (nickwallen)
> > closes
> > > apache/metron#1162
> > > METRON-1714 Create RPM Packaging for the Batch Profiler
> (nickwallen)
> > > closes apache/metron#1163
> > > METRON-1708 Run the Batch Profiler in Spark (nickwallen) closes
> > > apache/metron#1161
> > > METRON-1707 Port Profiler to Spark (nickwallen) closes
> > > apache/metron#1150
> > >     METRON-1705 Create ProfilePeriod Using Period ID (nickwallen)
> closes
> > > apache/metron#1148
> > > METRON-1706 HbaseClient.mutate should return the number of
> mutations
> > > (nickwallen) closes apache/metron#1147
> > > METRON-1704 Message Timestamp Logic Should be Shared (nickwallen)
> > > closes apache/metron#1146
> > > METRON-1703 Make Core Profiler Components Serializable (nickwallen)
> > > closes apache/metron#1145
> > >
> > >
> > > *apache/metron-bro-plugin-kafka*
> > > git log "master" "^tags/apache-metron-bro-plugin-kafka_0.2.0-release"
> > > --no-merges | grep -E "^[[:blank:]]+METRON" | sed 's/\[//g' | sed
> > 's/\]//g'
> > > | awk '!x[$1]++'
> > > METRON-1827 Update librdkafka in metron-bro-plugin-kafka (JonZeolla
> > via
> > > jonzeolla) closes apache/metron-bro-plugin-kafka#13
> > > METRON-1866 Improve metron-bro-plugin-kafka documentation
> (JonZeolla
> > > via jonzeolla) closes apache/metron-bro-plugin-kafka#17
> > > METRON-1304 Allow metron-bro-plugin-kafka to include or exclude
> logs
> > > (JonZeolla via nickwallen) closes apache/metron-bro-plugin-kafka#2
> > > METRON-1865 Fix metron-bro-plugin-kafka tests (JonZeolla via
> > jonzeolla)
> > > closes apache/metron-bro-plugin-kafka#16
> > > METRON-1828 Improve bro plugin contributing documentation
> (JonZeolla)
> > > closes apache/metron-bro-plugin-kafka#14
> > > METRON-1818 Remove config_files from bro-pkg.meta (JonZeolla)
> closes
> > > apache/metron-bro-plugin-kafka#11
> > > METRON-1800 Increment metron-bro-plugin-kafka version (JonZeolla
> via
> > > jonzeolla) closes apache/metron-bro-plugin-kafka#10
> > > METRON-1773 Bro plugin docs should refer to Apache Metron project
> > > (nickwallen) closes apache/metron-bro-plugin-kafka#9
> > >
> > > On Sun, Nov 18, 2018 at 7:52 AM zeo...@gmail.com 
> > wrote:
> > >
> > > > I'm good with that release schedule, and using version 0.7.0 for the
> > > > apache/metron release.
> > > >
> > > > I opened up METRON

Re: [ANNOUNCE] Shane Ardell is a committer

[zeo...@gmail.com]

Congrats Shane!

Jon

On Mon, Nov 19, 2018 at 10:43 AM Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> Many congratulations, Shane!
>
> Cheers,
> Anand
>
> On 11/19/18, 8:36 PM, "James Sirota"  wrote:
>
>
> The Project Management Committee (PMC) for Apache Metron has invited
> Shane Ardell to become a committer and we are pleased to announce that he
> has accepted.  I wanted to congratulate Shane on this achievement.
>
>
> Being a committer enables easier contribution to the project since
> there is no need to go via the patch submission process. This should enable
> better productivity. Being a PMC member enables assistance with the
> management and to guide the direction of the project.
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>
>
> --

Jon Zeolla

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

I'm good with that release schedule, and using version 0.7.0 for the
apache/metron release.

I opened up METRON-1881 and have a branch ready to PR for the 0.3
plugin upgrade.

Jon

On Fri, Nov 16, 2018 at 10:08 AM Otto Fowler 
wrote:

> Can you generate the jiras that would be included in the release?
>
>
> On November 16, 2018 at 10:05:50, Justin Leet (justinjl...@gmail.com)
> wrote:
>
> Given that we've had a couple major PRs (the ES client migration along with
> the Angular upgrade stuff, and I'm sure others), I'd be in favor of
> releasing both the plugin and the main repo.
>
> I'd be in favor of doing something like:
> metron-bro-plugin-kafka release 0.3.0
> PR to update full dev
> metron release 0.7.0 (given that things changed a good amount)
>
> I would also expect this process to most likely be kicked off post
> Thanksgiving break (Thursday 22nd and Friday 23rd for anybody not in the
> US).
>
> I can kick out another summarization thread if we want, but basically:
> * Nov 26 - Start the Bro plugin release process
> * Once that finishes, PR to update full dev with new plugin version
> * Once PR is in, start metron release process (hopefully) sometime the week
> of the 3rd?
>
> Are there any objections to staggering the releases like that? They could
> also be done together, but it means that we have to update full dev to
> match the plugin version post release.
>
> On Wed, Nov 14, 2018 at 10:29 AM zeo...@gmail.com 
> wrote:
>
> > In my opinion metron-bro-plugin-kafka is ready for a release. Anything
> > else people would want to see? Once it gets released, I would like to
> > update full dev to use the newest version prior to any future metron
> > release (0.6.1 or whatever we choose).
> >
> > Jon
> >
> > On Wed, Nov 7, 2018 at 8:07 PM zeo...@gmail.com 
> wrote:
> >
> > > So, about this release, anybody have time to review
> > > apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13?
> > >
> > > Jon
> > >
> > > On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > >> And I do think we will be ready to roll another Metron release in the
> > near
> > >> future as well.
> > >>
> > >> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet 
> > >> wrote:
> > >>
> > >> > I tend to agree with Mike. I think we could do a release of the main
> > >> > project and get benefit, but I think skipping this cycle (especially
> > >> since
> > >> > we had a release last month) let's us get things like the ES client
> > >> > migration in and settled and just generally make recommending a
> newer
> > >> > version easier.
> > >> >
> > >> > The main reason for doing a release for the Bro plugin is to address
> > >> > shortcomings that we discovered and are addressing.
> > >> >
> > >> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
> > >> > michael.miklav...@gmail.com> wrote:
> > >> >
> > >> > > Migrating the ES client, refactoring parser bolt. There are some
> > >> > interface
> > >> > > changes in flight right now that I think would be beneficial to
> see
> > in
> > >> > the
> > >> > > next release.
> > >> > >
> > >> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen 
> > >> wrote:
> > >> > >
> > >> > > > I am in favor of a release for both.
> > >> > > >
> > >> > > > There are a lot of really useful bug fixes, management of pcap
> > >> through
> > >> > > > Ambari, more flexibility for configuring JAAS in Ambari,
> increased
> > >> > > > Elasticsearch performance, the Syslog parser, and the Batch
> > >> Profiler,
> > >> > > among
> > >> > > > others. I would be happy with calling it a 0.6.1 point release.
> > >> > > >
> > >> > > > Mike - What is outstanding that you would like to see in the
> > >> release?
> > >> > > >
> > >> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
> > >> > > > michael.miklav...@gmail.com> wrote:
> > >> > > >
> > >> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin
> > release.
>

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

In my opinion metron-bro-plugin-kafka is ready for a release.  Anything
else people would want to see?  Once it gets released, I would like to
update full dev to use the newest version prior to any future metron
release (0.6.1 or whatever we choose).

Jon

On Wed, Nov 7, 2018 at 8:07 PM zeo...@gmail.com  wrote:

> So, about this release, anybody have time to review
> apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13?
>
> Jon
>
> On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> And I do think we will be ready to roll another Metron release in the near
>> future as well.
>>
>> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet 
>> wrote:
>>
>> > I tend to agree with Mike. I think we could do a release of the main
>> > project and get benefit, but I think skipping this cycle (especially
>> since
>> > we had a release last month) let's us get things like the ES client
>> > migration in and settled and just generally make recommending a newer
>> > version easier.
>> >
>> > The main reason for doing a release for the Bro plugin is to address
>> > shortcomings that we discovered and are addressing.
>> >
>> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
>> > michael.miklav...@gmail.com> wrote:
>> >
>> > > Migrating the ES client, refactoring parser bolt. There are some
>> > interface
>> > > changes in flight right now that I think would be beneficial to see in
>> > the
>> > > next release.
>> > >
>> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen 
>> wrote:
>> > >
>> > > > I am in favor of a release for both.
>> > > >
>> > > > There are a lot of really useful bug fixes, management of pcap
>> through
>> > > > Ambari, more flexibility for configuring JAAS in Ambari, increased
>> > > > Elasticsearch performance, the Syslog parser, and the Batch
>> Profiler,
>> > > among
>> > > > others. I would be happy with calling it a 0.6.1 point release.
>> > > >
>> > > > Mike - What is outstanding that you would like to see in the
>> release?
>> > > >
>> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
>> > > > michael.miklav...@gmail.com> wrote:
>> > > >
>> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin release.
>> It
>> > > > seems
>> > > > > like it's ready to go, and I think there are a few more things I'd
>> > like
>> > > > to
>> > > > > see get into our next Metron release so I'm good with holding off
>> > > there.
>> > > > >
>> > > > > Mike
>> > > > >
>> > > > > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet <
>> justinjl...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > As you might recall from a prior discussion about release
>> cadence,
>> > we
>> > > > > were
>> > > > > > interested in initiating release threads near our board reports
>> to
>> > > see
>> > > > if
>> > > > > > we wanted to do releases or not. Additionally, the work is done
>> to
>> > do
>> > > > two
>> > > > > > separate releases, so our options are releasing both, a single
>> one,
>> > > or
>> > > > > > neither.
>> > > > > >
>> > > > > > Having said that, a metron-bro-kafka-plugin 0.3.0 release came
>> up
>> > on
>> > > > this
>> > > > > > thread
>> > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
>> > > > > > >.
>> > > > > > In particular, the prospect of a release came up in the context
>> of
>> > > > > having a
>> > > > > > version with better (and working) testing.
>> > > > > >
>> > > > > > Version Number
>> > > > > > If we choose to do a co

Re: [DISCUSS] Slack Channel Use

[zeo...@gmail.com]

Spot on Justin, I totally agree.  My only nit is that often it's much
easier troubleshooting in Slack as opposed to the mailing lists, so I'm
game to allow some troubleshooting in Slack as long as the issue and
resolution makes it back to the lists.  Given that slack message history is
being kept (although to what degree I'm not sure), we could fairly easily
link to the start of a discussion in slack in the wrap-up mailing list
email for future reference.

Jon

On Mon, Nov 12, 2018 at 2:08 PM Casey Stella  wrote:

> Piling on, +1 to what Justin said.
>
> On Mon, Nov 12, 2018 at 12:42 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I'm also +1 to Justin's points.
> >
> > On Mon, Nov 12, 2018 at 10:38 AM Nick Allen  wrote:
> >
> > > +1 to all your points Justin.
> > >
> > > On Mon, Nov 12, 2018 at 10:08 AM Justin Leet 
> > > wrote:
> > >
> > > > I wanted to add back onto this thread after putting some more thought
> > > into
> > > > it.
> > > >
> > > > I like Slack for the type of small developer "what's going on here?"
> > type
> > > > discussions.  That's the kind of thing I like being real-time ("Hey,
> > full
> > > > dev is acting weird", "What's the basic layout of this stuff?",
> > "Anybody
> > > > else seen this test failure?", etc.).  I think we've been pretty good
> > > about
> > > > keeping our decision type dev discussions to the list (e.g. this
> exact
> > > > conversation).
> > > >
> > > > We've been doing this more, but I would like to see more of the user
> > and
> > > > troubleshooting move to the list.  I think we've gotten a bit better
> > > about
> > > > it as we've settled into Slack, but having that sort of helpful stuff
> > > > exposed and searchable for users who come in afterwards is a big
> > selling
> > > > point of the lists, imo.
> > > >
> > > > To add onto this, I'd probably like to see
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/METRON/Community+Resources#CommunityResources-ApacheMetronCommunityResources
> > > > (and any other relevant links) updated to emphasize a Slack focus on
> > > > developing Metron itself, and the user lists for configuration,
> > > > troubleshooting, etc.
> > > >
> > > > Essentially, I'm proposing:
> > > > Dev list / Jira / PRs as usual for any actual decisions + concrete
> > > feature
> > > > discussion/review.
> > > > Slack for Metron development "Hey, anyone seen this or have insight
> or
> > a
> > > > starting point?" and "I'm seeing something weird in our tests" type
> > stuff
> > > > User list for usage and troubleshooting questions.  Generally,
> > > discussions
> > > > like this in Slack should be redirected to the user list.
> > > >
> > > > Is this a reasonable way separate our concerns here?
> > > >
> > > > On Wed, Oct 24, 2018 at 11:37 AM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Yeah, I'm also surprised by that comment about the mailing list
> > > activity.
> > > > > Our dev/user list discussions are by far more active than they've
> > ever
> > > > > been. Just have a look at the list of DISCUSS threads that have
> come
> > up
> > > > in
> > > > > the past few months and it's clear that not only participation has
> > > > > increased, but diversity of topic and participant.
> > > > >
> > > > > On Wed, Oct 24, 2018 at 8:08 AM Casey Stella 
> > > wrote:
> > > > >
> > > > > > Not for nothing, but at least according to the last board report
> > > that I
> > > > > > submitted, the user@ traffic is up 100% and the dev list traffic
> > is
> > > > flat
> > > > > > as
> > > > > > compared to last quarter.  That's not to say that we couldn't
> stand
> > > > more
> > > > > > discussion on the lists, but a lot of the dev discussion happens
> on
> > > > > github
> > > > > > and JIRA and I'm happy to see an uptick in user traffic.
> > > > > >
> > > > > > On Wed, Oct 24, 2018 at 10:05 AM Otto Fowler <
> > > ottobackwa...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I wouldn’t be so quick to related the slack discussion with
> > > perceived
> > > > > > > activity on the list.
> > > > > > > That is more do to the other things that are bigger issues.
> > > > > > >
> > > > > > >
> > > > > > > On October 24, 2018 at 07:15:30, Nick Allen (
> n...@nickallen.org)
> > > > > wrote:
> > > > > > >
> > > > > > > > I have heard recently people thought Metron is sort of dead
> > just
> > > > > > because
> > > > > > > the mailing list is not so active anymore!
> > > > > > >
> > > > > > > That is exactly my concern.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Oct 24, 2018, 2:49 AM Ali Nazemian <
> > alinazem...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > I kind of expect to have Slack for more dev related
> discussions
> > > > > rather
> > > > > > > than
> > > > > > > > user QA. I guess it is quite common to expect mailing list to
> > be
> > > > used
> > > > > > for
> > > > > > > > the purpose of knowledge sharing to make sure it will be
> > > accessible
> 

Re: [DISCUSS] Knox SSO feature branch review and features

[zeo...@gmail.com]

Phew, that was quite the thread to catch up on.

I agree that this should be optional/pluggable to start, and I'm interested
to hear the issues as they relate to upgrading an existing cluster (given
the suggested approach) and exposing both legacy and knox URLs at the same
time.

Jon

On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic 
wrote:

> A couple more things, and I think this goes without saying - whatever we do
> with Knox should NOT
>
>1. Require unit and integration tests to use Knox
>2. Break fulldev
>
> Also, I don't know that I saw you mention this, but I'm unsure how we
> should leverage Knox as a core piece of the platform. i.e. should we make
> this required or optional? I'm open to hearing opinions on this, but I'm
> inclined to keep this a pluggable option.
>
> Mike
>
>
> On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Thanks for the update Ryan. Per my earlier comments, I thought it might
> be
> > the case that we could dramatically simplify this by leveraging Knox's
> > proxy capabilities, and per your research that appears to be the case.
> This
> > is a dramatic simplification and improvement of this feature imo, +1. I'm
> > also +1 on a couple distinct steps that you've laid out: fix the UI
> issues
> > in master, then add Knox for SSO. That should help mitigate issues with
> > merge conflicts with ongoing development.
> >
> > > I think it will be a challenge exposing the UIs through both the Knox
> > url and legacy urls at the same time.
> > I'm not sure I understand the issue here. Are you referring to this
> > comment? "Added a ng build option to build the UI with base href set to
> > Knox base path." Isn't it just a matter of URL rewriting/forwarding? I
> > thought we'd be exposing the URL's directly in one context, and through
> > Knox in the other. Either way, it seems like we should be able to
> provide a
> > dynamic base path through configuration in our web applications. I'd
> expect
> > to modify that property based on whether Knox is configured or not.
> >
> > > I'm also not clear on how one would use Knox with REST set to legacy
> > JDBC-based authentication. As far as I know Knox does not support JDBC so
> > there would be a mismatch between Knox and REST.
> > I'm OK with not having Knox work with JDBC. That's a feature of Knox and
> > probably not something we care much about.
> >
> > >We could initially make Knox an optional feature that requires setup
> with
> > the help of some documentation (like Kerberos) while keeping the system
> the
> > way it is now by default.
> > Sounds good to me.
> >
> > > I imagine we'll deprecate JDBC-based authentication at some point so
> > that may be a good time to switch.
> > I would like to announce deprecation in our next release and move to
> > remove it in a following release.
> >
> > Thanks for taking this on and great job laying things out.
> >
> > Thanks,
> > Mike
> >
> > On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman 
> wrote:
> >
> >> I have spent some time recently reviewing this discussion and the
> feature
> >> branch that Simon put out.  I think this is an important feature and
> want
> >> to move it forward.  I started another discussion on adding Knox to our
> >> stack but this discussion has a lot of good context so I will continue
> it
> >> here.
> >>
> >> I think the main point of contention was that this feature branch
> included
> >> several different architectural changes and it was unclear if they were
> >> needed and if so, could be done separately.  Fortunately LDAP
> >> authentication has been accepted into master so we can cross it off the
> >> list.  From my understanding of the points people have made, that leaves
> >> Knox related SSO changes and migrating expressjs to a different,
> JVM-based
> >> web server that includes proxying capabilities (Zuul).
> >>
> >> I think everyone agrees that if we can limit the scope to just Knox
> >> related
> >> SSO changes that would be ideal.  I believe I have found a way to do
> that
> >> while working on a small POC this week.  The key to this (Simon alluded
> to
> >> it earlier) is to put both REST and our UIs behind Knox.  I initially
> was
> >> focused on just adding REST as a service in Knox and decided to
> experiment
> >> with also adding our UIs.  After I did this it became clear that this
> >> simplifies things considerably:
> >>
> >>- The REST app and the UIs are now served from the same host so CORS
> >>concerns go away.
> >>- We no longer need to worry about proxying REST requests from the
> UIs
> >>with express or Zuul because Knox handles that for us.  This will
> make
> >> our
> >>express configuration even simpler.  In fact, all we need is a simple
> >> way
> >>to serve static UI assets.
> >>- We no longer need to check for SSO tokens and redirect in the UI
> >>web/app servers (or the REST app for that matter) because Knox
> handles
> >> that
> >>for us.
> >>- The

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

So, about this release, anybody have time to review
apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13?

Jon

On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> And I do think we will be ready to roll another Metron release in the near
> future as well.
>
> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet  wrote:
>
> > I tend to agree with Mike. I think we could do a release of the main
> > project and get benefit, but I think skipping this cycle (especially
> since
> > we had a release last month) let's us get things like the ES client
> > migration in and settled and just generally make recommending a newer
> > version easier.
> >
> > The main reason for doing a release for the Bro plugin is to address
> > shortcomings that we discovered and are addressing.
> >
> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Migrating the ES client, refactoring parser bolt. There are some
> > interface
> > > changes in flight right now that I think would be beneficial to see in
> > the
> > > next release.
> > >
> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen  wrote:
> > >
> > > > I am in favor of a release for both.
> > > >
> > > > There are a lot of really useful bug fixes, management of pcap
> through
> > > > Ambari, more flexibility for configuring JAAS in Ambari, increased
> > > > Elasticsearch performance, the Syslog parser, and the Batch Profiler,
> > > among
> > > > others. I would be happy with calling it a 0.6.1 point release.
> > > >
> > > > Mike - What is outstanding that you would like to see in the release?
> > > >
> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin release.
> It
> > > > seems
> > > > > like it's ready to go, and I think there are a few more things I'd
> > like
> > > > to
> > > > > see get into our next Metron release so I'm good with holding off
> > > there.
> > > > >
> > > > > Mike
> > > > >
> > > > > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet <
> justinjl...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > As you might recall from a prior discussion about release
> cadence,
> > we
> > > > > were
> > > > > > interested in initiating release threads near our board reports
> to
> > > see
> > > > if
> > > > > > we wanted to do releases or not. Additionally, the work is done
> to
> > do
> > > > two
> > > > > > separate releases, so our options are releasing both, a single
> one,
> > > or
> > > > > > neither.
> > > > > >
> > > > > > Having said that, a metron-bro-kafka-plugin 0.3.0 release came up
> > on
> > > > this
> > > > > > thread
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> > > > > > >.
> > > > > > In particular, the prospect of a release came up in the context
> of
> > > > > having a
> > > > > > version with better (and working) testing.
> > > > > >
> > > > > > Version Number
> > > > > > If we choose to do a core Metron release, I propose 0.6.1. For
> > > > > > metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the
> > > versioning
> > > > > for
> > > > > > the plugin is a bit different in that we need x.y for bro-pkg,
> > > instead
> > > > of
> > > > > > x.y.z, so we wouldn't release 0.2.1.
> > > > > >
> > > > > > I would personally be in favor of doing a plugin release, but I'm
> > > less
> > > > > > inclined to do a core Metron release.
> > > > > >
> > > > > > For Metron, I believe the main feature is the batch profiler
> work,
> > > but
> > > > > > there's a fair amount of fixes and improvements.
> > > > > >
> > > > > > For the plugin, I believe the main improvements would be around
> the
> > > > > fixing
> > > > > > our testing and more maintenance type work.
> > > > > >
> > > > > > JIRA status
> > > > > > *metron*
> > > > > > There are 24 opens PRs at https://github.com/apache/metron/pulls
> .
> > > If
> > > > we
> > > > > > do
> > > > > > decide to release, we should work on getting anything meriting
> > > > inclusion
> > > > > > closed out.
> > > > > >
> > > > > > There have been 49 commits since the 0.6.0 release (listed at the
> > end
> > > > of
> > > > > > the message). This assumes a release from master.
> > > > > >
> > > > > > *metron-bro-kafka-plugin*
> > > > > > There are 6 open PRs at
> > > > > > https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we
> do
> > > > > decide
> > > > > > to release, we should get anything we need closed out.
> > > > > >
> > > > > > There have been 2 commits since the 0.2.0 release (listed at the
> > end
> > > of
> > > > > the
> > > > > > message).  This assumes a release from master.
> > > > > >
> > > > > > *Metron changelog*
> > > > > > METRON-1801 Allow Customization of Elasticsearch Document ID
> > > > > > (nickwallen

Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

[zeo...@gmail.com]

+1 totally agree.

Jon

On Fri, Nov 2, 2018, 1:31 AM Anand Subramanian 
wrote:

> Piling on my +1 (non-binding) as well.
>
> On 11/2/18, 4:41 AM, "Ryan Merriman"  wrote:
>
> +1
>
> On Thu, Nov 1, 2018 at 5:38 PM Casey Stella 
> wrote:
>
> > +1
> > On Thu, Nov 1, 2018 at 18:34 Nick Allen  wrote:
> >
> > > +1
> > >
> > > On Thu, Nov 1, 2018, 6:27 PM Justin Leet 
> wrote:
> > >
> > > > +1, I haven't seen any case where the split-join topology isn't
> made
> > > > obsolete by the unified topology.
> > > >
> > > > On Thu, Nov 1, 2018 at 6:17 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Fellow Metronians,
> > > > >
> > > > > We've had the unified enrichment topology around for a number
> of
> > months
> > > > > now, it has proved itself stable, and there is yet to be a
> time that
> > I
> > > > have
> > > > > seen the split-join topology outperform the unified one. Here
> are
> > some
> > > > > simple reasons to deprecate the split-join topology.
> > > > >
> > > > >1. Unified topology performs better.
> > > > >2. The configuration, especially for performance tuning is
> much,
> > > much
> > > > >simpler in the unified model.
> > > > >3. The footprint within the cluster is smaller.
> > > > >4. One of the first activities for any install is that we
> spend
> > time
> > > > >instructing users to switch to the unified topology.
> > > > >5. One less moving part to maintain.
> > > > >
> > > > > I'd like to recommend that we deprecate the split-join
> topology and
> > > make
> > > > > the unified enrichment topology the new default.
> > > > >
> > > > > Best,
> > > > > Mike
> > > > >
> > > >
> > >
> >
>
>
> --

Jon Zeolla

Re: [DISCUSS] Day 1 User Experience - Getting Metron Running

[zeo...@gmail.com]

Yeah I would +1 katakoda.  I also think that it would help to start
distributing RPMs, DEBs, and the mpacks with the releases, as well as
consider a service like opensuse's build service for nightlies, etc.

Jon

On Fri, Oct 26, 2018 at 6:25 AM Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> Great idea! This will be a HUGE improvement in the user experience for
> first timers to Metron. Katakoda seems very interesting - simple and
> straight-forward. I loved the way you can provide instructions, commands
> (that can be directly clicked!), links, explanation and so on.
>
> Regards,
> Anand
>
> On 10/25/18, 7:49 PM, "Nick Allen"  wrote:
>
> We all know spinning up the development environment is a pain.
> Unfortunately, it is the only way for a new user to get a feel for
> Metron.
> We need a better way to introduce new users to Metron.
>
> I am hoping we can brainstorm ways to improve that experience.  Here
> are a
> few thoughts that might help start a discussion.
>
> (1) Create a *KataKoda* [1] based demo.  I ran across this after
> finding
> Apache Ozone's demo [2], which I think is great.
>
>
>- A user does not need to download or install anything.  It is a
>   completely hosted offering.
>   - Provides a step-by-step demo experience that could guide users
>   through creating an enrichment, defining a profile, managing
> alerts.
>   - Would require a Metron on Docker solution.
>
> (2) Create a *Vagrant Cloud* [3] hosted image of "Full Dev" with
> everything
> installed and ready to rock.  A user would just need to install
> Vagrant and
> run:
>
> vagrant init metron/0.6.0
>
> vagrant up
>
>
>- Reduces the number of dependencies needed to get Metron
> up-and-running.
>   - Significantly increases the success rate of new users getting
>   Metron running.
>   - Still results in "Full Dev" Metron which requires too many
>   resources for the average computer.
>
> Are these good options? What other approaches could we take?  Hopefully
> some JIRAs might fall out of this discussion.
>
> - Nick
>
>
> --
> [1] https://www.katacoda.com
> [2] https://www.katacoda.com/elek/scenarios/ozone101
> [3] https://app.vagrantup.com/boxes/search
>
>
> --

Jon Zeolla

Re: Invite to Slack Channel

[zeo...@gmail.com]

Invite sent

On Mon, Oct 22, 2018 at 9:26 AM Muhammed Irshad  wrote:

> Some one get me also the slack channel link ?
> Thanks,
> Muhammed Irshad
> Q*Burst*
> www.qburst.com
>
>
> On Wed, Oct 17, 2018 at 7:33 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Sent
> >
> > On Wed, Oct 17, 2018 at 7:23 AM Tibor Meller 
> > wrote:
> >
> > > Hi Guys,
> > > Can you add me to the apache metron slack chanel?
> > >
> > > Thanks,
> > >
> > > On Thu, Oct 4, 2018 at 1:14 PM Otto Fowler 
> > > wrote:
> > >
> > > > Done
> > > >
> > > >
> > > > On October 4, 2018 at 05:35:06, Tamás Fodor (ftamas.m...@gmail.com)
> > > wrote:
> > > >
> > > > Hello,
> > > >
> > > > Michael, can you add me as well?
> > > >
> > > > Thank you in advance!
> > > >
> > > > Tamas
> > > >
> > > > On Wed, Oct 3, 2018 at 4:27 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Sent
> > > > >
> > > > > On Wed, Oct 3, 2018 at 8:17 AM Shane Ardell <
> > shane.m.ard...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > Is it possible for someone to send me an invite to the Metron
> Slack
> > > > > > channel?
> > > > > >
> > > > > > Regards,
> > > > > > Shane
> > > > > >
> > > > >
> > > >
> > >
> >
>
-- 

Jon Zeolla

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[zeo...@gmail.com]

I agree with a metron-bro-plugin-kafka release of 0.3.0 (0.3 in bro-pkg),
assuming we can get apache/metron-bro-plugin-kafka#2 in.  I'm working on
adding travis to the metron-bro-plugin-kafka repo, but I'm not sure when I
will have enough time to finish my work there and wouldn't want to hold up
a release just for that.

Jon

On Tue, Oct 16, 2018 at 12:26 PM Justin Leet  wrote:

> Hi all,
>
> As you might recall from a prior discussion about release cadence, we were
> interested in initiating release threads near our board reports to see if
> we wanted to do releases or not. Additionally, the work is done to do two
> separate releases, so our options are releasing both, a single one, or
> neither.
>
> Having said that, a metron-bro-kafka-plugin 0.3.0 release came up on this
> thread
> <
> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> >.
> In particular, the prospect of a release came up in the context of having a
> version with better (and working) testing.
>
> Version Number
> If we choose to do a core Metron release, I propose 0.6.1. For
> metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the versioning for
> the plugin is a bit different in that we need x.y for bro-pkg, instead of
> x.y.z, so we wouldn't release 0.2.1.
>
> I would personally be in favor of doing a plugin release, but I'm less
> inclined to do a core Metron release.
>
> For Metron, I believe the main feature is the batch profiler work, but
> there's a fair amount of fixes and improvements.
>
> For the plugin, I believe the main improvements would be around the fixing
> our testing and more maintenance type work.
>
> JIRA status
> *metron*
> There are 24 opens PRs at https://github.com/apache/metron/pulls.  If we
> do
> decide to release, we should work on getting anything meriting inclusion
> closed out.
>
> There have been 49 commits since the 0.6.0 release (listed at the end of
> the message). This assumes a release from master.
>
> *metron-bro-kafka-plugin*
> There are 6 open PRs at
> https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we do decide
> to release, we should get anything we need closed out.
>
> There have been 2 commits since the 0.2.0 release (listed at the end of the
> message).  This assumes a release from master.
>
> *Metron changelog*
> METRON-1801 Allow Customization of Elasticsearch Document ID
> (nickwallen) closes apache/metron#1218
> METRON-1799 Remove outdated bylaws from site. (justinleet) closes
> apache/metron#1216
> METRON-1769 Script creation of a release candidate (justinleet) closes
> apache/metron#1188
> METRON-1761 Allow a grok statement to be applied to each line in a
> file. (ottobackwards) closes apache/metron#1184
> METRON-1813 Stellar REPL Not Initialized with Client JAAS (nickwallen)
> closes apache/metron#1232
> METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc)
> closes apache/metron#1230
> METRON-1811 Alert Search Fails When Sorting by Alert Status (merrimanr)
> closes apache/metron#1231
> METRON-1809 Support Column Oriented Input with Batch Profiler
> (nickwallen) closes apache/metron#1229
> METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via mmiklavc)
> closes apache/metron#1224
> METRON-1792 Simplify Profile Definitions in Integration Tests
> (nickwallen) closes apache/metron#1211
> METRON-1807 Auto populate the recommended values to some of the metron
> config parameters  (MohanDV via merrimanr) closes apache/metron#1227
> METRON-1808 Add Ansible created pyc to gitignore (justinleet) closes
> apache/metron#1228
> METRON-1695 Expose pcap properties through Ambari (anandsubbu) closes
> apache/metron#1207
> METRON-1771 Update REST endpoints to support eventually consistent UI
> updates (merrimanr) closes apache/metron#1190
> METRON-1791 Add GUID to Messages Produced by Profiler (nickwallen)
> closes apache/metron#1210
> METRON-1804 Update version to 0.6.1 (justinleet) closes
> apache/metron#1220
> METRON-1798 Add mpack support for parser aggregation (anandsubbu)
> closes apache/metron#1215
> METRON-1750 Create Parser for Syslog RFC 5424 Messages (ottobackwards)
> closes apache/metron#1175
> METRON-1794 Include User Details When Escalating Alerts (nickwallen)
> closes apache/metron#1212
> METRON-1782 Add Kafka Partition and Offset to Profiler Debug Logs
> (nickwallen) closes apache/metron#1202
> METRON-1758 Add support for Ansible 2.6 in dev (JonZeolla via
> jonzeolla) closes apache/metron#1179
> METRON-1784: Re-allow remote ssh and scp in Centos full dev (mmiklavc
> via mmiklavc) closes apache/metron#1204
> METRON-1787 Input Time Constraints for Batch Profiler (nickwallen)
> closes apache/metron#1209
> METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
> (nickwallen) closes apache/metron#1185
> METRON-1786 Pcap Topology Status Incorrect (MohanDV via nickwall

Re: Bro plugin unit tests failing

[zeo...@gmail.com]

All good suggestions.  I don't think I'll personally have time in the short
term to draft a release discussion email, but I do think it's a good idea.

I can take a stab at the README link to contributing and a PR template for
the plugin repo.

Jon

On Mon, Oct 15, 2018 at 8:09 AM Justin Leet  wrote:

> Should we just put out a discuss for releasing in general, with this
> discussion noted?  The outcome of the release cadence discussion was to put
> out a thread around each board report. I believe that's a few days from
> now, so it's good timing. As an aside, part of the post release activities
> for the plugin should probably be upping the plugin version in the main
> repo to use the new version.
>
> While we're on the topic, is there anything else we should do to improve
> the plugin repo?  Off the top of my head, setting up a version of the PR
> template would be helpful.  Maybe adding a section to the README.md linking
> to the CONTRIBUTING.md of the main repo?
>
> On Sun, Oct 14, 2018 at 11:14 AM Otto Fowler 
> wrote:
>
>> It is INFRA, see INFRA-17091 for example.
>>
>>
>> On October 12, 2018 at 20:47:24, zeo...@gmail.com (zeo...@gmail.com)
>> wrote:
>>
>> So it seems that the last commit before the 0.2 release of
>> metron-bro-plugin-kafka broke the one basic unit test that we had. Since
>> metron 0.6.0 pins to 0.1
>> <
>>
>> https://github.com/apache/metron/blob/Metron_0.6.0/metron-deployment/ansible/roles/bro/vars/main.yml#L30
>> >
>>
>> this wouldn't cause an obvious issue if you spun up the sensors in
>> full-dev, and even if we were to point metron to 0.2 of the plugin we
>> wouldn't have necessarily seen the issue because we do a --force
>> <
>>
>> https://github.com/apache/metron/blob/Metron_0.6.0/metron-deployment/ansible/roles/bro/tasks/metron-bro-plugin-kafka.yml#L35
>> >
>>
>> to remove interaction.
>>
>> After we get a chance to review/merge a recent PR
>> <https://github.com/apache/metron-bro-plugin-kafka/pull/2>, we should
>> have
>> a few more 'actual' unit tests, and I would like to get travis setup for
>> the repo <https://travis-ci.org/apache/metron-bro-plugin-kafka> to
>> actually
>> check those (PR incoming) so it's less likely this sort of thing would
>> happen again. However, it doesn't appear I have the right permissions to
>> be able to get travis moving on my own. Does anybody else have the right
>> access, or is this an asf infrastructure ticket?
>>
>> I'd also like us to consider doing a 0.3 release for the plugin so we
>> aren't distributing something that has broken tests. Thoughts on that?
>>
>> Jon
>> --
>>
>> Jon
>>
> --

Jon

Bro plugin unit tests failing

[zeo...@gmail.com]

So it seems that the last commit before the 0.2 release of
metron-bro-plugin-kafka broke the one basic unit test that we had.  Since
metron 0.6.0 pins to 0.1

this wouldn't cause an obvious issue if you spun up the sensors in
full-dev, and even if we were to point metron to 0.2 of the plugin we
wouldn't have necessarily seen the issue because we do a --force

to remove interaction.

After we get a chance to review/merge a recent PR
, we should have
a few more 'actual' unit tests, and I would like to get travis setup for
the repo  to actually
check those (PR incoming) so it's less likely this sort of thing would
happen again.  However, it doesn't appear I have the right permissions to
be able to get travis moving on my own.  Does anybody else have the right
access, or is this an asf infrastructure ticket?

I'd also like us to consider doing a 0.3 release for the plugin so we
aren't distributing something that has broken tests.  Thoughts on that?

Jon
-- 

Jon

Re: Bro plugin release process docs?

[zeo...@gmail.com]

I've updated the cwiki
<https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=66854770&selectedPageVersions=43&selectedPageVersions=40>
and opened a PR <https://github.com/apache/metron/pull/1236> for this, open
to feedback on any of the changes I made/suggested.  Since I assume we
wouldn't want to make any changes to releases retroactively, I added a note
to the cwiki to note the history (see "Historical Note" under section 5).
Thanks,

Jon

On Thu, Oct 11, 2018 at 11:05 AM zeo...@gmail.com  wrote:

> Okay, I'll PR something since I'm looking at it
>
> Jon
>
> On Thu, Oct 11, 2018 at 9:45 AM Justin Leet  wrote:
>
>> If I recall correctly, they were already different, presumably from the
>> more manual process, and I wanted keep things close to what they already
>> were before trying to clean that sort of thing up. We can definitely unify
>> them, and if we need to code in any exceptions to deal with pre-unified
>> versions, we can.  It just wasn't high on my priority list.
>>
>> In the script, the change to the separator itself is easy, but I'd have to
>> double check if there's anything else that needs to happen to make sure
>> tags and such line up.
>>
>> On Thu, Oct 11, 2018 at 9:18 AM zeo...@gmail.com 
>> wrote:
>>
>> > Is there a reason why the prefix for apache/metron ends with a -,
>> whereas
>> > the plugin ends with a _ separator?  I would like to see it more uniform
>> > using the _
>> >
>> > Jon
>> >
>> > On Thu, Oct 11, 2018 at 9:04 AM Justin Leet 
>> wrote:
>> >
>> > > That first half of that page will be rewritten in the next couple
>> days to
>> > > reflect the script to prepare release candidate; I didn't want to
>> change
>> > > that page until the PR actually went in.
>> > >
>> > > The release process, barring the KEYS issue (and a patch to the
>> script to
>> > > handle the bro plugin tag), is pretty much set to be managed
>> separately
>> > at
>> > > this point.  The docs just need an overhaul, so someone who's not me
>> > knows
>> > > what to do.
>> > >
>> > > On Wed, Oct 10, 2018 at 7:01 PM zeo...@gmail.com 
>> > wrote:
>> > >
>> > > > Yeah you're right when I looked closer to make the change it was
>> step
>> > 10.
>> > > > I pushed a manual 0.2 tag to metron-bro-plugin-kafka and did a quick
>> > > update
>> > > > to the cwiki, but it could use some additional love, especially if
>> we
>> > > want
>> > > > to split out the release processes generally.
>> > > >
>> > > > Jon
>> > > >
>> > > > On Wed, Oct 10, 2018 at 5:32 PM Justin Leet 
>> > > wrote:
>> > > >
>> > > > > Yeah, we need to update the release process instructions. It
>> should
>> > > > > actually be step 10 shouldn't it?  We want to keep the rc tag
>> as-is
>> > > > because
>> > > > > it may get rejected/cancelled, and update the final tag as the
>> > > release. I
>> > > > > already need to overhaul that page to reflect the addition of the
>> RC
>> > > > > creation script, so I can also update it.
>> > > > >
>> > > > > I didn't remember there was a specific reason for the git tagging,
>> > so I
>> > > > > just followed the convention we used for the main repo, which is
>> why
>> > we
>> > > > > have this problem.
>> > > > >
>> > > > > The release candidate script also needs to be updated (because it
>> > needs
>> > > > to
>> > > > > know the proper prior tag to pull). It's not hard to do, but we'll
>> > need
>> > > > to
>> > > > > open a Jira and update that portion of the script. It's pretty
>> easy
>> > to
>> > > > do,
>> > > > > it's just the portion here
>> > > > > <
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245
>> > > > > >
>> > > > > .
>> > > > >
>> > > > > On Wed, Oct 10, 2018 at 5:09 PM Michael Mikl

Re: Bro plugin release process docs?

[zeo...@gmail.com]

Okay, I'll PR something since I'm looking at it

Jon

On Thu, Oct 11, 2018 at 9:45 AM Justin Leet  wrote:

> If I recall correctly, they were already different, presumably from the
> more manual process, and I wanted keep things close to what they already
> were before trying to clean that sort of thing up. We can definitely unify
> them, and if we need to code in any exceptions to deal with pre-unified
> versions, we can.  It just wasn't high on my priority list.
>
> In the script, the change to the separator itself is easy, but I'd have to
> double check if there's anything else that needs to happen to make sure
> tags and such line up.
>
> On Thu, Oct 11, 2018 at 9:18 AM zeo...@gmail.com  wrote:
>
> > Is there a reason why the prefix for apache/metron ends with a -, whereas
> > the plugin ends with a _ separator?  I would like to see it more uniform
> > using the _
> >
> > Jon
> >
> > On Thu, Oct 11, 2018 at 9:04 AM Justin Leet 
> wrote:
> >
> > > That first half of that page will be rewritten in the next couple days
> to
> > > reflect the script to prepare release candidate; I didn't want to
> change
> > > that page until the PR actually went in.
> > >
> > > The release process, barring the KEYS issue (and a patch to the script
> to
> > > handle the bro plugin tag), is pretty much set to be managed separately
> > at
> > > this point.  The docs just need an overhaul, so someone who's not me
> > knows
> > > what to do.
> > >
> > > On Wed, Oct 10, 2018 at 7:01 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > Yeah you're right when I looked closer to make the change it was step
> > 10.
> > > > I pushed a manual 0.2 tag to metron-bro-plugin-kafka and did a quick
> > > update
> > > > to the cwiki, but it could use some additional love, especially if we
> > > want
> > > > to split out the release processes generally.
> > > >
> > > > Jon
> > > >
> > > > On Wed, Oct 10, 2018 at 5:32 PM Justin Leet 
> > > wrote:
> > > >
> > > > > Yeah, we need to update the release process instructions. It should
> > > > > actually be step 10 shouldn't it?  We want to keep the rc tag as-is
> > > > because
> > > > > it may get rejected/cancelled, and update the final tag as the
> > > release. I
> > > > > already need to overhaul that page to reflect the addition of the
> RC
> > > > > creation script, so I can also update it.
> > > > >
> > > > > I didn't remember there was a specific reason for the git tagging,
> > so I
> > > > > just followed the convention we used for the main repo, which is
> why
> > we
> > > > > have this problem.
> > > > >
> > > > > The release candidate script also needs to be updated (because it
> > needs
> > > > to
> > > > > know the proper prior tag to pull). It's not hard to do, but we'll
> > need
> > > > to
> > > > > open a Jira and update that portion of the script. It's pretty easy
> > to
> > > > do,
> > > > > it's just the portion here
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245
> > > > > >
> > > > > .
> > > > >
> > > > > On Wed, Oct 10, 2018 at 5:09 PM Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > > > +1 to all of that from me, Jon. Thanks for taking care of this.
> > > > > >
> > > > > > On Wed, Oct 10, 2018 at 2:34 PM zeo...@gmail.com <
> zeo...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > I wonder if we should also update the
> > > > > > >
> > https://cwiki.apache.org/confluence/display/METRON/Release+Process
> > > > > > > instructions
> > > > > > > to include tagging for the bro plugin, or if we were going to
> > split
> > > > out
> > > > > > the
> > > > > > > release processes?  I'd be happy to update the instructions
> (Step
> > > 8)
> > > > if
> > > > > > > that's 

Re: Bro plugin release process docs?

[zeo...@gmail.com]

Is there a reason why the prefix for apache/metron ends with a -, whereas
the plugin ends with a _ separator?  I would like to see it more uniform
using the _

Jon

On Thu, Oct 11, 2018 at 9:04 AM Justin Leet  wrote:

> That first half of that page will be rewritten in the next couple days to
> reflect the script to prepare release candidate; I didn't want to change
> that page until the PR actually went in.
>
> The release process, barring the KEYS issue (and a patch to the script to
> handle the bro plugin tag), is pretty much set to be managed separately at
> this point.  The docs just need an overhaul, so someone who's not me knows
> what to do.
>
> On Wed, Oct 10, 2018 at 7:01 PM zeo...@gmail.com  wrote:
>
> > Yeah you're right when I looked closer to make the change it was step 10.
> > I pushed a manual 0.2 tag to metron-bro-plugin-kafka and did a quick
> update
> > to the cwiki, but it could use some additional love, especially if we
> want
> > to split out the release processes generally.
> >
> > Jon
> >
> > On Wed, Oct 10, 2018 at 5:32 PM Justin Leet 
> wrote:
> >
> > > Yeah, we need to update the release process instructions. It should
> > > actually be step 10 shouldn't it?  We want to keep the rc tag as-is
> > because
> > > it may get rejected/cancelled, and update the final tag as the
> release. I
> > > already need to overhaul that page to reflect the addition of the RC
> > > creation script, so I can also update it.
> > >
> > > I didn't remember there was a specific reason for the git tagging, so I
> > > just followed the convention we used for the main repo, which is why we
> > > have this problem.
> > >
> > > The release candidate script also needs to be updated (because it needs
> > to
> > > know the proper prior tag to pull). It's not hard to do, but we'll need
> > to
> > > open a Jira and update that portion of the script. It's pretty easy to
> > do,
> > > it's just the portion here
> > > <
> > >
> >
> https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245
> > > >
> > > .
> > >
> > > On Wed, Oct 10, 2018 at 5:09 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > +1 to all of that from me, Jon. Thanks for taking care of this.
> > > >
> > > > On Wed, Oct 10, 2018 at 2:34 PM zeo...@gmail.com 
> > > wrote:
> > > >
> > > > > I wonder if we should also update the
> > > > > https://cwiki.apache.org/confluence/display/METRON/Release+Process
> > > > > instructions
> > > > > to include tagging for the bro plugin, or if we were going to split
> > out
> > > > the
> > > > > release processes?  I'd be happy to update the instructions (Step
> 8)
> > if
> > > > > that's the right place for now and I didn't miss a new place for
> the
> > > > plugin
> > > > > release instructions.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Wed, Oct 10, 2018 at 4:31 PM zeo...@gmail.com  >
> > > > wrote:
> > > > >
> > > > > > So I was poking around on the plugin today and noticed that we
> have
> > > > > > a apache-metron-bro-plugin-kafka_0.2.0-release and
> > > > > apache-metron-bro-plugin-kafka_0.2.0-rc1
> > > > > > tag, but no 0.2 (which is what bro-pkg would point to).  Anybody
> > have
> > > > any
> > > > > > concerns if I push the 0.2 tag as discussed above?  Then we could
> > > > update
> > > > > > the bro package manager, and finally update what the
> apache/metron
> > > > > full-dev
> > > > > > environment(s) point to (0.2 as opposed to 0.1).  Thanks,
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com <
> zeo...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> I did a bit of poking around and I don't believe we ever
> formally
> > > > wrote
> > > > > >> that down.  The last release happened as a combination of
> actions
> > > from
> > > > > >> mattf and myself (mo

Re: Bro plugin release process docs?

[zeo...@gmail.com]

Yeah you're right when I looked closer to make the change it was step 10.
I pushed a manual 0.2 tag to metron-bro-plugin-kafka and did a quick update
to the cwiki, but it could use some additional love, especially if we want
to split out the release processes generally.

Jon

On Wed, Oct 10, 2018 at 5:32 PM Justin Leet  wrote:

> Yeah, we need to update the release process instructions. It should
> actually be step 10 shouldn't it?  We want to keep the rc tag as-is because
> it may get rejected/cancelled, and update the final tag as the release. I
> already need to overhaul that page to reflect the addition of the RC
> creation script, so I can also update it.
>
> I didn't remember there was a specific reason for the git tagging, so I
> just followed the convention we used for the main repo, which is why we
> have this problem.
>
> The release candidate script also needs to be updated (because it needs to
> know the proper prior tag to pull). It's not hard to do, but we'll need to
> open a Jira and update that portion of the script. It's pretty easy to do,
> it's just the portion here
> <
> https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245
> >
> .
>
> On Wed, Oct 10, 2018 at 5:09 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > +1 to all of that from me, Jon. Thanks for taking care of this.
> >
> > On Wed, Oct 10, 2018 at 2:34 PM zeo...@gmail.com 
> wrote:
> >
> > > I wonder if we should also update the
> > > https://cwiki.apache.org/confluence/display/METRON/Release+Process
> > > instructions
> > > to include tagging for the bro plugin, or if we were going to split out
> > the
> > > release processes?  I'd be happy to update the instructions (Step 8) if
> > > that's the right place for now and I didn't miss a new place for the
> > plugin
> > > release instructions.
> > >
> > > Jon
> > >
> > > On Wed, Oct 10, 2018 at 4:31 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > So I was poking around on the plugin today and noticed that we have
> > > > a apache-metron-bro-plugin-kafka_0.2.0-release and
> > > apache-metron-bro-plugin-kafka_0.2.0-rc1
> > > > tag, but no 0.2 (which is what bro-pkg would point to).  Anybody have
> > any
> > > > concerns if I push the 0.2 tag as discussed above?  Then we could
> > update
> > > > the bro package manager, and finally update what the apache/metron
> > > full-dev
> > > > environment(s) point to (0.2 as opposed to 0.1).  Thanks,
> > > >
> > > > Jon
> > > >
> > > > On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com 
> > > wrote:
> > > >
> > > >> I did a bit of poking around and I don't believe we ever formally
> > wrote
> > > >> that down.  The last release happened as a combination of actions
> from
> > > >> mattf and myself (mostly mattf).
> > > >>
> > > >> The plugin has two new commits since the last release (1 bugfix 1
> > > >> feature) - if we want to couple version 0.2 of the plugin with a
> > metron
> > > >> 0.5.0 release we would need to make a 0.2 tag against HEAD of the
> > plugin
> > > >> repo, then increment the version in the ansible playbooks here
> > > >> <
> > >
> >
> https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30
> > > >
> > > >> .
> > > >>
> > > >> Or, if we just want to make a plugin release without changing
> > > >> apache/metron, we could just make a 0.2 tag in the plugin repo, and
> > then
> > > >> release it in a more disjointed way.  I know that's not super
> helpful
> > > since
> > > >> I don't have documentation for doing an apache release of the plugin
> > > other
> > > >> than hacking something together based on what's out there for
> > > apache/metron.
> > > >>
> > > >> Reference conversations:
> > > >>  -
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/2606166bd5e864f1b56db302099c9a8042cdadec8fa2692fef49493f@%3Cdev.metron.apache.org%3E
> > > >>  -
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/3aecbadbf3353e98c03ca4b680fcd998d0cd2bf5a4319238dd85ae75@%3Cdev.metron.apache.org%3E
> > > >>
> > > >> Jon
> > > >>
> > > >> On Fri, May 25, 2018 at 2:00 PM Justin Leet 
> > > >> wrote:
> > > >>
> > > >>> At the risk of exposing my ignorance, do we have the bro plugin
> > release
> > > >>> process documented anywhere?  We have a doc for the main release (
> > > >>> https://cwiki.apache.org/confluence/display/METRON/Release+Process
> ),
> > > >>> but I
> > > >>> haven't noticed one for the bro plugin.
> > > >>>
> > > >>> For the current RC, it's not included and it wasn't pushed for (it
> > has
> > > >>> less
> > > >>> changes for obvious reasons).  However, we should be making sure to
> > > >>> validate if its necessary to release and having the process
> > documented.
> > > >>>
> > > >>> Justin
> > > >>>
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > > --
> > > >
> > > > Jon
> > > >
> > > --
> > >
> > > Jon
> > >
> >
>
-- 

Jon

Re: Bro plugin release process docs?

[zeo...@gmail.com]

I wonder if we should also update the
https://cwiki.apache.org/confluence/display/METRON/Release+Process instructions
to include tagging for the bro plugin, or if we were going to split out the
release processes?  I'd be happy to update the instructions (Step 8) if
that's the right place for now and I didn't miss a new place for the plugin
release instructions.

Jon

On Wed, Oct 10, 2018 at 4:31 PM zeo...@gmail.com  wrote:

> So I was poking around on the plugin today and noticed that we have
> a apache-metron-bro-plugin-kafka_0.2.0-release and 
> apache-metron-bro-plugin-kafka_0.2.0-rc1
> tag, but no 0.2 (which is what bro-pkg would point to).  Anybody have any
> concerns if I push the 0.2 tag as discussed above?  Then we could update
> the bro package manager, and finally update what the apache/metron full-dev
> environment(s) point to (0.2 as opposed to 0.1).  Thanks,
>
> Jon
>
> On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com  wrote:
>
>> I did a bit of poking around and I don't believe we ever formally wrote
>> that down.  The last release happened as a combination of actions from
>> mattf and myself (mostly mattf).
>>
>> The plugin has two new commits since the last release (1 bugfix 1
>> feature) - if we want to couple version 0.2 of the plugin with a metron
>> 0.5.0 release we would need to make a 0.2 tag against HEAD of the plugin
>> repo, then increment the version in the ansible playbooks here
>> <https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30>
>> .
>>
>> Or, if we just want to make a plugin release without changing
>> apache/metron, we could just make a 0.2 tag in the plugin repo, and then
>> release it in a more disjointed way.  I know that's not super helpful since
>> I don't have documentation for doing an apache release of the plugin other
>> than hacking something together based on what's out there for apache/metron.
>>
>> Reference conversations:
>>  -
>> https://lists.apache.org/thread.html/2606166bd5e864f1b56db302099c9a8042cdadec8fa2692fef49493f@%3Cdev.metron.apache.org%3E
>>  -
>> https://lists.apache.org/thread.html/3aecbadbf3353e98c03ca4b680fcd998d0cd2bf5a4319238dd85ae75@%3Cdev.metron.apache.org%3E
>>
>> Jon
>>
>> On Fri, May 25, 2018 at 2:00 PM Justin Leet 
>> wrote:
>>
>>> At the risk of exposing my ignorance, do we have the bro plugin release
>>> process documented anywhere?  We have a doc for the main release (
>>> https://cwiki.apache.org/confluence/display/METRON/Release+Process),
>>> but I
>>> haven't noticed one for the bro plugin.
>>>
>>> For the current RC, it's not included and it wasn't pushed for (it has
>>> less
>>> changes for obvious reasons).  However, we should be making sure to
>>> validate if its necessary to release and having the process documented.
>>>
>>> Justin
>>>
>> --
>>
>> Jon
>>
> --
>
> Jon
>
-- 

Jon

Re: Bro plugin release process docs?

[zeo...@gmail.com]

So I was poking around on the plugin today and noticed that we have
a apache-metron-bro-plugin-kafka_0.2.0-release and
apache-metron-bro-plugin-kafka_0.2.0-rc1
tag, but no 0.2 (which is what bro-pkg would point to).  Anybody have any
concerns if I push the 0.2 tag as discussed above?  Then we could update
the bro package manager, and finally update what the apache/metron full-dev
environment(s) point to (0.2 as opposed to 0.1).  Thanks,

Jon

On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com  wrote:

> I did a bit of poking around and I don't believe we ever formally wrote
> that down.  The last release happened as a combination of actions from
> mattf and myself (mostly mattf).
>
> The plugin has two new commits since the last release (1 bugfix 1 feature)
> - if we want to couple version 0.2 of the plugin with a metron 0.5.0
> release we would need to make a 0.2 tag against HEAD of the plugin repo,
> then increment the version in the ansible playbooks here
> <https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30>
> .
>
> Or, if we just want to make a plugin release without changing
> apache/metron, we could just make a 0.2 tag in the plugin repo, and then
> release it in a more disjointed way.  I know that's not super helpful since
> I don't have documentation for doing an apache release of the plugin other
> than hacking something together based on what's out there for apache/metron.
>
> Reference conversations:
>  -
> https://lists.apache.org/thread.html/2606166bd5e864f1b56db302099c9a8042cdadec8fa2692fef49493f@%3Cdev.metron.apache.org%3E
>  -
> https://lists.apache.org/thread.html/3aecbadbf3353e98c03ca4b680fcd998d0cd2bf5a4319238dd85ae75@%3Cdev.metron.apache.org%3E
>
> Jon
>
> On Fri, May 25, 2018 at 2:00 PM Justin Leet  wrote:
>
>> At the risk of exposing my ignorance, do we have the bro plugin release
>> process documented anywhere?  We have a doc for the main release (
>> https://cwiki.apache.org/confluence/display/METRON/Release+Process), but
>> I
>> haven't noticed one for the bro plugin.
>>
>> For the current RC, it's not included and it wasn't pushed for (it has
>> less
>> changes for obvious reasons).  However, we should be making sure to
>> validate if its necessary to release and having the process documented.
>>
>> Justin
>>
> --
>
> Jon
>
-- 

Jon

Re: [DISCUSS] Split apart releases for core Metron and the Bro plugin

[zeo...@gmail.com]

Good catch.  I'm not sure what the right approach is there, I wonder what
other projects do for the same situation

Jon

On Mon, Oct 8, 2018 at 3:24 PM Justin Leet  wrote:

> https://github.com/apache/metron/pull/1188 scripts the creation of RC's.
> It does this split, and allows for independent releases. The main catch is
> a slight hangup with the KEYS file.
>
> The metron-bro-kafka-plugin repo doesn't contain a KEYS file, it currently
> has just been aligned with the main repo's KEYS file.The KEYS file doesn't
> need to live in each submodule's directory, just in the main folder (
> https://dist.apache.org/repos/dist/release/metron/).  This means it's not
> a
> problem for the plugin repo to be missing it.
>
> The main catch is that right now, we only publish the KEYS file when we
> release the main repo. In the (admittedly rare) situation where we have a
> new release manager who has to update the KEYS file, we'd currently need to
> do a metron release before we could do a plugin release.
>
> Should we split out and document a KEYS update from the main repo?
> Essentially, just run through the normal PR process and then have a post
> merge step of kicking up the updated KEYS file.  Then Release Process
> <https://cwiki.apache.org/confluence/display/METRON/Release+Process> would
> just get updated to reflect this process.
>
>
> On Fri, Sep 7, 2018 at 3:15 PM Casey Stella  wrote:
>
> > +1 to defer for this release and complete separation.  Good fences make
> > good submodules. ;)
> >
> > On Fri, Sep 7, 2018 at 2:33 PM zeo...@gmail.com 
> wrote:
> >
> > > +1 to defer for this release and +1 to Justin's suggested release/dist
> > > directory breakout and complete separation.
> > >
> > > Jon
> > >
> > > On Fri, Sep 7, 2018 at 1:43 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > +1 to deferring for this release and having the separation like NiFi.
> > > Since
> > > > we're bootstrapping from their process, what are they doing? I would
> > > assume
> > > > we'd want some sort of vote for the plugin version change as well.
> > > >
> > > > On Fri, Sep 7, 2018 at 10:15 AM Nick Allen 
> wrote:
> > > >
> > > > > +1 for complete separation as you've described.
> > > > >
> > > > > On Fri, Sep 7, 2018 at 11:31 AM Justin Leet  >
> > > > wrote:
> > > > >
> > > > > > I would like this to be a complete separation.  Complete with
> > > separate
> > > > > RCs,
> > > > > > separate call to vote, etc. There's a bit more overhead, but
> plugin
> > > > > > releases should be rarer and as the release infra gets improved
> and
> > > > > > scripted out more, I don't think it'll end up being much more
> than
> > > > > bundling
> > > > > > it together.
> > > > > >
> > > > > > On Fri, Sep 7, 2018 at 11:27 AM Nick Allen 
> > > wrote:
> > > > > >
> > > > > > > > Other projects, e.g. NiFi <http://apache.org/dist/nifi/>,
> > split
> > > > > apart
> > > > > > > these releases within their dist directories.
> > > > > > >
> > > > > > > I prefer the way Nifi organizes it.  Definitely seems more
> > > logically
> > > > > > > organized.
> > > > > > >
> > > > > > >
> > > > > > > > If we split them apart, we can make the releases
> independently.
> > > > This
> > > > > > > fixes the problem of aligning the versions (simply release the
> > > plugin
> > > > > > > first, update full-dev, release core Metron).
> > > > > > >
> > > > > > > Does this entail a complete separation; including a separate
> > > > > > call-to-vote?
> > > > > > > One vote for core Metron and a separate vote for plugin?
> > > > > > >
> > > > > > >
> > > > > > > > Do we want try to get this separation done after the current
> > > > release
> > > > > > > cycle is over?
> > > > > > >
> > > > > > > +1 Let's wait for the next release to hash this out.
> > > > > > >
> > > > > > &

Re: Metron dev environments moving to require Ansible 2.4+

[zeo...@gmail.com]

Hi All,

This has been pushed to master.  Please updated your Ansible
appropriately.  Thanks!

Jon

On Fri, Sep 28, 2018 at 12:18 PM Otto Fowler 
wrote:

> Yeah,  I thought we had more but maybe they where removed.
> Many places in *.md files referencing Ansible and versions too
>
>
> On September 28, 2018 at 11:45:14, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> Do you mean this
> <https://cwiki.apache.org/confluence/display/METRON/Downgrade+Ansible>?
> It was the only reference I could find on the wiki.  All of the READMEs
> should be updated as a part of the PR, but feel free to provide your input
> if I missed anything.
>
> Jon
>
> On Fri, Sep 28, 2018 at 10:15 AM Otto Fowler 
> wrote:
>
>> We should make sure the non-source documentation is updated
>>
>>
>> On September 28, 2018 at 09:32:52, zeo...@gmail.com (zeo...@gmail.com)
>> wrote:
>>
>> Hi All,
>>
>> As it currently sits, once METRON-1758
>> <https://github.com/apache/metron/pull/1179> is merged into the code
>> base, Ansible 2.4 or later will be required to use any of the Metron
>> ansible playbooks.  This is in contrast to the prior version requirements
>> outlined in Metron documentation which specifically point to 2.0.0.2 and
>> 2.2.0.0 as supported/recommended Ansible versions.  If you install Ansible
>> 2.5.0 exactly you should not experience any issues spinning up pre- and post-
>> merge versions of Metron.
>>
>> I am broadcasting this to both the user and dev communities in advance of
>> any changes to provide an opportunity to voice any concerns.  Thanks,
>>
>> Jon
>> --
>>
>> Jon
>>
>> --
>
> Jon
>
> --

Jon

Re: Metron dev environments moving to require Ansible 2.4+

[zeo...@gmail.com]

Do you mean this
<https://cwiki.apache.org/confluence/display/METRON/Downgrade+Ansible>?  It
was the only reference I could find on the wiki.  All of the READMEs should
be updated as a part of the PR, but feel free to provide your input if I
missed anything.

Jon

On Fri, Sep 28, 2018 at 10:15 AM Otto Fowler 
wrote:

> We should make sure the non-source documentation is updated
>
>
> On September 28, 2018 at 09:32:52, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> Hi All,
>
> As it currently sits, once METRON-1758
> <https://github.com/apache/metron/pull/1179> is merged into the code
> base, Ansible 2.4 or later will be required to use any of the Metron
> ansible playbooks.  This is in contrast to the prior version requirements
> outlined in Metron documentation which specifically point to 2.0.0.2 and
> 2.2.0.0 as supported/recommended Ansible versions.  If you install Ansible
> 2.5.0 exactly you should not experience any issues spinning up pre- and post-
> merge versions of Metron.
>
> I am broadcasting this to both the user and dev communities in advance of
> any changes to provide an opportunity to voice any concerns.  Thanks,
>
> Jon
> --
>
> Jon
>
> --

Jon

Metron dev environments moving to require Ansible 2.4+

[zeo...@gmail.com]

Hi All,

As it currently sits, once METRON-1758
 is merged into the code base,
Ansible 2.4 or later will be required to use any of the Metron ansible
playbooks.  This is in contrast to the prior version requirements outlined
in Metron documentation which specifically point to 2.0.0.2 and 2.2.0.0 as
supported/recommended Ansible versions.  If you install Ansible 2.5.0
exactly you should not experience any issues spinning up pre- and post-
merge versions of Metron.

I am broadcasting this to both the user and dev communities in advance of
any changes to provide an opportunity to voice any concerns.  Thanks,

Jon
-- 

Jon

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

Yup, your first paragraph is exactly right.

Jon

On Fri, Sep 7, 2018 at 1:23 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I'm a little rusty on my C++, but to summarize what I've gathered from this
> thread and linked code: The Bro plugin infrastructure does not support a
> patch version currently, only major.minor (x.y). But bro-pkg, the bro
> package installation mechanism, does support it now. Which is unfortunately
> moot bc unlike other 3rd party scripts that can support vX.Y.Z, this is a
> plugin and the API does not support the 'Y'.
>
> I think I'm in favor of keeping X.Y (0.2) for now. I would urge this
> approach by comparison to calculating significant figures - if my tooling
> is only able to calculate 2 decimal places, e.g. 1.23, I shouldn't be
> recording 1.230 or 1.23 as this isn't reflective of the tool's actual
> fidelity. Likewise, the Bro plugin API doesn't allow X.Y.Z right now, only
> X.Y per Jon's comments. Not strictly speaking an apples to apples
> comparison, but it's a justifiable approach imho. Once the plugin
> infrastructure supports it, we can add the extra patch version to reflect
> this additional available state info.
>
> Best,
> Mike
>
>
> On Thu, Sep 6, 2018 at 7:34 PM zeo...@gmail.com  wrote:
>
> > I'm not aware of the bro plugin artifacts being used in any way.
> >
> > Jon
> >
> > On Thu, Sep 6, 2018, 10:59 Justin Leet  wrote:
> >
> > > Do we use the artifacts directly at all? Or is it through bro-pkg only?
> > >
> > > Also, It's very possible I'm making a mountain out of a molehill here,
> > and
> > > if it's something that's not particularly important, it might be
> > worthwhile
> > > to just stick with what we did last time, and just table this
> discussion
> > > until post release.  It feels pretty nitpicky at this point, and if the
> > > practical implications are pretty minor, I'd rather just get an RC out.
> > >
> > > On Thu, Sep 6, 2018 at 10:02 AM zeo...@gmail.com 
> > wrote:
> > >
> > > > Either is fine with me.  If it's x.y in some parts of the app I
> prefer
> > to
> > > > keep it consistent throughout, but I'm also fine with lining up with
> > > > Apache/Metron where we can.
> > > >
> > > > I also refreshed myself on why we avoided x.y.z initially and it was
> > > > actually for this exact reason, we wanted consistent versioning
> > > throughout
> > > > a repo.  This issue is with the bro plugins themselves, not bro-pkg,
> > so I
> > > > submit a JIRA <https://bro-tracker.atlassian.net/browse/BIT-1985>.
> > > >
> > > > Jon
> > > >
> > > >
> > > > On Wed, Sep 5, 2018, 21:51 Justin Leet 
> wrote:
> > > >
> > > > > Makes sense. Do we have any objection to just going to the artifact
> > > being
> > > > > 0.2?  Or do we want to keep the mixed versioning and just live with
> > it,
> > > > at
> > > > > least for now?
> > > > >
> > > > > On Wed, Sep 5, 2018 at 8:58 PM zeo...@gmail.com 
> > > > wrote:
> > > > >
> > > > > > I think mattf-horton just did that as a part of convention.  He
> > > handled
> > > > > > that part, and I did the 0.1 tagging (as a prereq to this
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30
> > > > > > >)
> > > > > > last time the package was released.
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Wed, Sep 5, 2018 at 8:28 PM Justin Leet <
> justinjl...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Any idea why we released it as 0.1.0 in the artifacts version?
> > I'm
> > > > > fine
> > > > > > > with doing x.y if we need to, but I would like the artifact
> > > > versioning
> > > > > to
> > > > > > > be consistent if possible.
> > > > > > >
> > > > > > > On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com <
> > zeo...@gmail.com>
> > > > > > wrote:
> > > > > > &g

Re: [DISCUSS] Split apart releases for core Metron and the Bro plugin

[zeo...@gmail.com]

+1 to defer for this release and +1 to Justin's suggested release/dist
directory breakout and complete separation.

Jon

On Fri, Sep 7, 2018 at 1:43 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 to deferring for this release and having the separation like NiFi. Since
> we're bootstrapping from their process, what are they doing? I would assume
> we'd want some sort of vote for the plugin version change as well.
>
> On Fri, Sep 7, 2018 at 10:15 AM Nick Allen  wrote:
>
> > +1 for complete separation as you've described.
> >
> > On Fri, Sep 7, 2018 at 11:31 AM Justin Leet 
> wrote:
> >
> > > I would like this to be a complete separation.  Complete with separate
> > RCs,
> > > separate call to vote, etc. There's a bit more overhead, but plugin
> > > releases should be rarer and as the release infra gets improved and
> > > scripted out more, I don't think it'll end up being much more than
> > bundling
> > > it together.
> > >
> > > On Fri, Sep 7, 2018 at 11:27 AM Nick Allen  wrote:
> > >
> > > > > Other projects, e.g. NiFi , split
> > apart
> > > > these releases within their dist directories.
> > > >
> > > > I prefer the way Nifi organizes it.  Definitely seems more logically
> > > > organized.
> > > >
> > > >
> > > > > If we split them apart, we can make the releases independently.
> This
> > > > fixes the problem of aligning the versions (simply release the plugin
> > > > first, update full-dev, release core Metron).
> > > >
> > > > Does this entail a complete separation; including a separate
> > > call-to-vote?
> > > > One vote for core Metron and a separate vote for plugin?
> > > >
> > > >
> > > > > Do we want try to get this separation done after the current
> release
> > > > cycle is over?
> > > >
> > > > +1 Let's wait for the next release to hash this out.
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Sep 7, 2018 at 10:27 AM Justin Leet 
> > > wrote:
> > > >
> > > > > Right now, we tie together our main release and the Bro plugin, as
> > seen
> > > > in
> > > > > our 0.4.2 release
> > > > > https://archive.apache.org/dist/metron/0.4.2/ and the current RC.
> > > > >
> > > > > Other projects, e.g. NiFi , split
> > apart
> > > > > these
> > > > > releases within their dist directories.
> > > > >
> > > > > In our case this might look something like
> > > > > 0.5.0/
> > > > > metron-bro-plugin-kafka
> > > > > - 0.2.0/
> > > > >
> > > > > Right now, with the releases tied together, we aren't upgrading
> > > full-dev
> > > > > with the version of the plugin (because we're releasing
> > simultaneously
> > > > and
> > > > > can't update the version number).
> > > > >
> > > > > If we split them apart, we can make the releases independently.
> This
> > > > fixes
> > > > > the problem of aligning the versions (simply release the plugin
> > first,
> > > > > update full-dev, release core Metron).  The plugin also updates
> > > > > substantially less often and we can just do those releases at a
> > cadence
> > > > we
> > > > > choose.
> > > > >
> > > > > Any thoughts on doing this?
> > > > > Do we want try to get this separation done after the current
> release
> > > > cycle
> > > > > is over?
> > > > > If we do, do we have a preferred layout? I didn't see anything
> Apache
> > > > > preferred in a quick search, but I definitely could have missed
> > > something
> > > > > (and https://checker.apache.org/projs/nifi.html looks clean for
> > NiFi,
> > > > so I
> > > > > assume it's fine.)
> > > > >
> > > >
> > >
> >
>
-- 

Jon

Re: [DISCUSS] Feature branches post-merge

[zeo...@gmail.com]

Yeah I don't have a good reason to suggest we keep 'em. so +1 to deleting
old FBs.

Jon

On Fri, Sep 7, 2018 at 12:14 PM Nick Allen  wrote:

> +1 delete old feature branches.
>
> BTW, there is a branch out there called METRON-113 that we probably need to
> clean-up.  I'm not sure where that came from or why its still around.
> Probably a fat-finger from long ago.
>
> On Fri, Sep 7, 2018 at 12:00 PM Otto Fowler 
> wrote:
>
> > I would drop them.
> > I’ve already clean up FB’s around dead things.
> >
> >
> >
> > On September 6, 2018 at 13:42:55, Michael Miklavcic (
> > michael.miklav...@gmail.com) wrote:
> >
> > What are we doing with feature branches once they're complete and merged
> > into master? Is our expectation that we'll keep feature branches in
> > perpetuity, or should we plan to do some house cleaning once they've been
> > merged? I did a quick check of NiFi and Kafka and don't see much by way
> of
> > feature branches in their repos. I see plenty of RC's in both the
> branches
> > and tags listings, but nothing FB related. In previous discussions, we
> > talked quite a bit about us "trailblazing here," so it may be that this
> is
> > simply without much precedent and entirely for us to decide. I can
> > definitely see value in maintaining them for future reference, as it does
> > offer a nice bucket in which to collect the commits and discussion
> nicely,
> > but I wanted to get others' thoughts.
> >
> > Best,
> > Mike
> >
>
-- 

Jon

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

I'm not aware of the bro plugin artifacts being used in any way.

Jon

On Thu, Sep 6, 2018, 10:59 Justin Leet  wrote:

> Do we use the artifacts directly at all? Or is it through bro-pkg only?
>
> Also, It's very possible I'm making a mountain out of a molehill here, and
> if it's something that's not particularly important, it might be worthwhile
> to just stick with what we did last time, and just table this discussion
> until post release.  It feels pretty nitpicky at this point, and if the
> practical implications are pretty minor, I'd rather just get an RC out.
>
> On Thu, Sep 6, 2018 at 10:02 AM zeo...@gmail.com  wrote:
>
> > Either is fine with me.  If it's x.y in some parts of the app I prefer to
> > keep it consistent throughout, but I'm also fine with lining up with
> > Apache/Metron where we can.
> >
> > I also refreshed myself on why we avoided x.y.z initially and it was
> > actually for this exact reason, we wanted consistent versioning
> throughout
> > a repo.  This issue is with the bro plugins themselves, not bro-pkg, so I
> > submit a JIRA <https://bro-tracker.atlassian.net/browse/BIT-1985>.
> >
> > Jon
> >
> >
> > On Wed, Sep 5, 2018, 21:51 Justin Leet  wrote:
> >
> > > Makes sense. Do we have any objection to just going to the artifact
> being
> > > 0.2?  Or do we want to keep the mixed versioning and just live with it,
> > at
> > > least for now?
> > >
> > > On Wed, Sep 5, 2018 at 8:58 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > I think mattf-horton just did that as a part of convention.  He
> handled
> > > > that part, and I did the 0.1 tagging (as a prereq to this
> > > > <
> > > >
> > >
> >
> https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30
> > > > >)
> > > > last time the package was released.
> > > >
> > > > Jon
> > > >
> > > > On Wed, Sep 5, 2018 at 8:28 PM Justin Leet 
> > > wrote:
> > > >
> > > > > Any idea why we released it as 0.1.0 in the artifacts version?  I'm
> > > fine
> > > > > with doing x.y if we need to, but I would like the artifact
> > versioning
> > > to
> > > > > be consistent if possible.
> > > > >
> > > > > On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com 
> > > > wrote:
> > > > >
> > > > > > I lied, we didn't need to update our btests because it's limited
> > to a
> > > > > major
> > > > > > and minor version.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com <
> zeo...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I looked into x.y.z back when we released 0.1 and it was not
> > > possible
> > > > > in
> > > > > > > bro-pkg at the time but now it is
> > > > > > > <https://github.com/bro/package-manager/issues/32>.  In order
> to
> > > do
> > > > > > this,
> > > > > > > we'll also need to configure bro-pkg.meta to require the proper
> > > > version
> > > > > > of
> > > > > > > bro-pkg, as well as update the btests for the new version
> string.
> > > I
> > > > > will
> > > > > > > throw together a JIRA and PR to do all this in case we decide
> to
> > > > align
> > > > > > with
> > > > > > > x.y.z; we can trash it if we decide to stay with x.y.
> > > > > > >
> > > > > > > Jon
> > > > > > >
> > > > > > > On Wed, Sep 5, 2018 at 7:35 PM Justin Leet <
> > justinjl...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Long story short, while preparing the release candidate, I
> > > > discovered
> > > > > > our
> > > > > > >> metron-bro-plugin-kafka is inconsistently versioned.
> 

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

Either is fine with me.  If it's x.y in some parts of the app I prefer to
keep it consistent throughout, but I'm also fine with lining up with
Apache/Metron where we can.

I also refreshed myself on why we avoided x.y.z initially and it was
actually for this exact reason, we wanted consistent versioning throughout
a repo.  This issue is with the bro plugins themselves, not bro-pkg, so I
submit a JIRA <https://bro-tracker.atlassian.net/browse/BIT-1985>.

Jon


On Wed, Sep 5, 2018, 21:51 Justin Leet  wrote:

> Makes sense. Do we have any objection to just going to the artifact being
> 0.2?  Or do we want to keep the mixed versioning and just live with it, at
> least for now?
>
> On Wed, Sep 5, 2018 at 8:58 PM zeo...@gmail.com  wrote:
>
> > I think mattf-horton just did that as a part of convention.  He handled
> > that part, and I did the 0.1 tagging (as a prereq to this
> > <
> >
> https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30
> > >)
> > last time the package was released.
> >
> > Jon
> >
> > On Wed, Sep 5, 2018 at 8:28 PM Justin Leet 
> wrote:
> >
> > > Any idea why we released it as 0.1.0 in the artifacts version?  I'm
> fine
> > > with doing x.y if we need to, but I would like the artifact versioning
> to
> > > be consistent if possible.
> > >
> > > On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > I lied, we didn't need to update our btests because it's limited to a
> > > major
> > > > and minor version.
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34
> > > >
> > > > Jon
> > > >
> > > > On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com 
> > > wrote:
> > > >
> > > > > I looked into x.y.z back when we released 0.1 and it was not
> possible
> > > in
> > > > > bro-pkg at the time but now it is
> > > > > <https://github.com/bro/package-manager/issues/32>.  In order to
> do
> > > > this,
> > > > > we'll also need to configure bro-pkg.meta to require the proper
> > version
> > > > of
> > > > > bro-pkg, as well as update the btests for the new version string.
> I
> > > will
> > > > > throw together a JIRA and PR to do all this in case we decide to
> > align
> > > > with
> > > > > x.y.z; we can trash it if we decide to stay with x.y.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Wed, Sep 5, 2018 at 7:35 PM Justin Leet 
> > > > wrote:
> > > > >
> > > > >> Long story short, while preparing the release candidate, I
> > discovered
> > > > our
> > > > >> metron-bro-plugin-kafka is inconsistently versioned.
> > > > >>
> > > > >> In the repo, it's x.y (e.g. 0.2)
> > > > >> See:
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18
> > > > >>
> > > > >> In our released artifact, it's x.y.z (e.g. 0.1.0)
> > > > >> See http://archive.apache.org/dist/metron/0.4.2/
> > > > >>
> > > > >> Going forward from this release, I'd like the artifact to be
> > > consistent
> > > > >> with the repo.  I'd personally prefer x.y.z to be entirely
> > consistent
> > > > >> throughout Metron, but if there's a particular reason why it was
> x.y
> > > I'm
> > > > >> happy to entertain it.
> > > > >>
> > > > >> If we choose to move to x.y.z, I can provide a PR to update the
> > > version
> > > > to
> > > > >> 0.2.0 unless someone else wants to volunteer. Otherwise, I'd like
> to
> > > > >> release the artifact as apache-metron-bro-plugin-kafka_0.2.tar.gz
> > > > >>
> > > > >> Justin
> > > > >>
> > > > >> On Tue, Sep 4, 2018 at 12:15 PM Justin Leet <
> justinjl...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > As an update, I'll be working on starting the release process
> > > rolling
> > &

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

I think mattf-horton just did that as a part of convention.  He handled
that part, and I did the 0.1 tagging (as a prereq to this
<https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30>)
last time the package was released.

Jon

On Wed, Sep 5, 2018 at 8:28 PM Justin Leet  wrote:

> Any idea why we released it as 0.1.0 in the artifacts version?  I'm fine
> with doing x.y if we need to, but I would like the artifact versioning to
> be consistent if possible.
>
> On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com  wrote:
>
> > I lied, we didn't need to update our btests because it's limited to a
> major
> > and minor version.
> >
> >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34
> >
> > Jon
> >
> > On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com 
> wrote:
> >
> > > I looked into x.y.z back when we released 0.1 and it was not possible
> in
> > > bro-pkg at the time but now it is
> > > <https://github.com/bro/package-manager/issues/32>.  In order to do
> > this,
> > > we'll also need to configure bro-pkg.meta to require the proper version
> > of
> > > bro-pkg, as well as update the btests for the new version string.  I
> will
> > > throw together a JIRA and PR to do all this in case we decide to align
> > with
> > > x.y.z; we can trash it if we decide to stay with x.y.
> > >
> > > Jon
> > >
> > > On Wed, Sep 5, 2018 at 7:35 PM Justin Leet 
> > wrote:
> > >
> > >> Long story short, while preparing the release candidate, I discovered
> > our
> > >> metron-bro-plugin-kafka is inconsistently versioned.
> > >>
> > >> In the repo, it's x.y (e.g. 0.2)
> > >> See:
> > >>
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18
> > >>
> > >> In our released artifact, it's x.y.z (e.g. 0.1.0)
> > >> See http://archive.apache.org/dist/metron/0.4.2/
> > >>
> > >> Going forward from this release, I'd like the artifact to be
> consistent
> > >> with the repo.  I'd personally prefer x.y.z to be entirely consistent
> > >> throughout Metron, but if there's a particular reason why it was x.y
> I'm
> > >> happy to entertain it.
> > >>
> > >> If we choose to move to x.y.z, I can provide a PR to update the
> version
> > to
> > >> 0.2.0 unless someone else wants to volunteer. Otherwise, I'd like to
> > >> release the artifact as apache-metron-bro-plugin-kafka_0.2.tar.gz
> > >>
> > >> Justin
> > >>
> > >> On Tue, Sep 4, 2018 at 12:15 PM Justin Leet 
> > >> wrote:
> > >>
> > >> > As an update, I'll be working on starting the release process
> rolling
> > >> > today. The PCAP Query panel feature branch is in master and I
> haven't
> > >> heard
> > >> > of any other potential blockers. I'd appreciate everyone updating
> > their
> > >> > JIRAs with their state (complete, etc.) and version (0.6.0) as
> needed.
> > >> >
> > >> > Do we have anything that needs to go into the UPGRADING.md file?
> > >> >
> > >> > A PR is out for updating the version from 0.5.1 to 0.6.0. Please see
> > >> > https://github.com/apache/metron/pull/1183. I still need to spin up
> > >> full
> > >> > dev etc. before this is ready to be merged.
> > >> >
> > >> > Updated list of PRs that have made it into master as of Sept 4, 2018
> > >> >
> > >> > 6 days ago METRON-1751 Storm Profiler dies when consuming null
> message
> > >> > (nickwallen) closes apache/metron#1176
> > >> > 6 days ago METRON-1757 Storm Profiler Serialization Exception
> > >> (nickwallen)
> > >> > closes apache/metron#1178
> > >> > 6 days ago METRON-1743 CEF testPaloAltoCEF test using a confusing
> > >> variable
> > >> > name (JonZeolla via justinleet) closes apache/metron#1173
> > >> > 8 days ago METRON-1752 Prevent package.lock from changing during
> build
> > >> > (sardell via merrimanr) closes apache/metron#1177
> > >> > 8 days ago METRON-1724 Date/time validation missing in PCAP query
> > >> (tiborm
> > >> > via nickwallen) cl

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

I lied, we didn't need to update our btests because it's limited to a major
and minor version.

https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34

Jon

On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com  wrote:

> I looked into x.y.z back when we released 0.1 and it was not possible in
> bro-pkg at the time but now it is
> <https://github.com/bro/package-manager/issues/32>.  In order to do this,
> we'll also need to configure bro-pkg.meta to require the proper version of
> bro-pkg, as well as update the btests for the new version string.  I will
> throw together a JIRA and PR to do all this in case we decide to align with
> x.y.z; we can trash it if we decide to stay with x.y.
>
> Jon
>
> On Wed, Sep 5, 2018 at 7:35 PM Justin Leet  wrote:
>
>> Long story short, while preparing the release candidate, I discovered our
>> metron-bro-plugin-kafka is inconsistently versioned.
>>
>> In the repo, it's x.y (e.g. 0.2)
>> See:
>> https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18
>>
>> In our released artifact, it's x.y.z (e.g. 0.1.0)
>> See http://archive.apache.org/dist/metron/0.4.2/
>>
>> Going forward from this release, I'd like the artifact to be consistent
>> with the repo.  I'd personally prefer x.y.z to be entirely consistent
>> throughout Metron, but if there's a particular reason why it was x.y I'm
>> happy to entertain it.
>>
>> If we choose to move to x.y.z, I can provide a PR to update the version to
>> 0.2.0 unless someone else wants to volunteer. Otherwise, I'd like to
>> release the artifact as apache-metron-bro-plugin-kafka_0.2.tar.gz
>>
>> Justin
>>
>> On Tue, Sep 4, 2018 at 12:15 PM Justin Leet 
>> wrote:
>>
>> > As an update, I'll be working on starting the release process rolling
>> > today. The PCAP Query panel feature branch is in master and I haven't
>> heard
>> > of any other potential blockers. I'd appreciate everyone updating their
>> > JIRAs with their state (complete, etc.) and version (0.6.0) as needed.
>> >
>> > Do we have anything that needs to go into the UPGRADING.md file?
>> >
>> > A PR is out for updating the version from 0.5.1 to 0.6.0. Please see
>> > https://github.com/apache/metron/pull/1183. I still need to spin up
>> full
>> > dev etc. before this is ready to be merged.
>> >
>> > Updated list of PRs that have made it into master as of Sept 4, 2018
>> >
>> > 6 days ago METRON-1751 Storm Profiler dies when consuming null message
>> > (nickwallen) closes apache/metron#1176
>> > 6 days ago METRON-1757 Storm Profiler Serialization Exception
>> (nickwallen)
>> > closes apache/metron#1178
>> > 6 days ago METRON-1743 CEF testPaloAltoCEF test using a confusing
>> variable
>> > name (JonZeolla via justinleet) closes apache/metron#1173
>> > 8 days ago METRON-1752 Prevent package.lock from changing during build
>> > (sardell via merrimanr) closes apache/metron#1177
>> > 8 days ago METRON-1724 Date/time validation missing in PCAP query
>> (tiborm
>> > via nickwallen) closes apache/metron#1172
>> > 3 weeks ago METRON-1554 Pcap Query Panel (merrimanr) closes
>> > apache/metron#1169
>> > 3 weeks ago METRON-1739 UDP packets are not handled (merrimanr) closes
>> > apache/metron#1168
>> > 3 weeks ago METRON-1727: Alerts are not populated on the alerts UI after
>> > enabling X-pack for Elastic search (MohanDV via mmiklavc) closes
>> > apache/metron#1141
>> > 3 weeks ago METRON-1738: Pcap directories should have correct
>> permissions
>> > (merrimanr via mmiklavc) closes apache/metron#1166
>> > 3 weeks ago METRON-1737: Document Job cleanup (merrimanr via mmiklavc)
>> > closes apache/metron#1164
>> > 3 weeks ago METRON-1732: Fix job status liveness bug and parallelize
>> > finalizer file writing (mmiklavc via mmiklavc) closes apache/metron#1157
>> > 3 weeks ago METRON-1735 Empty print status option causes NPE (merrimanr)
>> > closes apache/metron#1160
>> > 3 weeks ago METRON-1733 PCAP UI - PCAP queries don't work on Safari
>> > (sardell via merrimanr) closes apache/metron#1158
>> > 3 weeks ago METRON-1734 Src and Dst port filters are incorrect after
>> > changing to empty (merrimanr) closes apache/metron#1159
>> > 4 weeks ago METRON-1725 Add ability to specify YARN queue for pcap jobs
>> > (merrimanr) closes apache/metron#1153
>> > 4 weeks ago METRON-1731: 

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

 ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago METRON-1617: Make threat triage score function with
> >> dots as
> >> > > > well as colons closes apache/incubator-metron#1062
> >> > > > 9 weeks ago METRON-1613 Metaalerts status update broken in Alerts
> UI
> >> > > > (merrimanr) closes apache/metron#1059
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago METRON-1588 Migrate storm-kafka-client to 1.2.1 closes
> >> > > > apache/incubator-metron#1039
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago Merge branch 'feature/METRON-1416-upgrade-solr' of
> >> > > > https://git-wip-us.apache.org/repos/asf/metron into
> >> > > > feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago Merge branch 'master' into
> >> feature/METRON-1416-upgrade-solr
> >> > > > 9 weeks ago METRON-1587 Make collection utility work for HDP
> search
> >> > > > (merrimanr) closes apache/metron#1043
> >> > > > 9 weeks ago METRON-1612 Fix website download links (justinleet)
> >> closes
> >> > > > apache/metron#1058
> >> > > > 9 weeks ago METRON-1608 Add configuration for threat.triage.field
> >> name
> >> > > > (merrimanr) closes apache/metron#1055
> >> > > > 10 weeks ago METRON-1585 SolrRetrieveLatestDao does not use the
> >> > > collection
> >> > > > lookup (justinleet via merrimanr) closes apache/metron#1050
> >> > > > 10 weeks ago METRON-1533 Create KAFKA_FIND Stellar Function
> >> > (nickwallen)
> >> > > > closes apache/metron#1025
> >> > > > 10 weeks ago METRON-1601: Rename metaalert alert nested field to
> >> > > > metron_alert to avoid collision closes
> apache/incubator-metron#1049
> >> > > > 10 weeks ago METRON-1572 Enhance KAFKA_PUT function (nickwallen)
> >> closes
> >> > > > apache/metron#1024
> >> > > > 10 weeks ago METRON-1607 update public web site to point at 0.5.0
> >> new
> >> > > > release (justinleet) closes apache/metron#1053
> >> > > > 10 weeks ago METRON-1568: Stellar should have a _ special variable
> >> > which
> >> > > > returns the message in map form closes
> apache/incubator-metron#1021
> >> > > > 2 months ago METRON-1594: KafkaWriter is asynchronous and may lose
> >> data
> >> > > on
> >> > > > node failure (mmiklavc via mmiklavc) closes apache/metron#1045
> >> > > > 2 months ago METRON-1603: Fix multivalue field errors in Bro Solr
> >> > schema
> >> > > > (mmiklavc via mmiklavc) closes apache/metron#1051
> >> > > > 2 months ago METRON-1584 Indexing Topology Crashes with Invalid
> >> Message
> >> > > > (nickwallen) closes apache/metron#1036
> >> > > > 2 months ago METRON-1547 Solr Comment Fields (justinleet) closes
> >> > > > apache/metron#1037
> >> > > > 2 months ago METRON-1553 Validate JIRA Script Error (nickwallen)
> >> closes
> >> > > > apache/metron#1013
> >> > > > 2 months ago METRON-1592 Unable to use third party parser with
> Storm
> >> > > > versions >= 1.1.0 (nickwallen) closes apache/metron#1042
> >> > > > 2 months ago METRON-1598 NoClassDefFoundError when running with
> >> > > > Elasticsearch X-Pack (nickwallen) closes apache/metron#1048
> >> > > > 2 months ago METRON-1589 '/api/v1/search/search' fails when 'Solr
> >> > > Zookeeper
> >> > > > Urls' has comma separated multiple zookeeper urls (justinleet)
> >> closes
> >> > > > apache/metron#1040
> >> > > > 2 months ago METRON-1593 Setting Metron rest additional classpath
> >> > removes
> >> > > > HBase and Hadoop configs from classpath (merrimanr) closes
> >> > > > apache/m

Re: IRC Channel -> OPS?

[zeo...@gmail.com]

Isn't it Casey?

Jon

On Wed, Aug 29, 2018, 08:41 Otto Fowler  wrote:

> Who has ops in the irc channel?
> Can you pop in and set the topic to something like:
> “There is an ASF slack with an active metron channel, please email
> dev@metron.apache.org and request an invite”
>
-- 

Jon

Re: [DISCUSS] Getting to a 1.0 release

[zeo...@gmail.com]

..@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 to separating developer docs and user docs. How should we
> approach
> > > > that.
> > > > > Have a separate doc book? I haven’t had a ton of time to contribute
> > to
> > > > code
> > > > > lately but I’d be happy to help write some of these.
> > > > >
> > > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen 
> > wrote:
> > > > >
> > > > > > Personally, I think the state of our docs and web presence is an
> > > > > inhibitor
> > > > > > to growing the Metron community.  Unless we can offer concise,
> > > > compelling
> > > > > > answers to the basic questions (What can I do with Metron?  Who
> > does
> > > it
> > > > > > help? How do I do that?), potential users and contributors are
> > unable
> > > > to
> > > > > > see the value of Metron.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen 
> > > > wrote:
> > > > > >
> > > > > > > I'd like to see us focus on improving our docs before a version
> > > 1.0.
> > > > > > > Right now we just stitch together a bunch of READMEs, which is
> a
> > > > great
> > > > > > > stride from where we started, but is not ideal.
> > > > > > >
> > > > > > > Our docs should focused on the user and use cases; What can I
> do
> > > with
> > > > > > > Metron?  Who does it help? How do I do that?
> > > > > > >
> > > > > > > The docs should be separate from the code base to allow for an
> > > > > > > organization that is focused on the user rather than the
> > > > > implementation.
> > > > > > > This allows the READMEs to focus on the developer and the
> > > > > implementation,
> > > > > > > which should make them more digestible too.  The docs should be
> > > > version
> > > > > > > controlled and maintained through PRs, just like the code.  We
> > > should
> > > > > > take
> > > > > > > just as much pride in our docs as we do in our code.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > > > > > > si...@simonellistonball.com> wrote:
> > > > > > >
> > > > > > >> Agreed, should we add TDE by default, and get the ranger
> > policies
> > > on
> > > > > by
> > > > > > >> default? That leaves secured in Kafka, which would have to be
> > > built
> > > > > into
> > > > > > >> the consumers and producers to encrypt into the on disk Kafka
> > > > topics.
> > > > > > Does
> > > > > > >> that seem necessary to people? It would have performance
> > > > implications
> > > > > > for
> > > > > > >> sure.
> > > > > > >>
> > > > > > >> Simon
> > > > > > >>
> > > > > > >> > On 15 Aug 2018, at 21:26, Otto Fowler <
> > ottobackwa...@gmail.com>
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > Well, I look at it like this.
> > > > > > >> >
> > > > > > >> > The Secure Vault was part of the original metron pitch, and
> > many
> > > > may
> > > > > > >> have used that as part of their evaluations.
> > > > > > >> > “Look, it is going to have a security vault type thing, it
> is
> > on
> > > > the
> > > > > > >> roadmap”.
> > > > > > >> >
> > > > > > >> > Regardless of the implementation, conceptually, security of
> > data
> > > > at
> > > > > > >> rest is important, and is a major outstanding item or the core
> > > > metron
> > > > > > >> proposition.
> > > > > > >> >
> > > > > > >> &

Re: [ANNOUNCE] - Apache Metron Slack channel

[zeo...@gmail.com]

Invite sent.

Jon

On Mon, Aug 27, 2018, 02:45 Ali Nazemian  wrote:

> Can I be invited as well?
>
> On Thu, Aug 16, 2018 at 4:37 AM Otto Fowler 
> wrote:
>
> > Done
> >
> >
> > On August 15, 2018 at 14:22:45, Vets, Laurens (laur...@daemon.be) wrote:
> >
> > Could I be invited?
> >
> > On 15-Aug-18 09:48, Michael Miklavcic wrote:
> > > + Metron user list
> > >
> > > On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > >> Turns out we are able to invite folks on an ad-hoc basis. See
> > instructions
> > >> here -
> > >>
> https://cwiki.apache.org/confluence/display/METRON/Community+Resources
> > >>
> > >>
> > >> On Wed, Aug 15, 2018 at 9:23 AM Michael Miklavcic <
> > >> michael.miklav...@gmail.com> wrote:
> > >>
> > >>> It's another option with different features. I imagine many people
> will
> > >>> use both.
> > >>>
> > >>> On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball <
> > >>> si...@simonellistonball.com> wrote:
> > >>>
> >  Since this is committers only, would it make more sense to stick to
> > IRC?
> >  Or
> >  is exclusivity the idea?
> > 
> >  On 15 August 2018 at 16:09, Nick Allen  wrote:
> > 
> > > Thanks for the instructions!
> > >
> > > On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > >> The Metron community has a Slack channel available for
> communication
> > >> (similar to the existing IRC channel, only on Slack).
> > >>
> > >> To join:
> > >>
> > >> 1. Go to slack.com.
> > >> 2. For organization/group, you'll enter "the-asf"
> > >> 3. Use your Apache email for your login
> > >> 4. Click "Channels" and look for #metron (Created by ottO June 15,
> > > 2018)
> > >> Best
> > >> Mike Miklavcic
> > >>
> > 
> > 
> >  --
> >  --
> >  simon elliston ball
> >  @sireb
> > 
> >
>
>
> --
> A.Nazemian
>
-- 

Jon

Re: Need a slack invite

[zeo...@gmail.com]

Invite sent.

Jon

On Mon, Aug 27, 2018, 03:36 Karthik D B  wrote:

> Hi Team,
> I’m a non-ASF committers, I Would like to join the Metron Slack Channel.
> pls. Send an invite.
> Thanks,
> Karthik DB

-- 

Jon

[DISCUSS] Getting to a 1.0 release

[zeo...@gmail.com]

So, as has been discussed in a few

other

recent dev list threads, I would like to discuss what a Metron 1.0 release
looks like.

In order to kick off the conversation, I would like to make a few
suggestions regarding "what 1.0 means to me," but I'm very interested to
hear everybody else's opinions.

In order to go 1.0 I believe we should have:
1. A clear, supported method of upgrading from one version of Metron to the
next.  We have attempted
 to make this
easier in the past, but it is currently not

supported

.
2. Authentication for all of the UIs and APIs should be secure and support
SSO.  I believe this is in progress via METRON-1663
.
3. Each of our personas

should
be well documented, understood, and supported.
 - The current state of documentation is, in my opinion, inadequate and I
admit I am partially to blame for this.  I suggest we define a strict
approach for documentation, align to it (such as perhaps migrating all
useful wiki documentation to git), and enforce it.
 - I would consider METRON-1699
 as a critical item for
a Security Data Scientist, but it is currently not clear to me where the
line exists between some of the other personas, or that each persona has
been sufficiently implemented.
4. A performance tuning guide should be available for all of the main
components, whether as an independent document or as a part of a larger
document.
5. Simple data ingest.
 - Similar to the ongoing conversation for NiFi integration
,
we should be able to say that we have broken down the barriers to getting
data into a Metron cluster in easy and efficient ways.  In addition to
NiFi, having support for other popular tools such as beats
, fluentd ,
etc.
 - Parsers should be pluggable, with independent tests and the ability to
make versioned modifications with roll-backs.

What else?  Are any of these items not necessary for a 1.0?

Jon
-- 

Jon

Re: [DISCUSS] Release cadence

[zeo...@gmail.com]

I'm a fan of a hybrid time/feature-based cadence.  Something like "When 3
months has passed since our last release, or a sufficiently complicated
change has been introduced to master (like merging a FB), a discuss thread
is started".  I'm primarily thinking of what the upgrade path looks like
(more on that in a "how do we get to 1.0" discuss).

Jon

On Wed, Aug 15, 2018 at 11:02 AM Justin Leet  wrote:

> Hi all,
>
> In concert with the discuss thread on a potential 0.6.0 release, I'd also
> like start a discussion about our release cadence.  We've generally been
> pretty relaxed around doing releases, and I'm curious what people's
> thoughts are on adopting a somewhat more regular schedule.
>
> Couple questions I think are relevant
> 1. Is this something we should work towards and, if we do, how do we want
> to go about it?
>
>- "Whenever someone feels like pushing out a discuss thread"?
>- "Let's just start a discuss thread every X and if we want to release
>we release"?
>- "let's try to get a release out every X and what's on the bus is on
>the bus"?
>- Something else?
>
> 2. Assuming we do want to do more regular releases, what's the timeframe
> we'd like to shoot for?
>
> Personally, I'd like to just start a discuss thread regularly, with the
> built-in expectation that not every thread should necessarily lead to a
> release. I don't want to be forcing release overhead when there's not
> enough to merit a release, but releasing more often than we often do now
> would provide a lot of values to users.
>
> In terms of timeframe, I tend to think a 2-3 month cadence for the threads
> is reasonable. It's long enough to potentially accrue enough features to
> merit a release, but short enough that when we pass on a release we're
> probably fine just waiting for another cycle to come around.  The last
> release was ~2 months ago and we have a good amount of stuff here, but I
> also don't expect two feature branches going in to be the norm.
>
> I'd expect whatever comes out of this thread to also be relatively
> informal. At least right now, I don't feel like we need a rigid schedule,
> and I'd still like people to feel encouraged to propose a release,
> particularly when there are a couple major features or critical fixes.
> Alternatively, I would expect some of these discuss threads to conclude,
> "We should do a release, but let's wait a couple waits for these tickets to
> finish up" (e.g. like the Pcap query panel).
>
> Justin
>
-- 

Jon

Re: [DISCUSS] Metron Release 0.6.0?

[zeo...@gmail.com]

I agree - I would love to see a release not long after the PCAP FB gets
into master, and 0.6.0 makes sense to me.

I'd also like to see a 0.2 release of metron-bro-plugin-kafka.  There is
one new commit, and I have a PR open which is waiting on some tests before
it's ready to be evaluated/merged.  I will try to get that work done asap.
As of right now metron's dev ansible scripts pin to a specific release of
metron-bro-plugin-kafka (0.1
<0.1https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml>),
and I'm fine leaving that as is until after the coming release, but we
could also do a metron-bro-plugin-kafka release first and then update
metron to point the dev environment to the new package prior to the
upcoming RC.

I would also like to discuss what the roadmap looks like for a 1.0 release
and perhaps a more regular release schedule.  I have some thoughts but
don't want to hijack this thread.

Jon

On Wed, Aug 15, 2018 at 9:11 AM Justin Leet  wrote:

> Hi all,
>
> It's been a little while since the last release, and a couple major items
> have gone in since then (or are hopefully close to going in!).  In
> particular, I'd personally like to see a release with our Solr work
>  and the
> close-to-completion PCAP Query Panel
> .  There is a thread
> <
> https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E
> >
> around what's left before merging the PCAP feature branch, I encourage you
> to take a look. There are also some nice-to-haves as well as some Apache
> cleanup around the RAT tool and typescript files
> .
>
> Version Number
> I'm proposing bumping to 0.6.0, in particular because of the Solr and PCAP
> efforts. We can adjust that as necessary.
>
> I'm proposing we release this from the Metron master branch, plus any
> commits the community considers necessary.  Note that I'm proposing that
> this release occur after the PCAP feature branch is merged into master.
>
> Proposed Timeframe
> I would tentatively like to start work on the RC Wednesday, September 5th.
> It's a little further out than usual, but I wanted to kick off the
> discussion before Labor Day and to give ongoing  time to settle. And also
> because I'll be unavailable around Labor Day.
>
> JIRA Status
> There are 31 open PRs at https://github.com/apache/metron/pulls. We should
> work on getting anything we feel merits inclusion closed out. Please
> respond with any tickets we'd like included.
>
> A couple of these are for the PCAP feature branch, and there will be at
> least one more for documentation.
>
> There will be updates necessary to get our Jira up to date.  I'll follow up
> on that, and ask that everyone double check their tickets.
>
> There have been 106 commits since the 0.5.0 release (listed at the end of
> message). There will be a few more when we pull in the PCAP feature branch.
>
> Completed PRs as of Aug 15 as generated by git log --pretty="%cr %s"
> tags/apache-metron-0.5.0-release..HEAD.
>
> 5 days ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
> via mmiklavc) closes apache/metron#1152
> 13 days ago METRON-1701 Update General notes on the installation of Pycapa
> on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
> 3 weeks ago METRON-1650 Packaging docker containers are too large
> (jameslamb via merrimanr) closes apache/metron#1091
> 3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
> management pack repo info closes apache/incubator-metron#1052
> 3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
> apache/incubator-metron#1126
> 3 weeks ago METRON-1694: Clean up Metron REST docs closes
> apache/incubator-metron#1131
> 4 weeks ago METRON-1606 Add a 'wrap' to incoming messages in the
> metron json parser (ottobackwards) closes apache/metron#1054
> 4 weeks ago METRON-1672 Add metron-alerts's UI unit tests to travis
> build process (justinleet) closes apache/metron#1106
> 4 weeks ago METRON-1684 Fix Markdown problems in 3rdPartyParser.md
> (justinleet) closes apache/metron#1110
> 4 weeks ago METRON-1657 Parser aggregation in storm (justinleet) closes
> apache/metron#1099
> 4 weeks ago METRON-1651 Fixing failing protractor e2e test (tiborm via
> merrimanr) closes apache/metron#1095
> 4 weeks ago METRON-1673 Fix Javadoc errors (justinleet) closes
> apache/metron#1107
> 4 weeks ago METRON-1620: Fixes for forensic clustering use case example
> (mmiklavc via mmiklavc) closes apache/metron#1065
> 4 weeks ago METRON-1659: The platform-info.sh should check for the vagrant
> hostmanager plugin closes apache/incubator-metron#1100
> 4 weeks ago METRON-1658: Upgrade bro to 2.5.4 closes
> apache/incubator-metron#1101
> 4 weeks ago METRON-1236 Add start/stop/restart commands that execute
> successfully, w

Re: Knox SSO feature branch PRs: a quick demo

[zeo...@gmail.com]

Nice run through Simon that was very helpful for me to catch up on the work
you've been doing.  Appreciate the focus on this too, when talking to
others about Metron I have heard a few times that they were interested in
features that it seems we will soon have.  Hopefully I'll have a chance to
take a swing through your FB/PRs next week as well.

Jon

On Thu, Aug 2, 2018, 15:10 Michael Miklavcic 
wrote:

> Nice! Thanks for the walk-through, Simon. Agreed that this should assist
> reviewers in understanding the feature branch and PR break-down.
>
> Cheers,
> Mike
>
> On Thu, Aug 2, 2018 at 8:51 AM larry mccay  wrote:
>
> > Hi Simon -
> >
> > I like how you walk through those various PRs and describe what is done
> at
> > each step.
> > Please feel free sure to bring any suggestions that you may have for
> > improvements in Knox and KnoxSSO to the community.
> > The public cert issue is a pain for sure - we do have a knoxcli command
> for
> > exporting it but you need to be on the Knox machine to do that.
> > I've been considering add an API to the admin API for retrieving it as
> well
> > - but of course it will need to be up and running for that.
> >
> > thanks,
> >
> > --larry
> >
> >
> > On Wed, Aug 1, 2018 at 11:34 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> > > I've recently put in a number of PRs on the Knox feature branch, and
> > > thought it might be useful to post a quick 'sprint demo' style
> > explanation
> > > of what the various PRs and functionality entails:
> > > https://youtu.be/9OJz6hg0N1I
> > >
> > > Hope this helps with review process. There are a couple of areas where
> > that
> > > need a little follow on improvement (Ambari mpack cosmetic oddness
> > mainly).
> > > Any thoughts and assistance on that would be very greatly appreciated.
> > >
> > > Simon
> > >
> >
>
-- 

Jon

Re: Bro plugin release process docs?

[zeo...@gmail.com]

I did a bit of poking around and I don't believe we ever formally wrote
that down.  The last release happened as a combination of actions from
mattf and myself (mostly mattf).

The plugin has two new commits since the last release (1 bugfix 1 feature)
- if we want to couple version 0.2 of the plugin with a metron 0.5.0 release we
would need to make a 0.2 tag against HEAD of the plugin repo, then
increment the version in the ansible playbooks here

.

Or, if we just want to make a plugin release without changing
apache/metron, we could just make a 0.2 tag in the plugin repo, and then
release it in a more disjointed way.  I know that's not super helpful since
I don't have documentation for doing an apache release of the plugin other
than hacking something together based on what's out there for apache/metron.

Reference conversations:
 -
https://lists.apache.org/thread.html/2606166bd5e864f1b56db302099c9a8042cdadec8fa2692fef49493f@%3Cdev.metron.apache.org%3E
 -
https://lists.apache.org/thread.html/3aecbadbf3353e98c03ca4b680fcd998d0cd2bf5a4319238dd85ae75@%3Cdev.metron.apache.org%3E

Jon

On Fri, May 25, 2018 at 2:00 PM Justin Leet  wrote:

> At the risk of exposing my ignorance, do we have the bro plugin release
> process documented anywhere?  We have a doc for the main release (
> https://cwiki.apache.org/confluence/display/METRON/Release+Process), but I
> haven't noticed one for the bro plugin.
>
> For the current RC, it's not included and it wasn't pushed for (it has less
> changes for obvious reasons).  However, we should be making sure to
> validate if its necessary to release and having the process documented.
>
> Justin
>
-- 

Jon

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

[zeo...@gmail.com]

We did discuss doing a release since there were two new commits, but I
don't think it was included in this round.

Jon

On Sat, May 26, 2018, 10:22 Otto Fowler  wrote:

> Is there a BRO RC # for this?
>
>
> On May 25, 2018 at 14:53:25, Nick Allen (n...@nickallen.org) wrote:
>
> +1 Release this package as Apache Metron 0.5.0-RC1
>
> Ran through all validation steps using the `metron-rc-check` script, which
> included running all the tests, license checks, and spun-up the CentOS dev
> environment successfully.
>
>
>
> On Fri, May 25, 2018 at 1:40 PM, Nick Allen  wrote:
>
> > This release does not contain an updated Bro plugin so the RC check
> script
> > does not currently work. Try using the patch at
> > https://github.com/apache/metron/pull/1034.
> >
> >
> > On Thu, May 24, 2018 at 3:23 PM, Justin Leet  wrote:
> >
> >> Hi all,
> >>
> >> This is a call to vote on releasing Apache Metron 0.5.0
> >>
> >> Full list of changes in this release:
> >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/CHANGES
> >>
> >> The tag/commit to be voted upon is:
> >>
> >> (apache/metron) apache-metron-0.5.0-rc1
> >>
> >> The source archive being voted upon can be found here:
> >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
> >> apache-metron-0.5.0-rc1.tar.gz
> >>
> >> Other release files, signatures and digests can be found here:
> >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
> >>
> >> The release artifacts are signed with the following key:
> >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/KEYS
> >>
> >> Please vote on releasing this package as Apache Metron 0.5.0-RC1
> >>
> >> When voting, please list the actions taken to verify the release.
> >>
> >> Recommended build validation and verification instructions are posted
> >> here:
> >> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >>
> >> This vote will be open until 4pm EDT on Tuesday May 29 2018, to account
> >> for
> >> the weekend.
> >>
> >> [ ] +1 Release this package as Apache Metron 0.3.0-RC1
> >>
> >> [ ] 0 No opinion
> >>
> >> [ ] -1 Do not release this package because...
> >>
> >
> >
>
-- 

Jon

Re: [DISCUSS] Pcap panel architecture

[zeo...@gmail.com]

I think baby steps are fine - admin gets access to all, otherwise you only
see your own pcaps, but we file a jira for a future add of API security,
which more mature SOCs that align with the Metron personas will need.

Jon

On Fri, May 11, 2018, 09:27 Ryan Merriman  wrote:

> That's a good point Jon.  There are different levels of effort associated
> with different options.  If we want to allow pcaps to be shared with
> specific users, we will need to introduce ACL security in our REST
> application using something like the ACL capability that comes with Spring
> Security or Ranger.  This would be more complex to design and implement.
> If we want something more broad like admin roles that can see all or
> allowing pcap files to become public, this would be less work.  Do you
> think ACL security is required or would the other options be acceptable?
>
> On Thu, May 10, 2018 at 2:47 PM, zeo...@gmail.com 
> wrote:
>
> > At the very least there needs to be the ability to share downloaded PCAPs
> > with other users and/or have roles that can see all pcaps.  A platform
> > engineer may want to clean up old pcaps after x time, or a manger may ask
> > an analyst to find all of the traffic that exhibits xyz behavior, dump a
> > pcap, and then point him to it so the manager can review.  Since the
> > pcap may be huge, we wouldn't want to try to push people to sending it
> via
> > email, uploading to a file server, finding an external hard drive, etc.
> >
> > Jon
> >
> > On Thu, May 10, 2018 at 10:16 AM Ryan Merriman 
> > wrote:
> >
> > > Mike, I believe the /pcapGetter/getPcapsByIdentifiers endpoint exposes
> > the
> > > fixed query option which we have covered.  I agree with you that
> > > deprecating the metron-api module should be a goal of this feature.
> > >
> > > On Wed, May 9, 2018 at 1:36 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > This looks like a pretty good start Ryan. Does the metadata endpoint
> > > cover
> > > > this https://github.com/apache/metron/tree/master/
> > > > metron-platform/metron-api#the-pcapgettergetpcapsbyidentifier
> > s-endpoint
> > > > from the original metron-api? If so, then we would be able to
> deprecate
> > > the
> > > > existing metron-api project. If we later go to micro-services, a pcap
> > > > module would spin back into the fold, but it would probably look
> > > different
> > > > from metron-api.
> > > >
> > > > I commented on the UI thread, but to reiterate for the purpose of
> > backend
> > > > functionality here I don't believe there is a way to "PAUSE" or
> > "SUSPEND"
> > > > jobs. That said, I think GET /api/v1/pcap/stop/ is sufficient
> > for
> > > > the job management operations.
> > > >
> > > > On Wed, May 9, 2018 at 11:00 AM, Ryan Merriman 
> > > > wrote:
> > > >
> > > > > Now that we are confident we can run submit a MR job from our
> current
> > > > REST
> > > > > application, is this the desired approach?  Just want to confirm.
> > > > >
> > > > > Next I think we should map out what the REST interface will look
> > like.
> > > > > Here are the endpoints I'm thinking about:
> > > > >
> > > > > GET /api/v1/pcap/metadata?basePath
> > > > >
> > > > > This endpoint will return metadata of pcap data stored in HDFS.
> This
> > > > would
> > > > > include pcap size, date ranges (how far back can I go), etc.  It
> > would
> > > > > accept an optional HDFS basePath parameter for cases where pcap
> data
> > is
> > > > > stored in multiple places and/or different from the default
> location.
> > > > >
> > > > > POST /api/v1/pcap/query
> > > > >
> > > > > This endpoint would accept a pcap request, submit a pcap query job,
> > and
> > > > > return a job id.  The request would be an object containing the
> > > > parameters
> > > > > documented here:  https://github.com/apache/metron/tree/master/
> > > > > metron-platform/metron-pcap-backend#query-filter-utility.  A
> > query/job
> > > > > would be associated with a user that submits it.  An exception will
> > be
> > > > > returned for violating constraints like too many queries submitted,
> > > query
> > > &g

Re: [DISCUSS] Pcap panel architecture

[zeo...@gmail.com]

At the very least there needs to be the ability to share downloaded PCAPs
with other users and/or have roles that can see all pcaps.  A platform
engineer may want to clean up old pcaps after x time, or a manger may ask
an analyst to find all of the traffic that exhibits xyz behavior, dump a
pcap, and then point him to it so the manager can review.  Since the
pcap may be huge, we wouldn't want to try to push people to sending it via
email, uploading to a file server, finding an external hard drive, etc.

Jon

On Thu, May 10, 2018 at 10:16 AM Ryan Merriman  wrote:

> Mike, I believe the /pcapGetter/getPcapsByIdentifiers endpoint exposes the
> fixed query option which we have covered.  I agree with you that
> deprecating the metron-api module should be a goal of this feature.
>
> On Wed, May 9, 2018 at 1:36 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > This looks like a pretty good start Ryan. Does the metadata endpoint
> cover
> > this https://github.com/apache/metron/tree/master/
> > metron-platform/metron-api#the-pcapgettergetpcapsbyidentifiers-endpoint
> > from the original metron-api? If so, then we would be able to deprecate
> the
> > existing metron-api project. If we later go to micro-services, a pcap
> > module would spin back into the fold, but it would probably look
> different
> > from metron-api.
> >
> > I commented on the UI thread, but to reiterate for the purpose of backend
> > functionality here I don't believe there is a way to "PAUSE" or "SUSPEND"
> > jobs. That said, I think GET /api/v1/pcap/stop/ is sufficient for
> > the job management operations.
> >
> > On Wed, May 9, 2018 at 11:00 AM, Ryan Merriman 
> > wrote:
> >
> > > Now that we are confident we can run submit a MR job from our current
> > REST
> > > application, is this the desired approach?  Just want to confirm.
> > >
> > > Next I think we should map out what the REST interface will look like.
> > > Here are the endpoints I'm thinking about:
> > >
> > > GET /api/v1/pcap/metadata?basePath
> > >
> > > This endpoint will return metadata of pcap data stored in HDFS.  This
> > would
> > > include pcap size, date ranges (how far back can I go), etc.  It would
> > > accept an optional HDFS basePath parameter for cases where pcap data is
> > > stored in multiple places and/or different from the default location.
> > >
> > > POST /api/v1/pcap/query
> > >
> > > This endpoint would accept a pcap request, submit a pcap query job, and
> > > return a job id.  The request would be an object containing the
> > parameters
> > > documented here:  https://github.com/apache/metron/tree/master/
> > > metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
> > > would be associated with a user that submits it.  An exception will be
> > > returned for violating constraints like too many queries submitted,
> query
> > > parameters out of limits, etc.
> > >
> > > GET /api/v1/pcap/status/
> > >
> > > This endpoint will return the status of a running job.  I imagine this
> is
> > > just a proxy to the YARN REST api.  We can discuss the implementation
> > > behind these endpoints later.
> > >
> > > GET /api/v1/pcap/stop/
> > >
> > > This endpoint would kill a running pcap job.  If the job has already
> > > completed this is a noop.
> > >
> > > GET /api/v1/pcap/list
> > >
> > > This endpoint will list a user's submitted pcap queries.  Items in the
> > list
> > > would contain job id, status (is it finished?), start/end time, and
> > number
> > > of pages.  Maybe there is some overlap with the status endpoint above
> and
> > > the status endpoint is not needed?
> > >
> > > GET /api/v1/pcap/pdml//
> > >
> > > This endpoint will return pcap results for the given page in pdml
> format
> > (
> > > https://wiki.wireshark.org/PDML).  Are there other formats we want to
> > > support?
> > >
> > > GET /api/v1/pcap/raw//
> > >
> > > This endpoint will allow a user to download raw pcap results for the
> > given
> > > page.
> > >
> > > DELETE /api/v1/pcap/
> > >
> > > This endpoint will delete pcap query results.  Not sure yet how this
> fits
> > > in with our broader cleanup strategy.
> > >
> > > This should get us started.  What did I miss and what would you change
> > > about these?  I did not include much detail related to security,
> cleanup
> > > strategy, or underlying implementation details but these are items we
> > > should discuss at some point.
> > >
> > > On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Sweet! That's great news. The pom changes are a lot simpler than I
> > > > expected. Very nice.
> > > >
> > > > On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman 
> > > wrote:
> > > >
> > > > > Finally figured it out.  Commit is here:
> > > > > https://github.com/merrimanr/incubator-metron/commit/
> > > > > 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
> > > > >
> > > > > It came down to figuring out the right combination of maven
> > > dependencies
> > > > > and passing i

Re: [DISCUSS] Release?

[zeo...@gmail.com]

I tend to like grouping the es changes into one release (i.e. include the
index name change) and solr into another (next release).

I think we go too long between releases myself and wouldn't be against
doing two releases just a couple of months apart.

Jon

On Wed, May 9, 2018, 14:47 Michael Miklavcic 
wrote:

> I don't have a strong opinion. The ES upgrade alone is a massive feature.
> It could make it easier to include the index change I mentioned along with
> Solr as a follow-up. I think if we did split, we could arguably start on
> the next release with Solr almost immediately.
>
> On Wed, May 9, 2018, 12:40 PM Nick Allen  wrote:
>
> > Simon brought up the idea of including the Solr enhancements (currently
> in
> > a feature branch) for the release.  What are people's opinions on this?
> Is
> > this something that is a blocker for the release?
> >
> > IMO, there is so much already in master waiting to be released that I
> don't
> > see a need to include it in the next release.  I'd be happy to see that
> > Solr work drive a follow-on release.
> >
> >
> >
> >
> > On Wed, May 9, 2018 at 2:16 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > One item we haven't gotten around to was redoing the index names to
> use a
> > > metron_ prefix. I'm the one that pushed the original DISCUSS thread on
> > > this, but haven't had a chance to advance it. Does anyone have any
> strong
> > > opinions on it? I originally thought it made sense to include alongside
> > the
> > > other major ES and Solr changes.
> > >
> > > On Wed, May 9, 2018 at 12:13 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > I'm also a +1 on 0.5.0. This is a fairly big release.
> > > >
> > > > On Wed, May 9, 2018 at 12:05 PM, Nick Allen 
> > wrote:
> > > >
> > > >> +1 to 0.5.0
> > > >>
> > > >> On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com 
> > > >> wrote:
> > > >>
> > > >> > I agree that it's probably time (more likely, overdue) for a
> > release.
> > > >> > Based off of looking at all of those changes I would also suggest
> > > going
> > > >> to
> > > >> > at least 0.5.x.
> > > >> >
> > > >> > It probably makes sense to take a look at Upgrading.md
> > > >> > <https://github.com/apache/metron/blob/master/Upgrading.md> (and
> > > >> related
> > > >> > docs) to make sure it's accurate as well.
> > > >> >
> > > >> > Jon
> > > >> >
> > > >> > On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
> > > >> > michael.miklav...@gmail.com> wrote:
> > > >> >
> > > >> > > Good call - I thought that made our last release, but this would
> > be
> > > >> the
> > > >> > 2nd
> > > >> > > follow-on from when Nick originally posed a breakdown
> > > >> > > 4 months ago METRON-939: Upgrade ElasticSearch and Kibana
> > (mmiklavc
> > > >> via
> > > >> > > mmiklavc) closes apache/metron#840
> > > >> > >
> > > >> > >
> > > >> > >
> > https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f
> > > >> > 36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
> > > >> > >
> > > >> > > On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com <
> > zeo...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > We should also mention the Upgrade of ElasticSearch and Kibana
> > > >> > > >
> > > >> > > > Jon
> > > >> > > >
> > > >> > > > On Wed, May 9, 2018 at 12:49 PM Nick Allen <
> n...@nickallen.org>
> > > >> wrote:
> > > >> > > >
> > > >> > > > > Oh, and also the Solr work that is currently in a feature
> > > >> branch.  We
> > > >> > > > would
> > > >> > > > > have to get the work finished up and merged though.  Sounds
> > like
> > > >> we
> > > &g

Re: [DISCUSS] Pcap UI user requirements

[zeo...@gmail.com]

Regarding the prioritization, that is what I was thinking as well, I just
wasn't as prescriptive with my suggestion.

I did look for a java implementation and failed to find one (the closest I
found was the Apache-licened bcc project <https://github.com/iovisor/bcc>).
Perhaps someone else's google-fu is better than mine though.

Not sure that BPF has the ability to depend on state (I feel it would be
very unlikely), but if it does I've never used it.  My 10 minute read of
the original paper (admittedly > 20 years old) and some other supplementary
info seems to support my prior thoughts.

Jon

On Wed, May 9, 2018 at 2:59 PM Casey Stella  wrote:

> A couple of thoughts on cluster overuse:
> * Definitely can't pause/resume MR jobs, unfortunately
> * The traditional approach to managing overuse of cluster resources and
> prioritization in Yarn is via the scheduler.  I'd suggest rather than
> building this ourselves, we allow users to be associated with yarn queues
> so their jobs are submitted to the correct queue and don't clobber the
> cluster.
>
> Beyond that, I like the BPF idea as a filter.  Any idea if there's a java
> BPF implementation?  Also, keep in mind that our query mechanism is a map
> and a reduce job, so any filtering system which depends on state (e.g.
> previous packets by time) is going to trigger another architecture.
>
> On Mon, May 7, 2018 at 4:05 PM zeo...@gmail.com  wrote:
>
> > From my perspective PCAP is primarily used as a follow-on to an alert or
> > meta-alert - people very rarely use PCAP for initial hunting.  I know
> this
> > has been brought up by Otto, Mike, and Ryan across the two related
> threads
> > and I think it's all spot on.  Going from an alert or meta-alert to
> pulling
> > PCAP would by far the primary use case for this in every SOC I've ever
> > worked in (not necessarily a representative sample).
> >
> > I also have some additional thoughts on the feature side after doing some
> > brainstorming and talking to two of the SOCs I work most with:
> >  - Limit the size of the PCAP, not just the date range, and maybe even
> have
> > a configurable cluster-wide admin max for PCAP retrieval, set to
> 0/infinite
> > by default.
> >  - Set priority of PCAP queries.  Perhaps there's an automated
> > pcap retrieval 'just in case', which should have a lower priority than an
> > interactive request via the UI.
> >  - Ability to pause/resume (not just cancel) jobs.
> >  - Configurable cluster-wide admin max # of current PCAP queries, set to
> > 0/infinite by default.
> >  - Ability to pull PCAP live off the wire and stream it into a file.
> >  - Ability to filter PCAP by providing a BPF filter to apply in
> server-side
> > post-processing (less efficient, but very versatile).
> >  - Request what PCAP data exists in the cluster (answering "how far back
> > can I go?")
> >  - This is obvious and is probably assumed, but queries based on any set
> of
> > the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.
> >
> > Jon
> >
> > On Fri, May 4, 2018 at 9:44 AM Otto Fowler 
> > wrote:
> >
> > > That is the ‘views’ part.
> > >
> > > We can have options on the data output, if you have output full data,
> > then
> > > we can have different views and interactions for inspection and level
> of
> > > detail.
> > >
> > >
> > >
> > > On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com)
> > wrote:
> > >
> > > It can be like a report but also to investigate some case where the
> user
> > > want to see the whole packet (all the bits and bytes). Like in
> wireshark,
> > > something interactive no?
> > >
> > > 2018-05-04 14:33 GMT+01:00 Otto Fowler :
> > >
> > > > The PCAP Query seems more like PCAP Report to me. You are generating
> a
> > > > report based on parameters.
> > > > That report is something that takes some time and external process to
> > > > generate… ie you have to wait for it.
> > > >
> > > > I can almost imagine a flow where you:
> > > >
> > > > * Are in the AlertUI
> > > > * Ask to generate a PCAP report based on some selected
> > alerts/meta-alert,
> > > > possibly picking from on or more report ‘templates’
> > > > that have query options etc
> > > > * The report request is ‘queued’, that is dispatched to be be
> > > > executed/generated
> > > > * You as a user have a ‘queue’ of your report

Re: [DISCUSS] Pcap UI user requirements

[zeo...@gmail.com]

I had a feeling it may be that way.  Unless anyone else knows of a better
approach, it's probably most reasonable to push that into a follow-on JIRA
and not over-complicate the current activities.

Jon

On Wed, May 9, 2018 at 2:33 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> We are limited by Yarn and MapReduce applications in the case of
> pause/resume - I could be wrong, but I don't think that's something that's
> supported unless you're talking about multiple MR jobs strung together.
>
>
> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application
>
> I don't see anything suggesting "SUSPENDED" or "PAUSED" as we have
> available in workflow engines like Oozie.
>
> "The valid application state can be one of the following:  ALL, NEW,
> NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED"
>
> Same goes for MR job commands:
>
> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html#job
>
> Mike
>
> On Mon, May 7, 2018 at 2:04 PM, zeo...@gmail.com  wrote:
>
> > From my perspective PCAP is primarily used as a follow-on to an alert or
> > meta-alert - people very rarely use PCAP for initial hunting.  I know
> this
> > has been brought up by Otto, Mike, and Ryan across the two related
> threads
> > and I think it's all spot on.  Going from an alert or meta-alert to
> pulling
> > PCAP would by far the primary use case for this in every SOC I've ever
> > worked in (not necessarily a representative sample).
> >
> > I also have some additional thoughts on the feature side after doing some
> > brainstorming and talking to two of the SOCs I work most with:
> >  - Limit the size of the PCAP, not just the date range, and maybe even
> have
> > a configurable cluster-wide admin max for PCAP retrieval, set to
> 0/infinite
> > by default.
> >  - Set priority of PCAP queries.  Perhaps there's an automated
> > pcap retrieval 'just in case', which should have a lower priority than an
> > interactive request via the UI.
> >  - Ability to pause/resume (not just cancel) jobs.
> >  - Configurable cluster-wide admin max # of current PCAP queries, set to
> > 0/infinite by default.
> >  - Ability to pull PCAP live off the wire and stream it into a file.
> >  - Ability to filter PCAP by providing a BPF filter to apply in
> server-side
> > post-processing (less efficient, but very versatile).
> >  - Request what PCAP data exists in the cluster (answering "how far back
> > can I go?")
> >  - This is obvious and is probably assumed, but queries based on any set
> of
> > the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.
> >
> > Jon
> >
> > On Fri, May 4, 2018 at 9:44 AM Otto Fowler 
> > wrote:
> >
> > > That is the ‘views’ part.
> > >
> > > We can have options on the data output, if you have output full data,
> > then
> > > we can have different views and interactions for inspection and level
> of
> > > detail.
> > >
> > >
> > >
> > > On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com)
> > wrote:
> > >
> > > It can be like a report but also to investigate some case where the
> user
> > > want to see the whole packet (all the bits and bytes). Like in
> wireshark,
> > > something interactive no?
> > >
> > > 2018-05-04 14:33 GMT+01:00 Otto Fowler :
> > >
> > > > The PCAP Query seems more like PCAP Report to me. You are generating
> a
> > > > report based on parameters.
> > > > That report is something that takes some time and external process to
> > > > generate… ie you have to wait for it.
> > > >
> > > > I can almost imagine a flow where you:
> > > >
> > > > * Are in the AlertUI
> > > > * Ask to generate a PCAP report based on some selected
> > alerts/meta-alert,
> > > > possibly picking from on or more report ‘templates’
> > > > that have query options etc
> > > > * The report request is ‘queued’, that is dispatched to be be
> > > > executed/generated
> > > > * You as a user have a ‘queue’ of your report results, and when the
> > > report
> > > > is done it is queued there
> > > > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > > > info/meta has the yarn details )
> > > > * You can select t

Re: [DISCUSS] Release?

[zeo...@gmail.com]

Nick - that was this
<https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9>
 and this
<https://github.com/apache/metron-bro-plugin-kafka/commit/4db999e82cbb91e989eaf00a88e94ffd2459f3a3>,
which just barely snuck it into the same release cycle as 0.4.2.  The
plugin has two new commits since then (1 bugfix 1 feature) - if we want to
include version 0.2 of the plugin with a metron 0.5.0 release we would need
to make a 0.2 tag against HEAD of the plugin repo, then increment the
version in the ansible playbooks here
<https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30>
.

Jon

On Wed, May 9, 2018 at 2:26 PM Otto Fowler  wrote:

> I think we should have that.
>
>
> On May 9, 2018 at 14:16:23, Michael Miklavcic (michael.miklav...@gmail.com
> )
> wrote:
>
> One item we haven't gotten around to was redoing the index names to use a
> metron_ prefix. I'm the one that pushed the original DISCUSS thread on
> this, but haven't had a chance to advance it. Does anyone have any strong
> opinions on it? I originally thought it made sense to include alongside the
> other major ES and Solr changes.
>
> On Wed, May 9, 2018 at 12:13 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I'm also a +1 on 0.5.0. This is a fairly big release.
> >
> > On Wed, May 9, 2018 at 12:05 PM, Nick Allen  wrote:
> >
> >> +1 to 0.5.0
> >>
> >> On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com 
> >> wrote:
> >>
> >> > I agree that it's probably time (more likely, overdue) for a release.
> >> > Based off of looking at all of those changes I would also suggest
> going
> >> to
> >> > at least 0.5.x.
> >> >
> >> > It probably makes sense to take a look at Upgrading.md
> >> > <https://github.com/apache/metron/blob/master/Upgrading.md> (and
> >> related
> >> > docs) to make sure it's accurate as well.
> >> >
> >> > Jon
> >> >
> >> > On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
> >> > michael.miklav...@gmail.com> wrote:
> >> >
> >> > > Good call - I thought that made our last release, but this would be
> >> the
> >> > 2nd
> >> > > follow-on from when Nick originally posed a breakdown
> >> > > 4 months ago METRON-939: Upgrade ElasticSearch and Kibana (mmiklavc
> >> via
> >> > > mmiklavc) closes apache/metron#840
> >> > >
> >> > >
> >> > > https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f
> >> > 36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
> >> > >
> >> > > On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com  >
> >> > > wrote:
> >> > >
> >> > > > We should also mention the Upgrade of ElasticSearch and Kibana
> >> > > >
> >> > > > Jon
> >> > > >
> >> > > > On Wed, May 9, 2018 at 12:49 PM Nick Allen 
> >> wrote:
> >> > > >
> >> > > > > Oh, and also the Solr work that is currently in a feature
> >> branch. We
> >> > > > would
> >> > > > > have to get the work finished up and merged though. Sounds like
> >> we
> >> > are
> >> > > > > real close on that.
> >> > > > >
> >> > > > > On Wed, May 9, 2018 at 12:47 PM, Nick Allen  >
> >> > > wrote:
> >> > > > >
> >> > > > > > The next part of the conversation would be, what should the
> >> version
> >> > > > > number
> >> > > > > > be? To help with that, I have tried to summarize the changes
> in
> >> > the
> >> > > > > > release. Of course, this is going to be heavily biased towards
> >> my
> >> > > own
> >> > > > > > interests, so please feel free to chime in if I have missed
> >> > anything.
> >> > > > > >
> >> > > > > >
> >> > > > > > - Significant performance improvements in Parsing,
> >> Enrichments,
> >> > > and
> >> > > > > > Indexing
> >> > > > > >
> >> > > > > >
> >> > > > > > - Support for Ubuntu installations
> >> > > > > >
> >

Re: [DISCUSS] Release?

[zeo...@gmail.com]

I agree that it's probably time (more likely, overdue) for a release.
Based off of looking at all of those changes I would also suggest going to
at least 0.5.x.

It probably makes sense to take a look at Upgrading.md
<https://github.com/apache/metron/blob/master/Upgrading.md> (and related
docs) to make sure it's accurate as well.

Jon

On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Good call - I thought that made our last release, but this would be the 2nd
> follow-on from when Nick originally posed a breakdown
> 4 months ago METRON-939: Upgrade ElasticSearch and Kibana (mmiklavc via
> mmiklavc) closes apache/metron#840
>
>
> https://lists.apache.org/thread.html/01fb18dd0ee10845588c0c1a4b3f2f36d7a107c66edd2247f61756c1@%3Cdev.metron.apache.org%3E
>
> On Wed, May 9, 2018 at 11:18 AM, zeo...@gmail.com 
> wrote:
>
> > We should also mention the Upgrade of ElasticSearch and Kibana
> >
> > Jon
> >
> > On Wed, May 9, 2018 at 12:49 PM Nick Allen  wrote:
> >
> > > Oh, and also the Solr work that is currently in a feature branch.  We
> > would
> > > have to get the work finished up and merged though.  Sounds like we are
> > > real close on that.
> > >
> > > On Wed, May 9, 2018 at 12:47 PM, Nick Allen 
> wrote:
> > >
> > > > The next part of the conversation would be, what should the version
> > > number
> > > > be?  To help with that, I have tried to summarize the changes in the
> > > > release.  Of course, this is going to be heavily biased towards my
> own
> > > > interests, so please feel free to chime in if I have missed anything.
> > > >
> > > >
> > > >- Significant performance improvements in Parsing, Enrichments,
> and
> > > >Indexing
> > > >
> > > >
> > > >- Support for Ubuntu installations
> > > >
> > > >
> > > >- Support for Storm 1.1 and HDP 2.6
> > > >
> > > >
> > > >- Event time processing in the Profiler
> > > >
> > > >
> > > >- Complex document support in the JSONMapParser
> > > >
> > > >
> > > >- Support for the Elasticsearch X-Pack
> > > >
> > > >
> > > >- Run Stellar in a Zeppelin Notebook
> > > >
> > > >
> > > >- Oodles of bug fixes and usability improvements
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 9, 2018 at 12:14 PM, Nick Allen 
> > wrote:
> > > >
> > > >> Something like this might be more digestible for these purposes.
> > > >>
> > > >> $git log --pretty="%cr %s" tags/apache-metron-0.4.2-release..HEAD
> > > >>
> > > >> 88 minutes ago METRON-1530 Default proxy config settings in
> > > >> metron-contrib need to be updated (sardell via merrimanr) closes
> > > >> apache/metron#998
> > > >> 5 days ago METRON-1545 Upgrade Spring and Spring Boot (merrimanr)
> > closes
> > > >> apache/metron#1008
> > > >> 7 days ago METRON-1543 Unable to Set Parser Output Topic in Sensor
> > > Config
> > > >> (nickwallen) closes apache/metron#1007
> > > >> 2 weeks ago METRON-1539: Specialized RENAME field transformer closes
> > > >> apache/incubator-metron#1002
> > > >> 2 weeks ago METRON-1520: Add caching for stellar field
> transformations
> > > >> closes apache/incubator-metron#990
> > > >> 2 weeks ago METRON-1529 CONFIG_GET Fails to Retrieve Latest Config
> > When
> > > >> Run in Zeppelin REPL (nickwallen) closes apache/metron#997
> > > >> 2 weeks ago METRON-1511 Unable to Serialize Profiler Configuration
> > > >> (nickwallen) closes apache/metron#982
> > > >> 3 weeks ago METRON-1528: Fix missing file in metron.spec (mmiklavc
> via
> > > >> mmiklavc) closes apache/metron#996
> > > >> 3 weeks ago METRON-1445: Update performance tuning guide with more
> > > >> explicit parameter instructions (mmiklavc via mmiklavc) closes
> > > >> apache/metron#988
> > > >> 3 weeks ago METRON-1502 Upgrade Doxia plugin to 1.8 (justinleet)
> > closes
> > > >> apache/metron#974
> > > >> 3 weeks ago METRON-1527: Remove dead test file sitting in source
> > folder
> > > >> (mmiklavc via mmiklavc) clo

Re: [DISCUSS] Pcap panel architecture

[zeo...@gmail.com]

This looks really great and gets me excited to maybe revisit some old
conversations about PCAP capture in Metron.  The only thing that I think
it's missing is the ability to filter using bpf.  I think the same thing
can technically be accomplished by using packet_filter and I wouldn't throw
a fit if that's considered a follow-on, but bpf is the standard language
that people who do packet munging for a living know.

Jon

On Wed, May 9, 2018 at 1:00 PM Ryan Merriman  wrote:

> Now that we are confident we can run submit a MR job from our current REST
> application, is this the desired approach?  Just want to confirm.
>
> Next I think we should map out what the REST interface will look like.
> Here are the endpoints I'm thinking about:
>
> GET /api/v1/pcap/metadata?basePath
>
> This endpoint will return metadata of pcap data stored in HDFS.  This would
> include pcap size, date ranges (how far back can I go), etc.  It would
> accept an optional HDFS basePath parameter for cases where pcap data is
> stored in multiple places and/or different from the default location.
>
> POST /api/v1/pcap/query
>
> This endpoint would accept a pcap request, submit a pcap query job, and
> return a job id.  The request would be an object containing the parameters
> documented here:  https://github.com/apache/metron/tree/master/
> metron-platform/metron-pcap-backend#query-filter-utility.  A query/job
> would be associated with a user that submits it.  An exception will be
> returned for violating constraints like too many queries submitted, query
> parameters out of limits, etc.
>
> GET /api/v1/pcap/status/
>
> This endpoint will return the status of a running job.  I imagine this is
> just a proxy to the YARN REST api.  We can discuss the implementation
> behind these endpoints later.
>
> GET /api/v1/pcap/stop/
>
> This endpoint would kill a running pcap job.  If the job has already
> completed this is a noop.
>
> GET /api/v1/pcap/list
>
> This endpoint will list a user's submitted pcap queries.  Items in the list
> would contain job id, status (is it finished?), start/end time, and number
> of pages.  Maybe there is some overlap with the status endpoint above and
> the status endpoint is not needed?
>
> GET /api/v1/pcap/pdml//
>
> This endpoint will return pcap results for the given page in pdml format (
> https://wiki.wireshark.org/PDML).  Are there other formats we want to
> support?
>
> GET /api/v1/pcap/raw//
>
> This endpoint will allow a user to download raw pcap results for the given
> page.
>
> DELETE /api/v1/pcap/
>
> This endpoint will delete pcap query results.  Not sure yet how this fits
> in with our broader cleanup strategy.
>
> This should get us started.  What did I miss and what would you change
> about these?  I did not include much detail related to security, cleanup
> strategy, or underlying implementation details but these are items we
> should discuss at some point.
>
> On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Sweet! That's great news. The pom changes are a lot simpler than I
> > expected. Very nice.
> >
> > On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman 
> wrote:
> >
> > > Finally figured it out.  Commit is here:
> > > https://github.com/merrimanr/incubator-metron/commit/
> > > 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
> > >
> > > It came down to figuring out the right combination of maven
> dependencies
> > > and passing in the HDP version to REST as a Java system property.  I
> also
> > > included some HDFS setup tasks.  I tested this in full dev and can now
> > > successfully run a pcap query and get results.  All you should have to
> do
> > > is generate some pcap data first.
> > >
> > > On Tue, May 8, 2018 at 4:17 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > @Ryan - pulled your branch and experimented with a few things. In
> doing
> > > so,
> > > > it dawned on me that by adding the yarn and hadoop classpath, you
> > > probably
> > > > didn't introduce a new classpath issue, rather you probably just
> moved
> > > onto
> > > > the next classpath issue, ie hbase per your exception about hbase
> jaxb.
> > > > Anyhow, I put up a branch with some pom changes worth trying in
> > > conjunction
> > > > with invoking the rest app startup via "/usr/bin/yarn jar"
> > > >
> > > > https://github.com/mmiklavc/metron/tree/ryan-rest-test
> > > >
> > > > https://github.com/mmiklavc/metron/commit/
> > 5ca23580fc6e043fafae2327c80b65
> > > > b20ca1c0c9
> > > >
> > > > Mike
> > > >
> > > >
> > > > On Tue, May 8, 2018 at 7:44 AM, Simon Elliston Ball <
> > > > si...@simonellistonball.com> wrote:
> > > >
> > > > > That would be a step closer to something more like a micro-service
> > > > > architecture. However, I would want to make sure we think about the
> > > > > operational complexity, and mpack implications of having another
> > server
> > > > > installed and running somewhere on the cluster (also, ssl,
> kerberos,
> > > etc
> > > >

Re: [DISCUSS] Release?

[zeo...@gmail.com]

We should also mention the Upgrade of ElasticSearch and Kibana

Jon

On Wed, May 9, 2018 at 12:49 PM Nick Allen  wrote:

> Oh, and also the Solr work that is currently in a feature branch.  We would
> have to get the work finished up and merged though.  Sounds like we are
> real close on that.
>
> On Wed, May 9, 2018 at 12:47 PM, Nick Allen  wrote:
>
> > The next part of the conversation would be, what should the version
> number
> > be?  To help with that, I have tried to summarize the changes in the
> > release.  Of course, this is going to be heavily biased towards my own
> > interests, so please feel free to chime in if I have missed anything.
> >
> >
> >- Significant performance improvements in Parsing, Enrichments, and
> >Indexing
> >
> >
> >- Support for Ubuntu installations
> >
> >
> >- Support for Storm 1.1 and HDP 2.6
> >
> >
> >- Event time processing in the Profiler
> >
> >
> >- Complex document support in the JSONMapParser
> >
> >
> >- Support for the Elasticsearch X-Pack
> >
> >
> >- Run Stellar in a Zeppelin Notebook
> >
> >
> >- Oodles of bug fixes and usability improvements
> >
> >
> >
> >
> > On Wed, May 9, 2018 at 12:14 PM, Nick Allen  wrote:
> >
> >> Something like this might be more digestible for these purposes.
> >>
> >> $git log --pretty="%cr %s" tags/apache-metron-0.4.2-release..HEAD
> >>
> >> 88 minutes ago METRON-1530 Default proxy config settings in
> >> metron-contrib need to be updated (sardell via merrimanr) closes
> >> apache/metron#998
> >> 5 days ago METRON-1545 Upgrade Spring and Spring Boot (merrimanr) closes
> >> apache/metron#1008
> >> 7 days ago METRON-1543 Unable to Set Parser Output Topic in Sensor
> Config
> >> (nickwallen) closes apache/metron#1007
> >> 2 weeks ago METRON-1539: Specialized RENAME field transformer closes
> >> apache/incubator-metron#1002
> >> 2 weeks ago METRON-1520: Add caching for stellar field transformations
> >> closes apache/incubator-metron#990
> >> 2 weeks ago METRON-1529 CONFIG_GET Fails to Retrieve Latest Config When
> >> Run in Zeppelin REPL (nickwallen) closes apache/metron#997
> >> 2 weeks ago METRON-1511 Unable to Serialize Profiler Configuration
> >> (nickwallen) closes apache/metron#982
> >> 3 weeks ago METRON-1528: Fix missing file in metron.spec (mmiklavc via
> >> mmiklavc) closes apache/metron#996
> >> 3 weeks ago METRON-1445: Update performance tuning guide with more
> >> explicit parameter instructions (mmiklavc via mmiklavc) closes
> >> apache/metron#988
> >> 3 weeks ago METRON-1502 Upgrade Doxia plugin to 1.8 (justinleet) closes
> >> apache/metron#974
> >> 3 weeks ago METRON-1527: Remove dead test file sitting in source folder
> >> (mmiklavc via mmiklavc) closes apache/metron#994
> >> 3 weeks ago METRON-1499 Enable Configuration of Unified Enrichment
> >> Topology via Ambari (nickwallen) closes apache/metron#984
> >> 3 weeks ago METRON-1515: Errors loading stellar functions currently bomb
> >> the entire topology, they should be recoverable closes
> >> apache/incubator-metron#985
> >> 4 weeks ago METRON-1522 Fix the typo errors at profile debugger readme
> >> (MohanDV via nickwallen) closes apache/metron#992
> >> 4 weeks ago METRON-1519 Indexing Error Topic Property Not Displayed in
> >> MPack (nickwallen) closes apache/metron#987
> >> 4 weeks ago METRON-1347: Indexing Topology should fail tuples without a
> >> source.type (cstella via mmiklavc) closes apache/metron#863
> >> 4 weeks ago Add support for Vagrant Cachier plugin if present
> >> (simonellistonball via mmiklavc) closes apache/metron#993
> >> 4 weeks ago METRON-1521: JSONMapParser is no longer serializable closes
> >> apache/incubator-metron#991
> >> 4 weeks ago METRON-1516 Support for Ansible 2.5.0 (ottobackwards) closes
> >> apache/metron#989
> >> 4 weeks ago METRON-1494 Profiler Emits Messages to Kafka When Not Needed
> >> (nickwallen) closes apache/metron#967
> >> 4 weeks ago METRON-1510 Update Metron website to include info about
> >> github update subscription (anandsubbu) closes apache/metron#981
> >> 4 weeks ago METRON-1518 Build Failure When Using Profile HDP-2.5.0.0
> >> (nickwallen) closes apache/metron#986
> >> 4 weeks ago METRON-1465:Support for Elasticsearch X-pack (wardbekker via
> >> mmiklavc) closes apache/metron#946
> >> 4 weeks ago METRON-1504: Enriching missing values does not match the
> >> semantics between the new enrichment topology and old closes
> >> apache/incubator-metron#976
> >> 5 weeks ago METRON-1505 Intermittent Profiler Integration Test Failure
> >> (nickwallen) closes apache/metron#977
> >> 5 weeks ago METRON-1449 Set Zookeeper URL for Stellar Running in
> Zeppelin
> >> Notebook (nickwallen) closes apache/metron#931
> >> 5 weeks ago METRON-1462: Separate ES and Kibana from Metron Mpack
> >> (mmiklavc via mmiklavc) closes apache/metron#943
> >> 5 weeks ago METRON-1501 Parser messages that fail to validate are
> dropped
> >> silently (cestella via justinleet) closes apache/metron#972
> >> 5 weeks 

Re: [DISCUSS] Pcap UI user requirements

[zeo...@gmail.com]

>From my perspective PCAP is primarily used as a follow-on to an alert or
meta-alert - people very rarely use PCAP for initial hunting.  I know this
has been brought up by Otto, Mike, and Ryan across the two related threads
and I think it's all spot on.  Going from an alert or meta-alert to pulling
PCAP would by far the primary use case for this in every SOC I've ever
worked in (not necessarily a representative sample).

I also have some additional thoughts on the feature side after doing some
brainstorming and talking to two of the SOCs I work most with:
 - Limit the size of the PCAP, not just the date range, and maybe even have
a configurable cluster-wide admin max for PCAP retrieval, set to 0/infinite
by default.
 - Set priority of PCAP queries.  Perhaps there's an automated
pcap retrieval 'just in case', which should have a lower priority than an
interactive request via the UI.
 - Ability to pause/resume (not just cancel) jobs.
 - Configurable cluster-wide admin max # of current PCAP queries, set to
0/infinite by default.
 - Ability to pull PCAP live off the wire and stream it into a file.
 - Ability to filter PCAP by providing a BPF filter to apply in server-side
post-processing (less efficient, but very versatile).
 - Request what PCAP data exists in the cluster (answering "how far back
can I go?")
 - This is obvious and is probably assumed, but queries based on any set of
the network 5 tuple (IPs, Ports, Protocol) with at least 1 required.

Jon

On Fri, May 4, 2018 at 9:44 AM Otto Fowler  wrote:

> That is the ‘views’ part.
>
> We can have options on the data output, if you have output full data, then
> we can have different views and interactions for inspection and level of
> detail.
>
>
>
> On May 4, 2018 at 09:37:13, Michel Sumbul (michelsum...@gmail.com) wrote:
>
> It can be like a report but also to investigate some case where the user
> want to see the whole packet (all the bits and bytes). Like in wireshark,
> something interactive no?
>
> 2018-05-04 14:33 GMT+01:00 Otto Fowler :
>
> > The PCAP Query seems more like PCAP Report to me. You are generating a
> > report based on parameters.
> > That report is something that takes some time and external process to
> > generate… ie you have to wait for it.
> >
> > I can almost imagine a flow where you:
> >
> > * Are in the AlertUI
> > * Ask to generate a PCAP report based on some selected alerts/meta-alert,
> > possibly picking from on or more report ‘templates’
> > that have query options etc
> > * The report request is ‘queued’, that is dispatched to be be
> > executed/generated
> > * You as a user have a ‘queue’ of your report results, and when the
> report
> > is done it is queued there
> > * We ‘monitor’ the report/queue press through the yarn rest ( report
> > info/meta has the yarn details )
> > * You can select the report from your queue and view it either in a new
> UI
> > or custom component
> > * You can then apply a different ‘view’ to the report or work with the
> > report data
> > * You can print / save etc
> > * You can associate the report with the alerts ( again in the report info
> )
> > with…. a ‘case’ or ‘ticket’ or investigation something or other
> >
> >
> > We can introduce extensibility into the report templates, report views (
> > thinks that work with the json data of the report )
> >
> > Something like that.
> >
> >
> > On May 4, 2018 at 09:19:15, Ryan Merriman (merrim...@gmail.com) wrote:
> >
> > Continuing a discussion that started in a discuss thread about exposing
> > Pcap query capabilities in the back end. How should we expose this
> feature
> > to users? Should it be integrated into the Alerts UI or be separate
> > standalone UI?
> >
> > To summarize the general points made in the other thread:
> >
> > - Adding this capability to the Alerts UI will make it more of a
> > composite app. Is that really what we want since we have separate UIs for
> > Alerts and management?
> > - Would it be better to bring it in on it's own so it can be released
> > with qualifiers and tested with the right expectations without affecting
> > the Alerts UI?
> > - There are some use cases that begin with an infosec analyst doing a
> > search on alerts
> > followed by them going to query pcap data corresponding to the
> > threats they're investigating. Would having these features in the same
> > UI streamline this process?
> >
> > There was also mention of some features we should consider:
> >
> > - Pcap queries should be made asynchronous via the UI
> > - Take care that a user doesn't hit refresh or POST multiple times and
> kick
> > off 50 mapreduce jobs
> > - Options for managing the YARN queue that is used
> > - Provide a "cancel" option that kills the MR job, or tell the user to
> > go to the CLI to kill their job
> > - Managing data if multiple users run queries
> > - Strategy for cleaning up jobs and implementing a TTL (I think this one
> > will be tricky and definitely needs discussion)
> > - Date range or other query limits
> >
>

Re: [VOTE] Development Guidelines Addendum on Inactive Pull Requests

[zeo...@gmail.com]

+1 (non-binding)

On Fri, Apr 20, 2018 at 9:42 AM Michel Sumbul 
wrote:

> +1
>
> 2018-04-20 14:40 GMT+01:00 Otto Fowler :
>
> > +1
> >
> >
> > On April 20, 2018 at 09:30:30, Nick Allen (n...@nickallen.org) wrote:
> >
> > I am proposing the following addition to the project's development
> > guidelines [1]. Based on these guidelines, an abandoned pull request can
> > be closed in roughly 6 weeks time (4 weeks of inactivity plus 2 weeks to
> > respond to a committer's request.)
> >
> > Please vote +1, 0, or -1 and also indicate if your vote is binding or
> > non-binding. More information on voting can be found in the Apache Metron
> > By-Laws [2].
> >
> > This vote will remain open for at least 72 hours, excluding this weekend.
> > I plan to close the vote no sooner than Wednesday, April 25, 2018 at 8:00
> > AM EST.
> >
> > The discuss thread that preceeded this vote can be found here [3].
> >
> > --
> >
> > 2.6.1 Inactive Pull Requests
> >
> >
> > Contributions can often take a significant amount of time to complete the
> > code review process. This process requires active participation from the
> > contributor. If the contributor is unable to actively participate, the
> > pull request is unlikely to successfully complete this process.
> >
> > Pull Requests that have failed to receive active participation from the
> > contributor for an extended period of time risk being abandoned. Any
> > committer can submit a request for Apache Infra to close a pull request
> > that has been abandoned according to the following guidelines.
> >
> >
> > - A pull request is 'inactive' if no comments or updates have been made
> > by the contributor in the previous 4 weeks.
> >
> >
> > - For any 'inactive' pull request, a committer can request from the
> > contributor justification for keeping the pull request open.
> >
> >
> > - The committer's request should be made as a public comment on the pull
> > request. The committer should refer the contributor to these development
> > guidelines for inactive pull requests.
> >
> >
> > - If the contributor publically responds to the request, the pull
> > request is no longer consider 'inactive'.
> >
> >
> > - If the contributor does not respond to the request within 2 weeks, the
> > pull request is considered 'abandoned'.
> >
> >
> > - A committer can cast a -1 vote on any 'abandoned' pull request using
> > these development guidelines as justification.
> >
> >
> > - A committer can submit a request to Apache Infra to close the
> > 'abandoned' pull request based on this -1 vote.
> >
> > --
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> >
> > [2] https://cwiki.apache.org/confluence/display/METRON/
> > Apache+Metron+Bylaws
> >
> > [3]
> > https://lists.apache.org/thread.html/a4e72af67994c8e818f843a9ea8cc2
> > 86d81b5c72002fd011d66111f6@%3Cdev.metron.apache.org%3E
> >
>
-- 

Jon

Re: GeoLite deprecating legacy DBs

[zeo...@gmail.com]

Okay great, sorry I missed that conversation.  I did a quick search earlier
on my phone of where we pull the DBs from and if you strip out the file
GeoLite2-City.mmdb.gz it redirects to the legacy page, but I guess that's
just a strange redirection and not an indication that that download area
only holds the legacy DBs.

Jon

On Fri, Apr 13, 2018 at 7:03 AM Nick Allen  wrote:

> See also https://issues.apache.org/jira/browse/METRON-1517
>
> On Fri, Apr 13, 2018, 5:40 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > Don’t we already use the GeoLite2 database? Mine are all
> > /apps/metron/geo/default/GeoLite2-City.mmdb.gz downloaded from
> > http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
> > which seems to match the new format page.
> >
> > Am I missing something Jon, or are you referring to the old geo
> enrichment?
> >
> > Simon
> >
> >
> > > On 13 Apr 2018, at 10:27, zeo...@gmail.com  wrote:
> > >
> > > Looks like we will need to update the Geo DBs that we use for
> enrichment.
> > >
> > >
> > > Updated versions of the GeoLite Legacy databases are now only available
> > to
> > > redistribution license customers, although anyone can continue to
> > download
> > > the March 2018 GeoLite Legacy builds. Starting January 2, 2019, the
> last
> > > build will be removed from our website. GeoLite Legacy database users
> > will
> > > need to switch to the GeoLite2 or commercial GeoIP databases and update
> > > their integrations by January 2, 2019.
> > >
> > > New Database Format Available: ... For our latest database format,
> please
> > > see our GeoLite2 Databases.
> > >
> > > https://dev.maxmind.com/geoip/legacy/geolite/
> > >
> > > Jon
> > > --
> > >
> > > Jon
> >
> >
>
-- 

Jon

GeoLite deprecating legacy DBs

[zeo...@gmail.com]

Looks like we will need to update the Geo DBs that we use for enrichment.


Updated versions of the GeoLite Legacy databases are now only available to
redistribution license customers, although anyone can continue to download
the March 2018 GeoLite Legacy builds. Starting January 2, 2019, the last
build will be removed from our website. GeoLite Legacy database users will
need to switch to the GeoLite2 or commercial GeoIP databases and update
their integrations by January 2, 2019.

New Database Format Available: ... For our latest database format, please
see our GeoLite2 Databases.

https://dev.maxmind.com/geoip/legacy/geolite/

Jon
-- 

Jon

Re: Secure code analysis

[zeo...@gmail.com]

I would like to volunteer some effort to see how we might be able to
integrate Veracode scans with the ASF Jenkins instance to see how it could
be useful, but in order to do so I need to get some additional
authorization.  *Would a PMC member mind getting me access* so I can take a
look, given that nobody seems to have had an issue with this?  For
reference, from my prior email:

The ASF seems to support giving non-PMC committers access
<https://wiki.apache.org/general/Jenkins#How_do_I_get_an_account> to
Jenkins, but it requires that the PMC chair do some work, and generally it
looks like they want admins
<https://wiki.apache.org/general/Jenkins#FAQ_For_Administrators>/PMC
<https://wiki.apache.org/general/Jenkins#FAQ_For_PMCs> members to be
involved (I also don't have access to the builds JIRA project
<https://issues.apache.org/jira/projects/BUILDS>, if it really exists).

Jon

On Sun, Jan 7, 2018 at 8:16 AM Nadir Hajiyani 
wrote:

> Here is the documentation for various Veracode integrations -
> https://help.veracode.com/reader/QJgoLlv~uqsO6Zvu9jG9pw/
> h2NG_xyaRqXJtAUioBS2SA
>
> A few options can be explored here, like:
>
>- Sending the scans directly via the IDE (Eclipse, IntelliJ, Visual
>Studio)
>- Utilizing the API Wrapper
>- Using the Upload API (Easier said than done)
>
>
> On Sun, Dec 24, 2017 at 9:58 AM, Nick Allen  wrote:
>
> > > 3) I have been manually making submissions dating back to 2017-02-13,
> but
> >
> > Oh, great.
> > ​So your general impression based on those submissions is that this would
> > be useful for us?
> >
> > I didn't realize that you had already been reviewing the output of the
> tool
> > over a period of time.
> >
> > Thanks, Jon
> >
> >
> > On Dec 23, 2017 8:32 PM, "zeo...@gmail.com"  wrote:
> >
> > Sure, not a problem.
> >
> > (1) I went to an event where a presenter from Veracode was calling out
> some
> > bugs in open source projects, and that Veracode wanted to be a part of
> the
> > solution.  As such, they offered to give free analysis to open source
> > projects that reach out.  At this point the account that I have access to
> > is just for the Apache Metron project, but it is possible that the
> > relationship could grow if it makes sense for other projects.  For
> > instance, this <
> https://twitter.com/PeteChestna/status/943845893597483008
> > >.
> >
> > (2) No specific reason - in the past I looked at Coverity (see below in
> > this thread) but was deterred from personally setting it up due to some
> of
> > their policies about who can register new scans (i.e. I was not a
> committer
> > at the time I believe, and that level of involvement was requested).  I
> > have used Veracode in the past, along with others (AppScan, Fortify,
> etc.),
> > and had a good experience albeit in a very different setting than this.
> I
> > would be more than happy to play around with any of these kinds of
> services
> > and no affinity to one or the other, but right now the only thing I
> > actually have access to is Veracode and free options like Coverity.
> >
> > Veracode is a proprietary cloud-hosted platform that has dynamic and
> static
> > scan offerings, and they have various integrations
> > <https://community.veracode.com/s/integrations> with build systems
> (maven,
> > Jenkins, Bamboo, etc.) and IDEs (IntelliJ, Eclipse, etc.).  They also
> > appear to have opened up their training materials
> > <https://community.veracode.com/s/education-and-training>, which are
> handy
> > to point to from time to time.  I've worked with it in the past and
> things
> > largely seem to work as you would expect, although it has been 5 years
> > since I really used their products regularly.
> >
> > (3) I have been manually making submissions dating back to 2017-02-13,
> but
> > because the file transfer is uploaded from my home Internet (upload
> speeds
> > of ~6Mbps), it takes quite a while and so I don't do it very frequently.
> > Usually just around releases.
> >
> > Jon
> >
> > On Sat, Dec 23, 2017 at 11:13 AM Nick Allen  wrote:
> >
> > > > Veracode has provided us with a 100% free portal to scan the Metron
> > code
> > > with, but in order to integrate, the safest option is probably to use
> the
> > > ASF's jenkins server
> > >
> > > (1) Can you describe this more?   How has this been provided?  Is this
> > for
> > > all Apache projects; just Metron?  Was this based on a relationship you
> > > have 

Re: [DISCUSS] Generic Syslog Parsing capability for parsers

[zeo...@gmail.com]

So I've kept my ear to the ground regarding this topic for a while now, and
had some conversations a year or so ago about the idea as well.  At the
very least, I think having the concept of a pre-parser is a good one, if
not chaining an arbitrary number of parsers together.  I see this as an
important way to reduce the complexity of implementing new parsers and
getting more community involvement/contributions.

Syslog headers are a solid use case to start with because a lot of
implementations fail to properly implement it on the sending side, at least
in the real world scenarios that I've seen.  Having a way to extend the
parser to easily handle incorrect implementations of syslog would be great,
but anything that can pre-parse or trim the syslog headers to make parsing
further along in the pipeline more simple would help.

Another idea that would be attractive would be the ability to do
opportunistic parsing given an ordered list of parsers and some criteria
for successful parsing (which I admittedly am not sure how to solve) which
(at least in my mind) would require similar logic to parser chaining.  In
some highly decentralized organizations this would be helpful as it takes
the configuration effort off of the team sending the logs (and thus makes
them more willing to send logs _at all_) and pushes it onto the team
parsing and/or storing them.

I'm not suggesting we attempt to crack that second nut here, I would love
to see that use case in mind during discussions.

TL;DR:  +1

Jon

On Tue, Mar 20, 2018 at 6:14 PM Otto Fowler  wrote:

> I think the chaining of parsers, or ability to compose parsers is a good
> idea, but with reference to the pr mentioned, I would have some number of
> StellarChainLinks as opposed re-implementing stellar in chainlinks.
> Although it is NiFi-y.  But since I write Processors too, that is fine.
>
>
> On March 20, 2018 at 18:05:12, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> It seems like parser chaining is becomes a hot topic on the repo too with
> https://github.com/apache/metron/pull/969#partial-pull-merging <
> https://github.com/apache/metron/pull/969#partial-pull-merging>
>
> I would like to discuss the option, and how we might architect, of
> configuring parsers to operate on the output of parsers. This may also give
> us the opportunity to be more efficient in scenarios where people have
> large numbers of sources, and so use up a lot of slots for lower volume
> parsers for example.
>
> I have a bunch of ideas around this, but am more keen to hear what everyone
> else thinks at this stage. How should we go about fixing parser config so
> that it’s clearer (removing the need for people to reinvent the parser
> wheel as we’ve seen in a few places) and also more concise and powerful
> (consolidating the parsing of transports such as syslog and content such as
> application logs, or types of device logs).
>
> If this can lead to a more efficient way of handling both the syslog
> problem, and the kind of problem that leads to switching between grok
> statements in something like our ASA parser then all the better. I suspect
> that there might also be a case for multi-level chaining here too, since
> some things are embedded in multiple transports, or might have complex
> fields that want ‘sub-parsing’.
>
> Of course one of the key values of Metron is its speed, so maybe
> formalising some of the microbenchmarking approaches a few of us have been
> working on might help here too. I’ve got a few bits of micro-benching
> infrastructure around CEF and ASA, and I believe there’s also been some
> work to load and perf test things like enrichment that might be leveraged.
>
> Thoughts on a dev board?
>
> Simon
>
> > On 20 Mar 2018, at 21:47, Otto Fowler  wrote:
> >
> > I entered METRON–1453  >
> a
> > little while ago while working on the PR#579
> > .
> >
> > "We have several parsers now, with many imaginable that are based on
> > syslog, where the format is SYSLOG HEADER MESSAGE.
> >
> > With message being in a different format. It would be great is we had a
> way
> > to generically handle syslog headers, such that ANY parser data could
> come
> > over syslog.
> >
> > Either you could have a custom parser, or configure CSV or JSON such that
> > they could be the payload, such that you can handle JSON over syslog by
> > configuration only."
> >
> > The idea would be that the parser bolt would use the configuration to
> > trigger parsing the incoming message as syslog formatted, and pass the
> > message part to the parser, and put the syslog parts in the message(s)
> > after parsing.
> >
> > As part of this I did some work on parsing syslog, using both grok and a
> > DSL that I did from the spec :
> https://github.com/ottobackwards/grok-v-antlr
> >
> > The DSL is slower, but grok cannot handle multiple structured data
> entries,
> > and the DSL can. I’m not good enoug

Re: [DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron

[zeo...@gmail.com]

I agree, the first approach makes the most sense to me.

Jon

On Wed, Feb 21, 2018 at 11:45 AM Nick Allen  wrote:

> +1 to the first approach, as you've laid it out.  That makes the most sense
> to me.  We need a way to rev the version of the ES Mpack independent of the
> ES version.
>
> On Wed, Feb 21, 2018 at 11:30 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Anyone have any opinions on how we should version the ES/Kibana MPack?
> >
> > We currently rev the Metron one based on current Metron version and apply
> > it to both the overall MPack as well as the individual service versions,
> > e.g. metron_mpack-0.4.3.0/common-services/METRON/0.4.3. For ES we have
> > been
> > keeping the service version matched to the current ES version because
> it's
> > independent of Metron, ie 5.6.2. The upshot is that we can handle this
> one
> > of two ways.
> >
> >1. Keep the same approach after splitting off ES and Kibana - ES MPack
> >version is set to the current Metron version (0.4.3) and the service
> > itself
> >is set to the ES version (5.6.2).
> >2. Use ES version for both the MPack version and service version
> > (5.6.2).
> >
> >
> > I personally recommend and prefer the first approach because it allows us
> > to make changes to the mpack itself without necessarily changing the
> > version of ES or Kibana, which is something that is likely to happen.
> It's
> > also consistent and seamless with our current versioning approach.
> Lastly,
> > I don't believe the ES versions provide much contextual sense in the
> Metron
> > world for an MPack version - setting the service version definitely makes
> > sense to indicate what exact ES version we're using, but the MPack is
> > really our way of providing custom functionality that wraps a specific
> > version of ES/Kibana. Hope this makes sense.
> >
> > Mike
> >
> >
> > On Sat, Feb 17, 2018 at 11:13 AM, Nick Allen  wrote:
> >
> > > +1 I agree with Otto's idea.  It makes a lot of sense for Elasticsearch
> > and
> > > Kibana to live in a separate Mpack.
> > >
> > > This provides us a path forward to support additional indexers like
> Solr.
> > >
> > > We should also not force our users to install an external component
> (like
> > > Elasticsearch) using the Mpack.  There are just too many installation
> > > configurations for us to reasonably support; especially on larger
> > > installations.  Supporting the Elasticsearch MPack is a project unto
> > > itself.  That being said, the functionality will still be there for
> those
> > > that want to use it.
> > >
> > >
> > >
> > > On Fri, Feb 16, 2018 at 4:10 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > This came up earlier when discussing work around the ES upgrade:
> > > > https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd
> > > > 74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E
> > > > https://lists.apache.org/thread.html/8ec83b6a3ef39057c9466ff72a2f63
> > > > c9308452f1ebc1804e67cb495b@%3Cdev.metron.apache.org%3E
> > > >
> > > > Looks like Otto made this suggestion and Kyle is on board. I was
> > > originally
> > > > opposed to this because it did not seem worth the effort to support 2
> > > > separate MPacks. However, now that we are working on the Solr
> upgrade,
> > it
> > > > seems like an appropriate solution for enabling us to make the
> indexing
> > > > piece pluggable. I propose that we commence with this solution.
> > > >
> > > > Cheers,
> > > > Mike
> > > >
> > >
> >
>
-- 

Jon

Re: metron-bro-plugin kafka

[zeo...@gmail.com]

Okay, great.  It's possible that you need to do something like the
following to get known devices:

 echo "redef Software::asset_tracking = ALL_HOSTS;" >>
/usr/local/bro/share/bro/site/local.bro

These snippets are from my testing instructions related to adding support
for bro 2.5.2 logs (link <https://github.com/apache/metron/pull/844>).
They should find their way into the plugin README eventually.

Jon

On Tue, Feb 13, 2018 at 6:35 AM bharath phatak 
wrote:

> Hi Jon,
>
> Other than Known::DEVICES_LOG rest all worked.
>
> Thanks,
> Bharath
> On Tue, Feb 13, 2018, 4:15 PM zeo...@gmail.com  wrote:
>
> > Try
> >
> > redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG,
> > FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG,
> > Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG,
> > Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);
> >
> > Note that you usually wouldn't want to send reporter.log, as that's where
> > errors get sent and it could become an infinite loop.
> >
> > Jon
> >
> > On Tue, Feb 13, 2018, 05:26 bharath phatak 
> > wrote:
> >
> > > Hi Team,
> > >
> > > Can some one help me out on the list of
> > > redef Kafka::logs_to_send values?
> > >
> > > I want to push all logs generated by bro to Kafka.
> > >
> > > I tried adding log file name but getting bro is crashing
> > >
> > > Ex weird::LOG, Files::LOG
> > >
> > > Thanks,
> > > Bharath
> > >
> >
> >
> > --
> >
> > Jon
> >
>
-- 

Jon

Re: metron-bro-plugin kafka

[zeo...@gmail.com]

Try

redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG,
FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG,
Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG,
Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);

Note that you usually wouldn't want to send reporter.log, as that's where
errors get sent and it could become an infinite loop.

Jon

On Tue, Feb 13, 2018, 05:26 bharath phatak  wrote:

> Hi Team,
>
> Can some one help me out on the list of
> redef Kafka::logs_to_send values?
>
> I want to push all logs generated by bro to Kafka.
>
> I tried adding log file name but getting bro is crashing
>
> Ex weird::LOG, Files::LOG
>
> Thanks,
> Bharath
>


-- 

Jon

DataWorks Summit San Jose

[zeo...@gmail.com]

Hi All,

Just a heads up that *the San Jose DataWorks Summit's call for papers is
coming to a close soon *(February 9th, in 2 days!).  If you are doing
anything cool with open source big data and security that you want to talk
about, please submit to the Cyber Security track.  I'm hoping to attend and
I'm sure there will be others from the community in attendance as well.

Feel free to reply here (or to me directly) if you have any questions about
submitting or even if you just want to meet up at the event.  For more
information check out the website here
.

Jon
-- 

Jon

Re: [DISCUSS] Profiler Enhancement

[zeo...@gmail.com]

Scenario 2 is one that I'm specifically interested in, I have that exact
use case right now.  I can see Scenario 1 being useful in the future as
well.

I'm also interested in a conversation along the lines of what Otto brought
up (i.e. I would like to re-ingest data to redo parsing, enrichments, etc.)
but happy to keep that conversation separate or for the future.

Really just wanted to comment that this work effort has a huge +1 from me
and is something I've been following.

This work should interact nicely with METRON-1397
, because users may need to
ingest bulk data from time to time that is a result of a system export.

Jon

On Mon, Feb 5, 2018 at 9:38 AM Otto Fowler  wrote:

> I think that is fine,  we can use that and work out the UX to manage new or
> replace.  Maybe we can do Profile Compare down the line?
>
> On February 5, 2018 at 09:28:16, Nick Allen (n...@nickallen.org) wrote:
>
> > If we replay a set of data with a new version of a profile I think it
> will always have to be a new profile and not ‘replace’ the old one.
> Series1, Seriers2  etc?
>
> As part of this effort (unless there is a compelling reason) I wouldn't
> change that behavior.  The profile data is stored based on profile name +
> entity + timestamp (I'm glossing over some of the details, but that's
> effectively what happens).  If you change the definition of a profile, but
> the name does not change, then you would replace the existing profile
> data.  If you do not want to replace, then you should change the name of
> the profile.
>
> Now is this the best way to store the data?  I am not sure.  It is a
> complex discussion all by itself, but is something that I would rather
> handle as a separate effort.
>
>
>
> On Fri, Feb 2, 2018 at 5:42 PM, Otto Fowler 
> wrote:
>
> > You know, I am going to back this up.
> > I usually thing of replay as replay, profiler or not, but that is not
> true.
> > Replay of data through the full pipeline (parsers/enrichement) has more
> > consequences or concerns, so we can drop this.
> > I don’t want to expand the scope of your idea.  We can reuse/refactor to
> > the other case (parser + enrichment) later.
> > Sorry.
> >
> >
> > ——
> >
> > So, about re-writing.
> > If we replay a set of data with a new version of a profile I think it
> will
> > always have to be a new profile and not ‘replace’
> > the old one.   Series1, Seriers2  etc?
> >
> >
> >
> >
> > On February 2, 2018 at 17:24:46, Nick Allen (n...@nickallen.org) wrote:
> >
> > I think that is definitely a reasonable extension.
> >
> > In this case would we need any additional actions to indicate that data
> > will be overwritten?
> >
> > I am trying to think of other additional needs that this use case has
> over
> > the others.
> >
> > On Feb 2, 2018 12:38 PM, "Otto Fowler"  wrote:
> >
> >> Scenario 3:
> >> As a Security ?  I have modified a profile or parser configuration (
> >> replay is replay ), and I want to run the new version
> >> against my old data.
> >>
> >>
> >>
> >> On February 2, 2018 at 12:19:54, Nick Allen (n...@nickallen.org) wrote:
> >>
> >> I have been thinking about an enhancement to the Profiler for quite some
> >> time. Actually, my first pass at defining this was called "Replay
> >> Telemetry through Profiler" back in METRON-594 [1].
> >>
> >> I'd like to first discuss the use case to make sure we start out on the
> >> right foot. Here is how I would define the use cases for this
> >> functionality.
> >>
> >> *> Scenario 1: Model Development*
> >>
> >> As a Security Data Scientist, I want to understand the historical
> >> behaviors
> >> and trends of a profile that I have created so that I can understand if
> it
> >> is valuable for model building.
> >>
> >> There are two possible negative outcomes that the Security Data
> Scientist
> >> must be aware of when creating profiles.
> >>
> >>
> >> - The profile might have been defined incorrectly resulting in a feature
> >> set that does not match reality (a bug in the profile definition).
> >>
> >>
> >> - The profile might have been defined correctly, but the feature set
> >> itself has no predictive value.
> >>
> >> Analyzing the profile over archived, historical telemetry allows the
> >> Security Data Scientist to better to mitigate both of these negative
> >> outcomes.
> >>
> >>
> >> *> Scenario 2: Model Deployment*
> >>
> >> As a Security Platform Engineer, I want to generate a profile using
> >> archived telemetry when I deploy a new model to production so that
> models
> >> depending on that profile can begin to function on day 1.
> >>
> >>
> >>
> >> (Q) Do these make sense? Am I missing anything? Too broad or too narrow?
> >>
> >> Once we nail down the use case(s), I'll delete the old JIRA and create a
> >> new JIRA with the use cases. That would give us a place to start on the
> >> technical details of the implementation.
> >>
> >> [1] https://issues.apache.org/jira/browse/METRON-594
> >>
> >>
>
-- 

Jon

[REQUEST] Add Ian as an Assignee in JIRA

[zeo...@gmail.com]

Can someone add Ian Abreu as a potential assignee on JIRA?  He has a PR
 open against his
ticket  in the bro
plugin repo.  Thanks,

Jon
-- 

Jon

Re: Metron User Community Meeting Call

[zeo...@gmail.com]

Thanks Otto, I'm in to attend at that time/place.

Jon

On Thu, Jan 25, 2018, 14:45 Otto Fowler  wrote:

> I would like to propose a Metron user community meeting. I propose that we
> set the meeting next week, and will throw out Wednesday, January 31st at
> 09:30AM PST, 12:30 on the East Coast and 5:30 in London Towne. This meeting
> will be held over a web-ex, the details of which will be included in the
> actual meeting notice.
> Topics
>
> We have a volunteer for a community member presentation:
>
> Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR -
> Cybersecurity Operations Center Carleton University - cugcr.com
>
> Ahmed would like to talk to the community about
>
>-
>
>Who the GCR group is
>-
>
>How they use Metron 0.4.1
>-
>
>Walk through their dashboards, UI management screen, nifi
>-
>
>Challenges we faced up until now
>
> I would like to thank Ahmed for stepping forward for this meeting.
>
> If you have something you would like to present or talk about please reply
> here! Maybe we can have people ask for “A better explanation of feature
> X” type things?
> Metron User Community Meetings
>
> User Community Meetings are a means for realtime discussion of experiences
> with Apache Metron, or demonstration of how the community is using or will
> be using Apache Metron.
>
> These meetings are geared towards:
>
>-
>
>Demonstrations and knowledge sharing as opposed to technical
>discussion or implementation details from members of the Apache Metron
>Community
>-
>
>Existing Feature demonstrations
>-
>
>Proposed Feature demonstrations
>-
>
>Community feedback
>
> These meetings are *not* for :
>
>-
>
>Support discussions. Those are best left to the mailing lists.
>-
>
>Development discussions. There is another type of meeting for that.
>
>
>
>

-- 

Jon

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

[zeo...@gmail.com]

I agree with having a metron_ prefix for ES indexes, and the timing.

Jon

On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> With the completion of https://github.com/apache/metron/pull/840
> (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings for a
> major release rev of Metron in the upcoming release (currently slotted to
> 0.4.3, I believe). Since there are non-backwards compatible changes
> pertaining to ES indexing, it seems like a good opportunity to revisit our
> index naming standards.
>
> I propose we add a simple prefix "metron_" to all Metron indexes. There are
> numerous reasons for doing so
>
>- removes the likelihood of index name collisions when we perform
>operations on index wildcard names, e.g. "enrichment_*, indexing_*,
> etc.".
>- ie, this allows us to be more friendly in a multi-tenant ES
>environment for relatively low engineering cost.
>- simplifies the Kibana dashboard a bit. We currently needed to create a
>special index pattern in order to accommodate multi-index pattern
> matching
>across all metron-specific indexes. Using metron_* would be much simpler
>and less prone to error.
>- easier for customers to debug and identify Metron-specific indexes and
>associated data
>
>
> The reason for making these changes now is that we already have breaking
> changes with ES. Leveraging existing indexed data rather than deleting
> indexes and starting from scractch already requires a re-indexing/migration
> step, so there is no additional effort on the part of users if they choose
> to attempt a migration. It further makes sense with our current work
> towards upgrading Solr.
>
> We already have a battery of integration and manual tests after the ES
> upgrade work that can be leveraged to validate the changes.
>
> Mike Miklavcic
>


-- 

Jon

Re: [DISCUSS] Time to remove github updates from dev?

[zeo...@gmail.com]

I would give that  +1 as well.

Jon

On Fri, Jan 19, 2018 at 3:32 PM Casey Stella  wrote:

> I could get behind that.
>
> On Fri, Jan 19, 2018 at 3:31 PM, Andre  wrote:
>
> > Folks,
> >
> > May I suggest Metron follows the NiFi mailing list strategy (we got
> > inspired by another project but I don't recall the name) and remove the
> > github comments from the dev list?
> >
> > Within NiFi we have both the dev and the issues lists. dev is for humans,
> > issues is for JIRA and github commits.[1]
> >
> > This allows the list thread list to be cleaner and is particularly
> helpful
> > for those reading the list from a list aggregation service.
> >
> > Cheers
> >
> >
> > [1] https://lists.apache.org/list.html?iss...@nifi.apache.org
> >
>


-- 

Jon

Re: Anand is a new Committer!

[zeo...@gmail.com]

Welcome aboard, Anand!  Congrats

Jon

On Thu, Jan 11, 2018 at 10:41 AM Otto Fowler 
wrote:

> Congratulations and welcome Anand!
>
>
> On January 11, 2018 at 09:29:24, Casey Stella (ceste...@gmail.com) wrote:
>
> The Project Management Committee (PMC) for Apache Metron has invited Anand
> Subramanian to become a committer and we are pleased to announce that they
> have accepted.
>
> Congratulations and welcome, Anand!
>


-- 

Jon

Re: Secure code analysis

[zeo...@gmail.com]

Sure, not a problem.

(1) I went to an event where a presenter from Veracode was calling out some
bugs in open source projects, and that Veracode wanted to be a part of the
solution.  As such, they offered to give free analysis to open source
projects that reach out.  At this point the account that I have access to
is just for the Apache Metron project, but it is possible that the
relationship could grow if it makes sense for other projects.  For
instance, this <https://twitter.com/PeteChestna/status/943845893597483008>.

(2) No specific reason - in the past I looked at Coverity (see below in
this thread) but was deterred from personally setting it up due to some of
their policies about who can register new scans (i.e. I was not a committer
at the time I believe, and that level of involvement was requested).  I
have used Veracode in the past, along with others (AppScan, Fortify, etc.),
and had a good experience albeit in a very different setting than this.  I
would be more than happy to play around with any of these kinds of services
and no affinity to one or the other, but right now the only thing I
actually have access to is Veracode and free options like Coverity.

Veracode is a proprietary cloud-hosted platform that has dynamic and static
scan offerings, and they have various integrations
<https://community.veracode.com/s/integrations> with build systems (maven,
Jenkins, Bamboo, etc.) and IDEs (IntelliJ, Eclipse, etc.).  They also
appear to have opened up their training materials
<https://community.veracode.com/s/education-and-training>, which are handy
to point to from time to time.  I've worked with it in the past and things
largely seem to work as you would expect, although it has been 5 years
since I really used their products regularly.

(3) I have been manually making submissions dating back to 2017-02-13, but
because the file transfer is uploaded from my home Internet (upload speeds
of ~6Mbps), it takes quite a while and so I don't do it very frequently.
Usually just around releases.

Jon

On Sat, Dec 23, 2017 at 11:13 AM Nick Allen  wrote:

> > Veracode has provided us with a 100% free portal to scan the Metron code
> with, but in order to integrate, the safest option is probably to use the
> ASF's jenkins server
>
> (1) Can you describe this more?   How has this been provided?  Is this for
> all Apache projects; just Metron?  Was this based on a relationship you
> have within CA?
>
>
> (2) Why Veracode?  Can you describe this platform more?  Is it open source
> or proprietary?  Why is this better than alternatives?
>
>
> (3) I have no objection to experimenting with the service to see if it
> provides actionable results, but is there no simpler way to do this?  It
> doesn't seem like we should have to mess with a bunch of Apache
> infrastructure to see if the service works at a basic level.  Can't we
> manually submit master and/or previous releases to Veracode to see if we
> get actionable results?
>
>
>
>
>
> On Thu, Dec 21, 2017 at 10:48 AM, zeo...@gmail.com 
> wrote:
>
> > Just following up on this conversation again -
> >
> > I have discussed this ad-hoc with a few PMC members recently and wanted
> to
> > bring it up on the list.  Veracode has provided us with a 100% free
> portal
> > to scan the Metron code with, but in order to integrate, the safest
> option
> > is probably to use the ASF's jenkins server (as I'm not aware of a safe
> way
> > to automatically pass API creds to Veracode from GitHub).  My long-term
> > interest here would be to scan and clean up the code base generally, and
> > then to try and scan PRs for concerns (non-blocking).  Perhaps at some
> > point, if we identify that these scans are actually useful and not
> > false-positive prone/onerous, we could turn this into a blocking
> > requirement for contributions.  Being a security project, I feel that we
> > should be doing as much as we can to ensure that what we're providing is
> > safe.
> >
> > I looked briefly at the Veracode Jenkins integrations, and the ASF
> Jenkins
> > setup.  It looks like Veracode has a Jenkins plugin
> > <https://help.veracode.com/reader/PgbNZUD7j8aY7iG~hQZWxQ/
> > _4G8gT1rhWMgVVtCI1C57A>,
> > Jenkins has a plugin for Veracode in its plugin repo
> > <https://plugins.jenkins.io/veracode-scanner> (not supported by
> Veracode),
> > the ASF supports adding plugins
> > <https://wiki.apache.org/general/Jenkins#How_do_I_
> > install_a_new_Jenkins_plugin.3F>
> > to their Jenkins servers (although I think
> > <http://What_do_Administrators_do.3F> the admins are supposed to do
> this),
> > and Metron is not yet set up <https://bui

Re: Secure code analysis

[zeo...@gmail.com]

Just following up on this conversation again -

I have discussed this ad-hoc with a few PMC members recently and wanted to
bring it up on the list.  Veracode has provided us with a 100% free portal
to scan the Metron code with, but in order to integrate, the safest option
is probably to use the ASF's jenkins server (as I'm not aware of a safe way
to automatically pass API creds to Veracode from GitHub).  My long-term
interest here would be to scan and clean up the code base generally, and
then to try and scan PRs for concerns (non-blocking).  Perhaps at some
point, if we identify that these scans are actually useful and not
false-positive prone/onerous, we could turn this into a blocking
requirement for contributions.  Being a security project, I feel that we
should be doing as much as we can to ensure that what we're providing is
safe.

I looked briefly at the Veracode Jenkins integrations, and the ASF Jenkins
setup.  It looks like Veracode has a Jenkins plugin
<https://help.veracode.com/reader/PgbNZUD7j8aY7iG~hQZWxQ/_4G8gT1rhWMgVVtCI1C57A>,
Jenkins has a plugin for Veracode in its plugin repo
<https://plugins.jenkins.io/veracode-scanner> (not supported by Veracode),
the ASF supports adding plugins
<https://wiki.apache.org/general/Jenkins#How_do_I_install_a_new_Jenkins_plugin.3F>
to their Jenkins servers (although I think
<http://What_do_Administrators_do.3F> the admins are supposed to do this),
and Metron is not yet set up <https://builds.apache.org/view/M-R/> on the
ASF Jenkins server.  The ASF seems to support giving non-PMC committers
access <https://wiki.apache.org/general/Jenkins#How_do_I_get_an_account> to
Jenkins, but it requires that the PMC chair do some work, and generally it
looks like they want admins
<https://wiki.apache.org/general/Jenkins#FAQ_For_Administrators>/PMC
<https://wiki.apache.org/general/Jenkins#FAQ_For_PMCs> members to be
involved (I also don't have access to the builds JIRA project
<https://issues.apache.org/jira/projects/BUILDS>, if it really exists).

I'm happy to play around with this and see how it could be useful, but in
order to do so I need to get some additional authorization.  Does anybody
have any concerns with delegating this access to me, or with this general
approach?

Jon

On Fri, Dec 16, 2016 at 11:39 AM James Sirota  wrote:

> That would be great. I can work with them
>
> 15.12.2016, 18:38, "zeo...@gmail.com" :
> > I recently discussed this topic with Veracode regarding the metron
> project
> > and they mentioned there may be interest in providing free services,
> > however they would need to work with an official project rep. If there's
> > interest in pursuing this please let me know.
> >
> > On Thu, Jun 2, 2016, 21:17 zeo...@gmail.com  wrote:
> >
> >>  Per the other discussion it is possible that this conflicts with the
> >>  Apache stance for vulnerability disclosure/management. I'm going to
> hold
> >>  off on any additional effort until I know more.
> >>
> >>  Jon
> >>
> >>  On Tue, May 31, 2016, 16:07 James Sirota  wrote:
> >>
> >>  Jon, would it be possible for you to scan Metron from your own branch?
> >>  I'd like to know if this is useful at all. If we get value out of it
> I'll
> >>  run this down and see how we can get it hooked up.
> >>
> >>  31.05.2016, 10:08, "Nick Allen" :
> >>  > I connect Travis to my own personal fork of Metron so that the CI
> builds
> >>  > run on my own branches before I submit PRs. Thinking you could do the
> >>  same
> >>  > with this. Maybe I'm wrong.
> >>  >
> >>  > On Tue, May 31, 2016 at 1:06 PM, zeo...@gmail.com 
> >>  wrote:
> >>  >
> >>  >> To register project on Coverity Scan, you must be contributor or
> >>  maintainer
> >>  >> of the project.
> >>  >>
> >>  >> It may also be worth mentioning that there are a ton of Apache
> projects
> >>  >> already registered, including Ambari, Drill, Flume, Hadoop, HBase,
> >>  NiFi,
> >>  >> Oozie, Ranger, Sqoop, Spark, Storm, Tez, etc. See
> >>  >> https://scan.coverity.com/projects?page=2
> >>  >>
> >>  >> Jon
> >>  >>
> >>  >> On Tue, May 31, 2016 at 12:52 PM Nick Allen 
> >>  wrote:
> >>  >>
> >>  >> > You could set it up on your own fork of Metron in Github. Then you
> >>  can
> >>  >> > tell us if it is useful at all.
> >>  >> >
> >>  >> > On Sat, May 28, 2016 at 2:36 PM, zeo...@gmail.c