Re: [VOTE] Move Apache Metron to the Apache Attic and Dissolve PMC

[Justin Leet]

The result of the vote is:

7 binding +1s
1 non-binding +1

The vote passes to move Apache Metron to the Attic, and the next step is to
inform the board of the vote and add a resolution to the next board meeting.

Thanks,
Justin

On Mon, Nov 16, 2020 at 6:26 PM Ryan Merriman  wrote:

> +1
>
> > On Nov 16, 2020, at 5:19 PM, David Lyle  wrote:
> >
> > +1
> >
> >> On Mon, Nov 16, 2020 at 3:10 PM Michael Miklavcic <
> >> michael.miklav...@gmail.com> wrote:
> >>
> >> +1
> >>
> >>> On Mon, Nov 16, 2020 at 7:01 AM Justin Leet 
> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> This is a vote thread to retire Metron to the Attic, and dissolve the
> >> PMC.
> >>> This follows a discussion thread on the dev list ([DISCUSS] Retire
> Metron
> >>> to the Attic
> >>> <
> >>>
> >>
> https://lists.apache.org/thread.html/reb31f643fac20d3ad09521fd702b19922412b7a4e8e08062968268c5%40%3Cdev.metron.apache.org%3E
> >>>> ).
> >>> More details can be found in that discussion, but the most relevant
> link
> >> is
> >>> the specific process at Moving a project to the Attic
> >>> <http://attic.apache.org/process.html>.
> >>>
> >>> As noted in the process page, this is a PMC vote. As usual, feel
> >> encouraged
> >>> to contribute non-binding votes.
> >>>
> >>> The vote will run 72 hours, until Nov 19th at 9:00 am EST.
> >>>
> >>> Thank you,
> >>> Justin
> >>>
> >>
>

Re: [VOTE] Move Apache Metron to the Apache Attic and Dissolve PMC

[Justin Leet]

+1

Just to be explicit about my support.

On Mon, Nov 16, 2020 at 9:01 AM Justin Leet  wrote:

> Hi all,
>
> This is a vote thread to retire Metron to the Attic, and dissolve the PMC.
> This follows a discussion thread on the dev list ([DISCUSS] Retire Metron
> to the Attic
> <https://lists.apache.org/thread.html/reb31f643fac20d3ad09521fd702b19922412b7a4e8e08062968268c5%40%3Cdev.metron.apache.org%3E>).
> More details can be found in that discussion, but the most relevant link is
> the specific process at Moving a project to the Attic
> <http://attic.apache.org/process.html>.
>
> As noted in the process page, this is a PMC vote. As usual, feel
> encouraged to contribute non-binding votes.
>
> The vote will run 72 hours, until Nov 19th at 9:00 am EST.
>
> Thank you,
> Justin
>

[VOTE] Move Apache Metron to the Apache Attic and Dissolve PMC

[Justin Leet]

Hi all,

This is a vote thread to retire Metron to the Attic, and dissolve the PMC.
This follows a discussion thread on the dev list ([DISCUSS] Retire Metron
to the Attic
).
More details can be found in that discussion, but the most relevant link is
the specific process at Moving a project to the Attic
.

As noted in the process page, this is a PMC vote. As usual, feel encouraged
to contribute non-binding votes.

The vote will run 72 hours, until Nov 19th at 9:00 am EST.

Thank you,
Justin

[DISCUSS] Retire Metron to the Attic

[Justin Leet]

Hi all,

I want to start a discussion in the community to consider retiring Metron
to the Apache Attic and dissolve the project management committee. For
anyone unaware, an overview is available at Apache Attic
. In short, it's a way to provide a wind-down
process for projects reaching their end of life. This process, and
potential next steps, can be found here:
http://attic.apache.org/process.html.

We've seen a substantial decrease in contributions (both code and mailing
lists) over the past year or so (see:
https://github.com/apache/metron/commits/master). There's been some
interest in the project, but in my personal opinion, not enough to build or
sustain real momentum. Additionally, there's been an inability to close the
loop on paring down the project's scope and building a solid foundation for
future work (see: Development Activity has dropped to effectively 0, what
should we do?
).
Finally, the last release was over a year ago and I don't see a release
naturally building anytime soon. We haven't seen much active development or
participation for a substantial period of time, which to me implies we've
reached a natural end of life.

The most obvious practical effects would be no new releases, setting source
read-only, mailing lists closed down, automated processes ended, etc. In
addition, the PMC would also be dissolved as part of this process.
Community members may still fork all or part of Metron, but should keep in
mind that the Metron trademark remains with the ASF.

What opinions are there either in favor of or against moving the project to
the Attic? Any questions about the process (or comments/corrections anyone
would like to add)?

Thanks,
Justin

Re: Development Activity has dropped to effectively 0, what should we do?

[Justin Leet]

ld take many shapes, but in my opinion
> it
> > would be a blocker for continuing any development on Apache Metron,
> > unfortunately.
> >
> > Please do let me know if anyone disagrees or can think of an
> > alternative
> > approach that would allow the current Ambari MPack to remain viable.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Apr 16, 2020 at 4:34 PM Dima Kovalyov 
> > wrote:
> >
> > >   - Dropping Ambari.
> > >
> > > I like the progress that Apache did with Ambari in 2.7. And I don't
> > know a
> > > better installer/manager for all the services (we use other Hadoop
> > eco
> > > services besides Metron).
> > >
> > > Sometimes its buggy, agents get stuck or server needs reboot from
> > time to
> > > time, mpacks brake some functionality. But overall I feel this is
> the
> > > direction for central management and orchestration.
> > >
> > > - Dima
> > >
> > > On Wed, Apr 15, 2020, 12:45 Justin Leet 
> > wrote:
> > >
> > > > This is a bit off the top of my head, but I'd I agree with pretty
> > much
> > > all
> > > > of points on what's bringing a lot of overhead.  There's probably
> > also a
> > > > worthwhile discussion about what value we're shooting for the
> > project to
> > > > provide to people that influences what stays/goes.
> > > >
> > > > Thinking out loud a bit
> > > >
> > > >- Dropping Storm and moving to Spark drops the very hard to
> > > >tune/manage/troubleshoot Storm.
> > > >- Dropping the UIs (and making SQL the external interface)
> > pretty much
> > > >implies dropping the REST APIs and ES/Solr.  ES/Solr have been
> > a giant
> > > >source of dev heartache on the project and they exist
> primarily
> > for
> > > the
> > > >real time use case.  People can build whatever UIs or use
> > existing
> > > tools
> > > >against Parquet/Hive/whatever.
> > > >- Dropping Ambari. It's a complex beast to install because of
> > how many
> > > >components we have. Dropping the above makes our install much
> > easier
> > > and
> > > >should alleviate the need for a complex installer.
> > > >
> > > > At that point, we're basically left with
> > > >
> > > >- Some Spark for parse -> enrich -> output
> > > >- The profiler
> > > >- Stellar
> > > >- Probably some other misc stuff (sensors, bro kafka plugging,
> > etc.)
> > > >
> > > > At a glance, that seems almost an order of magnitude smaller than
> > what we
> > > > currently try to handle.
> > > >
> > > > I'm not really sure what an appropriate way to handle the
> profiler
> > is.
> > > I've
> > > > barely touched the code for it, so I anything I say is a vague
> > guess.
> > > >
> > > > On Wed, Apr 8, 2020 at 7:38 PM Yerex, Tom 
> > wrote:
> > > >
> > > > > To me Metron is big and broad in the scope of technology
> > required to
> > > get
> > > > > it running. If things were more modular that would go a long
> way
> > to
> > > > > reducing the learning curve or at least putting it into smaller
> > bites
> > > > (and
> > > > > it might encourage more people to get involved).
> > > > >
> > > > > If the UI were an add-on module in another project, it would
> > have made
> > > it
> > > > > easier for me and it could also encourage my hypothetical buddy
> > who is
> > > a
> > > > > web developer expert to get involved since he could focus on
> the
> > web-ui
> > > > > module instead of trying to tackle all the other pieces that
> are
> > > probably
> > > > > not part of his bailiwick.
> > > > >
> > > > > Stellar is very intriguing, maybe that is not unique to Metron?
> > The
> > > &g

Re: Development Activity has dropped to effectively 0, what should we do?

[Justin Leet]

This is a bit off the top of my head, but I'd I agree with pretty much all
of points on what's bringing a lot of overhead.  There's probably also a
worthwhile discussion about what value we're shooting for the project to
provide to people that influences what stays/goes.

Thinking out loud a bit

   - Dropping Storm and moving to Spark drops the very hard to
   tune/manage/troubleshoot Storm.
   - Dropping the UIs (and making SQL the external interface) pretty much
   implies dropping the REST APIs and ES/Solr.  ES/Solr have been a giant
   source of dev heartache on the project and they exist primarily for the
   real time use case.  People can build whatever UIs or use existing tools
   against Parquet/Hive/whatever.
   - Dropping Ambari. It's a complex beast to install because of how many
   components we have. Dropping the above makes our install much easier and
   should alleviate the need for a complex installer.

At that point, we're basically left with

   - Some Spark for parse -> enrich -> output
   - The profiler
   - Stellar
   - Probably some other misc stuff (sensors, bro kafka plugging, etc.)

At a glance, that seems almost an order of magnitude smaller than what we
currently try to handle.

I'm not really sure what an appropriate way to handle the profiler is. I've
barely touched the code for it, so I anything I say is a vague guess.

On Wed, Apr 8, 2020 at 7:38 PM Yerex, Tom  wrote:

> To me Metron is big and broad in the scope of technology required to get
> it running. If things were more modular that would go a long way to
> reducing the learning curve or at least putting it into smaller bites (and
> it might encourage more people to get involved).
>
> If the UI were an add-on module in another project, it would have made it
> easier for me and it could also encourage my hypothetical buddy who is a
> web developer expert to get involved since he could focus on the web-ui
> module instead of trying to tackle all the other pieces that are probably
> not part of his bailiwick.
>
> Stellar is very intriguing, maybe that is not unique to Metron? The
> architecture of Metron with respect to parsing, enriching, etc., makes a
> lot of sense to anyone I talk with. These two aspects of Metron seem like
> standout examples that make for a powerful platform to develop on.
>
> Thanks for continuing this discussion,
>
> Tom.
>
>
> On 2020-04-08 15:32:46-07:00 Casey Stella wrote:
>
> As far as I know there is no minimum bar of development activity to keep a
> project open.  I think we would all be grateful for any investment that you
> or your organization would want to make.
> It also occurs to me that your observation is absolutely spot on: we have
> a LOT of moving parts.
> I see some deficiencies here:
>
>   *   We depend on a lot of the various hadoop ecosystem projects and they
> have to work together very precisely:
>  *   This makes for a system that is hard to install.
>  *   This also makes for a system which is hard to tune/manage
>   *   We have a large surface area of coverage
>  *   We have an installer, backend system and front-end UI, which
> stretches our developers a bit thin, especially since there isn't even
> interest in those systems
>
> Perhaps a reconsideration of the scope and technologies that we use would
> be merited?  If we were to decide to, for instance:
>
>   *   Consolidate scope: focus on a viable backend/API rather than a UI
>   *   Consolidate technology: reposition ourselves on top of Spark as a
> consolidated streaming/batch system
>   *   Make SQL our external interface: write out to parquet + the Hive
> metastore and let users pin up presto tables or hive tables as they see fit
>
> This might reduce some of our surface area and make it more viable to get
> started?
> Anyway, just some thoughts.
> Casey
>
> On Wed, Apr 8, 2020 at 6:20 PM Yerex, Tom  tom.ye...@ubc.ca>> wrote:
> Hi Casey,
>
> I'm new here and new to contributing to an open source project. Thus far
> my contribution has been questions, however the steep learning curve has
> had me working to understand all the moving parts for the last 18 months
> and I see that as a big investment by my organization.
>
> What is a level that would be viable?
>
> If my organization were to contribute I don't know that it would be soon
> enough or at the volume that is recognized as viable, which is why I ask
> the question.
>
>
> On 2020-04-08 15:05:51-07:00 Casey Stella wrote:
>
> Hi all,
>
> When composing the board report today, I realized that we have effectively
> had no development in the last quarter on this project.  Please be aware
> that I say this without a shred of blame or judgement (especially so
> considering I have not contributed in a long time).  That being said, I
> would like to pose the question to the community:
>
> Do we feel that this project is viable?  If so, how are we going to spur
> new contributions?  If not, then should we begin the process to fold the
> project?
>
>
> 

Re: Centos6 and Centos7 instructions

[Justin Leet]

I believe you should just have to make a similar README to what's in the
Centos6 dir. If you go to the site-book dir and run `mvn site`, you should
be able to open the output in your browser and ensure that your changes are
there.

If I recall correctly, the intent was to eventually drop Centos6 and just
use 7.  It might be worth revisiting that and just dropping 6, moving the
README to 7 and just support 7 going forward.

As a note, the page gets deployed at release time with the latest site-book.

On Wed, Apr 15, 2020 at 1:24 PM Yerex, Tom  wrote:

> Good morning,
>
> How does one go about updating the documentation at
> https://metron.apache.org/current-book/metron-deployment/development/centos6/index.html
> ?
>
> I would like to add a similar page for Centos7, which I see is in the
> repo. Is there any reason to keep the CentOS 6 page, or should it come down?
>
> Cheers,
>
> Tom.
>

Re: [DISCUSS] Next Release - Life After 0.7.1

[Justin Leet]

I mentioned it earlier in the thread, but Casey is right about dependabot
per https://issues.apache.org/jira/browse/LEGAL-491.

On Wed, Jan 15, 2020 at 8:13 PM Casey Stella  wrote:

> I'd recommend pulling this into a separate thread and tagging the question
> with [MENTORS].  FWIW, I'm of the opinion that you should just denote in
> the commit that it was a dependabot contribution, squash like we normally
> do and not rewrite the user for attribution.  dependabot does not appear to
> have a username, so I think that's ok.
>
> On Wed, Jan 15, 2020 at 6:18 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Ok, I've tested and +1'ed https://github.com/apache/metron/pull/1552.
> > Anyone have any idea how to properly merge and attribute a PR like this?
> > Did a quick search on "apache attribution dependabot" and nothing useful
> > showed up on that pass.
> >
> > M
> >
> > On Tue, Jan 14, 2020 at 11:36 AM Otto Fowler 
> > wrote:
> >
> > > yes
> > >
> > >
> > >
> > >
> > > On January 14, 2020 at 13:05:17, Michael Miklavcic (
> > > michael.miklav...@gmail.com) wrote:
> > >
> > > We should probably also get this resolved for this release.
> > > https://issues.apache.org/jira/browse/METRON-2340. Thoughts?
> > >
> > > On Mon, Dec 16, 2019 at 2:19 PM Shane Ardell  >
> > > wrote:
> > >
> > > > Both PR #1527 <https://github.com/apache/metron/pull/1527> and #1533
> > > > <https://github.com/apache/metron/pull/1533> are now merged into
> > master.
> > > >
> > > > On Fri, Dec 13, 2019 at 3:57 PM Justin Leet 
> > > wrote:
> > > >
> > > > > I also brought up https://github.com/apache/metron/pull/1282 and
> > > > > https://github.com/apache/metron/pull/1552 if anyone has any
> > thoughts
> > > > on
> > > > > them.
> > > > >
> > > > > On Fri, Dec 13, 2019 at 11:58 AM Shane Ardell <
> > > shane.m.ard...@gmail.com>
> > >
> > > > > wrote:
> > > > >
> > > > > > Quick update from my end: I just left a +1 on
> > > > > > https://github.com/apache/metron/pull/1527 and will merge it
> into
> > > > master
> > > > > > shortly. We are actively looking into the cause of the bug I
> > > > encountered
> > > > > in
> > > > > > https://github.com/apache/metron/pull/1533. It would be nice to
> > have
> > > > > this
> > > > > > in the release, but I would not categorize it as critical like
> > #1527.
> > > > I'm
> > > > > > optimistic we will have this resolved and merged into master by
> > this
> > > > > > weekend, but I'm fine reducing scope to not include it.
> > > > > >
> > > > > > On Fri, Dec 13, 2019 at 11:24 AM Nick Allen 
> > > > wrote:
> > > > > >
> > > > > > > Are we just waiting on the following PRs as release blockers?
> Any
> > > > > > others?
> > > > > > >
> > > > > > > - https://github.com/apache/metron/pull/1533
> > > > > > > - https://github.com/apache/metron/pull/1527
> > > > > > >
> > > > > > > Being towards the end of the year, people are going to be on
> > > holiday.
> > > > > It
> > > > > > > would be great if we could focus on reducing scope and getting
> a
> > > > > release
> > > > > > > cut.
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Justin Leet <
> > justinjl...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > https://github.com/apache/metron/pull/1568 and
> > > > > > > > https://github.com/apache/metron/pull/1554 are in master
> now.
> > > > > > > >
> > > > > > > > On Fri, Dec 6, 2019 at 7:16 PM Justin Leet <
> > > justinjl...@gmail.com>
> > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I'd like to throw
> https://github.com/apache/metron/pull/1552
> > > on
> > > > > the
> > > > > > > > pile.
> &

Re: [DISCUSS] GeoLite database licensing change, master build broken…..

[Justin Leet]

To clarify a bit, on Otto's comment

> I have not read the new license, but the idea I believe is yes, whomever
> downloads and accepts the new license is then responsible for adherence to
> the applicable law,
> Which is why the Apache Foundation cannot be that entity I would think


My concern is less the legal concern about anything regarding Metron or
ASF, I 100% agree the ASF isn't responsible for users fulfilling their
legal obligations. My concern there about the practical ability of users to
even be able to do the sort of recalculation necessary to properly act on
such a requirement. The minimum is probably to call out they need to clean
it up, but that may cause a lot of friction for our users.

On Mon, Jan 13, 2020 at 5:52 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 to Apache Legal advice
>
> On Mon, Jan 13, 2020 at 3:30 PM Otto Fowler 
> wrote:
>
> > Justin,
> >
> > I have not read the new license, but the idea I believe is yes, whomever
> > downloads and accepts the new license is then responsible for adherence
> to
> > the applicable law,
> > Which is why the Apache Foundation cannot be that entity I would think.
> >
> > We may want to send this past Apache Legal?
> >
> >
> >
> > On January 13, 2020 at 17:22:21, Justin Leet (justinjl...@gmail.com)
> > wrote:
> >
> > On the whole, I agree. I think the immediate focus should be to rip out
> > maxmind from default usage in master, even if it's done a bit roughly.
> > Getting master at least building for people would probably be a good
> first
> > step.
> >
> > Couple further thoughts
> >
> > - JUnit 5 supports a variety of conditional testing conditions. It might
> > be possible to do "Test is enabled if property
> > maxmind.geo.database.location is set", and just provide instructions to
> > users on actually doing it instead of having to manually enable/disable
> the
> > test via code.
> > - Remove DB download/install from Ambari. I agree with removing the
> > actual dl/load, but still keeping the default location setup.
> > - I'm inclined to think it should be removed from the default demo
> > enrichments. We might be able to replace the GEO_GET with some dummy
> > stellar and just produce similar outputs, since our demo data has a
> limited
> > set of IPs, iirc. Obviously, we'd want to document that fairly well and
> > also let users manually set it up for non-demo data.
> > - CLI tool should be fine for letting users load, it's more or less why
> > it existed in the first place. Maybe we also add a check if someone tries
> > using the old URL and let them know.
> > - We need to document that these changes occurred for CCPA reasons, so
> > that users can evaluate the consequences.
> > - Given that I am not even slightly a lawyer, my understanding here is
> > likely wrong, but are there further implications here? Say a user runs a
> > bunch of data through Metron (both enrichments and profiler) using these
> > IPs. A subset of those IPs is removed as it shouldn't be used. Would a
> user
> > then have to remove all data derived from these IPs (e.g. anything in
> > ES/Solr, anything in HDFS, any profiles using the data?). If they do have
> > to remove it, I assume that's not directly out problem (since they have
> to
> > do cleanup), but we're probably not making it easy on them in terms of
> > providing the ability to clean up that sort of information and rerun
> > profiles and such.
> >
> >
> > On Mon, Jan 13, 2020 at 5:20 PM Otto Fowler 
> > wrote:
> >
> > > I agree with all of that, my only thinking on contrib is, if that
> > component
> > > is not tested and has no coverage, and needs manual steps, then we may
> > want
> > > to separate it.
> > > We would have to have some handle on full dev as well right?
> > >
> > > On January 13, 2020 at 16:57:13, Michael Miklavcic (
> > > michael.miklav...@gmail.com) wrote:
> > >
> > > Hey Otto,
> > >
> > > As I mentioned above, we have had this issue with other components
> > before,
> > > e.g. mysql. I don't see a compelling reason to discontinue or push this
> > > component to contrib just yet - it's a type of enrichment that happens
> to
> > > require an additional manual step. Per the article (
> > >
> > >
> >
> >
> https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
> > > ),
> > >
> > > Maxmind is not requiring purchase, merely a user to register due

Re: [DISCUSS] GeoLite database licensing change, master build broken…..

[Justin Leet]

  On the whole, I agree.  I think the immediate focus should be to rip out
maxmind from default usage in master, even if it's done a bit roughly.
Getting master at least building for people would probably be a good first
step.

Couple further thoughts

   - JUnit 5 supports a variety of conditional testing conditions. It might
   be possible to do "Test is enabled if property
   maxmind.geo.database.location is set", and just provide instructions to
   users on actually doing it instead of having to manually enable/disable the
   test via code.
   - Remove DB download/install from Ambari. I agree with removing the
   actual dl/load, but still keeping the default location setup.
   - I'm inclined to think it should be removed from the default demo
   enrichments. We might be able to replace the GEO_GET with some dummy
   stellar and just produce similar outputs, since our demo data has a limited
   set of IPs, iirc. Obviously, we'd want to document that fairly well and
   also let users manually set it up for non-demo data.
   - CLI tool should be fine for letting users load, it's more or less why
   it existed in the first place. Maybe we also add a check if someone tries
   using the old URL and let them know.
   - We need to document that these changes occurred for CCPA reasons, so
   that users can evaluate the consequences.
   - Given that I am not even slightly a lawyer, my understanding here is
   likely wrong, but are there further implications here? Say a user runs a
   bunch of data through Metron (both enrichments and profiler) using these
   IPs. A subset of those IPs is removed as it shouldn't be used. Would a user
   then have to remove all data derived from these IPs (e.g. anything in
   ES/Solr, anything in HDFS, any profiles using the data?). If they do have
   to remove it, I assume that's not directly out problem (since they have to
   do cleanup), but we're probably not making it easy on them in terms of
   providing the ability to clean up that sort of information and rerun
   profiles and such.


On Mon, Jan 13, 2020 at 5:20 PM Otto Fowler  wrote:

> I agree with all of that, my only thinking on contrib is, if that component
> is not tested and has no coverage, and needs manual steps, then we may want
> to separate it.
> We would have to have some handle on full dev as well right?
>
> On January 13, 2020 at 16:57:13, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> As I mentioned above, we have had this issue with other components before,
> e.g. mysql. I don't see a compelling reason to discontinue or push this
> component to contrib just yet - it's a type of enrichment that happens to
> require an additional manual step. Per the article (
>
> https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
> ),
>
> Maxmind is not requiring purchase, merely a user to register due to GDPR
> and CCPA requirements. That being said, it does cause issues for the
> integration tests. I think we should minimally add an @Ignore annotation to
> the testLoadGeoIpDatabase test in MaxmindDbEnrichmentLoaderTest and
> provide manual instructions for running the integration test if/when
> someone submits a PR affecting this code.
>
> Users can still use the geolite DB by uploading it to HDFS. (Incidentally,
> this even provides a crude mechanism for versioning, should a user choose
> to use the config path that way). We separated the deployment side of the
> geolite DB from the consumption side
>
> https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabase.java#L135
> .
>
> All that needs to happen is the user would drop the file in HDFS. This had
> previously happened via Ambari - see here -
>
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py#L95
> .
>
> We should probably do a couple things:
>
> 1. Minimally, remove the DB download and install from Ambari - we might
> choose to keep around the HDFS path creation so that we have a reasonable
> default prepped and ready.
> 2. Add documentation to manually install the geolite DB (register with
> maxmind) OR remove it from our default demo enrichments shipped with the
> platform.
>
> Manual DB file loading would be managed using the CLI tool. You should be
> able to load it via local file URL (
> https://en.wikipedia.org/wiki/File_URI_scheme) or via a custom hosted
> solution via the --geo_url option. See more details here -
>
> https://github.com/apache/metron/tree/master/metron-platform/metron-data-management#geolite2-loader
>
> Anything else I'm missin

Re: [DISCUSS] Next Release - Life After 0.7.1

[Justin Leet]

I also brought up https://github.com/apache/metron/pull/1282 and
https://github.com/apache/metron/pull/1552  if anyone has any thoughts on
them.

On Fri, Dec 13, 2019 at 11:58 AM Shane Ardell 
wrote:

> Quick update from my end: I just left a +1 on
> https://github.com/apache/metron/pull/1527 and will merge it into master
> shortly. We are actively looking into the cause of the bug I encountered in
> https://github.com/apache/metron/pull/1533. It would be nice to have this
> in the release, but I would not categorize it as critical like #1527. I'm
> optimistic we will have this resolved and merged into master by this
> weekend, but I'm fine reducing scope to not include it.
>
> On Fri, Dec 13, 2019 at 11:24 AM Nick Allen  wrote:
>
> > Are we just waiting on the following PRs as release blockers?  Any
> others?
> >
> >- https://github.com/apache/metron/pull/1533
> >- https://github.com/apache/metron/pull/1527
> >
> > Being towards the end of the year, people are going to be on holiday. It
> > would be great if we could focus on reducing scope and getting a release
> > cut.
> >
> >
> > On Sat, Dec 7, 2019 at 10:04 AM Justin Leet 
> wrote:
> >
> > > https://github.com/apache/metron/pull/1568 and
> > > https://github.com/apache/metron/pull/1554 are in master now.
> > >
> > > On Fri, Dec 6, 2019 at 7:16 PM Justin Leet 
> > wrote:
> > >
> > > > I'd like to throw https://github.com/apache/metron/pull/1552 on the
> > > pile.
> > > > Per https://issues.apache.org/jira/browse/LEGAL-491, we should just
> > note
> > > > the contribution comes from dependabot. Would someone more familiar
> > with
> > > > the implications of upgrading that be able to review it, or give some
> > > > advice on what we should be looking for in the review?
> > > >
> > > > On Thu, Dec 5, 2019 at 12:06 PM Shane Ardell <
> shane.m.ard...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Speaking on the UI-related PRs that Justin mentioned, I also would
> > like
> > > to
> > > >> see both of them merged before a release. At the moment, #1527 does
> > not
> > > >> address a few "stale data state" message inconsistencies that become
> > > >> apparent as a result of that PR's work (you can read more about it
> in
> > > the
> > > >> PR comments). That said, I think those inconsistencies can be
> tracked
> > > and
> > > >> addressed separately from the current PR.
> > > >>
> > > >> On Thu, Dec 5, 2019 at 11:51 AM Michael Miklavcic <
> > > >> michael.miklav...@gmail.com> wrote:
> > > >>
> > > >> > I think the junit upgrade should go in also. I'm almost finished
> > > >> reviewing
> > > >> > that.
> > > >> >
> > > >> > On Thu, Dec 5, 2019, 8:50 AM Justin Leet 
> > > wrote:
> > > >> >
> > > >> > >  If we're going to do a bug fix release, I'd like to see some of
> > the
> > > >> low
> > > >> > > hanging fix PRs get finished and merged prior to the release.
> > We've
> > > >> been
> > > >> > > lax about getting them cleaned up, so I'd like to use a release
> as
> > > an
> > > >> > > opportunity to whittle the PRs down and put out a really solid
> > > >> release.
> > > >> > >
> > > >> > > https://github.com/apache/metron/pull/1568 should be in before
> > > >> release.
> > > >> > It
> > > >> > > addresses an issue with our validation of
> > dependencies_with_url.csv
> > > >> and
> > > >> > > it's validation.
> > > >> > > Should https://github.com/apache/metron/pull/1282 be in? Seems
> > like
> > > >> that
> > > >> > > should have been merged awhile ago.
> > > >> > >
> > > >> > > There's also a couple UI performance / bug fixes PRs (e.g.
> > > >> > > https://github.com/apache/metron/pull/1533 and
> > > >> > > https://github.com/apache/metron/pull/1527) that have been
> > sitting
> > > >> > awhile.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Dec 5, 2019 at 10:32 AM Nick Allen 
&

JUnit 5 PR merged into master

[Justin Leet]

Hi all,

The JUnit 5 migration PR has been merged to master. From this point
forward, please use the newer interfaces and methods.  There are plenty of
examples through the code, and for more information, check out see
https://junit.org/junit5/.

Brief list of things to be aware of when writing JUnit 5 tests:

   - Generally, use org.junit.jupiter.api imports.
   - There are some minor interface changes, e.g. @BeforeAll
   replaces @BeforeClass
   - Failure messages are now at the end of the argument list.
   - Exception checking idiom has been improved with JUnit 5. Please see an
   example at UpdateDaoTest.java#L60
   

   .
   - Parameterized tests function differently now. Please see an example at
   ByteArrayMatchingUtilTest.java#L85
   

   .
   - No more PowerMock. Please reconsider your code design if you need to
   mock a static classes.
   - The replacement for @TemporaryFolder is still experimental, and as
   such has generally stayed. @EnableRuleMigrationSupport at the class level
   allows this to continue, as @Rule is generally no longer used.


Thanks!

Re: [DISCUSS] Next Release - Life After 0.7.1

[Justin Leet]

https://github.com/apache/metron/pull/1568 and
https://github.com/apache/metron/pull/1554 are in master now.

On Fri, Dec 6, 2019 at 7:16 PM Justin Leet  wrote:

> I'd like to throw https://github.com/apache/metron/pull/1552 on the pile.
> Per https://issues.apache.org/jira/browse/LEGAL-491, we should just note
> the contribution comes from dependabot. Would someone more familiar with
> the implications of upgrading that be able to review it, or give some
> advice on what we should be looking for in the review?
>
> On Thu, Dec 5, 2019 at 12:06 PM Shane Ardell 
> wrote:
>
>> Speaking on the UI-related PRs that Justin mentioned, I also would like to
>> see both of them merged before a release. At the moment, #1527 does not
>> address a few "stale data state" message inconsistencies that become
>> apparent as a result of that PR's work (you can read more about it in the
>> PR comments). That said, I think those inconsistencies can be tracked and
>> addressed separately from the current PR.
>>
>> On Thu, Dec 5, 2019 at 11:51 AM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>> > I think the junit upgrade should go in also. I'm almost finished
>> reviewing
>> > that.
>> >
>> > On Thu, Dec 5, 2019, 8:50 AM Justin Leet  wrote:
>> >
>> > >  If we're going to do a bug fix release, I'd like to see some of the
>> low
>> > > hanging fix PRs get finished and merged prior to the release. We've
>> been
>> > > lax about getting them cleaned up, so I'd like to use a release as an
>> > > opportunity to whittle the PRs down and put out a really solid
>> release.
>> > >
>> > > https://github.com/apache/metron/pull/1568 should be in before
>> release.
>> > It
>> > > addresses an issue with our validation of dependencies_with_url.csv
>> and
>> > > it's validation.
>> > > Should https://github.com/apache/metron/pull/1282 be in? Seems like
>> that
>> > > should have been merged awhile ago.
>> > >
>> > > There's also a couple UI performance / bug fixes PRs (e.g.
>> > > https://github.com/apache/metron/pull/1533 and
>> > > https://github.com/apache/metron/pull/1527) that have been sitting
>> > awhile.
>> > >
>> > >
>> > >
>> > > On Thu, Dec 5, 2019 at 10:32 AM Nick Allen 
>> wrote:
>> > >
>> > > > Hello Metron'ers -
>> > > >
>> > > > I would like to make the case that it is time for us to cut the next
>> > > Apache
>> > > > Metron release.
>> > > >
>> > > >- Our last release was 0.7.1 on May 15th
>> > > ><
>> > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/e2e532cbb63be757d0875718b082c069a268f57a9087510f196be09b%40%3Cdev.metron.apache.org%3E
>> > > > >.
>> > > >It has been *~6 months* since this release.
>> > > >
>> > > >
>> > > >- We have *102 changes* in master since the last release. This
>> > figure
>> > > >excludes the two feature branches currently undergoing active
>> > > > development.
>> > > >
>> > > >
>> > > >- We should cut a release *prior to merging in any other
>> significant
>> > > >changes* from either feature branch.  The two active feature
>> > branches
>> > > >include ~47 other changes at this point in time.
>> > > >
>> > > >
>> > > >- These 102 changes include some very nice *bug fixes and
>> usability
>> > > >improvements*. I would propose that we treat this as a bug-fix
>> > release
>> > > >and label it as *Metron 0.7.2*.
>> > > >
>> > > > Please let me know if you agree or disagree with this call for a
>> > release.
>> > > >
>> > > > For those interested, here are the 102 unreleased changes in master.
>> > > >
>> > > > METRON-2323 Increase unit test coverage for Alerts List (sardell)
>> > closes
>> > > > apache/metron#1567
>> > > > METRON-2208 [UI] Increase unit test coverage for Alert Details
>> > (sardell)
>> > > > closes apache/metron#1479
>> > > > METRON-2316 [UI] Drag drop sorting for the selected fields in the
>> > Alerts
>> > > UI
>> > > >

Re: Discuss: Time to update bundled SOLR support?

[Justin Leet]

Someone working more on the feature branch can correct me if I'm wrong, but
I believe that's occurring as part of the general "Upgrade HDP version"
branch, since that involves a lot of major upgrades to more supported
versions of basically all the components. Specifically, it looks like it
occurs here

.

Having said that, especially for feature branches, having more eyes on it
is generally pretty helpful if you're interested in hopping in to catch any
issues or opportunities.  I believe the people most involved are Mike
Miklavcic, Nick Allen, and Ryan Merriman, so they may have some more
input or have seen some opportunities if you're looking to contribute
something there.

On Wed, Nov 13, 2019 at 2:46 AM Dale Richardson 
wrote:

> At the moment, Metron currently ships with Solr 6.6.2 support (I think
> that matches HDP Search 3).
> Horton Works HDP Search 4 and 5 is based off Apache Solr 7.4
> Cloudera search is based off Apache Solr 7.4 as well.
> Lucidworks Solr is based on 7.x, soon to be 8.x (if not already) - they
> always seem to push the bleeding edge with SOLR in production.
>
> Does anybody have any issues if I upgrade the bundled SOLR support to Solr
> 7.4?  I'm hoping most people that use Metron/Solr in production use a
> supported distribution, and thus we should try and keep up with the
> supported Solr versions as much as possible.
>
> Regards,
> Dale.
>

[ANNOUNCE] Apache Metron-bro-plugin-kafka release 0.3.0

[Justin Leet]

Hi all,

I’m pleased to announce the release of Metron 0.3.0!  It's been a little
while coming, but there's a good number of improvements and fixes, both
around functionality and testing.  Thanks to everyone who's contributing to
and using the plugin!

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/metron-bro-plugin-kafka/0.3.0/

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/metron-bro-plugin-kafka/0.3.0/

The release branch in GitHub is
https://github.com/apache/metron-bro-plugin-kafka/tree/Metron-bro-plugin-kafka_0.3.0
(tag
apache-metron-bro-plugin-kafka_0.3.0-release)

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

CHANGES (in reverse chronological order):

METRON-2269 Cannot run Docker tests if src is not a git repo
(ottobackwards) closes apache/metron-bro-plugin-kafka#37
METRON-2069 Add btests for bro plugin topic_name selection
(ottobackwards) closes apache/metron-bro-plugin-kafka#36
METRON-2045 Pass a version argument to the bro plugin docker
scripts (JonZeolla) closes apache/metron-bro-plugin-kafka#35
METRON-2003 Bro plugin topic should fall back to the log writer's
path (JonZeolla) closes apache/metron-bro-plugin-kafka#26
METRON-2025 Bro Kafka Plugin Docker should yum clean
(ottobackwards) closes apache/metron-bro-plugin-kafka#33
METRON-2021 Add screen to bro docker image (ottobackwards) closes
apache/metron-bro-plugin-kafka#32
METRON-2013 The bro plugin docker scripts topic name should
be configurable (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#27
METRON-2020 Running run_end_to_end.sh with docker give warning if
bash  4.0 (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#31
METRON-1991 Bro plugin docker scripts should exit nonzero when bro
and kafka counts differ (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#29
METRON-2017 The Bro plugin docker data processing script
incorrectly runs bro (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#30
METRON-1990 Bro plugin docker should exit nonzero if it encounters
issues (JonZeolla) closes apache/metron-bro-plugin-kafka#28
METRON-2004 Bro plugin kafka docker_execute_shell.sh workdir
should be unspecified (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#25
METRON-2000 Fix bro plugin docker line counting for BRO_COUNT
(JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#24
METRON-1992 Support sending a log to multiple topics (JonZeolla)
closes apache/metron-bro-plugin-kafka#23
METRON-1910 bro plugin segfaults on src/KafkaWriter.cc:72
(JonZeolla) closes apache/metron-bro-plugin-kafka#20
METRON-1911 Create Docker based test environment for Bro Kafka
Plugin (ottobackwards) closes apache/metron-bro-plugin-kafka#21
METRON-1885 Remove version from bro plugin btest (JonZeolla)
closes apache/metron-bro-plugin-kafka#19
METRON-1827 Update librdkafka in metron-bro-plugin-kafka
(JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#13
METRON-1866 Improve metron-bro-plugin-kafka documentation
(JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#17
METRON-1304 Allow metron-bro-plugin-kafka to include or exclude
logs (JonZeolla via nickwallen) closes
apache/metron-bro-plugin-kafka#2
METRON-1865 Fix metron-bro-plugin-kafka tests (JonZeolla via
jonzeolla) closes apache/metron-bro-plugin-kafka#16
METRON-1828 Improve bro plugin contributing documentation
(JonZeolla) closes apache/metron-bro-plugin-kafka#14
METRON-1818 Remove config_files from bro-pkg.meta (JonZeolla)
closes apache/metron-bro-plugin-kafka#11
METRON-1800 Increment metron-bro-plugin-kafka version (JonZeolla
via jonzeolla) closes apache/metron-bro-plugin-kafka#10
METRON-1773 Bro plugin docs should refer to Apache Metron project
(nickwallen) closes apache/metron-bro-plugin-kafka#9

[RESULT][VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC3

[Justin Leet]

The vote has passed.  Including my +1, the voting was:
3 binding +1’s
no 0’s
no -1’s.

I'll start working on the remainder of the release process and will notify
the dev and users lists once completed.

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC2

[Justin Leet]

Alright, I'd make another RC either this afternoon or early tomorrow.
Thanks!

On Thu, Oct 10, 2019 at 8:15 AM zeo...@gmail.com  wrote:

> From my perspective, yes
>
> Jon Zeolla
>
> On Thu, Oct 10, 2019, 7:24 AM Justin Leet  wrote:
>
> > Are we at the point where we're comfortable spinning up another RC?
> >
> > On Thu, Oct 3, 2019 at 5:04 PM Justin Leet  wrote:
> >
> > > The vote has failed. The voting was:
> > > 2 -1’s (binding)
> > >
> > > A new RC will be created once we're satisfied the latest fix has
> resolved
> > > issues.
> > >
> > > On Tue, Oct 1, 2019, 2:47 PM zeo...@gmail.com 
> wrote:
> > >
> > >> -1 as well, validated the issue that Otto was seeing.
> > >>
> > >> I'm also testing to ensure that the fix properly addressed the issue
> and
> > >> will respond if I see any issues that would block a fast follow RC3.
> > >>
> > >> - Jon Zeolla
> > >> zeo...@gmail.com
> > >>
> > >>
> > >> On Tue, Oct 1, 2019 at 3:27 PM Otto Fowler 
> > >> wrote:
> > >>
> > >> > The fix has landed.
> > >> >
> > >> > Also, here is a quick script to try for validation
> > >> >
> > >> >
> > >> >
> > >>
> >
> https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/bro-kafka/metron-bro-kafka-rc-check
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On October 1, 2019 at 07:41:02, Otto Fowler (
> ottobackwa...@gmail.com)
> > >> > wrote:
> > >> >
> > >> > https://github.com/apache/metron-bro-plugin-kafka/pull/37
> > >> >
> > >> >
> > >> >
> > >> > On September 30, 2019 at 18:44:29, Otto Fowler (
> > ottobackwa...@gmail.com
> > >> )
> > >> > wrote:
> > >> >
> > >> > –1 binding.
> > >> >
> > >> > A new feature to our docker testing to add support for the git
> > version /
> > >> > branch of the plugin to test was added during development.
> > >> >
> > >> > This feature however makes it impossible to run the docker testing
> > suite
> > >> > against the signed src tarball for verification.
> > >> >
> > >> > I have created METRON–2269 for this issue.
> > >> >
> > >> >
> > >> >
> > >> > On September 30, 2019 at 09:56:54, Justin Leet (l...@apache.org)
> > wrote:
> > >> >
> > >> > This is a call to vote on releasing Apache Metron-bro-plugin-kafka
> > 0.3.0
> > >> > The release candidate is available at:
> > >> >
> > >> >
> > >>
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
> > >> > Full list of changes in this release:
> > >> >
> > >> >
> > >>
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/CHANGES
> > >> >
> > >> > The tag to be voted upon is:
> > >> > apache-metron-bro-plugin-kafka_0.3.0-rc2
> > >> > <
> > >> >
> > >> >
> > >>
> >
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc2
> > >> > >
> > >> >
> > >> > The source archives being voted upon can be found here:
> > >> >
> > >> >
> > >>
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/apache-metron-bro-plugin-kafka_0.3.0-rc2.tar.gz
> > >> >
> > >> > Other release files, signatures and digests can be found here:
> > >> >
> > >> >
> > >>
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
> > >> >
> > >> > The release artifacts are signed with the following key:
> > >> > *https://dist.apache.org/repos/dist/release/metron/KEYS*
> > >> > <https://dist.apache.org/repos/dist/release/metron/KEYS>
> > >> > Please vote on releasing this package as Apache
> > Metron-bro-plugin-kafka
> > >> > 0.3.0
> > >> >
> > >> > When voting, please list the actions taken to verify the release.
> > >> >
> > >> > Recommended build validation and verification instructions are
> posted
> > >> here:
> > >> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > >> >
> > >> > This is only for the plugin, so the VM validation is not required.
> > >> >
> > >> > This vote will be open for until 10am EDT on Thursday September 3
> > 2019.
> > >> >
> > >> > [ ] +1 Release this package as Apache Metron 0.3.0-RC2
> > >> >
> > >> > [ ] 0 No opinion
> > >> >
> > >> > [ ] -1 Do not release this package because...
> > >> >
> > >>
> > >
> >
>

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC2

[Justin Leet]

Are we at the point where we're comfortable spinning up another RC?

On Thu, Oct 3, 2019 at 5:04 PM Justin Leet  wrote:

> The vote has failed. The voting was:
> 2 -1’s (binding)
>
> A new RC will be created once we're satisfied the latest fix has resolved
> issues.
>
> On Tue, Oct 1, 2019, 2:47 PM zeo...@gmail.com  wrote:
>
>> -1 as well, validated the issue that Otto was seeing.
>>
>> I'm also testing to ensure that the fix properly addressed the issue and
>> will respond if I see any issues that would block a fast follow RC3.
>>
>> - Jon Zeolla
>> zeo...@gmail.com
>>
>>
>> On Tue, Oct 1, 2019 at 3:27 PM Otto Fowler 
>> wrote:
>>
>> > The fix has landed.
>> >
>> > Also, here is a quick script to try for validation
>> >
>> >
>> >
>> https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/bro-kafka/metron-bro-kafka-rc-check
>> >
>> >
>> >
>> >
>> > On October 1, 2019 at 07:41:02, Otto Fowler (ottobackwa...@gmail.com)
>> > wrote:
>> >
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/37
>> >
>> >
>> >
>> > On September 30, 2019 at 18:44:29, Otto Fowler (ottobackwa...@gmail.com
>> )
>> > wrote:
>> >
>> > –1 binding.
>> >
>> > A new feature to our docker testing to add support for the git version /
>> > branch of the plugin to test was added during development.
>> >
>> > This feature however makes it impossible to run the docker testing suite
>> > against the signed src tarball for verification.
>> >
>> > I have created METRON–2269 for this issue.
>> >
>> >
>> >
>> > On September 30, 2019 at 09:56:54, Justin Leet (l...@apache.org) wrote:
>> >
>> > This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
>> > The release candidate is available at:
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
>> > Full list of changes in this release:
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/CHANGES
>> >
>> > The tag to be voted upon is:
>> > apache-metron-bro-plugin-kafka_0.3.0-rc2
>> > <
>> >
>> >
>> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc2
>> > >
>> >
>> > The source archives being voted upon can be found here:
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/apache-metron-bro-plugin-kafka_0.3.0-rc2.tar.gz
>> >
>> > Other release files, signatures and digests can be found here:
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
>> >
>> > The release artifacts are signed with the following key:
>> > *https://dist.apache.org/repos/dist/release/metron/KEYS*
>> > <https://dist.apache.org/repos/dist/release/metron/KEYS>
>> > Please vote on releasing this package as Apache Metron-bro-plugin-kafka
>> > 0.3.0
>> >
>> > When voting, please list the actions taken to verify the release.
>> >
>> > Recommended build validation and verification instructions are posted
>> here:
>> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>> >
>> > This is only for the plugin, so the VM validation is not required.
>> >
>> > This vote will be open for until 10am EDT on Thursday September 3 2019.
>> >
>> > [ ] +1 Release this package as Apache Metron 0.3.0-RC2
>> >
>> > [ ] 0 No opinion
>> >
>> > [ ] -1 Do not release this package because...
>> >
>>
>

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC2

[Justin Leet]

The vote has failed. The voting was:
2 -1’s (binding)

A new RC will be created once we're satisfied the latest fix has resolved
issues.

On Tue, Oct 1, 2019, 2:47 PM zeo...@gmail.com  wrote:

> -1 as well, validated the issue that Otto was seeing.
>
> I'm also testing to ensure that the fix properly addressed the issue and
> will respond if I see any issues that would block a fast follow RC3.
>
> - Jon Zeolla
> zeo...@gmail.com
>
>
> On Tue, Oct 1, 2019 at 3:27 PM Otto Fowler 
> wrote:
>
> > The fix has landed.
> >
> > Also, here is a quick script to try for validation
> >
> >
> >
> https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/bro-kafka/metron-bro-kafka-rc-check
> >
> >
> >
> >
> > On October 1, 2019 at 07:41:02, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > https://github.com/apache/metron-bro-plugin-kafka/pull/37
> >
> >
> >
> > On September 30, 2019 at 18:44:29, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > –1 binding.
> >
> > A new feature to our docker testing to add support for the git version /
> > branch of the plugin to test was added during development.
> >
> > This feature however makes it impossible to run the docker testing suite
> > against the signed src tarball for verification.
> >
> > I have created METRON–2269 for this issue.
> >
> >
> >
> > On September 30, 2019 at 09:56:54, Justin Leet (l...@apache.org) wrote:
> >
> > This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
> > The release candidate is available at:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
> > Full list of changes in this release:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/CHANGES
> >
> > The tag to be voted upon is:
> > apache-metron-bro-plugin-kafka_0.3.0-rc2
> > <
> >
> >
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc2
> > >
> >
> > The source archives being voted upon can be found here:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/apache-metron-bro-plugin-kafka_0.3.0-rc2.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
> >
> > The release artifacts are signed with the following key:
> > *https://dist.apache.org/repos/dist/release/metron/KEYS*
> > <https://dist.apache.org/repos/dist/release/metron/KEYS>
> > Please vote on releasing this package as Apache Metron-bro-plugin-kafka
> > 0.3.0
> >
> > When voting, please list the actions taken to verify the release.
> >
> > Recommended build validation and verification instructions are posted
> here:
> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >
> > This is only for the plugin, so the VM validation is not required.
> >
> > This vote will be open for until 10am EDT on Thursday September 3 2019.
> >
> > [ ] +1 Release this package as Apache Metron 0.3.0-RC2
> >
> > [ ] 0 No opinion
> >
> > [ ] -1 Do not release this package because...
> >
>

[VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC2

[Justin Leet]

This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
The release candidate is available at:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/CHANGES

The tag to be voted upon is:
apache-metron-bro-plugin-kafka_0.3.0-rc2


The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/apache-metron-bro-plugin-kafka_0.3.0-rc2.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/

The release artifacts are signed with the following key:
*https://dist.apache.org/repos/dist/release/metron/KEYS*

Please vote on releasing this package as Apache Metron-bro-plugin-kafka
0.3.0

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This is only for the plugin, so the VM validation is not required.

This vote will be open for until 10am EDT on Thursday September 3 2019.

[ ] +1 Release this package as Apache Metron 0.3.0-RC2

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[Justin Leet]

Jon actually wouldn't be able to cut a release candidate immediately
anyway; he doesn't have an entry in the KEYS file.

As backgound noted in:
https://cwiki.apache.org/confluence/display/METRON/Release+Process:

"Currently, the KEYS file is only pushed with a main Metron release.  This
> means that, when there is a new release manager, they will be unable to
> release apache/metron-bro-kafka-plugin without a release of apache/metron."


Nobody's particularly cared about fixing this, given that the plugin
releases more rarely than the main repo.


I should be able to cut an RC tomorrow, but when is a bit more
questionable. The other catch is that I'm doing some travel late this week,
so my ability to actually push the release post vote (assuming success) or
cut a new RC (assuming failure) is pretty limited. I'll leave it up to the
community if everyone would rather live with the possibility that there's a
delay post vote or if we'd rather start next week.

On Sun, Sep 29, 2019 at 12:47 PM zeo...@gmail.com  wrote:

> Justin Leet was running this release previously
>
> Jon Zeolla
>
> On Sun, Sep 29, 2019, 12:07 PM Otto Fowler 
> wrote:
>
> > If you are doing the RM duties, just go a head and cut the RC.
> >
> >
> >
> >
> > On September 29, 2019 at 11:35:10, zeo...@gmail.com (zeo...@gmail.com)
> > wrote:
> >
> > Hi everyone,
> >
> > This took much longer than expected, but we recently cleaned up the last
> > blocker to getting a 0.3 release out. As a reminder, currently the latest
> > apache/metron-bro-plugin-kafka release has a bug causing the tests to
> > fail. Master no longer has this, and in fact has a fair amount of
> > improvements both in features and automated testing which went in since
> > this rc. I would like to see them all included in the new 0.3 rc (i.e.
> > align the release with master).
> >
> > We may also want to consider a fast following release that includes the
> two
> > current open PRs (after review) that were waiting on the bro/zeek 3.0
> > release (which recently happened). Because these cause breaking changes I
> > do not think they should be in the 0.3 release, but rather are good
> > candidates for a 1.0.0 release (moving from x.y to x.y.z versioning as
> > well). At that time we may also want to consider a project rename due to
> > the bro project rename to zeek, but I'll leave that for a separate thread
> > for after we get 0.3 out.
> >
> > Jon Zeolla
> >
> > On Fri, Nov 30, 2018, 6:14 PM Justin Leet  wrote:
> >
> > > The vote has failed. The voting was:
> > > 1 -1’s (binding)
> > >
> > > A new RC will be created once this issue is resolved.
> > >
> > > On Wed, Nov 28, 2018 at 10:49 AM zeo...@gmail.com 
> > > wrote:
> > >
> > > > -1
> > > >
> > > > In my testing it appears that an issue was introduced in 0.2 which is
> > > > causing a segfault on the destructor (
> > > >
> > > >
> > >
> >
> >
> https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57
> > > > ).
> > > > I've opened METRON-1910 and am testing a fix now.
> > > >
> > > > On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:
> > > >
> > > > > This is a call to vote on releasing Apache Metron-bro-plugin-kafka
> > > 0.3.0
> > > > > The release candidate is available at:
> > > > >
> > > > >
> > > >
> > >
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> > > > >
> > > > > Full list of changes in this release:
> > > > >
> > > > >
> > > >
> > >
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES
> > > > >
> > > > > The tag to be voted upon is:
> > > > > apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > > > <
> > > > >
> > > >
> > >
> >
> >
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > > > >
> > > > >
> > > > > The source archives being voted upon can be found here:
> > > > >
> > > > >
> > > >
> > >
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/apache-metron-bro-plugin-kafka_0.3.0-rc1.ta

Re: PRs done but Jiras not updated to reflect completion

[Justin Leet]

If you want a quick way to check, you can abuse one of the release utils a
bit.

E.g. this will basically do the prerelease check, and spit out a list of
tickets. Any ticket with a FIX link to the Jira can be updated.
./dev-utilities/release-utils//validate-jira-for-release -v="Next + 1"
-s="tags/apache-metron_0.7.1-release" | grep "Justin Leet"

Here's an unsorted, current list for anyone interested. Sidenote, I created
https://jira.apache.org/jira/browse/METRON-2242 if anyone wants to improve
the script's usability a bit. It's just to improve the script output so
it's easily sorted by contributor and such.

METRON-2179 In Progress   Shane
Ardell  https://issues.apache.org/jira/browse/METRON-2179
METRON-2205 In Progress   Tibor
Meller  https://issues.apache.org/jira/browse/METRON-2205
METRON-2192Done
Unassigned  https://issues.apache.org/jira/browse/METRON-2192
METRON-2191Done   Tibor
Meller  https://issues.apache.org/jira/browse/METRON-2191
METRON-2172   To Do Nick
Allen  https://issues.apache.org/jira/browse/METRON-2172
METRON-2185DoneOtto
Fowler  https://issues.apache.org/jira/browse/METRON-2185
METRON-2174Done   Shane
Ardell  https://issues.apache.org/jira/browse/METRON-2174
METRON-2161Done   Shane
Ardell  https://issues.apache.org/jira/browse/METRON-2161
METRON-2183Done   Shane
Ardell  https://issues.apache.org/jira/browse/METRON-2183
METRON-2148   To Do
Unassigned  https://issues.apache.org/jira/browse/METRON-2148
METRON-2150Done   Tibor
Meller  https://issues.apache.org/jira/browse/METRON-2150
METRON-2168   To Do Nick
Allen  https://issues.apache.org/jira/browse/METRON-2168
METRON-2061   To Do
Unassigned  https://issues.apache.org/jira/browse/METRON-2061
METRON-2164 In Progress Nick
Allen  https://issues.apache.org/jira/browse/METRON-2164
METRON-2155   To Do Nick
Allen  https://issues.apache.org/jira/browse/METRON-2155
METRON-2156   To Do
Unassigned  https://issues.apache.org/jira/browse/METRON-2156
METRON-2142   To Do
 Mohan  https://issues.apache.org/jira/browse/METRON-2142
METRON-1253   To DoTamas
Fodor  https://issues.apache.org/jira/browse/METRON-1253
METRON-2073   To Do
Unassigned  https://issues.apache.org/jira/browse/METRON-2073
METRON-2092   To DoTamas
Fodor  https://issues.apache.org/jira/browse/METRON-2092
METRON-2153   To Do
Unassigned  https://issues.apache.org/jira/browse/METRON-2153
METRON-2087Done  Ryan
Merriman  https://issues.apache.org/jira/browse/METRON-2087
METRON-2143   To Do Nick
Allen  https://issues.apache.org/jira/browse/METRON-2143
METRON-2109   To Do
Unassigned  https://issues.apache.org/jira/browse/METRON-2109
METRON-2085   To DoTamas
Fodor  https://issues.apache.org/jira/browse/METRON-2085
METRON-2058   To DoTamas
Fodor  https://issues.apache.org/jira/browse/METRON-2058
METRON-1997   To DoTamas
Fodor  https://issues.apache.org/jira/browse/METRON-1997
METRON-2089 In Progress   Tibor
Meller  https://issues.apache.org/jira/browse/METRON-2089
METRON-2106   To Do  Ryan
Merriman  https://issues.apache.org/jira/browse/METRON-2106

On Thu, Aug 29, 2019 at 12:47 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hey all,
>
> I'm seeing quite a few Jiras that have been completed but not closed out
> and marked as done. Can the committers do a quick spin through their PRs
> and Jiras and check?:
>
>1. The reporter and assignee are correct (assignee should be the
>committer or contributor that did the PR - if you mark a Jira as done,
> it
>will set the assignee to your name automatically, so be sure to override
>that if it should be someone else)
>2. Version is set to Next +1 for upcoming release, or correct version if
>included in a previous release
>3. Marked as Done
>
> The release manager does a final walk through, but this should make
> Justin's life a lot easier next time we're ready to spin up another
> release.
>
> Cheers,
> Mike
>

Re: METRON-614 Developer Implications

[Justin Leet]

The PR is now in master, so please update your PRs and branches accordingly.

Justin

On Fri, Aug 16, 2019 at 2:29 PM Justin Leet  wrote:

> Hi all,
>
> I will shortly be working on merging PR-1341
> <https://github.com/apache/metron/pull/1341> for METRON-614
> <https://jira.apache.org/jira/browse/METRON-614>.
>
> There are a couple of implications to this that developers should be aware
> of, in addition to changes documented in UPGRADING.md for users.
>
>
>- All methods that use a default Charset will now result in
>compilation errors. You MUST provide a Charset. For most uses, this will be
>StandardCharsets.UTF_8. *I strongly recommend merging master into your
>current PRs.* This will help ensure that your build functions as
>expected.
>- The two main sources of this compilation error can be set in
>IntelliJ via Preferences -> Editor -> Inspections, and setting "Standard
>Charset object can be used" and "Implicit use of the platform's default
>Charset" to error.
>- Thanks to Mike Miklavcic, our parsers now support defining a Charset
>on a per parser basis. This allows for mixed Charset sources (e.g. one
>parser may be UTF_8, one may be ISO_8859_1, etc. In addition to the
>implications in UPGRADING.md, this means developers working with parsers
>may need to be aware of this configurability to ensure things like logging
>are properly handling the raw data.
>- Our general build output (including Travis) should be substantially
>cleaner now.  I believe this was the single largest source of warning
>output.
>
> Thanks,
> Justin
>

METRON-614 Developer Implications

[Justin Leet]

Hi all,

I will shortly be working on merging PR-1341
 for METRON-614
.

There are a couple of implications to this that developers should be aware
of, in addition to changes documented in UPGRADING.md for users.


   - All methods that use a default Charset will now result in compilation
   errors. You MUST provide a Charset. For most uses, this will be
   StandardCharsets.UTF_8. *I strongly recommend merging master into your
   current PRs.* This will help ensure that your build functions as
   expected.
   - The two main sources of this compilation error can be set in IntelliJ
   via Preferences -> Editor -> Inspections, and setting "Standard Charset
   object can be used" and "Implicit use of the platform's default Charset" to
   error.
   - Thanks to Mike Miklavcic, our parsers now support defining a Charset
   on a per parser basis. This allows for mixed Charset sources (e.g. one
   parser may be UTF_8, one may be ISO_8859_1, etc. In addition to the
   implications in UPGRADING.md, this means developers working with parsers
   may need to be aware of this configurability to ensure things like logging
   are properly handling the raw data.
   - Our general build output (including Travis) should be substantially
   cleaner now.  I believe this was the single largest source of warning
   output.

Thanks,
Justin

Re: [DISCUSS] Travis CI Build Time Limits

[Justin Leet]

Yeah, it'd be a one time thing, but there's no guarantee our builds
(including master) actually run after invalidating the caches.
Unfortunately, I don't really have time in the immediate future to play
around with it a whole lot to find out if it's viable without additional
work (e.g. kick off some runs that just ignore the cache).

Keep in mind, the cache is pushed after the script phase, so if we time out
and the build is killed, the cache doesn't get pushed. So we can't might
not be able to just ignore a failed build to rebuild the cache, it'll could
broken until successful cache push. However, the cache push could
potentially occur from the other parallel builds (I'm honestly not sure),
in which case it might just be one failure. At that point, the integration
tests might pick up the cache like normal and the failure doesn't actually
matter.  It's just something that potentially needs some playing around
with.  I really don't want to end up with a broken master build on my PR,
because again, I don't have time in the immediate future to fix it.

If someone else wants to do the work of running that out in their personal
fork / Travis, I'd be super appreciative, otherwise it's gonna be a little
bit.

On Thu, May 23, 2019 at 11:34 AM Otto Fowler 
wrote:

> Yes, that is what I’m proposing, but if we want to try to break things down
> to speed them up, we can still do that ;)
>
>
> On May 23, 2019 at 11:28:51, Michael Miklavcic (
> michael.miklav...@gmail.com)
> wrote:
>
> I think we're talking about killing the cache just 1 time for the purpose
> of verifying that the https:// repo endpoints will work. It's possible I'm
> misunderstanding how we've configured our caches, but it seems like we'd
> just need to kick travis 1x after the first failed build that did all the
> downloading, and then the cache will have been populated.
>
> On Wed, May 22, 2019 at 7:21 AM Nick Allen  wrote:
>
> > > Justin Leet said
> > <https://github.com/apache/metron/pull/1417#issuecomment-494464795>
> > [1]: Looking
> > at our build times, I'm actually concerned that if we kill the caches our
> > builds won't complete. The integration tests take >45 minutes and it's
> very
> > possible redownloading everything goes over our remaining time.
> >
> > To recap, our current Travis CI build is composed of multiple jobs.
> Travis'
> > time limit of 50 minutes
> > <https://docs.travis-ci.com/user/customizing-the-build/#build-timeouts>
> > [2]
> > is per job, rather than the total build time. While our total build time
> > is on the order of 2.5 hours, it is only our integration-test job which
> is
> > coming close to that 50 minute limit.
> >
> > We could try splitting our integration tests into multiple jobs. Each of
> > these would have the opportunity to run in parallel (given whatever
> > resources Travis can allocate to us), but more importantly each one has
> its
> > own 50 minute limit. For example...
> >
> > - Parser Integration Tests:
> > - `time mvn surefire:test@integration-tests -pl
> > "metron-platform/metron-parsing/metron-parsing-storm,
> > metron-platform/metron-parsing/metron-parsers,
> > metron-platform/metron-parsing/metron-parsers-common/"`
> > - Enrichment Integration Tests
> > - `time mvn surefire:test@integration-tests -pl
> >
> >
>
> "metron-platform/metron-enrichment/metron-enrichment-common/,metron-platform/metron-enrichment/metron-enrichment-storm"`
>
> > - etc, etc
> >
> >
> > We would just need to determine how to logically split the tests up. If
> > this sounds reasonable, I've already got a start on a POC.
> >
> > ---
> > [1] https://github.com/apache/metron/pull/1417#issuecomment-494464795
> > [2]
> https://docs.travis-ci.com/user/customizing-the-build/#build-timeouts
> >
>

Re: [DISCUSS] Build RPM/DEBs in Travis?

[Justin Leet]

Yep, that's all I was doing.  I'm in favor of adding it to Travis.

On Wed, May 22, 2019 at 10:28 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I think Justin was just giving rationale, not suggesting we shouldn't make
> the improvements. I think at one point we also had some ambiguity about
> whether or how Docker was going to run in Travis, but I believe some folks
> have found a path through that? I agree with you both - let's get this
> added to Travis. Good suggestion, Nick. +1.
>
> On Wed, May 22, 2019 at 7:53 AM Nick Allen  wrote:
>
> > Yes, running up Full Dev is a manual verification that is required.  And
> as
> > a manual verification sometimes that will get missed.  And in this
> specific
> > case, tt does seem a bit silly that the addition of a parser should
> require
> > the contributor to run up Full Dev.
> >
> > That being said, anything we can do to reduce the amount of manual
> > verification that is required is a good thing.  We should be pushing
> > ourselves to an end state where no manual verification is required for
> any
> > Metron PR.  I think building the RPMs/DEBs as part of the Travis build is
> > at least a small step in the right direction.
> >
> >
> > On Wed, May 22, 2019 at 9:34 AM Justin Leet 
> wrote:
> >
> > > Theoretically, we didn't need to before there were both RPMs and DEBs
> > since
> > > running dev up (which necessitates building those) is part of the build
> > > process. Since they've been split apart, I agree we probably should be
> > > building them, because nobody is going to run both unless they
> > specifically
> > > done something they'd expect to affect both.
> > >
> > > On Wed, May 22, 2019 at 9:30 AM Nick Allen  wrote:
> > >
> > > > In light of issues like this
> > https://github.com/apache/metron/pull/1419,
> > > > has anyone looked into building our RPMs and DEBs in Travis?  This
> is a
> > > > very common and easy mistake to make and our CI builds really should
> be
> > > > able to catch this.
> > > >
> > >
> >
>

Re: [DISCUSS] Build RPM/DEBs in Travis?

[Justin Leet]

Theoretically, we didn't need to before there were both RPMs and DEBs since
running dev up (which necessitates building those) is part of the build
process. Since they've been split apart, I agree we probably should be
building them, because nobody is going to run both unless they specifically
done something they'd expect to affect both.

On Wed, May 22, 2019 at 9:30 AM Nick Allen  wrote:

> In light of issues like this https://github.com/apache/metron/pull/1419,
> has anyone looked into building our RPMs and DEBs in Travis?  This is a
> very common and easy mistake to make and our CI builds really should be
> able to catch this.
>

Re: Master build - failed due to Maven download fail

[Justin Leet]

I saw the same thing on https://github.com/apache/metron/pull/1407, but
bouncing Travis fixed it. Not sure if it's a connection issue or what, but
it seems like we should be able to cache it to avoid downloading every time.

On Mon, May 20, 2019 at 6:14 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> FYI I kicked off a rebuild for master. For some reason the wget command to
> grab Maven 3.3.9 failed.
>

[ANNOUNCE] Apache Metron release 0.7.1

[Justin Leet]

Hi all,

I’m pleased to announce the release of Metron 0.7.1! There's been a lot of
work in put in for everything including fixes, improvements, documentation,
refactoring, and discussion. Thanks to everyone who's contributed, and
thanks to our users!

There is a callout for users regarding the development environment, and the
use of parser aggregation by default. This functionality is used by default
in the environment to fit more sensors into the limited resources, but
management of this feature is not yet available in the management UI. How
to work with and disable it as needed, are detailed in Upgrading.md

and Parser
Aggregation Feature

.

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/0.7.1

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/0.7.1

The release branch in github is
https://github.com/apache/metron/tree/Metron_0.7.1 (tag
apache-metron_0.7.1-release)

The release doc book is at http://metron.apache.org/current-book/index.html
The Apache Metron web site at http://metron.apache.org/ has been updated;
please refresh your web browser cache if the new links do not immediately
appear.

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

CHANGES (in reverse chronological order):

METRON-2100 Update developer documentation for full dev management
UI parser aggregation feature gap (mmiklavc) closes apache/metron#1398
METRON-2018 Update prepare-commit to add Bro plugin tests
(ottobackwards) closes apache/metron#1348
METRON-2093 Metron RC check script is outdated (justinleet) closes
apache/metron#1394
METRON-2094 Create CentOS 7 Development Environment (nickwallen)
closes apache/metron#1395
METRON-2090 Full dev is failing with missing
org.mortbay.jetty:jetty-util:jar:6.1.26.hwx dependency (merrimanr)
closes apache/metron#1391
METRON-2091 SimpleHBaseEnrichmentWriterTest should be included in
tests (merrimanr) closes apache/metron#1392
METRON-2078 Remove Storm dependency from metron-writer (merrimanr)
closes apache/metron#1386
METRON-2065 Setting Parser Output Topic in Sensor Config is broken
(merrimanr) closes apache/metron#1377
METRON-2067 Maven pom file duplicate dependency fixes (mmiklavc)
closes apache/metron#1379
METRON-2074 Script to handle TGT renewal with Storm and Kerberos
enabled (mmiklavc) closes apache/metron#1382
METRON-2082 Update the README document steps to run Batch Profiler
(MohanDV via nickwallen) closes apache/metron#1387
METRON-2006 Reenable JacoCo code coverage (justinleet via
nickwallen) closes apache/metron#1339
METRON-2071 Add MAP_PUT and MAP_MERGE to Stellar (mmiklavc) closes
apache/metron#1385
METRON-2014 Add architectural documentation for metron-writer
(merrimanr) closes apache/metron#1381
METRON-2026 Remove Storm dependency from metron-common (merrimanr)
closes apache/metron#1351
METRON-2062 Metron Alerts: Accidentally commited 'fdescribe' in
unit tests (ruffle1986 via mmiklavc) closes apache/metron#1372
METRON-2050 Automatically populate a list of enrichments from
HBase (mmiklavc) closes apache/metron#1365
METRON-2060 Improving Alerts table config pane (tiborm via
mmiklavc) closes apache/metron#1375
METRON-2064 Metron REST API overwriting global.json values
(merrimanr) closes apache/metron#1376
METRON-2066 Documentation and logging corrections (mmiklavc)
closes apache/metron#1378
METRON-1654 findOne request after an alert patch returns with the
original state of the alert item (merrimanr) closes apache/metron#1344
METRON-2053 Refactor metron-enrichment to decouple Storm
dependencies (mmiklavc) closes apache/metron#1368
METRON-2022 Metron rest creates large number of connections to ZK
which causes subsequent connection to zk fail (merrimanr) closes
apache/metron#1367
METRON-2056 Support LDAP Bind Authentication (nickwallen) closes
apache/metron#1371
METRON-2039 Time range queries do not work with Solr (merrimanr)
closes apache/metron#1359
METRON-2052 UI Changing default query time range to 15 minutes
(tiborm via sardell) closes apache/metron#1369
METRON-2051 Improve stellar-zeppelin documentation (mmiklavc)
closes apache/metron#1366
METRON-2023 UI Covering Grok Parser Creation with Cypress tests
(tiborm via sardell) closes apache/metron#1364
METRON-2046 IS_EMPTY stellar functions fails on empty map
(anandsubbu) closes apache/metron#1363
METRON-2029 Configure Table should have filter (sardell) closes
apache/metron#1356
METRON-2032 Create summary table having list of stellar functions
in README (anandsubbu) closes 

Re: [RESULT][VOTE] Metron Release Candidate 0.7.1-RC2

[Justin Leet]

Sure, I'll call out the deployment README when the announcement goes out.

On Tue, May 14, 2019 at 7:48 AM Otto Fowler  wrote:

> I would like the announcement to state to look into the new deployment
> readme for issues with the configuration ui.
>
> I was holding my +1 for discussion on this point, and the release notes.
>
>
> On May 13, 2019 at 22:34:58, Michael Miklavcic (
> michael.miklav...@gmail.com)
> wrote:
>
> 
>
> On Mon, May 13, 2019, 8:28 PM Justin Leet  wrote:
>
> > The vote has passed. Including my +1, the voting was:
> > 3 binding +1’s
> > no 0’s
> > no -1’s.
> >
> > I'll work on finishing out the release process tomorrow and will notify
> the
> > dev and user lists.
> >
>

[RESULT][VOTE] Metron Release Candidate 0.7.1-RC2

[Justin Leet]

The vote has passed.  Including my +1, the voting was:
3 binding +1’s
no 0’s
no -1’s.

I'll work on finishing out the release process tomorrow and will notify the
dev and user lists.

Re: [VOTE] Metron Release Candidate 0.7.1-RC2

[Justin Leet]

No, to the best of my knowledge we don't. I'm in favor of making a Jira for
handling more complete release notes in some manner, though.  Might be
worth a (hopefully) quick discuss on what we want (e.g. Just notable things
or down to jiras broken into "New Feature", "Improvement, "Bug Fix", etc.),
how we'd want to implement them, how we'd want to distribute them (On the
site itself as part of the post-release activities), etc.

On Mon, May 13, 2019 at 12:17 PM Otto Fowler 
wrote:

> We don’t have release notes do we?  Is our only recourse for pointing out
> things to look out for the release announcement?
>
>
> On May 10, 2019 at 20:58:40, Michael Miklavcic (
> michael.miklav...@gmail.com)
> wrote:
>
> +1 binding
>
> Validated same as Nick.
>
> Mike
>
> On Thu, May 9, 2019 at 5:54 PM Nick Allen  wrote:
>
> > +1 binding
> >
> > I validated the release tarball, ran the full test suite and validated
> the
> > CentOS 6 development environment. Everything looks solid. Let's ship it.
> >
> > On Wed, May 8, 2019 at 6:50 PM Justin Leet 
> wrote:
> >
> > > This is a call to vote on releasing Apache Metron 0.7.1
> > >
> > > Full list of changes in this release:
> > > https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/CHANGES
> > > The tag to be voted upon is:
> > > apache-metron_0.7.1-rc2
> > >
> > > The source archives being voted upon can be found here:
> > >
> > >
> >
>
> https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/apache-metron_0.7.1-rc2.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > > https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/
> > >
> > > The release artifacts are signed with the following key:
> > > https://dist.apache.org/repos/dist/release/metron/KEYS
> > > Please vote on releasing this package as Apache Metron 0.7.1-RC2
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > Recommended build validation and verification instructions are posted
> > > here:
> > > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > >
> > > This vote will be open for until 7pm EDT on Monday May 13 2019, to
> > account
> > > for the weekend.
> > >
> > > [ ] +1 Release this package as Apache Metron 0.7.1-RC2
> > >
> > > [ ] 0 No opinion
> > >
> > > [ ] -1 Do not release this package because...
> > >
> >
>

[VOTE] Metron Release Candidate 0.7.1-RC2

[Justin Leet]

This is a call to vote on releasing Apache Metron 0.7.1

Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/CHANGES
The tag to be voted upon is:
apache-metron_0.7.1-rc2

The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/apache-metron_0.7.1-rc2.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/

The release artifacts are signed with the following key:
https://dist.apache.org/repos/dist/release/metron/KEYS
Please vote on releasing this package as Apache Metron 0.7.1-RC2

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted
here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This vote will be open for until 7pm EDT on Monday May 13 2019, to account
for the weekend.

[ ] +1 Release this package as Apache Metron 0.7.1-RC2

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Justin Leet]

Sure, I can kick it off.  I'd expect that to drop either tonight or
tomorrow morning, depending on when I can dedicate a bit of time.

Thanks to everyone who's helped (and continued to help) with working on
this!

On Wed, May 8, 2019 at 12:23 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hey everyone,
>
> METRON-2100 has been merged - https://github.com/apache/metron/pull/1398,
> and the parser aggregation UI work and feature branch is under way.
>
> Justin, can we kick off an RC2?
>
> Cheers,
> Mike
>
> On Fri, May 3, 2019 at 6:06 AM Otto Fowler 
> wrote:
>
> > Despite the name, we *have* been using it as both for quite some amount
> of
> > time.  It *is* both dev and demo, and we recommend it as such on the list
> > all the time.
> >
> > So there isn’t a decision to be made here as far as the status quo -> we
> > use full dev as both dev and demo.
> >
> >
> >
> >
> > On May 2, 2019 at 18:53:37, Michael Miklavcic (
> michael.miklav...@gmail.com
> > )
> > wrote:
> >
> > Whether or not full dev is, first and foremost, "dev" I think your
> > questions being up a good point. If not full_dev for introducing new
> users,
> > then what? If we want to provide a different env for letting people
> tinker
> > and try it out than we do for development, that's completely fine. But we
> > don't have that right now. So we can treat full_dev as multipurpose, or
> we
> > can stop directing non-devs to it, or we can add something new. I
> honestly
> > don't have any recommendations here. We've talked about docker instances
> > for replacing in-memory components, but I'm still not sure that solves
> this
> > problem, or adds more complexity. Given the current options on the table,
> > I'm inclined to go with "full_dev" serves both dev and demo purposes.
> Otto,
> > what do you think?
> >
> > On Thu, May 2, 2019, 4:32 PM Otto Fowler 
> wrote:
> >
> > > I’ve commented on the PR, and I won’t repeat it here as well, I will
> > > however ask again if we know and can list all of the usability issues
> > that
> > > surround this problem. IE. All the things that can happen or may happen
> > > for people who are not Metron developers and committers who are using
> > > full dev, because we keep recommending it.
> > >
> > >
> > >
> > > On May 2, 2019 at 17:38:30, Michael Miklavcic (
> > michael.miklav...@gmail.com
> > > )
> > > wrote:
> > >
> > > PR is up. I added the doc change to the metron-deployment README since
> > this
> > > serves as the gateway doc for all the VM instances. All of which would
> be
> > > affected by the feature gap.
> > >
> > > https://github.com/apache/metron/pull/1398
> > >
> > > On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Here's the ticket I created to track it, which also references the
> Jira
> > > > for the new UI feature.
> > > > https://issues.apache.org/jira/browse/METRON-2100
> > > >
> > > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > >> :-)
> > > >>
> > > >> I expect to have #2 out sometime today.
> > > >>
> > > >> On Thu, May 2, 2019, 12:11 PM Justin Leet 
> > > wrote:
> > > >>
> > > >>> >
> > > >>> > I personally
> > > >>> > don't like this feature gap in full dev. It seems Otto agrees,
> and
> > > >>> Casey at
> > > >>> > the very least sees it as enough of an issue to gate us from 0.8.
> > > >>> >
> > > >>>
> > > >>> +1 on all of this. I don't like it either.
> > > >>>
> > > >>>
> > > >>> > Our vote landed 2-2. We are having a discussion about what to do
> > with
> > > >>> the
> > > >>> > release. This is that discussion.
> > > >>>
> > > >>>
> > > >>> I'm going to be honest, my response was a combination of misreading
> > > what
> > > >>> you said and thinking you were proposing delaying the release more
> > > >>> seriously and feeling a bit blindsided by a perceived move from the
> > > >>> initial
> > > >&g

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Justin Leet]

>
> I personally
> don't like this feature gap in full dev. It seems Otto agrees, and Casey at
> the very least sees it as enough of an issue to gate us from 0.8.
>

+1 on all of this. I don't like it either.


> Our vote landed 2-2. We are having a discussion about what to do with the
> release. This is that discussion.


I'm going to be honest, my response was a combination of misreading what
you said and thinking you were proposing delaying the release more
seriously and feeling a bit blindsided by a perceived move from the initial
"take more time than originally anticipated" (which in my head I took as a
couple days) to "versus next week, or the week after" (where delaying
things weeks is something I personally would like not buried so far down in
the thread). Totally my bad, sorry about that.

Other than that, it sounds like we're pretty much in agreement.

Here's my current understanding of the state and consensus as of right now
(which is subject to change as more discussion happens):

   - Most of the people in the thread are in favor of #2 for 0.7.1 and #3
   for 0.8.0.
  - I don't believe I've seen an explicit response from Otto on what he
  thinks about doing this, and from a personal perspective like to see what
  his opinion is as the person who originally brought it up.
   - Mike said he's going to kick out a PR that addresses #2
   - After that undergoes the normal review process and is merged, we
   proceed normally and cut RC2.


On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I think your later point about using a project release version, from the
> example of using other projects, is a valid one.  To exactly that point, a
> community member (Otto) brought up an issue/bug they found through testing
> that they were previously unaware of and did not find documented. Which was
> argued would be confusing to someone wanting to use a published release. We
> discussed the implications of this bug/feature gap. And incidentally, it
> sounds like some people see full dev as more useful than just a dev box,
> others do not, independent of what we chose to name it. That came from our
> discussion about it.
>
> The expectation I had from my discussion with the contributors was that
> this fix for aggregation was ready. So to your point about whether it
> belonged or not, I'm inclined to say yes, had it been ready. I personally
> don't like this feature gap in full dev. It seems Otto agrees, and Casey at
> the very least sees it as enough of an issue to gate us from 0.8. New
> information about that feature has changed my mind about what to do about
> it in the short term. I think we should move forward.
>
> Our vote landed 2-2. We are having a discussion about what to do with the
> release. This is that discussion.
>
> On Thu, May 2, 2019, 10:52 AM Justin Leet  wrote:
>
> > @Mike
> > I have a different question: Why is this enough to consider delaying a
> > release in the first place for a fairly involved fix?
> >
> > There was a discuss thread, where the general agreement was that we had
> > enough value to do a release (Over a month ago. And more things have gone
> > into master since then). There's a good number of fixes, and not just
> > trivial ones either. The general consensus here seems to be that the
> > management UI issue is fairly minor for a point release (after all,
> there's
> > been multiple people who think option 2 is sufficient), but becomes
> > important if we want to release a minor version. The question I asked
> > myself about this was ""Does this issue detract enough value that a
> release
> > isn't worthwhile?" and my answer was, and still is, "No, we have enough
> > value to do a meaningful release".
> >
> > I'm fine with delaying or cancelling a release because we find issues
> that
> > are severe enough or we don't think there's enough value anymore, but to
> be
> > entirely honest, I'm absolutely shocked this issue has blown up so much.
> > However, if you want to have a discuss thread to reevaluate if it's
> > worthwhile to do a release, go for it.  The communities' calculus on the
> > "Does this issue detract enough value that a release isn't worthwhile?"
> may
> > be different than mine.
> >
> > Having said all that, to a large extent, I think you're right. It really
> > doesn't matter* that much* if we release next week or the week after or
> > whenever. But at the same time I personally get super frustrated when I
> go
> > to use a project, find a bug, it's already known and fixed, but it just
> > never puts out a released version.  Every cutoff is largely arbitrary,
> but
> 

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Justin Leet]

>
> > > >
> > > >
> > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com)
> wrote:
> > > >
> > > > In PR#1360 we introduced a new state management strategy involving a
> > new
> > > > module called Ngrx. We had a discussion thread on this a few months
> ago
> > > and
> > > > we successfully convinced you about the benefits. This is one of the
> > > > reasons why this PR is going to be still huge after cleaning up the
> > > commit
> > > > history. After you having a look at the changes and the feature
> itself,
> > > > there's likely have questions about why certain parts work as they
> do.
> > > The
> > > > thing what I'd like to point out is that, yes, it probably takes more
> > > time
> > > > to get it in.
> > > >
> > > > In order to being able to release the RC, wouldn't it be an easy and
> > > quick
> > > > fix on the backend if it sent the aggregated parsers to the client as
> > > they
> > > > were one sensor? It's just an idea, it might be wrong, but at least
> we
> > > > shouldn't have to wait until the aforementioned PR gets ready to be
> > > merged
> > > > to the master.
> > > >
> > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet 
> > > wrote:
> > > >
> > > > > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for
> > > 0.8.0.
> > > > > #3 seems like a total waste of time and effort.
> > > > >
> > > > > The wall of text version:
> > > > > I agree this isn't "just the wrong thing shown", but for completely
> > > > > different reasons.
> > > > >
> > > > > To be extremely clear about what the problem is: Our "dev"
> > environment
> > > > > (whose very name implies the audience is develops) uses a
> > > > performance-based
> > > > > advanced feature to ensure that all our of sample flows are
> regularly
> > > run
> > > > > and produce data. This feature has a bare minimal implementation to
> > be
> > > > > enabled via Ambari, which it currently is by default. This is
> because
> > > of
> > > > > the limited resources available that previously resulted in us
> > turning
> > > > off
> > > > > Yaf, and therefore testing it during regular full dev runs. Right
> now
> > > > > however, this feature is not exposed through the management UI, and
> > > > > therefore it isn't obvious what the implications are. Am I missing
> > > > anything
> > > > > here?
> > > > >
> > > > > For users actually choosing to use the parser aggregation feature
> in
> > a
> > > > > non-full-dev environment, I'd expect substantially more care to be
> > > > involved
> > > > > given the lack of easy configuration for it (after all, why would
> you
> > > > > bother running the aggregated parser alongside the regular parser?
> > This
> > > > > could be more explicitly stated, but again that feels like a doc
> > > problem.
> > > > > Right now I could essentially provide two of the same parser and
> > create
> > > > the
> > > > > same problem, so right now aggregation is only special because it
> > runs
> > > on
> > > > > dev by default). This is, in my opinion, primarily a first
> impression
> > > > > problem and likely one of many areas that could use improved
> > > > documentation.
> > > > >
> > > > > Quite frankly, I think the issue pointed out here could mostly be
> > > > resolved
> > > > > by documenting how the current aggregation is done in dev, and
> > telling
> > > > how
> > > > > to change it. Especially for a 0.x.1 release, which is primarily
> bug
> > > > > fixes. As can be inferred from my vote, I don't think this problem
> > is a
> > > > > problem that needs solving in a point release. I would support
> > > improving
> > > > > the documentation, both full-dev and for aggregation in general for
> > the
> > > > > 0.7.1 point release, while making a 0.8.0 release contingent upon
> the
> > > > > outstanding PRs to enable it in the management UI.
> > > > &g

Re: [DISCUSS] Full-dev role in PR testign

[Justin Leet]

>
> I know this has been a perennial thread for us. I can dig up the original
> discuss threads on this as well if folks think they're still pertinent, but
> I think we're pretty far removed now from that original point in time and
> should just move forward with fresh perspectives.
>
> On Wed, May 1, 2019 at 9:21 AM Justin Leet  wrote:
>
> > Re: the integration testing point above, a strategy I used recently to
> > alleviate a similar problem was to exploit JUnit 5's extensions.  I
> haven't
> > played with this at all in Metron's code, so 1) Take this with a several
> > grains of salt and 2) Feel encouraged to point out anything
> > wrong/problematic/ludicrous.  My use case was pretty tame and easy to
> > implement.
> >
> > Essentially, you can set up an extension backed by a singleton cluster
> (in
> > the case I was doing, I had two extensions, one for a Kafka cluster and
> one
> > for MiniDfs).  The backing clusters expose some methods (i.e. create
> topic,
> > get brokers, get file system, etc.), which lets the tests classes setup
> > whatever they need. Classes that need them just use an annotation and can
> > be setup under the hood to init only when tests in the current run suite
> > actually use them and spin down after all are done. This saved ~1 min on
> a
> > 4 minute build.  It ends up being a pretty clean way to use them,
> although
> > we might need to be a bit more sophisticated, since my case was less
> > complicated. The main messiness is that this necessarily invites tests to
> > interfere with each other (i.e. if multiple tests use the kafka topic
> > "test", problems will be involved). We might be able to improve on this.
> >
> > We could likely do something similar with our InMemoryComponents. Right
> now
> > these are spun up on a per-test basis, rather than the overall suite.
> This
> > involves some setup in any class that wants them, just to be able to get
> a
> > cluster.  There's also some definition of spin up order and so on, which
> is
> > already taken care of with the extensions (declaration order).  The other
> > catch in our case is that we have way more code covered by JUnit 4, which
> > doesn't have the same capabilities.  That migration doesn't necessarily
> > have to be done all at once, but it is a potential substantial pain
> point.
> >
> > Basically, I'd hope to push management of our abstraction further back
> into
> > actual JUnit so they're get more reuse across runs and it's easily
> > understood what a test needs and uses right up front.  In our case, the
> > InMemoryComponents likely map to extensions.
> >
> > If we did something like this, it potentially solves a few problems
> > * Makes it easy to build an integration test that says "Give me these
> > components".
> > * Keeps it alive across any test that needs them
> > * Only spins it up if there are tests running that do need them. Only
> spins
> > up the necessary components.
> > * Lower our build times.
> > * Interaction with components remains primarily through the same methods
> > you'd normally interact (you can build producers/consumers, or whatever).
> > * IMO, easier to explain and have people use.  I've had a couple people
> > pretty easily pick up on it.
> >
> > I can't share the code, but essentially it looks like (And I'm sure email
> > will butcher this, so I'm sorry in advance):
> > @ExtendWith({ > ...
> > @BeforeAll
> > public static void beforeAll() {
> >// Test specific setup, similar to before. But without the cluster
> > creation work.
> > }
> >
> > Everything else gets handled under the covers, with the caveat that what
> I
> > needed to do was less involved than some of our tests, so there may be
> some
> > experimenting.
> >
> > On Wed, May 1, 2019 at 10:59 AM Justin Leet 
> wrote:
> >
> > > Hi all,
> > >
> > > I wanted to start a discussion on something near and dear to all of our
> > > hearts: The role of full-dev in our testing cycle.
> > >
> > > Right now, we require all PRs to have spun up the full-dev environment
> > and
> > > make sure that things flow through properly. In some cases, this is a
> > > necessity, and in other cases, it's a fairly large burden on current
> and
> > > potential contributors.
> > >
> > > So what do we need to do to reduce our dependence on full dev and
> > increase
> > > our confidence in our CI process?
> > >
> > > My initial thoughts on this encompass 

Re: [DISCUSS] Full-dev role in PR testign

[Justin Leet]

Re: the integration testing point above, a strategy I used recently to
alleviate a similar problem was to exploit JUnit 5's extensions.  I haven't
played with this at all in Metron's code, so 1) Take this with a several
grains of salt and 2) Feel encouraged to point out anything
wrong/problematic/ludicrous.  My use case was pretty tame and easy to
implement.

Essentially, you can set up an extension backed by a singleton cluster (in
the case I was doing, I had two extensions, one for a Kafka cluster and one
for MiniDfs).  The backing clusters expose some methods (i.e. create topic,
get brokers, get file system, etc.), which lets the tests classes setup
whatever they need. Classes that need them just use an annotation and can
be setup under the hood to init only when tests in the current run suite
actually use them and spin down after all are done. This saved ~1 min on a
4 minute build.  It ends up being a pretty clean way to use them, although
we might need to be a bit more sophisticated, since my case was less
complicated. The main messiness is that this necessarily invites tests to
interfere with each other (i.e. if multiple tests use the kafka topic
"test", problems will be involved). We might be able to improve on this.

We could likely do something similar with our InMemoryComponents. Right now
these are spun up on a per-test basis, rather than the overall suite. This
involves some setup in any class that wants them, just to be able to get a
cluster.  There's also some definition of spin up order and so on, which is
already taken care of with the extensions (declaration order).  The other
catch in our case is that we have way more code covered by JUnit 4, which
doesn't have the same capabilities.  That migration doesn't necessarily
have to be done all at once, but it is a potential substantial pain point.

Basically, I'd hope to push management of our abstraction further back into
actual JUnit so they're get more reuse across runs and it's easily
understood what a test needs and uses right up front.  In our case, the
InMemoryComponents likely map to extensions.

If we did something like this, it potentially solves a few problems
* Makes it easy to build an integration test that says "Give me these
components".
* Keeps it alive across any test that needs them
* Only spins it up if there are tests running that do need them. Only spins
up the necessary components.
* Lower our build times.
* Interaction with components remains primarily through the same methods
you'd normally interact (you can build producers/consumers, or whatever).
* IMO, easier to explain and have people use.  I've had a couple people
pretty easily pick up on it.

I can't share the code, but essentially it looks like (And I'm sure email
will butcher this, so I'm sorry in advance):
@ExtendWith({ wrote:

> Hi all,
>
> I wanted to start a discussion on something near and dear to all of our
> hearts: The role of full-dev in our testing cycle.
>
> Right now, we require all PRs to have spun up the full-dev environment and
> make sure that things flow through properly. In some cases, this is a
> necessity, and in other cases, it's a fairly large burden on current and
> potential contributors.
>
> So what do we need to do to reduce our dependence on full dev and increase
> our confidence in our CI process?
>
> My initial thoughts on this encompass a few things
> * Increasing our ability to easily write automated tests. In particular, I
> think our integration testing is fairly hard and has some structural issues
> (e.g. our tests spin up components per Test class, not for the overall test
> suite being run, which also increases build times, etc.). How do we make
> this easier and have less boilerplate?  I have one potential idea I'll
> reply to this thread with.
> * Unit tests in general. I'd argue that any area we thing need full-dev
> spun up to be confident in needs more testing. Does anyone have any
> opinions on which areas we have the least confidence in?  Which areas of
> code are people afraid to touch?
> * Our general procedures. Should we not be requiring full-dev on all PRs,
> but maybe just asking for a justification why not (e.g. "I don't need
> full-dev for this, because I added unit tests here and here and integration
> tests for the these components?"). And then a reviewer can request a
> full-dev spin up if needed?  Could we stage this rollout (e.g.
> metron-platform modules require it, but others don't, etc.?) This'll add
> more pressure onto the release phase, so maybe that involves fleshing out a
> checklist of critical path items to check.
> * Do we need to improve our docs? Publish Javadocs? After all, if docs are
> better, hopefully we'd see less issues introduced and be more confident in
> changes coming in.
> * What environment do we even want testing to occur in?
> https://github.com/apache/metron/pull/1261 has seen a lot of work, and
> getting it across the finish line may help make the dev environment less
> 

[DISCUSS] Full-dev role in PR testign

[Justin Leet]

Hi all,

I wanted to start a discussion on something near and dear to all of our
hearts: The role of full-dev in our testing cycle.

Right now, we require all PRs to have spun up the full-dev environment and
make sure that things flow through properly. In some cases, this is a
necessity, and in other cases, it's a fairly large burden on current and
potential contributors.

So what do we need to do to reduce our dependence on full dev and increase
our confidence in our CI process?

My initial thoughts on this encompass a few things
* Increasing our ability to easily write automated tests. In particular, I
think our integration testing is fairly hard and has some structural issues
(e.g. our tests spin up components per Test class, not for the overall test
suite being run, which also increases build times, etc.). How do we make
this easier and have less boilerplate?  I have one potential idea I'll
reply to this thread with.
* Unit tests in general. I'd argue that any area we thing need full-dev
spun up to be confident in needs more testing. Does anyone have any
opinions on which areas we have the least confidence in?  Which areas of
code are people afraid to touch?
* Our general procedures. Should we not be requiring full-dev on all PRs,
but maybe just asking for a justification why not (e.g. "I don't need
full-dev for this, because I added unit tests here and here and integration
tests for the these components?"). And then a reviewer can request a
full-dev spin up if needed?  Could we stage this rollout (e.g.
metron-platform modules require it, but others don't, etc.?) This'll add
more pressure onto the release phase, so maybe that involves fleshing out a
checklist of critical path items to check.
* Do we need to improve our docs? Publish Javadocs? After all, if docs are
better, hopefully we'd see less issues introduced and be more confident in
changes coming in.
* What environment do we even want testing to occur in?
https://github.com/apache/metron/pull/1261 has seen a lot of work, and
getting it across the finish line may help make the dev environment less
onerous, even if still needed.

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Justin Leet]

Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for 0.8.0.
#3 seems like a total waste of time and effort.

The wall of text version:
I agree this isn't "just the wrong thing shown", but for completely
different reasons.

To be extremely clear about what the problem is: Our "dev" environment
(whose very name implies the audience is develops) uses a performance-based
advanced feature to ensure that all our of sample flows are regularly run
and produce data. This feature has a bare minimal implementation to be
enabled via Ambari, which it currently is by default.  This is because of
the limited resources available that previously resulted in us turning off
Yaf, and therefore testing it during regular full dev runs. Right now
however, this feature is not exposed through the management UI, and
therefore it isn't obvious what the implications are. Am I missing anything
here?

For users actually choosing to use the parser aggregation feature in a
non-full-dev environment, I'd expect substantially more care to be involved
given the lack of easy configuration for it (after all, why would you
bother running the aggregated parser alongside the regular parser? This
could be more explicitly stated, but again that feels like a doc problem.
Right now I could essentially provide two of the same parser and create the
same problem, so right now aggregation is only special because it runs on
dev by default).  This is, in my opinion, primarily a first impression
problem and likely one of many areas that could use improved documentation.

Quite frankly, I think the issue pointed out here could mostly be resolved
by documenting how the current aggregation is done in dev, and telling how
to change it.  Especially for a 0.x.1 release, which is primarily bug
fixes. As can be inferred from my vote, I don't think this problem is a
problem that needs solving in a point release.  I would support improving
the documentation, both full-dev and for aggregation in general for the
0.7.1 point release, while making a 0.8.0 release contingent upon the
outstanding PRs to enable it in the management UI.

There are a couple deeper issues, imo, that I care substantially more about
than this in particular
* The dev environment is being used as our intro for users, because it's
convenient for us to not maintain more environments (which has been a major
pain point in the past). Worse, the dev environment strongly implies it's
for Metron developers, rather than people looking to build on top of
Metron. We need an actual strategy for providing end users a clean
impression of Metron (this could be clarifying what the expectations of
full dev are, renaming it to something like "full-demo", something more
involved, etc.). This is something that we've needed for awhile in general,
and includes larger topics like improving our website, potentially
improving the site book, actually publishing our Javadocs somewhere so
people can develop things easier, publishing out info about Stellar
functions in a better manner, etc.
* The fact that parsers are handled in Ambari at all. It's awful and leads
to situations like this. To the best of my knowledge, once we can do
chaining and aggregation in the Management UI, we should be able to
entirely divorce these two overlapping domains.  I'd love to see parsers
ripped out of Ambari, then full-dev manages all the setup via REST. At that
point, we can easily tell everyone to just use the management UI.

On Wed, May 1, 2019 at 7:23 AM Otto Fowler  wrote:

> I think it would help if the full consequences of having the UI show the
> wrong status where listed.
>
> Someone trying metron, will, by default , see the wrong thing in the UI for
> the ONLY sensors they have that are running and doing data.
>
> What happens when they try to start them to make them work? One, two or
> all?
> What happens when he edits them or try to add transformations? One, two or
> all?
> What other things can you do with the sensors in the ui?  What happens?
>
> Are we recommending aggregation on the list and elsewhere for users?  Are
> we recommending something that is going to ensure they get into this
> situation?
>
> I think this is more than ‘just the wrong thing shown’ in the ui.
>
>
>
>
> On April 30, 2019 at 20:48:10, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> The vote for RC1 did not pass and I'd like to kickstart some discussion
> about what we should do.
>
> I started taking a look at PR#1360 and it looks like this isn't quite as
> close to being able go in as I had originally expected. I want to talk
> about options here. It seems to me that we can:
>
> 1. Wait for PR#1360 to go in, but this is likely going to take more time
> than originally anticipated
> 2. Accept the issue in full dev, but add some notes in the developer
> docs about the current feature gap and why sensors aren't showing status in
> the management UI when aggregation is enabled.
> 3. Find some other workable UI solution.
> 4. 

Re: [RESULT][VOTE] Metron Release Candidate 0.7.1-RC1

[Justin Leet]

Mike is right, this is what I get for copy pasting, then not double
checking the email before I sent it.

I'd open up a discuss thread, rather than have it on the vote result thread.

On Tue, Apr 30, 2019 at 6:59 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> btw, it's 2 "binding" -1’s (Otto, Mike)
>
> I started taking a look at that PR and it looks like this isn't quite as
> close to being able go in as I had originally expected. I want to talk
> about options here. It seems to me that we can:
>
>1. Wait for PR#1360 to go in, but this is likely going to take more time
>than originally anticipated
>2. Accept the issue in full dev, but add some notes in the developer
>docs about the current feature gap and why sensors aren't showing
> status in
>the management UI when aggregation is enabled.
>3. Find some other workable UI solution.
>4. Other option?
>
> All things considered, I'm personally leaning towards #2 in the short-term,
> but I think we should probably talk about this a bit before deciding what
> RC2 should be.
>
> Best,
> Mike
>
>
>
>
>
> On Tue, Apr 30, 2019 at 4:33 PM Justin Leet  wrote:
>
> > The vote has failed.  The voting was:
> >
> > 2 binding +1’s (Justin, Nick)
> > 2 non-binding -1’s (Otto, Mike)
> > no 0’s
> >
> > As discussed in the release thread, any further RC's are pending the
> review
> > and merge of PR#1360 <https://github.com/apache/metron/pull/1360>
> (Parser
> > aggregation UI implementation).
> >
> > Thanks,
> > Justin
> >
>

[RESULT][VOTE] Metron Release Candidate 0.7.1-RC1

[Justin Leet]

The vote has failed.  The voting was:

2 binding +1’s (Justin, Nick)
2 non-binding -1’s (Otto, Mike)
no 0’s

As discussed in the release thread, any further RC's are pending the review
and merge of PR#1360  (Parser
aggregation UI implementation).

Thanks,
Justin

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

[Justin Leet]

Mike is correct, that is because of the combination of full dev
restrictions and the lack of support in the configuration UI for parser
aggregation.  This was introduced in
https://github.com/apache/metron/pull/1207 and also was true of the last
release. Currently, parser aggregation is an advanced/manual feature whose
(bare minimum) configuration can be done via Ambari, out of convenience.

I haven't looked into it, but https://github.com/apache/metron/pull/1360 is
likely the work for this (and need additional work before merging).

I'm personally letting my binding +1 stand, although I would support either
ensuring we get that PR cleaned up and in and/or additional documentation
regarding the current limitations of this feature.


On Sat, Apr 27, 2019 at 2:38 PM Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> I can confirm that I've seen the Mgmt UI shows the sensor status correctly
> when they run as single topologies.
>
> -Anand
>
> On 4/27/19, 11:37 PM, "Michael Miklavcic" 
> wrote:
>
> I believe that is bc of parser aggregation. The UI does not support it
> currently. IIRC there was a PR to change the bro, snort, and yaf
> sensors to
> aggregated bc full dev didn't have enough resources. The upshot is
> that the
> UI still works for single sensors, but the feature for enabling
> aggregated
> sensors has not yet been completed.
>
> On Sat, Apr 27, 2019, 11:33 AM Otto Fowler 
> wrote:
>
> > -1
> >
> > Ran the script and ran full dev, all good.
> > In the configuration ui, the status of the sensors is not correct.
> It
> > does not show any running, but they are running in storm and the
> data was
> > moved correctly.
> >
> >
> > On April 26, 2019 at 09:58:02, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Curious Anand,
> > are your steps for bringing up an open stack cluster something we
> could
> > script like the AWS stuff?
> >
> >
> > On April 26, 2019 at 09:35:29, Anand Subramanian (
> > asubraman...@hortonworks.com) wrote:
> >
> > +1 (non-binding)
> >
> > * Built RPMs and mpacks.
> > * Brought up Metron stack on 12-node CentOS 7 openstack cluster.
> > * Ran sensor-stubs and validated events in the Alerts UI for the
> default
> > sensors.
> > * Management UI, Alerts UI and Swagger UI sanity check
> >
> > Regards,
> > Anand
> >
> > On 4/26/19, 5:18 AM, "Nick Allen"  wrote:
> >
> > +1 Verified release with all documented steps and ran up Full Dev.
> >
> > On Thu, Apr 25, 2019 at 6:10 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Ok cool, just finished the validation and updated the steps in the
> doc to
> > > reflect the current code base.
> > >
> > > On Thu, Apr 25, 2019 at 3:45 PM Nick Allen 
> wrote:
> > >
> > > > No voting required. Those are just docs. Whoever is willing to
> correct
> > > > and has access, should be able to. Good catch.
> > > >
> > > > On Thu, Apr 25, 2019 at 4:32 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > We're also not "incubator-metron" any longer. Do we require
> any kind
> > of
> > > > > voting or +1 on that verification page to make corrections to
> it?
> > > > >
> > > > > On Thu, Apr 25, 2019 at 2:29 PM Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > > > fyi, the steps in this doc have changed slightly per this
> naming
> > > > > > convention change as well -
> > > > > >
> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Apr 25, 2019 at 1:25 PM Justin Leet <
> justinjl...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> For everyone taking the time to validate and vote on the
> RC, there
> > > is
> > > > a
> > > > > >> caveat. The naming conventions for the two repos are now
> aligned
> > > > > >> (_, instead of being '-

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

[Justin Leet]

For everyone taking the time to validate and vote on the RC, there is a
caveat.  The naming conventions for the two repos are now aligned
(_, instead of being '-' in the main repo and '_' in
the plugin repo) along with the location of the KEYS file, I have a PR out
to update the metron-rc-check script (
https://github.com/apache/metron/pull/1394).

This accounts for both of these changes, and should allow the script to be
run normally.

On Thu, Apr 25, 2019 at 3:22 PM Justin Leet  wrote:

> This is a call to vote on releasing Apache Metron 0.7.1
>
> Full list of changes in this release:
> https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/CHANGES
> The tag to be voted upon is:
> apache-metron_0.7.1-rc1
>
> The source archives being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/apache-metron_0.7.1-rc1.tar.gz
>
> Other release files, signatures and digests can be found here:
> https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/
>
> The release artifacts are signed with the following key:
> https://dist.apache.org/repos/dist/release/metron/KEYS
> Please vote on releasing this package as Apache Metron 0.7.1-RC1
>
> When voting, please list the actions taken to verify the release.
>
> Recommended build validation and verification instructions are posted
> here:
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
> This vote will be open for until 4pm EDT on Tuesday April 30 2019, to
> account for the weekend..
>
> [ ] +1 Release this package as Apache Metron 0.7.1-RC1
>
> [ ] 0 No opinion
>
> [ ] -1 Do not release this package because...
>

[VOTE] Metron Release Candidate 0.7.1-RC1

[Justin Leet]

This is a call to vote on releasing Apache Metron 0.7.1

Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/CHANGES
The tag to be voted upon is:
apache-metron_0.7.1-rc1

The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/apache-metron_0.7.1-rc1.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/

The release artifacts are signed with the following key:
https://dist.apache.org/repos/dist/release/metron/KEYS
Please vote on releasing this package as Apache Metron 0.7.1-RC1

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted
here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This vote will be open for until 4pm EDT on Tuesday April 30 2019, to
account for the weekend..

[ ] +1 Release this package as Apache Metron 0.7.1-RC1

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [DISCUSS] Next Release

[Justin Leet]

Great, thanks Nick!

I'll work on cutting the release candidate later today. That'll probably
happen in the afternoon (EDT), and I'll let everyone know here and in the
Slack room when it's available.

On Thu, Apr 25, 2019 at 8:41 AM Nick Allen  wrote:

> Justin - I cleaned up the last bit of stragglers in JIRA.  JIRA should be
> ready for the release.
>
> $ /dev-utilities/release-utils/validate-jira-for-release --version=0.7.1
> --start=tags/apache-metron_0.7.0-release
> ...
>JIRA  STATUS FIX VERSION
>  ASSIGNEEFIX
> METRON-2091Done   0.7.1  Ryan
> Merriman
> METRON-2078Done   0.7.1  Ryan
> Merriman
> METRON-2065Done   0.7.1  Ryan
> Merriman
> METRON-2067Done   0.7.1  Michael
> Miklavcic
> METRON-2074Done   0.7.1  Michael
> Miklavcic
> METRON-2082Done   0.7.1
> Mohan
> METRON-2006    Done   0.7.1Justin
> Leet
> METRON-2071Done   0.7.1  Michael
> Miklavcic
> METRON-2014Done   0.7.1  Ryan
> Merriman
> METRON-2026Done   0.7.1  Ryan
> Merriman
> METRON-2062Done   0.7.1  Michael
> Miklavcic
> METRON-2050Done   0.7.1  Michael
> Miklavcic
> METRON-2060Done   0.7.1  Michael
> Miklavcic
> METRON-2064Done   0.7.1  Ryan
> Merriman
> METRON-2066Done   0.7.1  Michael
> Miklavcic
> METRON-1654Done   0.7.1  Ryan
> Merriman
> METRON-2053Done   0.7.1  Michael
> Miklavcic
> METRON-2022Done   0.7.1  Ryan
> Merriman
> METRON-2056Done   0.7.1 Nick
> Allen
> METRON-2023Done   0.7.1   Tibor
> Meller
> METRON-2039Done   0.7.1  Ryan
> Merriman
> METRON-2052Done   0.7.1   Tibor
> Meller
> METRON-2051Done   0.7.1  Michael
> Miklavcic
> METRON-2023Done   0.7.1   Tibor
> Meller
> METRON-2046Done   0.7.1  Anand
> Subramanian
> METRON-2029Done   0.7.1   Shane
> Ardell
> METRON-2032Done   0.7.1  Anand
> Subramanian
> METRON-2038Done   0.7.1 Nick
> Allen
> METRON-2035Done   0.7.1 Nick
> Allen
> METRON-2041Done   0.7.1  Michael
> Miklavcic
> METRON-2036Done   0.7.1  Michael
> Miklavcic
> METRON-2030Done   0.7.1  Ryan
> Merriman
> METRON-2031Done   0.7.1   Tibor
> Meller
> METRON-2012Done   0.7.1 Nick
> Allen
> METRON-1971Done   0.7.1   Shane
> Ardell
> METRON-1940Done   0.7.1
> Mohan
> METRON-2019Done   0.7.1  Ryan
> Merriman
> METRON-2016Done   0.7.1  Ryan
> Merriman
> METRON-1987Done   0.7.1   Shane
> Ardell
> METRON-1968Done   0.7.1  Ryan
> Merriman
> METRON-1778Done   0.7.1 Nick
> Allen
> METRON-1996Done   0.7.1
> Mohan
> METRON-1944Done   0.7.1    Tamas
> Fodor
> METRON-2010Done       0.7.1 Nick
> Allen
> METRON-1998Done   0.7.1  Ryan
> Merriman
> METRON-2009Done   0.7.1Justin
> Leet
> METRON-2005Done   0.7.1Justin
> Leet
> METRON-2007Done   0.7.1  Ryan
> Merriman
> METRON-1986Done   0.7.1 Nick
> Allen
> METRON-1993Done   0.7.1  Ryan
> Merriman

Re: [DISCUSS] Next Release

[Justin Leet]

Thanks Nick!

I've updated all the "Next + 1" tickets to "0.7.1", and cleaned up the
unassigned tickets (and possibly my own).  Everyone else should have emails
outlining what to update at this point.

On Tue, Apr 23, 2019 at 3:54 PM Nick Allen  wrote:

> I was able to create the "0.7.1" release version in JIRA.
>
> On Tue, Apr 23, 2019 at 1:41 PM Justin Leet  wrote:
>
> > Absolutely. It'll probably be tomorrow before that gets into full swing.
> >
> > I don't believe we have a "0.7.1" release in Jira, and I (oddly enough)
> > don't believe I have permissions to create it.  We'll need someone to get
> > that created (James, maybe?).
> >
> > Until then, I'd like to ask everyone to make sure Jira is up to date
> > (particularly with assignee, release version (or at least next+1 which is
> > easy to find), along with status). Typically, I've taken care of getting
> > Jira up to date in the last couple releases, but it would be helpful if
> > everyone took a few minutes to update their own tickets, so I'm just
> doing
> > touch-ups (and updating my own tickets).  If I can grab some time, I'll
> try
> > to get a list of tickets to update to the appropriate contributors.
> >
> > Thanks,
> > Justin
> >
> >
> >
> > On Tue, Apr 23, 2019, 09:54 Nick Allen  wrote:
> >
> > > Justin - Can you kick-off the release process?
> > >
> > > On Wed, Apr 17, 2019 at 11:51 AM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > I don't think it should hold up the release. It's *extremely* rare,
> > even
> > > > though it's come up twice. I wanted to be preemptive about bringing
> up
> > > the
> > > > intermittent failure list because we agreed as a community on this
> > being
> > > a
> > > > regular release cycle review step. I think all those voting on the
> > > release
> > > > should have a full awareness of issues seen and vote accordingly as
> to
> > > > whether we should move forward or address a problem first. Having
> > > reviewed
> > > > the test-failures list, I'm in favor of moving forward with the
> > release.
> > > >
> > > > On Wed, Apr 17, 2019 at 9:29 AM Nick Allen 
> wrote:
> > > >
> > > > > Oh, and that failure from 21 days ago was caused by a downstream
> > issue
> > > > > brought in by PR #1364, that we then later reverted [1].  So at
> least
> > > > that
> > > > > particular issue has been identified and addressed and should no
> > longer
> > > > > impacts builds.
> > > > >
> > > > > ---
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron/commit/cad679a5948ce0ee9866e61bbf75b1f5f255682c
> > > > >
> > > > > On Wed, Apr 17, 2019 at 11:25 AM Nick Allen 
> > > wrote:
> > > > >
> > > > > > Are you suggesting the release should wait on this?  Personally,
> I
> > > > don't
> > > > > > feel that we have any *persistently intermittent* test failures
> > that
> > > > > would
> > > > > > block a release.  The last build failure on master I see was from
> > 21
> > > > days
> > > > > > ago.
> > > > > >
> > > > > > Otherwise, I'd really like to kick off the next release.
> > > > > >
> > > > > > On Tue, Apr 16, 2019 at 11:25 AM Michael Miklavcic <
> > > > > > michael.miklav...@gmail.com> wrote:
> > > > > >
> > > > > >> FYI, I just saw one of our reported intermittent test failures
> pop
> > > up
> > > > > >> again
> > > > > >> today - https://issues.apache.org/jira/browse/METRON-1814
> > > > > >>
> > > > > >> On Mon, Apr 15, 2019 at 2:22 PM Michael Miklavcic <
> > > > > >> michael.miklav...@gmail.com> wrote:
> > > > > >>
> > > > > >> > I want to thread the needle on this. I just reviewed a PR from
> > > Ryan
> > > > > that
> > > > > >> > addresses this and gave it a +1.
> > > > > >> > https://github.com/apache/metron/pull/1381
> > > > > >> >
> > > > > >> > I t

Re: [DISCUSS] Next Release

[Justin Leet]

Absolutely. It'll probably be tomorrow before that gets into full swing.

I don't believe we have a "0.7.1" release in Jira, and I (oddly enough)
don't believe I have permissions to create it.  We'll need someone to get
that created (James, maybe?).

Until then, I'd like to ask everyone to make sure Jira is up to date
(particularly with assignee, release version (or at least next+1 which is
easy to find), along with status). Typically, I've taken care of getting
Jira up to date in the last couple releases, but it would be helpful if
everyone took a few minutes to update their own tickets, so I'm just doing
touch-ups (and updating my own tickets).  If I can grab some time, I'll try
to get a list of tickets to update to the appropriate contributors.

Thanks,
Justin



On Tue, Apr 23, 2019, 09:54 Nick Allen  wrote:

> Justin - Can you kick-off the release process?
>
> On Wed, Apr 17, 2019 at 11:51 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I don't think it should hold up the release. It's *extremely* rare, even
> > though it's come up twice. I wanted to be preemptive about bringing up
> the
> > intermittent failure list because we agreed as a community on this being
> a
> > regular release cycle review step. I think all those voting on the
> release
> > should have a full awareness of issues seen and vote accordingly as to
> > whether we should move forward or address a problem first. Having
> reviewed
> > the test-failures list, I'm in favor of moving forward with the release.
> >
> > On Wed, Apr 17, 2019 at 9:29 AM Nick Allen  wrote:
> >
> > > Oh, and that failure from 21 days ago was caused by a downstream issue
> > > brought in by PR #1364, that we then later reverted [1].  So at least
> > that
> > > particular issue has been identified and addressed and should no longer
> > > impacts builds.
> > >
> > > ---
> > > [1]
> > >
> > >
> >
> https://github.com/apache/metron/commit/cad679a5948ce0ee9866e61bbf75b1f5f255682c
> > >
> > > On Wed, Apr 17, 2019 at 11:25 AM Nick Allen 
> wrote:
> > >
> > > > Are you suggesting the release should wait on this?  Personally, I
> > don't
> > > > feel that we have any *persistently intermittent* test failures that
> > > would
> > > > block a release.  The last build failure on master I see was from 21
> > days
> > > > ago.
> > > >
> > > > Otherwise, I'd really like to kick off the next release.
> > > >
> > > > On Tue, Apr 16, 2019 at 11:25 AM Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > >> FYI, I just saw one of our reported intermittent test failures pop
> up
> > > >> again
> > > >> today - https://issues.apache.org/jira/browse/METRON-1814
> > > >>
> > > >> On Mon, Apr 15, 2019 at 2:22 PM Michael Miklavcic <
> > > >> michael.miklav...@gmail.com> wrote:
> > > >>
> > > >> > I want to thread the needle on this. I just reviewed a PR from
> Ryan
> > > that
> > > >> > addresses this and gave it a +1.
> > > >> > https://github.com/apache/metron/pull/1381
> > > >> >
> > > >> > I think Jon Zeolla and Justin Leet should have an opportunity to
> > chime
> > > >> in
> > > >> > as they also had commented about this request in the original PR.
> > > >> >
> > > >> > One other thing to note - we had agreed last time around to do a
> > scan
> > > >> over
> > > >> > any intermittent test failures encountered and assess whether or
> not
> > > >> they
> > > >> > were still valid. I'd like to ask the community to take a look and
> > > >> comment
> > > >> > on whether any of these has appeared for them. From what I can
> tell
> > > >> after
> > > >> > having looked them over, it looks like random Travis issues. I
> > haven't
> > > >> > experienced any of these locally in quite some time.
> > > >> >
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/METRON-2077?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-failure%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > >> >
> > > >> > Best,
> > > >> > Mike
> > > >> >
> > > >> >

Re: Problems with Dev deployment.

[Justin Leet]

It's a guess, I bet removing files that were outright deleted still exist
when constructing a new site-book. So older links to things like the
codelab stuff still exist to link to, but aren't actually linked within the
site-book.

On Wed, Apr 10, 2019 at 12:08 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Dale - to spin back around to what would best help you, for now you could
> reference docs from latest Metron master. You can either open the READMEs
> directly on github or just build them locally, ie.
> metron$ cd site-book
> metron/site-book$ mvn clean site && open target/site/index.html
>
> I'm not sure what's going on with the main product docs - I think we may
> have some old pages that didn't get cleaned up and are still being cached
> by Google, et al. For example, this shows our latest documentation
> correctly -
> https://metron.apache.org/current-book/metron-deployment/index.html.
> Actually, as long as you're seeing version 0.7.0 in the menu bar, you're
> good to go. No need to traverse the READMEs or build locally. Just follow
> the TOC there.
>
> Best,
> Mike
>
> On Wed, Apr 10, 2019 at 10:00 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > So does anyone know, have we not been updating the site-book when we do a
> > release? That documentation is wy outdated - v0.4.2??? - and should
> > either be removed or setup as that version number in the URL as other
> > projects do. e.g. Spark - https://spark.apache.org/documentation.html
> >
> > On Wed, Apr 10, 2019 at 9:34 AM zeo...@gmail.com 
> wrote:
> >
> >> Wow, I didn't realize that never got finalized/merged.  Looks like there
> >> is
> >> a failure in travis on that PR, if you get that wrapped up I would think
> >> we
> >> should take another look at that and maybe get it merged.  It has been a
> >> while but I recall I was pretty happy with it after my review cycle with
> >> you.
> >>
> >> - Jon Zeolla
> >> zeo...@gmail.com
> >>
> >>
> >> On Wed, Apr 10, 2019 at 9:17 AM Otto Fowler 
> >> wrote:
> >>
> >> > These issues are the reason
> https://github.com/apache/metron/pull/1261
> >> was
> >> > done.  It would be nice if we could get by them.
> >> >
> >> >
> >> > On April 10, 2019 at 08:13:04, Dale Richardson (
> tigerqu...@outlook.com)
> >> > wrote:
> >> >
> >> > Older pre-req versions are mentioned at:
> >> >
> >> >
> >> >
> >>
> https://metron.apache.org/current-book/metron-deployment/vagrant/codelab-platform/index.html
> >> > Metron – Developer Image for Apache Metron on Virtualbox<
> >> >
> >> >
> >>
> https://metron.apache.org/current-book/metron-deployment/vagrant/codelab-platform/index.html
> >> > >
> >> >
> >> > Developer Image for Apache Metron on Virtualbox. This image is a fully
> >> > functional Metron installation that has been pre-loaded with Ambari,
> HDP
> >> > and Metron.
> >> > metron.apache.org
> >> >
> >> >
> >> >
> >>
> https://metron.apache.org/current-book/metron-deployment/vagrant/full-dev-platform/index.html
> >> > Metron – Full Development Platform<
> >> >
> >> >
> >>
> https://metron.apache.org/current-book/metron-deployment/vagrant/full-dev-platform/index.html
> >> > >
> >> >
> >> > Full Development Platform. This project fully automates the
> provisioning
> >> > and deployment of Apache Metron and all necessary prerequisites on a
> >> > single, virtualized host running on Virtualbox.
> >> > metron.apache.org
> >> >
> >> >
> >> >
> >> >
> >>
> https://metron.apache.org/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
> >> > Metron – Quick Development Platform<
> >> >
> >> >
> >>
> https://metron.apache.org/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
> >> > >
> >> >
> >> > Quick Development Platform. This project fully automates the
> >> provisioning
> >> > and deployment of Apache Metron and all necessary prerequisites on a
> >> > single, virtualized host running on Virtualbox.
> >> > metron.apache.org
> >> >
> >> >
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
> >> >
> >> > Metron Install on Ubuntu/Debian single-node VM with Vagrant and
> Ambari -
> >> > Metron - Apache Software Foundation - cwiki.apache.org<
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
> >> >
> >> > Contributed by Umesh Kaushik < umesh.kaus...@bhujang.net >
> >> Introduction.
> >> > These instructions are for an Ubuntu host, with occasional comments
> >> about
> >> > how to do similar tasks in CentOS.
> >> > cwiki.apache.org
> >> >
> >> > Which links to
> >> > https://gist.github.com/dpalomar/96b826dac5c2e8b62cbf4c86dbd1c9df
> >> >
> >> >
> >> >
> >> > 
> >> > From: Michael Miklavcic 
> >> > Sent: Wednesday, 10 April 2019 4:15 AM
> >> > To: dev@metron.apache.org
> >> > Subject: Re: Problems with Dev deployment.
> >> >
> >> > Where are you seeing 2.0.0.2 for Ansible? Should be 2.4.0+ now.
> >> >
> >> >
> >>
> 

Re: [DISCUSS] Next Release

[Justin Leet]

I'm in favor doing a release, pending the ticket Mike pointed out (and
anything else someone comes up with).

To the best of my knowledge, I think 0.7.1 is sufficient, but if someone
comes up with something, it's not hard to pivot.

On Wed, Mar 13, 2019, 13:08 Michael Miklavcic 
wrote:

> I'd like to see this fixed for the next release.
> https://issues.apache.org/jira/browse/METRON-2036. Even though it's a
> non-prod issue, this is a core part of our infrastructure/development
> lifecycle that is currently broken and fits with our previous agreements of
> holding a release until all intermittent test failures are addressed.
>
> On Wed, Mar 13, 2019 at 11:33 AM Nick Allen  wrote:
>
> > I would like to open a discussion in regards to the next release. Our
> last
> > 0.7.0 release was on Dec 11th.
> >
> > I believe we have a significant number of bug fixes and performance
> > improvements that would make a worthy point release; 0.7.1.  Although, we
> > should review the change log and see if there are any breaking changes
> that
> > would require a bump to the minor version.
> >
> > Thoughts?
> >
> > $ git log --format=%B  tags/apache-metron_0.7.0-release..HEAD | grep
> > METRON-
> > Merge remote-tracking branch 'apache/master' into METRON-2035
> > METRON-2035 Allow User to Configure Role Names for Access Control
> > METRON-2030 SensorParserGroupControllerIntegrationTest intermittent
> errors
> > (merrimanr via mmiklavc) closes apache/metron#1352
> > METRON-2031 [UI] Turning off initial search request and polling by
> default
> > on Alerts UI (tiborm via mmiklavc) closes apache/metron#1353
> > METRON-2012 Unable to Execute Stellar Functions Against HBase in the REPL
> > (nickwallen) closes apache/metron#1345
> > METRON-1971 Short timeout value in Cypress may cause build failures
> > (sardell) closes apache/metron#1323
> > METRON-1940 Check if not and install Elastic search templates / Solr
> > collections when indexing server is restarted (MohanDV) closes
> > apache/metron#1305
> > METRON-2019 Improve Metron REST Logging (merrimanr) closes
> > apache/metron#1347
> > METRON-2016 Parser aggregate groups should be persisted and available
> > through REST (merrimanr) closes apache/metron#1346
> > METRON-1987 Upgrade Alert UI to stable Bootstrap 4 (sardell) closes
> > apache/metron#1336
> > METRON-1968 Messages are lost when a parser produces multiple messages
> and
> > batch size is greater than 1 (merrimanr) closes apache/metron#1330
> > METRON-1778 Out-of-order timestamps may delay flush in Storm Profiler
> > (nickwallen) closes apache/metron#1197
> > METRON-1996 Solr search throws NPE for group search if the group
> parameter
> > is null or empty (MohanDV via nickwallen) closes apache/metron#1333
> > METRON-1944 Unable to Delete a Comment in Alerts UI (ruffle1986 via
> > sardell) closes apache/metron#1307
> > METRON-2010 Unable to Build Metron Due to Inaccessible Repository
> > (nickwallen) closes apache/metron#1343
> > METRON-1998 Only one sensor is flushed by tick tuple (merrimanr) closes
> > apache/metron#1335
> > METRON-2009 Address Javadoc checkstyle issues in metron-common
> (justinleet)
> > closes apache/metron#1342
> > METRON-2005 Batch Writer writes 0-byte files to HDFS on rotation
> > (justinleet) closes apache/metron#1338
> > METRON-2007 Management UI not loading grok statements correctly
> (merrimanr)
> > closes apache/metron#1340
> > METRON-1986 Batch Profiler Fails to Resolve Stats Stellar Functions
> > (nickwallen) closes apache/metron#1328
> > METRON-1993 Stellar REST_GET should handle responses when content length
> is
> > less than zero (merrimanr) closes apache/metron#1331
> > METRON-1999 Adding validation against special characters to parser name
> > field (tiborm via sardell) closes apache/metron#1337
> > METRON-1985 Improve Error Handling When Cannot Connect to HBase
> > (nickwallen) closes apache/metron#1327
> > METRON-1974 Batch Profiler Should Handle Errant Profiles Better
> > (nickwallen) closes apache/metron#1326
> > METRON-1970 Add Metadata to Error Messages Generated During Parsing
> > (nickwallen) closes apache/metron#1325
> > METRON-1995 Arrow icon in date range selector moved to a wrong position
> > (ruffle1986 via sardell) closes apache/metron#1332
> > METRON-1973 Upgrade Alert UIs webpack-dev-server to 3.1.14 (tiborm
> > via sardell) closes apache/metron#1324
> > METRON-1948 Dropped messages from REGEX_SELECT parser field
> transformation
> > are not acked in Storm (merrimanr) closes apache/metron#1321
> > METRON-1969 Adding Cypress documentation to Alert UIs README.md
> > (tiborm via sardell) closes apache/metron#1322
> > METRON-1933 Improve build-utils helper scripts (JonZeolla via
> > ottobackwards) closes apache/metron#1297
> > METRON-1962 Make entering JDBC details in REST config to be optional
> > (anandsubbu) closes apache/metron#1318
> > METRON-1929 Build GET_ASN Stellar function (justinleet) closes
> > apache/metron#1299
> > METRON-1956 prepare-commit does not run 

Re: [DISCUSS] Architecture documentation

[Justin Leet]

Re: Labeling in Jira, I'm fine with having a be "Next + 1" from a release
management perspective, but I'd still consider at least taking action on
followup to be the relevant party's responsibility (implementer or whatever
the case may be).  We probably should have a more clear way to tag things
like this, but I don't believe we do right now. If there's not a harder
dependency than my memory, chances are high it gets
overlooked/missed/whatever.

On Mon, Feb 25, 2019 at 4:32 PM Otto Fowler  wrote:

> I really like the idea of architecture.md -> **/architecture.md.
>
> We overall do not have javadoc in a lot of areas, and could maybe start
> working on it as we go and think about asking for it in reviews.
> We are also missing the Parser Programmer’s Guide, how to add a parser to
> the metron system/install etc and other things.
>
>
>
> On February 25, 2019 at 15:22:47, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> I feel like the code itself is pretty well documented. I updated existing
> javadocs and added javadocs to classes that didn't have them before this
> PR. In my opinion the level of documentation for these classes has
> increased significantly.
>
> On Mon, Feb 25, 2019 at 1:52 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Tentatively agreed on further clarification of what we consider in/out of
> > scope for documentation re: document something that wasn't documented
> > before. Ryan, can you give a quick summary of what you *have*
> added/updated
> > in documentation on this PR vs what you want to leave out?
> >
> > My initial concern in punting on docs right now is that part of what made
> > this PR/task more challenging in the first place was not having
> > documentation. We risk losing context and detail again if we don't do
> this
> > immediately. Would it be reasonable to split it up as follows?:
> >
> > 1. Additional overarching documentation feels out of scope - make it a
> > follow on (see comments below).
> > 2. Adding documentation to our existing README's and java code comments
> > that describe the new/modified functionality should be in scope because
> > it's part of the unit of work. I expect that a developer should be able
> > to
> > look at the code, tests, comments, and README's and understand how this
> > code functions without having to start from scratch.
> >
> > The way we've handled follow-on work before, at least as far as feature
> > branches are concerned, was to create Jiras and link them to the
> > appropriate discussions for context. Maybe we can take that one step
> > further and do the release manager a favor by also labeling the
> > required/requested release on the Jira as a gating factor. This follows
> our
> > pattern for intermittent test failure reporting, e.g.
> >
> >
>
> https://issues.apache.org/jira/browse/METRON-1946?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-failure%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > .
> >
> > I'm also in favor of continuing to document architecture and technical
> > details as part of the code base as Ryan and Jon have suggested. I think
> we
> > should have an "architecture.md" in metron root that replaces this -
> >
> >
>
> https://github.com/apache/metron/blob/d7d4fd9afb19e2bd2e66babb7e1514a19eae07d0/README.md#navigating-the-architecture
> > and covers the broad architecture with links to the appropriate modules
> for
> > detail. Minimally, it would be nice if we had a simple diagram showing
> the
> > basic flow of data in Metron. I think we probably want an updated version
> > of this wiki entry from back in the day -
> > https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture
> >
> > Best,
> > Mike
> >
> >
> > On Mon, Feb 25, 2019 at 7:18 AM Nick Allen  wrote:
> >
> > > I don't think we should hold up this work to document something that
> > wasn't
> > > previously documented. A follow-on is sufficient.
> > >
> > > On Mon, Feb 25, 2019 at 8:50 AM Ryan Merriman 
> > wrote:
> > >
> > > > Recently I submitted a PR <
> https://github.com/apache/metron/pull/1330>
>
> > > > that
> > > > introduces a large number of changes to a critical part of our code
> > base.
> > > > Reviewers feel like it is significant enough to document at an
> > > > architectural level (and I agree). There are a couple points I would
> > > like
> > > > to clarify.
> > > >
> > > > Generally architectural documentation lives in the README of the
> > > > appropriate module. Do we want to continue documenting architecture
> > > here?
> > > > I think it makes sense because it will be versioned along with the
> > code.
> > > > Just wanted to confirm there are no objections to continuing this
> > > practice.
> > > >
> > > > A reviewer suggested we could accept the PR as is and leave the
> > > > architectural documentation as a follow on. I think this makes sense
> > > > because it can be tedious to maintain a large PR as other smaller
> > commits
> > > > are 

[DISCUSS] Charset PR merge effects

[Justin Leet]

Hey all,

I wanted to bring a bit of attention to a change and its effects before I
push to master.

PR#1341  Removes all uses of
the default charset (which is platform dependent) and moves everything to
UTF-8. This PR currently has a +1, but obviously any new input is more than
welcome.

Going forward, it will be a compiler error to use the default charset (e.g.
getBytes without StandardCharsets.UTF_8, or using APIs that don't allow an
encoding to be passed. I want to call attention to the fact that existing
PRs may break after this merge.

Post merging this PR, I would send another email to the list (and Slack and
so on), to note that existing PRs should merge in master (or at least
determine if they need to).

Does anyone have any objections or concerns? Should I be more proactive
about a heads up on PRs, or are we okay with what I'm proposing?

[ANNOUNCE] Apache Metron release 0.7.0

[Justin Leet]

Hi all,

I’m pleased to announce the release of Metron 0.7.0! There's been a lot of
work on improvements, upgrades, discussion, and more. Thanks to everyone
who's contributed, and thank you to our users.

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/0.7.0

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/0.7.0

The release branches in github is
https://github.com/apache/metron/tree/Metron_0.7.0 (tag
apache-metron_0.7.0-release)

The release doc book is at http://metron.apache.org/current-book/index.html
The Apache Metron web site at http://metron.apache.org/ has been updated;
please refresh your web browser cache if the new links do not immediately
appear.

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

CHANGES (in reverse chronological order):

METRON-1928 Bump Metron version to 0.7.0 for release. (justinleet)
closes apache/metron#1293
METRON-1931 Update dev utilities to support new repo location
(rlenferink via justinleet) closes apache/metron#1295
METRON-1922 Escaping incorrectly handled in current aesh version
(justinleet) closes apache/metron#1291
METRON-1867 Remove `/api/v1/update/replace` endpoint (nickwallen)
closes apache/metron#1284
METRON-1810 Storm Profiler Intermittent Test Failure (nickwallen)
closes apache/metron#1289
METRON-1909 Remove http filter from release utils changelog
generation (justinleet) closes apache/metron#1283
METRON-1869 Unable to Sort an Escalated Meta Alert (nickwallen)
closes apache/metron#1280
METRON-1889: Add any missing timestamp fields to unified
enrichment topology (mmiklavc via mmiklavc) closes apache/metron#1286
METRON-1913 metron-alert UI - Build broken by missing transitive
dependency (tiborm via sardell) closes apache/metron#1285
METRON-1845 Correct Test Data Load in Elasticsearch Integration
Tests (nickwallen) closes apache/metron#1247
METRON-1888 Default Topology Settings in MPack Cause Profiler to
Stall (nickwallen) closes apache/metron#1276
METRON-1887: Add logging to the ClasspathFunctionResolver
(mmiklavc via mmiklavc) closes apache/metron#1274
METRON-1873 Update Bootstrap version in Management UI (sardell)
closes apache/metron#1267
METRON-1825 Upgrade bro to 2.5.5 (JonZeolla via nickwallen) closes
apache/metron#1237
METRON-1890 Metron Vagrant should disable audio (ottobackwards)
closes apache/metron#1277
METRON-1874 Create a Parser Debugger (nickwallen) closes apache/metron#1265
METRON-1880 Use Caffeine for Profiler Caching (nickwallen) closes
apache/metron#1270
METRON-1877 Nested IF ELSE statements can cause parse errors in
Stellar (justinleet) closes apache/metron#1268
METRON-1872 Move rat plugin away from snapshot version
(justinleet) closes apache/metron#1264
METRON-1875 Expose configurable global settings in the Alerts UI
(merrimanr) closes apache/metron#1266
METRON-1834: Migrate Elasticsearch from TransportClient to new
Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
METRON-1834: Migrate Elasticsearch from TransportClient to new
Java REST API (cstella via mmiklavc)
METRON-1749 Update Angular to latest release in Management UI
(sardell via nickwallen) closes apache/metron#1217
METRON-1870 Intermittent Stellar REST test failures (merrimanr via
nickwallen) closes apache/metron#1263
METRON-1868 metron-committer-common incorrectly checking REPO_NAME
(JonZeolla via jonzeolla) closes apache/metron#1260
METRON-1740 Improve Palo Alto parser to handle CONFIG and SYSTEM
syslog messages (liuy-tnz via nickwallen) closes apache/metron#1171
METRON-1847 Create reusable script with functions from
prepare-commit (ottobackwards) closes apache/metron#1248
METRON-1850 Stellar REST function (merrimanr) closes apache/metron#1250
METRON-1858 BasicFireEyeParser check style cleanup and
optimization (ottobackwards) closes apache/metron#1255
METRON-1864 Stellar date format test fails after daylight saving
(ottobackwards) closes apache/metron#1258
METRON-1861 METRON-1861: REST fails to start when LDAP enabled and
'Active Spring profiles' config is empty (anandsubbu via justinleet)
closes apache/metron#1256
METRON-1853: Add shutdown hook to Stellar BaseFunctionResolver
(mmiklavc via mmiklavc) closes apache/metron#1251
METRON-1857 Fix Metaalert Nested Alert Field Name in Index
Template (nickwallen) closes apache/metron#1253
METRON-1855: Make unified enrichment topology the default and
deprecate split-join (mmiklavc via mmiklavc) closes apache/metron#1252
METRON-1790 Unsubscribe from every observable in the pcap panel UI
component (ruffle via nickwallen) closes apache/metron#1208
METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc)

[RESULT][VOTE] Metron Release Candidate 0.7.0-RC1

[Justin Leet]

The vote has passed.  Including my +1, the voting was:
3 binding +1’s
no 0’s
no -1’s.

Re: [VOTE] Metron Release Candidate 0.7.0-RC1

[Justin Leet]

I haven't seen another project include a KEYS file in the main release
itself, and since the KEYS file hasn't changed I just linked to the one in
dist. They either seem to include it at the root level (so
https://dist.apache.org/repos/dist/dev/metron/ for us) or they don't
include it at all (except maybe during releases and it just appeared empty
when I looked).

The management of the KEYS file came up in relation to the plugin repo and
in the original PR for the RC script, but nobody seems to really have
strong opinions. Specifically, we can trivially include it with the main
repo, but not the plugin repo. We'd need to pull the KEYS file from
somewhere with the plugin and if that's only getting updated on release of
the main repo, it causes friction if we're only releasing the plugin and
changing release managers (who need to add the key to the file).

I'm happy to revisit this with a more general solution (e.g. having a
script to publish just the KEYS file?) . Given that this causes problems
with the RC check, it seems like we need to update something one way or the
other. It might just be updating the RC check script in the short term.


On Tue, Dec 11, 2018 at 3:20 PM Nick Allen  wrote:

> Should there be a KEYS file with the release candidate at
> https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/KEYS? Or was that
> changed to https://dist.apache.org/repos/dist/release/metron/KEYS ?
>
> ```
> $ ~/Development/metron/dev-utilities/release-utils/metron-rc-check
> --version=0.7.0 --candidate=1
> Metron Version 0.7.0
> Release Candidate rc1
> Metron RC Distribution Root is
> https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1
> Working directory /Users/nallen/tmp/metron-0.7.0-rc1
> Downloading https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/KEYS
> --2018-12-11
> <https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/KEYS--2018-12-11>
> 15:18:49--
> https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/KEYS
> Resolving dist.apache.org (dist.apache.org)... 209.188.14.144
> Connecting to dist.apache.org (dist.apache.org)|209.188.14.144|:443...
> connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2018-12-11 15:18:50 ERROR 404: Not Found.
>
> [ERROR] Failed to download
> https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/KEYS
> ```
>
> On Tue, Dec 11, 2018 at 2:43 PM Justin Leet  wrote:
>
> > This is a call to vote on releasing Apache Metron 0.7.0
> >
> > Full list of changes in this release:
> > https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/CHANGES
> > The tag to be voted upon is:
> > apache-metron-0.7.0-rc1
> >
> > The source archives being voted upon can be found here:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/apache-metron-0.7.0-rc1.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> > https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/
> >
> > The release artifacts are signed with the following key:
> > https://dist.apache.org/repos/dist/release/metron/KEYS
> > Please vote on releasing this package as Apache Metron 0.7.0-RC1
> >
> > When voting, please list the actions taken to verify the release.
> >
> > Recommended build validation and verification instructions are posted
> > here:
> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >
> > This vote will be open for until 3pm EDT on Friday December 14 2018.
> >
> > [ ] +1 Release this package as Apache Metron 0.7.0-RC1
> >
> > [ ] 0 No opinion
> >
> > [ ] -1 Do not release this package because...
> >
>

[VOTE] Metron Release Candidate 0.7.0-RC1

[Justin Leet]

This is a call to vote on releasing Apache Metron 0.7.0

Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/CHANGES
The tag to be voted upon is:
apache-metron-0.7.0-rc1

The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/apache-metron-0.7.0-rc1.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/

The release artifacts are signed with the following key:
https://dist.apache.org/repos/dist/release/metron/KEYS
Please vote on releasing this package as Apache Metron 0.7.0-RC1

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted
here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This vote will be open for until 3pm EDT on Friday December 14 2018.

[ ] +1 Release this package as Apache Metron 0.7.0-RC1

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [DISCUSS] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Justin Leet]

"merge pull request" just offers a couple ways to commit directly within
GitHub (plain merge of all commits, squash down into one commit, and rebase
the PR and merge it).  I don't recall it allowing you to enforce a format
on the PRs for the commit (e.g.  ( via )
closes apache/metron#), which unfortunately makes it less than
ideal for us, since it's prone to typos. If you don't have that
requirement, it's great, I've used it before on other projects.

If someone knows if it's possible to set it up to autobuild the appropriate
commit message, I'd probably be for it.

On Mon, Dec 10, 2018 at 5:12 PM Nick Allen  wrote:

> Another thing to note here is that committers have admin access to the
> Github account.  This access is granted once you link your accounts using
> the URL that Roy sent.  It took about an hour for mine to sync.
>
> This means we can do things like close abandoned pull requests ourselves.
> This also exposes the "merge pull request" button in Github.  I am not
> quite sure how that might differ from our current 'prepare-commit' script.
> I would suggest we keep using the script until we figure that out.
>
>
>
>
> On Mon, Dec 10, 2018 at 3:29 PM Roy Lenferink 
> wrote:
>
> > The repo has just moved to gitbox. Keep in mind the (ASF) remote has
> > changed. The GitHub remote remains the same.
> > I've opened a pull request to adapt the scripts to the gitbox URLs.
> >
> > In order to fully use the GitBox functionality, it is needed to link your
> > ASF and GitHub account which can be done here:
> > https://gitbox.apache.org/setup/
> >
> > When having any questions, don't hesitate to ask!
> >
> > - Roy
> >
> > Op ma 10 dec. 2018 om 20:11 schreef Roy Lenferink  >:
> >
> > > I have created the issue with INFRA to move over from git-wip-us to
> > > GitBox.
> > > METRON-1931 is created for updating the scripts with the new GitBox
> > > location. I'll start with this once the repositories are moved.
> > >
> > > - Roy
> > >
> > > Op ma 10 dec. 2018 om 15:52 schreef Nick Allen :
> > >
> > >> +1  Thanks for the heads up.  We will do whatever we need to to help
> > with
> > >> the transition.
> > >>
> > >> On Sun, Dec 9, 2018 at 11:03 AM Otto Fowler 
> > >> wrote:
> > >>
> > >>> +1
> > >>>
> > >>> We will need jiras and PR’s for updating our scripts post move
> however.
> > >>>
> > >>
> >
>

Please update Git Apache remote URL

[Justin Leet]

If you haven't been watching this thread
,
the ASF remote for our Git instance has changed as part of a mandatory
relocation.

GitHub remote is the same and does not need any updates.

Kudos to Roy Lenferink for submitting a PR to update our scripts for us and
making everything go smoothly!

Old:
https://git-wip-us.apache.org/repos/asf/metron
New:
http://gitbox.apache.org/repos/asf/metron.git

An example command to update git is (assuming the remote is named 'apache'):
git remote set-url apache https://gitbox.apache.org/repos/asf/metron.git

Re: Authorization for Configuration

[Justin Leet]

You're right, I didn't really think through the specific configs that each
sensor has across enrichment and indexing (because even indexing has the
various batch tuning configurations). I was thinking in terms of topology,
which isn't complete. Really it should (probably) be in terms of sensor all
the way through.

Here's some thoughts on that.

   - Users have permissions based on sensor throughout (parsing,
   enrichment, and indexing).
  - Ideally, this is fine-grained enough that administrators might be
  able to restrict at each level (e.g. I don't want a user to say the batch
  size is either 1 or MAX_INT, that could cause global performance issues
  even if it's just on a single sensor).
  - Do we need to be more fine-grained in the first cut (e.g. allow
  users to do indexing level configs, but
   - Global configs also need to be able to have permissions to apply to
   them (after all, I don't want a user to change something for everyone with
   no notice). I would expect a lot of these to be more admin-level things. If
   they aren't, I question if they should actually be in the global config.
   - I'm personally fine with either holding off or doing read perms. Right
   now, there aren't read perms, and it's obviously much harder for a user to
   cause larger scale issues with read perms on configuration itself. I guess
   I could potentially see users not wanting other tenants to see their
   transforms, but that seems fairly unlikely for most use cases. Read perms
   on data an obviously be applied outside of Metron on the data stores
   themselves as necessary.


On Mon, Dec 3, 2018 at 4:15 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> > @Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
> pending this exact conversation winding down.
>
> Yup, understood.
>
> On Fri, Nov 30, 2018 at 7:36 AM Justin Leet  wrote:
>
> > To start with, I'm thinking just the configuration, in particular
> anything
> > that touches the ConfigurationsUtils.  I think for the first pass, we
> could
> > leave reading from configs out of it and focus on who can write configs
> > (since that's more disruptive to the system).
> >
> > To the best of my knowledge, this is:
> >
> >- Configuration through the Management UI
> >- Configuration through the CLI (e.g. the push script + Stellar + any
> >other scripts that have to do it).
> >
> > In terms of how fine-grained, I would essentially expect this to be at
> the
> > users+groups level and the individual sensor / topology level (e.g.
> parsers
> > can be authorized individually by users/group, indexing is on the whole).
> > Essentially, I don't propose trying to rearchitect things to try to have
> > multiple indexing topologies for complete separation, etc.  Just to take
> > the existing setup and be able to apply authorization on top of it.
> >
> > I would expect anything covered by the ZK configs to be done as part of
> > this effort (possibly in stages).  As noted, I would expect this to be a
> > feature branch rather than piecemeal replacement.
> >
> > @Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
> > pending this exact conversation winding down.
> >
> > On Tue, Nov 20, 2018 at 12:08 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Justin, It probably makes sense to provide a list of these
> configuration
> > > items as subtasks on the FB Jira so that we can crosscheck what entry
> > > points have been implemented against the test scripts. Do you think
> this
> > > will impact streaming enrichments or the profiler at all? That is to
> say,
> > > as Ali asked, just how far are you looking to take the fine grained
> auth
> > > scope for this?
> > >
> > > M
> > >
> > > On Mon, Nov 19, 2018 at 11:37 PM Ali Nazemian 
> > > wrote:
> > >
> > > > Hi Justin,
> > > >
> > > > By configuration do you mean the sensor related configurations only?
> > Are
> > > > you limiting the scope of this activity to the management-UI or also
> > > > Alert-UI as well? For example, defining different roles (pre-defined
> > > > or customizable) and the fine-grained integration with Ranger?
> > > >
> > > > Cheers,
> > > > Ali
> > > >
> > > > On Wed, Nov 14, 2018 at 1:25 AM Justin Leet 
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Right now, our various configs can be modified by anyone with
> access
> > to
> > > > the
&

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Justin Leet]

I'm inclined to do move forward with the core repo release. Having said
that, there's a few test bugs and such I'd like to see addressed, either
"won't fix" or preferably with PRs, before creating an RC (as noted earlier
in the thread).  It's probably a good opportunity to ask again if there's
anything outstanding we'd like to see in the release. Is there anything
we'd like taken care of that's relatively close to going in?

If the plugin gets fixed before we're set to move forward with a core
release (or we choose not to fix it, given the current version is
affected), I'm happy to put out a new RC.

On Mon, Dec 3, 2018 at 4:12 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 Nick
>
> On Mon, Dec 3, 2018 at 2:04 PM Nick Allen  wrote:
>
> > OK, well either way, I see no need to hold up Metron 0.6.1.
> >
> > On Mon, Dec 3, 2018 at 3:51 PM zeo...@gmail.com 
> wrote:
> >
> > > I believe that 0.2.0 is impacted by the bug.
> > >
> > > Jon
> > >
> > > On Mon, Dec 3, 2018 at 3:50 PM Nick Allen  wrote:
> > >
> > > > In light of this comment [1], I propose that we move forward with
> > another
> > > > Metron release and forgo the Metron Bro Plugin 0.3.0 release until we
> > can
> > > > resolve METRON-1910 [2].  There is no need to rush the fix as the
> > current
> > > > 0.2.0 release of the Bro Plugin is not impacted by the bug. We do
> have
> > a
> > > > good amount of other Metron functionality to release though.  I do
> not
> > > see
> > > > a need to hold-up the release.
> > > >
> > > > ---
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/pull/20#issuecomment-443481440
> > > > [2] https://github.com/apache/metron-bro-plugin-kafka/pull/20
> > > >
> > > > On Thu, Nov 29, 2018 at 10:29 AM Justin Leet 
> > > > wrote:
> > > >
> > > > > There's a few issues I would like to see at least triaged and
> > > preferably
> > > > > addressed prior to the release of the main repo. In Jira, we have a
> > > > > "test-failures" label, that has a few things attached to it. If we
> > know
> > > > of
> > > > > any other Jiras that should have this label attached, please do so
> > and
> > > > I'd
> > > > > appreciate it if you replied to the thread.  See test-failures
> > > > > <
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/METRON-1851?jql=project%20%3D%20METRON%20AND%20labels%20%3D%20test-failure
> > > > > >
> > > > > .
> > > > >
> > > > > The Jiras are:
> > > > > METRON-1810 <https://issues.apache.org/jira/browse/METRON-1810>
> > > > > METRON-1814 <https://issues.apache.org/jira/browse/METRON-1814>
> > > > > METRON-1851 <https://issues.apache.org/jira/browse/METRON-1851>
> > > > >
> > > > > On Wed, Nov 21, 2018 at 2:20 PM zeo...@gmail.com  >
> > > > wrote:
> > > > >
> > > > > > A metron-bro-plugin-kafka 0.3 release is good to go from my side.
> > > > Thanks
> > > > > > for all of the reviews Nick
> > > > > >
> > > > > > On Wed, Nov 21, 2018 at 11:16 AM Nick Allen 
> > > > wrote:
> > > > > >
> > > > > > > Ha.  Yes, that definitely counts and makes a ton of sense.
> > Thanks!
> > > > > > >
> > > > > > > On Wed, Nov 21, 2018 at 11:00 AM Justin Leet <
> > > justinjl...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Does "I forgot to pull master fresh before running the
> command"
> > > > count
> > > > > > as
> > > > > > > a
> > > > > > > > reason?
> > > > > > > >
> > > > > > > > The missing Jiras are:
> > > > > > > >
> > > > > > > > METRON-1890 Metron Vagrant should disable audio
> > > (ottobackwards)
> > > > > > > closes
> > > > > > > > apache/metron#1277
> > > > > > > > METRON-1874 Create a Parser Debugger (nickwallen) closes
> > > > > > > > apach

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[Justin Leet]

The vote has failed. The voting was:
1 -1’s (binding)

A new RC will be created once this issue is resolved.

On Wed, Nov 28, 2018 at 10:49 AM zeo...@gmail.com  wrote:

> -1
>
> In my testing it appears that an issue was introduced in 0.2 which is
> causing a segfault on the destructor (
>
> https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57
> ).
> I've opened METRON-1910 and am testing a fix now.
>
> On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:
>
> > This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
> > The release candidate is available at:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> >
> > Full list of changes in this release:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES
> >
> > The tag to be voted upon is:
> > apache-metron-bro-plugin-kafka_0.3.0-rc1
> > <
> >
> https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc1
> > >
> >
> > The source archives being voted upon can be found here:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/apache-metron-bro-plugin-kafka_0.3.0-rc1.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> >
> https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> >
> > The release artifacts are signed with the following key:
> > *https://dist.apache.org/repos/dist/release/metron/KEYS*
> > <https://dist.apache.org/repos/dist/release/metron/KEYS>
> > Please vote on releasing this package as Apache Metron-bro-plugin-kafka
> > 0.3.0
> >
> > When voting, please list the actions taken to verify the release.
> >
> > Recommended build validation and verification instructions are posted
> here:
> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >
> > This is only for the plugin, so the VM validation is not required.
> >
> > This vote will be open for until 3pm EDT on Friday November 27 2018.
> >
> > [ ] +1 Release this package as Apache Metron 0.3.0-RC1
> >
> > [ ] 0 No opinion
> >
> > [ ] -1 Do not release this package because...
> >
> --
>
> Jon Zeolla
>

Re: Authorization for Configuration

[Justin Leet]

To start with, I'm thinking just the configuration, in particular anything
that touches the ConfigurationsUtils.  I think for the first pass, we could
leave reading from configs out of it and focus on who can write configs
(since that's more disruptive to the system).

To the best of my knowledge, this is:

   - Configuration through the Management UI
   - Configuration through the CLI (e.g. the push script + Stellar + any
   other scripts that have to do it).

In terms of how fine-grained, I would essentially expect this to be at the
users+groups level and the individual sensor / topology level (e.g. parsers
can be authorized individually by users/group, indexing is on the whole).
Essentially, I don't propose trying to rearchitect things to try to have
multiple indexing topologies for complete separation, etc.  Just to take
the existing setup and be able to apply authorization on top of it.

I would expect anything covered by the ZK configs to be done as part of
this effort (possibly in stages).  As noted, I would expect this to be a
feature branch rather than piecemeal replacement.

@Mike Yeah, I agree. The Jira for that doesn't exist yet, pretty much
pending this exact conversation winding down.

On Tue, Nov 20, 2018 at 12:08 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Justin, It probably makes sense to provide a list of these configuration
> items as subtasks on the FB Jira so that we can crosscheck what entry
> points have been implemented against the test scripts. Do you think this
> will impact streaming enrichments or the profiler at all? That is to say,
> as Ali asked, just how far are you looking to take the fine grained auth
> scope for this?
>
> M
>
> On Mon, Nov 19, 2018 at 11:37 PM Ali Nazemian 
> wrote:
>
> > Hi Justin,
> >
> > By configuration do you mean the sensor related configurations only? Are
> > you limiting the scope of this activity to the management-UI or also
> > Alert-UI as well? For example, defining different roles (pre-defined
> > or customizable) and the fine-grained integration with Ranger?
> >
> > Cheers,
> > Ali
> >
> > On Wed, Nov 14, 2018 at 1:25 AM Justin Leet 
> wrote:
> >
> > > Hi all,
> > >
> > > Right now, our various configs can be modified by anyone with access to
> > the
> > > various scripts. I'd like to start a discussion around building out
> some
> > > authorization to be able to add some more fine grained controls around
> > > this.
> > >
> > > Other projects have some variants on how to accomplish this.
> Typically,
> > > this follows a pattern of calling out to a interface/class that takes
> in
> > > the operation and context (user and other info) and returns true/false
> if
> > > something is authorized.
> > >
> > > In my mind, what we would need out of this is
> > >
> > >- Ability to apply fine grained permissions
> > >- The various scripts and UI should flow this authorization
> framework.
> > >I believe most (if not all) of our configuration flows through
> > >ConfigurationsUtils.  Anything that doesn't should either be hooked
> in
> > > or
> > >refactored to flow through the same codepaths.
> > >- Pluggability. We shouldn't force only one authorizer.
> > >
> > > In particular, I'm proposing we use Apache Ranger
> > > <https://ranger.apache.org/> as a supported authorization framework,
> > > implementing it alongside the authorization framework to validate what
> we
> > > build. In my mind, the main catch with Ranger is that, based on my
> > > understanding, we won't be able to restrict users with direct access to
> > > ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into
> ZK's
> > > ACLs).  I believe this is a reasonable restriction, especially as the
> > > management UI gets improved to handle more of the configuration burden
> > and
> > > the number of users with access to ZK CLI begins to decrease.  Users
> can
> > > still add ZK ACLs separately to enforce that access there.
> > >
> > > For anyone not familiar with Ranger, essentially you build a plugin
> that
> > > hooks into the existing component's authorization framework (e.g. for
> > > Storm, the plugin runs through the IAuthorizer
> > > <
> > >
> >
> https://storm.apache.org/releases/1.2.2/javadocs/org/apache/storm/security/auth/IAuthorizer.html
> > > >
> > > interface, for Yarn it runs through YarnAuthorizationProvider
> > > <
> > >
> >
> http://hadoop.apache.org/docs/sta

Re: [DISCUSS] Managing intermittent test failures

[Justin Leet]

Well, since I just put out pretty much this same request on the dev thread,
I'm +1.

On Thu, Nov 29, 2018 at 10:26 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Every now and then we see intermittent test failures, and rather than
> sweeping them under the rug, we should have a simple method to track and
> handle them. I started creating Jiras for tests that I've seen fail, but
> that don't fail consistently, or even fail more than once. For example,
> https://issues.apache.org/jira/browse/METRON-1851.
>
> I think we're all taking steps to varying degrees already, but I want to
> call it out formally. I propose we create a ticket and add the label
> "test-failure." It might also make sense to send a quick note to the dev
> list or Slack channel, so attention can be brought to it and anyone else
> that may have run into an issue with the test can chime in. We can clean
> them out every few months - maybe do a review going into a release and
> close any that have not been reproduced for some time. What do you all
> think?
>
> Mike
>

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Justin Leet]

There's a few issues I would like to see at least triaged and preferably
addressed prior to the release of the main repo. In Jira, we have a
"test-failures" label, that has a few things attached to it. If we know of
any other Jiras that should have this label attached, please do so and I'd
appreciate it if you replied to the thread.  See test-failures
<https://issues.apache.org/jira/browse/METRON-1851?jql=project%20%3D%20METRON%20AND%20labels%20%3D%20test-failure>
.

The Jiras are:
METRON-1810 <https://issues.apache.org/jira/browse/METRON-1810>
METRON-1814 <https://issues.apache.org/jira/browse/METRON-1814>
METRON-1851 <https://issues.apache.org/jira/browse/METRON-1851>

On Wed, Nov 21, 2018 at 2:20 PM zeo...@gmail.com  wrote:

> A metron-bro-plugin-kafka 0.3 release is good to go from my side.  Thanks
> for all of the reviews Nick
>
> On Wed, Nov 21, 2018 at 11:16 AM Nick Allen  wrote:
>
> > Ha.  Yes, that definitely counts and makes a ton of sense.  Thanks!
> >
> > On Wed, Nov 21, 2018 at 11:00 AM Justin Leet 
> > wrote:
> >
> > > Does "I forgot to pull master fresh before running the command" count
> as
> > a
> > > reason?
> > >
> > > The missing Jiras are:
> > >
> > > METRON-1890 Metron Vagrant should disable audio (ottobackwards)
> > closes
> > > apache/metron#1277
> > > METRON-1874 Create a Parser Debugger (nickwallen) closes
> > > apache/metron#1265
> > > METRON-1880 Use Caffeine for Profiler Caching (nickwallen) closes
> > > apache/metron#1270
> > > METRON-1877 Nested IF ELSE statements can cause parse errors in
> > Stellar
> > > (justinleet) closes apache/metron#1268
> > > METRON-1872 Move rat plugin away from snapshot version (justinleet)
> > > closes apache/metron#1264
> > >
> > > On Wed, Nov 21, 2018 at 10:59 AM Nick Allen 
> wrote:
> > >
> > > > Also, I'd like to get this one included in the release.  This is
> really
> > > > annoying for people just wanting to try out the Profiler.  And this
> was
> > > > 'broken' after the last release, so there currently is no release
> with
> > > this
> > > > problem and I'd like to keep it that way. :)
> > > >
> > > > https://github.com/apache/metron/pull/1276
> > > >
> > > > On Wed, Nov 21, 2018 at 10:11 AM Justin Leet 
> > > > wrote:
> > > >
> > > > > Realized I'd never sent the updated list of Jiras.  I changed the
> > > command
> > > > > slightly (to remove a clause I thought we'd already removed re:
> http,
> > > and
> > > > > added the awk to remove dupes resulting from multiple commits for a
> > > > single
> > > > > Jira. I'll do a PR for these changes).
> > > > >
> > > > > *apache/metron*
> > > > > git log "master" "^tags/apache-metron-0.6.0-release" --no-merges |
> > grep
> > > > -E
> > > > > "^[[:blank:]]+METRON" | sed 's/\[//g' | sed 's/\]//g' | awk
> > '!x[$1]++'
> > > > > METRON-1875 Expose configurable global settings in the Alerts
> UI
> > > > > (merrimanr) closes apache/metron#1266
> > > > > METRON-1834: Migrate Elasticsearch from TransportClient to new
> > Java
> > > > > REST API (mmiklavc via mmiklavc) closes apache/metron#1242
> > > > > METRON-1749 Update Angular to latest release in Management UI
> > > > (sardell
> > > > > via nickwallen) closes apache/metron#1217
> > > > > METRON-1870 Intermittent Stellar REST test failures (merrimanr
> > via
> > > > > nickwallen) closes apache/metron#1263
> > > > > METRON-1868 metron-committer-common incorrectly checking
> > REPO_NAME
> > > > > (JonZeolla via jonzeolla) closes apache/metron#1260
> > > > > METRON-1740 Improve Palo Alto parser to handle CONFIG and
> SYSTEM
> > > > syslog
> > > > > messages (liuy-tnz via nickwallen) closes apache/metron#1171
> > > > > METRON-1847 Create reusable script with functions from
> > > prepare-commit
> > > > > (ottobackwards) closes apache/metron#1248
> > > > > METRON-1850 Stellar REST function (merrimanr) closes
> > > > apache/metron#1250
> > > > > METRON-1858 BasicFireEyeParser check style cleanup and
> > optimization
> > > > > (ottobackwards) closes apa

[VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[Justin Leet]

This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
The release candidate is available at:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/

Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES

The tag to be voted upon is:
apache-metron-bro-plugin-kafka_0.3.0-rc1


The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/apache-metron-bro-plugin-kafka_0.3.0-rc1.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/

The release artifacts are signed with the following key:
*https://dist.apache.org/repos/dist/release/metron/KEYS*

Please vote on releasing this package as Apache Metron-bro-plugin-kafka
0.3.0

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This is only for the plugin, so the VM validation is not required.

This vote will be open for until 3pm EDT on Friday November 27 2018.

[ ] +1 Release this package as Apache Metron 0.3.0-RC1

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Justin Leet]

Does "I forgot to pull master fresh before running the command" count as a
reason?

The missing Jiras are:

METRON-1890 Metron Vagrant should disable audio (ottobackwards) closes
apache/metron#1277
METRON-1874 Create a Parser Debugger (nickwallen) closes
apache/metron#1265
METRON-1880 Use Caffeine for Profiler Caching (nickwallen) closes
apache/metron#1270
METRON-1877 Nested IF ELSE statements can cause parse errors in Stellar
(justinleet) closes apache/metron#1268
METRON-1872 Move rat plugin away from snapshot version (justinleet)
closes apache/metron#1264

On Wed, Nov 21, 2018 at 10:59 AM Nick Allen  wrote:

> Also, I'd like to get this one included in the release.  This is really
> annoying for people just wanting to try out the Profiler.  And this was
> 'broken' after the last release, so there currently is no release with this
> problem and I'd like to keep it that way. :)
>
> https://github.com/apache/metron/pull/1276
>
> On Wed, Nov 21, 2018 at 10:11 AM Justin Leet 
> wrote:
>
> > Realized I'd never sent the updated list of Jiras.  I changed the command
> > slightly (to remove a clause I thought we'd already removed re: http, and
> > added the awk to remove dupes resulting from multiple commits for a
> single
> > Jira. I'll do a PR for these changes).
> >
> > *apache/metron*
> > git log "master" "^tags/apache-metron-0.6.0-release" --no-merges | grep
> -E
> > "^[[:blank:]]+METRON" | sed 's/\[//g' | sed 's/\]//g' | awk '!x[$1]++'
> > METRON-1875 Expose configurable global settings in the Alerts UI
> > (merrimanr) closes apache/metron#1266
> > METRON-1834: Migrate Elasticsearch from TransportClient to new Java
> > REST API (mmiklavc via mmiklavc) closes apache/metron#1242
> > METRON-1749 Update Angular to latest release in Management UI
> (sardell
> > via nickwallen) closes apache/metron#1217
> > METRON-1870 Intermittent Stellar REST test failures (merrimanr via
> > nickwallen) closes apache/metron#1263
> > METRON-1868 metron-committer-common incorrectly checking REPO_NAME
> > (JonZeolla via jonzeolla) closes apache/metron#1260
> > METRON-1740 Improve Palo Alto parser to handle CONFIG and SYSTEM
> syslog
> > messages (liuy-tnz via nickwallen) closes apache/metron#1171
> > METRON-1847 Create reusable script with functions from prepare-commit
> > (ottobackwards) closes apache/metron#1248
> > METRON-1850 Stellar REST function (merrimanr) closes
> apache/metron#1250
> > METRON-1858 BasicFireEyeParser check style cleanup and optimization
> > (ottobackwards) closes apache/metron#1255
> > METRON-1864 Stellar date format test fails after daylight saving
> > (ottobackwards) closes apache/metron#1258
> > METRON-1861 METRON-1861: REST fails to start when LDAP enabled and
> > 'Active Spring profiles' config is empty (anandsubbu via justinleet)
> closes
> > apache/metron#1256
> > METRON-1853: Add shutdown hook to Stellar BaseFunctionResolver
> > (mmiklavc via mmiklavc) closes apache/metron#1251
> > METRON-1857 Fix Metaalert Nested Alert Field Name in Index Template
> > (nickwallen) closes apache/metron#1253
> > METRON-1855: Make unified enrichment topology the default and
> deprecate
> > split-join (mmiklavc via mmiklavc) closes apache/metron#1252
> > METRON-1790 Unsubscribe from every observable in the pcap panel UI
> > component (ruffle via nickwallen) closes apache/metron#1208
> > METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc)
> closes
> > apache/metron#1226
> > METRON-1844 Allow for LDAP to be used for authentication and roles
> > (justinleet) closes apache/metron#1246
> > METRON-1830 Re-implement Alerts dialog box without jQuery (sardell
> via
> > merrimanr) closes apache/metron#1240
> > METRON-1826 Update librdkafka and devtoolset (JonZeolla via
> jonzeolla)
> > closes apache/metron#1238
> > METRON-1839 Install Elasticsearch MPack Step in Ansible Not
> Idempotent
> > (nickwallen) closes apache/metron#1244
> > METRON-1833: Management UI incorrectly displaying sensor topology
> > latency units as seconds instead of millis (mmiklavc via mmiklavc) closes
> > apache/metron#1241
> > METRON-1829 Large Error Message Causes Slow Search Performance
> > (merrimanr) closes apache/metron#1239
> > METRON-1831 Project Version Substitution Not Working (nickwallen)
> > closes apache/metron#1243
> > METRON-1816 Date format Stellar function (merrimanr) closes
> > apache/metron#1233
> > METRON-1681 Decouple the ParserBolt from the Parse e

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Justin Leet]

t; I'm good with that release schedule, and using version 0.7.0 for the
> apache/metron release.
>
> I opened up METRON-1881 and have a branch ready to PR for the 0.3
> plugin upgrade.
>
> Jon
>
> On Fri, Nov 16, 2018 at 10:08 AM Otto Fowler 
> wrote:
>
>> Can you generate the jiras that would be included in the release?
>>
>>
>> On November 16, 2018 at 10:05:50, Justin Leet (justinjl...@gmail.com)
>> wrote:
>>
>> Given that we've had a couple major PRs (the ES client migration along
>> with
>> the Angular upgrade stuff, and I'm sure others), I'd be in favor of
>> releasing both the plugin and the main repo.
>>
>> I'd be in favor of doing something like:
>> metron-bro-plugin-kafka release 0.3.0
>> PR to update full dev
>> metron release 0.7.0 (given that things changed a good amount)
>>
>> I would also expect this process to most likely be kicked off post
>> Thanksgiving break (Thursday 22nd and Friday 23rd for anybody not in the
>> US).
>>
>> I can kick out another summarization thread if we want, but basically:
>> * Nov 26 - Start the Bro plugin release process
>> * Once that finishes, PR to update full dev with new plugin version
>> * Once PR is in, start metron release process (hopefully) sometime the
>> week
>> of the 3rd?
>>
>> Are there any objections to staggering the releases like that? They could
>> also be done together, but it means that we have to update full dev to
>> match the plugin version post release.
>>
>> On Wed, Nov 14, 2018 at 10:29 AM zeo...@gmail.com 
>> wrote:
>>
>> > In my opinion metron-bro-plugin-kafka is ready for a release. Anything
>> > else people would want to see? Once it gets released, I would like to
>> > update full dev to use the newest version prior to any future metron
>> > release (0.6.1 or whatever we choose).
>> >
>> > Jon
>> >
>> > On Wed, Nov 7, 2018 at 8:07 PM zeo...@gmail.com 
>> wrote:
>> >
>> > > So, about this release, anybody have time to review
>> > > apache/metron-bro-plugin-kafka#2 and
>> apache/metron-bro-plugin-kafka#13?
>> > >
>> > > Jon
>> > >
>> > > On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic <
>> > > michael.miklav...@gmail.com> wrote:
>> > >
>> > >> And I do think we will be ready to roll another Metron release in the
>> > near
>> > >> future as well.
>> > >>
>> > >> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet 
>> > >> wrote:
>> > >>
>> > >> > I tend to agree with Mike. I think we could do a release of the
>> main
>> > >> > project and get benefit, but I think skipping this cycle
>> (especially
>> > >> since
>> > >> > we had a release last month) let's us get things like the ES client
>> > >> > migration in and settled and just generally make recommending a
>> newer
>> > >> > version easier.
>> > >> >
>> > >> > The main reason for doing a release for the Bro plugin is to
>> address
>> > >> > shortcomings that we discovered and are addressing.
>> > >> >
>> > >> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
>> > >> > michael.miklav...@gmail.com> wrote:
>> > >> >
>> > >> > > Migrating the ES client, refactoring parser bolt. There are some
>> > >> > interface
>> > >> > > changes in flight right now that I think would be beneficial to
>> see
>> > in
>> > >> > the
>> > >> > > next release.
>> > >> > >
>> > >> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen 
>> > >> wrote:
>> > >> > >
>> > >> > > > I am in favor of a release for both.
>> > >> > > >
>> > >> > > > There are a lot of really useful bug fixes, management of pcap
>> > >> through
>> > >> > > > Ambari, more flexibility for configuring JAAS in Ambari,
>> increased
>> > >> > > > Elasticsearch performance, the Syslog parser, and the Batch
>> > >> Profiler,
>> > >> > > among
>> > >> > > > others. I would be happy with calling it a 0.6.1 point release.
>> > >> > > >
>> > >> &g

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Justin Leet]

Given that we've had a couple major PRs (the ES client migration along with
the Angular upgrade stuff, and I'm sure others), I'd be in favor of
releasing both the plugin and the main repo.

I'd be in favor of doing something like:
metron-bro-plugin-kafka release 0.3.0
PR to update full dev
metron release 0.7.0 (given that things changed a good amount)

I would also expect this process to most likely be kicked off post
Thanksgiving break (Thursday 22nd and Friday 23rd for anybody not in the
US).

I can kick out another summarization thread if we want, but basically:
* Nov 26 - Start the Bro plugin release process
* Once that finishes, PR to update full dev with new plugin version
* Once PR is in, start metron release process (hopefully) sometime the week
of the 3rd?

Are there any objections to staggering the releases like that?  They could
also be done together, but it means that we have to update full dev to
match the plugin version post release.

On Wed, Nov 14, 2018 at 10:29 AM zeo...@gmail.com  wrote:

> In my opinion metron-bro-plugin-kafka is ready for a release.  Anything
> else people would want to see?  Once it gets released, I would like to
> update full dev to use the newest version prior to any future metron
> release (0.6.1 or whatever we choose).
>
> Jon
>
> On Wed, Nov 7, 2018 at 8:07 PM zeo...@gmail.com  wrote:
>
> > So, about this release, anybody have time to review
> > apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13?
> >
> > Jon
> >
> > On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> And I do think we will be ready to roll another Metron release in the
> near
> >> future as well.
> >>
> >> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet 
> >> wrote:
> >>
> >> > I tend to agree with Mike. I think we could do a release of the main
> >> > project and get benefit, but I think skipping this cycle (especially
> >> since
> >> > we had a release last month) let's us get things like the ES client
> >> > migration in and settled and just generally make recommending a newer
> >> > version easier.
> >> >
> >> > The main reason for doing a release for the Bro plugin is to address
> >> > shortcomings that we discovered and are addressing.
> >> >
> >> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
> >> > michael.miklav...@gmail.com> wrote:
> >> >
> >> > > Migrating the ES client, refactoring parser bolt. There are some
> >> > interface
> >> > > changes in flight right now that I think would be beneficial to see
> in
> >> > the
> >> > > next release.
> >> > >
> >> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen 
> >> wrote:
> >> > >
> >> > > > I am in favor of a release for both.
> >> > > >
> >> > > > There are a lot of really useful bug fixes, management of pcap
> >> through
> >> > > > Ambari, more flexibility for configuring JAAS in Ambari, increased
> >> > > > Elasticsearch performance, the Syslog parser, and the Batch
> >> Profiler,
> >> > > among
> >> > > > others. I would be happy with calling it a 0.6.1 point release.
> >> > > >
> >> > > > Mike - What is outstanding that you would like to see in the
> >> release?
> >> > > >
> >> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
> >> > > > michael.miklav...@gmail.com> wrote:
> >> > > >
> >> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin
> release.
> >> It
> >> > > > seems
> >> > > > > like it's ready to go, and I think there are a few more things
> I'd
> >> > like
> >> > > > to
> >> > > > > see get into our next Metron release so I'm good with holding
> off
> >> > > there.
> >> > > > >
> >> > > > > Mike
> >> > > > >
> >> > > > > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet <
> >> justinjl...@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > > As you might recall from a prior discussion about release
> >> cadence,
> >> > we
> >> > > > 

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

[Justin Leet]

I took a look at this with Mike a bit, and it seems like it's pretty
painful and without a clear way to avoid remerging conflicts.  If the
latest attempt doesn't work, I'm in favor of getting it in and just getting
it down to as few commits as reasonably possible.

On Thu, Nov 15, 2018 at 4:12 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I'm attempting 1 more option, which would be to do a "git merge
> --strategy-option theirs" after having done the commit wrangling in the PR
> branch. Will reply back with results.
>
> On Thu, Nov 15, 2018 at 2:02 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Yes, definitely.
> >
> > On Thu, Nov 15, 2018 at 2:01 PM Casey Stella  wrote:
> >
> >> Can you at least rename your commits to have METRON-1834 prefixing them?
> >> On Thu, Nov 15, 2018 at 15:19 Michael Miklavcic <
> >> michael.miklav...@gmail.com>
> >> wrote:
> >>
> >> > https://github.com/apache/metron/pull/1242
> >> >
> >> > TL;DR
> >> > I'd like to discuss the best option to merge METRON-1834 into master.
> I
> >> > want to propose handling this like a feature branch and merging it
> >> as-is.
> >> > ---
> >> >
> >> > I'm sure most folks' initial reaction will be some skepticism akin to
> >> "have
> >> > you tried turning it off again," as this was my initial reaction as
> >> well.
> >> > It does not seem like this should be difficult. And I'm hoping that
> this
> >> > may be some esoteric thing on my system, though I believe this is a
> real
> >> > problem. A rather tedious explanation follows of what I've tried and
> the
> >> > problems encountered along the way. What seemed like a really simple
> >> > problem instead appears to be a bit much for Git to handle without
> >> > requiring redoing merges and another full round of testing. I'd much
> >> prefer
> >> > to avoid that in this instance.
> >> >
> >> > This PR is ready to be merged into master. It's recent and very close
> to
> >> > fully up to date in the branch. Latest master merges cleanly. There is
> >> an
> >> > attribution to Casey Stella for the base point of this PR that I need
> to
> >> > include when getting this into master. When I created my branch, I
> >> > collapsed his initial set of commits into a single squashed commit on
> >> > master at the time, and I started to work from there. Over time, I
> made
> >> a
> >> > number of additional commits and merges from master. Now for the
> issues.
> >> >
> >> > Originally, my expectation was that I could have 2 commits - the
> >> original
> >> > squashed commit from Casey along with all my additional commits (and
> the
> >> > merges with master) right on top. Nice clean history on master. Turns
> >> out,
> >> > this doesn't work as cleanly as expected because a combination of the
> >> > multiple merges and the need to keep the original commit with
> >> attribution
> >> > to Casey's work. A normal git pull --squash works fine, as expected,
> >> but we
> >> > lose the base commit, and therefore the requisite attribution. Here
> are
> >> > some other things I've tried, to no avail.
> >> >
> >> >1. Git pull --squash after a merge with master. This will squash
> the
> >> >entire tree back to the branch point. No good.
> >> >2. Git rebase -i master. Allows you to cleanly apply changes, but
> >> then
> >> >it ends up having problems with a clean rebase and shows
> conflicts. I
> >> >expect this is because of the merge history being necessary.
> >> >3. Checking out a branch from the base point squashed commit from
> >> Casey,
> >> >and attempt to apply my changes on top. Numerous methods for
> >> >squashing/rebasing my changes on top applies nicely in the branch.
> >> But
> >> > then
> >> >it once again causes merge conflicts when I attempt to get this
> onto
> >> >master. Things I attempted include: manually copying files,
> rebasing
> >> > all my
> >> >commits plus merges on top of the base commit, git merge --squash,
> >> >intimidation.
> >> >
> >> > For one example of the result I'm talking about, this looks "good" but
> >> it's
> >> > missing a ton of recent commits because they get caught up in the
> rebase
> >> > and get squashed in with my commit. When you attempt to merge this
> onto
> >> > master, it is just plain wrong (see example below with merge
> conflicts).
> >> > * 22c3b3bc 2018-11-15 | METRON-1834: Migrate Elasticsearch from
> >> > TransportClient to new Java REST API (mmiklavc via mmiklavc) closes
> >> > apache/metron#1242 (HEAD -> stella-es-base2) [mmiklavc]
> >> > * 84232e90 2018-10-08 | METRON-1834: Elasticsearch rest client
> migration
> >> > base work starting point for apache/metron#1242 (cstella via mmiklavc)
> >> > [cstella]
> >> > * 5bfc08c5 2018-10-08 | METRON-1792 Simplify Profile Definitions in
> >> > Integration Tests (nickwallen) closes apache/metron#1211 [nickwallen]
> >> >
> >> > Here's 1 merge conflict (say what??)
> >> > CONFLICT (rename/rename): Rename
> >> >
> >> >
> >>
> 

[DISCUSS] Authorization for Configuration

[Justin Leet]

Hi all,

Sorry for the second copy of this email, I forgot the  [DISCUSS] tag on the
original.  Otherwise, this has the same content.

Right now, our various configs can be modified by anyone with access to the
various scripts. I'd like to start a discussion around building out some
authorization to be able to add some more fine grained controls around this.

Other projects have some variants on how to accomplish this.  Typically,
this follows a pattern of calling out to a interface/class that takes in
the operation and context (user and other info) and returns true/false if
something is authorized.

In my mind, what we would need out of this is

   - Ability to apply fine grained permissions
   - The various scripts and UI should flow this authorization framework.
   I believe most (if not all) of our configuration flows through
   ConfigurationsUtils.  Anything that doesn't should either be hooked in or
   refactored to flow through the same codepaths.
   - Pluggability. We shouldn't force only one authorizer.

In particular, I'm proposing we use Apache Ranger
 as a supported authorization framework,
implementing it alongside the authorization framework to validate what we
build. In my mind, the main catch with Ranger is that, based on my
understanding, we won't be able to restrict users with direct access to
ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into ZK's
ACLs).  I believe this is a reasonable restriction, especially as the
management UI gets improved to handle more of the configuration burden and
the number of users with access to ZK CLI begins to decrease.  Users can
still add ZK ACLs separately to enforce that access there.

For anyone not familiar with Ranger, essentially you build a plugin that
hooks into the existing component's authorization framework (e.g. for
Storm, the plugin runs through the IAuthorizer

interface,
for Yarn it runs through YarnAuthorizationProvider
).
Additionally, Ranger provides auditing capabilities for this authorization
and has plugins for a good portion of our stack already (so users can
already setup ACLs on HDFS, Storm, etc.). Checkout the Ranger Github
 for a list of the plugins they have
built in.

What this means for Metron is building out an authorization setup similar
to Storm or Yarn or whatever we choose. We'll want this anyway, to allow
our solution to be pluggable.  At that point, we build an implementation of
the authorizer compatible with Ranger along with the plugin.

I think this could probably be a fairly small feature branch, which I'm
suggesting primarily to do the Ranger implementation alongside the general
authorization work to validate what's being built.  I think the main
tasking would be something similar to:

   - Build out pluggable authorization for our configs.
   - This includes testing (and possibly doing something similar to Storm,
   where they have a some testing IAuthorizers, e.g. NoopAuthorizer,
   DenyAuthorizer, etc.)
   - Ensure that all the code paths consistently flow through this
   Authorization.
   - Build a Ranger compatible version of this.
   - Define the Ranger plugin for this.
   - Make sure auditing is defined.
   - Integration testing (particularly with Kerberos. After all, if they
   want to do authorization and auditing, they're almost certainly using
   Kerberos).

Is there anything missing that we'd need or want for this?  Are there any
other concerns we'd want to make sure are taken care of?

Authorization for Configuration

[Justin Leet]

Hi all,

Right now, our various configs can be modified by anyone with access to the
various scripts. I'd like to start a discussion around building out some
authorization to be able to add some more fine grained controls around this.

Other projects have some variants on how to accomplish this.  Typically,
this follows a pattern of calling out to a interface/class that takes in
the operation and context (user and other info) and returns true/false if
something is authorized.

In my mind, what we would need out of this is

   - Ability to apply fine grained permissions
   - The various scripts and UI should flow this authorization framework.
   I believe most (if not all) of our configuration flows through
   ConfigurationsUtils.  Anything that doesn't should either be hooked in or
   refactored to flow through the same codepaths.
   - Pluggability. We shouldn't force only one authorizer.

In particular, I'm proposing we use Apache Ranger
 as a supported authorization framework,
implementing it alongside the authorization framework to validate what we
build. In my mind, the main catch with Ranger is that, based on my
understanding, we won't be able to restrict users with direct access to
ZooKeeper via it's CLI (e.g. Ranger can't mirror it's ACLs down into ZK's
ACLs).  I believe this is a reasonable restriction, especially as the
management UI gets improved to handle more of the configuration burden and
the number of users with access to ZK CLI begins to decrease.  Users can
still add ZK ACLs separately to enforce that access there.

For anyone not familiar with Ranger, essentially you build a plugin that
hooks into the existing component's authorization framework (e.g. for
Storm, the plugin runs through the IAuthorizer

interface, for Yarn it runs through YarnAuthorizationProvider
).
Additionally, Ranger provides auditing capabilities for this authorization
and has plugins for a good portion of our stack already (so users can
already setup ACLs on HDFS, Storm, etc.). Checkout the Ranger Github
 for a list of the plugins they have
built in.

What this means for Metron is building out an authorization setup similar
to Storm or Yarn or whatever we choose. We'll want this anyway, to allow
our solution to be pluggable.  At that point, we build an implementation of
the authorizer compatible with Ranger along with the plugin.

I think this could probably be a fairly small feature branch, which I'm
suggesting primarily to do the Ranger implementation alongside the general
authorization work to validate what's being built.  I think the main
tasking would be something similar to:

   - Build out pluggable authorization for our configs.
   - This includes testing (and possibly doing something similar to Storm,
   where they have a some testing IAuthorizers, e.g. NoopAuthorizer,
   DenyAuthorizer, etc.)
   - Ensure that all the code paths consistently flow through this
   Authorization.
   - Build a Ranger compatible version of this.
   - Define the Ranger plugin for this.
   - Make sure auditing is defined.
   - Integration testing (particularly with Kerberos. After all, if they
   want to do authorization and auditing, they're almost certainly using
   Kerberos).

Is there anything missing that we'd need or want for this?  Are there any
other concerns we'd want to make sure are taken care of?

Re: [DISCUSS] Slack Channel Use

[Justin Leet]

t; > On Mon, Oct 22, 2018 at 5:05 PM Otto Fowler <
> > ottobackwa...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > According to Greg Stein, an infra admin on the NiFi slack, the
> > ASF
> > > > > slack
> > > > > > > that metron is in IS the standard plan, not the free one and is
> > > > > > searchable
> > > > > > > past 10,000 messages.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On October 22, 2018 at 15:35:51, Michael Miklavcic (
> > > > > > > michael.miklav...@gmail.com) wrote:
> > > > > > >
> > > > > > > ...From an archival and broader reach point of view, I do think
> > > > there's
> > > > > > > something to be said about using the mailing list. It's also
> > easier
> > > > to
> > > > > > link
> > > > > > > to Q/A threads from the mailing list archives and do
> searches...
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> >
> https://lists.apache.org/thread.html/1aa85bc13d41e04a1f85c3100c2b803abe35d79b54062bbeaab83ace@%3Cdev.metron.apache.org%3E
> > > > > > >
> > > > > > > How very Inception.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 22, 2018 at 1:32 PM Michael Miklavcic <
> > > > > > > michael.miklav...@gmail.com> wrote:
> > > > > > >
> > > > > > > > I just want to point out that we currently have 32 members in
> > the
> > > > > > Metron
> > > > > > > > Slack channel which I personally think is a great sign. This
> is
> > > > good
> > > > > > from
> > > > > > > a
> > > > > > > > community perspective and helps foster interactive sessions
> > where
> > > > > > > required.
> > > > > > > > From an archival and broader reach point of view, I do think
> > > > there's
> > > > > > > > something to be said about using the mailing list. It's also
> > > easier
> > > > > to
> > > > > > > link
> > > > > > > > to Q/A threads from the mailing list archives and do
> searches.
> > As
> > > > > > such, I
> > > > > > > > would also go along with Nick's suggestion and urge members
> to
> > > > prefer
> > > > > > the
> > > > > > > > user/dev list where possible.
> > > > > > > >
> > > > > > > > On Mon, Oct 22, 2018 at 10:51 AM Justin Leet <
> > > > justinjl...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> If we want to push more discussion to the dev list, my
> obvious
> > > > > follow
> > > > > > up
> > > > > > > >> question then is "What are we hoping to get out of
> > > Slack/irc/other
> > > > > > > >> interactive medium?". What discussion would we even want on
> > > there,
> > > > > if
> > > > > > we
> > > > > > > >> can't have decisions and don't want usage/support?
> > > > > > > >>
> > > > > > > >> On Mon, Oct 22, 2018 at 12:44 PM Casey Stella <
> > > ceste...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > I am of 2 minds, but I tend to agree. On the one hand,
> it's
> > > > > > definitely
> > > > > > > >> the
> > > > > > > >> > preference that we use the mailing lists for the reasons
> you
> > > > > stated
> > > > > > > (and
> > > > > > > >> > also because not everyone has access to slack generally).
> On
> > > the
> > > > > > other
> > > > > > > >> > hand, I think an interactive medium like Slack has a lot
> of
> > > > > > advantages
> > > > > > > >> in
> > > > > > > >> > terms of user satisfaction. Ultimately, though, we may
> > satisfy
> > > 1
> > > > > > user
> > > > > > > >> at
> > > > > > > >> > the cost of not persisting the discussion and satisfying
> > many
> > > > > users.
> > > > > > > >> >
> > > > > > > >> > I'll go along with a specific preference to drive more
> > > > discussion
> > > > > to
> > > > > > > the
> > > > > > > >> > mailing list.
> > > > > > > >> >
> > > > > > > >> > Casey
> > > > > > > >> >
> > > > > > > >> > On Mon, Oct 22, 2018 at 12:18 PM Nick Allen <
> > > n...@nickallen.org
> > > > >
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > It seems that we are seeing a lot of Metron usage and
> > > support
> > > > > > > >> questions
> > > > > > > >> > on
> > > > > > > >> > > the Slack Channel.
> > > > > > > >> > > These are questions that previously would have been
> > directed
> > > > to
> > > > > > the
> > > > > > > >> User
> > > > > > > >> > or
> > > > > > > >> > > Dev mailing lists. Since this is occurring in the Slack
> > > > Channel,
> > > > > > the
> > > > > > > >> > > conversations are not archived.
> > > > > > > >> > >
> > > > > > > >> > > In my opinion, this is not good for the Metron
> community.
> > > > Having
> > > > > > > this
> > > > > > > >> > > persisted in a discoverable form (like a mailing list
> > > archive)
> > > > > not
> > > > > > > >> only
> > > > > > > >> > > helps support current users, but also helps *potential*
> > > users
> > > > > > > >> understand
> > > > > > > >> > > how Metron is being used.
> > > > > > > >> > >
> > > > > > > >> > > Does anyone else agree or disagree? At a minimum, I feel
> > we
> > > > need
> > > > > > to
> > > > > > > >> do
> > > > > > > >> > > something to direct these conversations back to the
> > mailing
> > > > > list.
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > A.Nazemian
> > > >
> > >
> >
>

Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

[Justin Leet]

+1, I haven't seen any case where the split-join topology isn't made
obsolete by the unified topology.

On Thu, Nov 1, 2018 at 6:17 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Fellow Metronians,
>
> We've had the unified enrichment topology around for a number of months
> now, it has proved itself stable, and there is yet to be a time that I have
> seen the split-join topology outperform the unified one. Here are some
> simple reasons to deprecate the split-join topology.
>
>1. Unified topology performs better.
>2. The configuration, especially for performance tuning is much, much
>simpler in the unified model.
>3. The footprint within the cluster is smaller.
>4. One of the first activities for any install is that we spend time
>instructing users to switch to the unified topology.
>5. One less moving part to maintain.
>
> I'd like to recommend that we deprecate the split-join topology and make
> the unified enrichment topology the new default.
>
> Best,
> Mike
>

Re: [DISCUSS] Slack Channel Use

[Justin Leet]

If we want to push more discussion to the dev list, my obvious follow up
question then is "What are we hoping to get out of Slack/irc/other
interactive medium?".  What discussion would we even want on there, if we
can't have decisions and don't want usage/support?

On Mon, Oct 22, 2018 at 12:44 PM Casey Stella  wrote:

> I am of 2 minds, but I tend to agree.  On the one hand, it's definitely the
> preference that we use the mailing lists for the reasons you stated (and
> also because not everyone has access to slack generally).  On the other
> hand, I think an interactive medium like Slack has a lot of advantages in
> terms of user satisfaction.  Ultimately, though, we may satisfy 1 user at
> the cost of not persisting the discussion and satisfying many users.
>
> I'll go along with a specific preference to drive more discussion to the
> mailing list.
>
> Casey
>
> On Mon, Oct 22, 2018 at 12:18 PM Nick Allen  wrote:
>
> > It seems that we are seeing a lot of Metron usage and support questions
> on
> > the Slack Channel.
> > These are questions that previously would have been directed to the User
> or
> > Dev mailing lists.  Since this is occurring in the Slack Channel, the
> > conversations are not archived.
> >
> > In my opinion, this is not good for the Metron community.  Having this
> > persisted in a discoverable form (like a mailing list archive) not only
> > helps support current users, but also helps *potential* users understand
> > how Metron is being used.
> >
> > Does anyone else agree or disagree?  At a minimum, I feel we need to do
> > something to direct these conversations back to the mailing list.
> >
>

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Justin Leet]

I tend to agree with Mike. I think we could do a release of the main
project and get benefit, but I think skipping this cycle (especially since
we had a release last month) let's us get things like the ES client
migration in and settled and just generally make recommending a newer
version easier.

The main reason for doing a release for the Bro plugin is to address
shortcomings that we discovered and are addressing.

On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Migrating the ES client, refactoring parser bolt. There are some interface
> changes in flight right now that I think would be beneficial to see in the
> next release.
>
> On Tue, Oct 16, 2018 at 2:01 PM Nick Allen  wrote:
>
> > I am in favor of a release for both.
> >
> > There are a lot of really useful bug fixes, management of pcap through
> > Ambari, more flexibility for configuring JAAS in Ambari, increased
> > Elasticsearch performance, the Syslog parser, and the Batch Profiler,
> among
> > others. I would be happy with calling it a 0.6.1 point release.
> >
> > Mike - What is outstanding that you would like to see in the release?
> >
> > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > I'd be +1 on going with just the metron-bro-kafka-plugin release. It
> > seems
> > > like it's ready to go, and I think there are a few more things I'd like
> > to
> > > see get into our next Metron release so I'm good with holding off
> there.
> > >
> > > Mike
> > >
> > > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > As you might recall from a prior discussion about release cadence, we
> > > were
> > > > interested in initiating release threads near our board reports to
> see
> > if
> > > > we wanted to do releases or not. Additionally, the work is done to do
> > two
> > > > separate releases, so our options are releasing both, a single one,
> or
> > > > neither.
> > > >
> > > > Having said that, a metron-bro-kafka-plugin 0.3.0 release came up on
> > this
> > > > thread
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> > > > >.
> > > > In particular, the prospect of a release came up in the context of
> > > having a
> > > > version with better (and working) testing.
> > > >
> > > > Version Number
> > > > If we choose to do a core Metron release, I propose 0.6.1. For
> > > > metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the
> versioning
> > > for
> > > > the plugin is a bit different in that we need x.y for bro-pkg,
> instead
> > of
> > > > x.y.z, so we wouldn't release 0.2.1.
> > > >
> > > > I would personally be in favor of doing a plugin release, but I'm
> less
> > > > inclined to do a core Metron release.
> > > >
> > > > For Metron, I believe the main feature is the batch profiler work,
> but
> > > > there's a fair amount of fixes and improvements.
> > > >
> > > > For the plugin, I believe the main improvements would be around the
> > > fixing
> > > > our testing and more maintenance type work.
> > > >
> > > > JIRA status
> > > > *metron*
> > > > There are 24 opens PRs at https://github.com/apache/metron/pulls.
> If
> > we
> > > > do
> > > > decide to release, we should work on getting anything meriting
> > inclusion
> > > > closed out.
> > > >
> > > > There have been 49 commits since the 0.6.0 release (listed at the end
> > of
> > > > the message). This assumes a release from master.
> > > >
> > > > *metron-bro-kafka-plugin*
> > > > There are 6 open PRs at
> > > > https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we do
> > > decide
> > > > to release, we should get anything we need closed out.
> > > >
> > > > There have been 2 commits since the 0.2.0 release (listed at the end
> of
> > > the
> > > > message).  This assumes a release from master.
> > > >
> > > > *Metron changelog*
> > > > METRON-1801 Allow Customization of Elasticsearch Do

Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Justin Leet]

Hi all,

As you might recall from a prior discussion about release cadence, we were
interested in initiating release threads near our board reports to see if
we wanted to do releases or not. Additionally, the work is done to do two
separate releases, so our options are releasing both, a single one, or
neither.

Having said that, a metron-bro-kafka-plugin 0.3.0 release came up on this
thread
.
In particular, the prospect of a release came up in the context of having a
version with better (and working) testing.

Version Number
If we choose to do a core Metron release, I propose 0.6.1. For
metron-bro-kafka-plugin, I propose 0.3.0.  Keep in mind, the versioning for
the plugin is a bit different in that we need x.y for bro-pkg, instead of
x.y.z, so we wouldn't release 0.2.1.

I would personally be in favor of doing a plugin release, but I'm less
inclined to do a core Metron release.

For Metron, I believe the main feature is the batch profiler work, but
there's a fair amount of fixes and improvements.

For the plugin, I believe the main improvements would be around the fixing
our testing and more maintenance type work.

JIRA status
*metron*
There are 24 opens PRs at https://github.com/apache/metron/pulls.  If we do
decide to release, we should work on getting anything meriting inclusion
closed out.

There have been 49 commits since the 0.6.0 release (listed at the end of
the message). This assumes a release from master.

*metron-bro-kafka-plugin*
There are 6 open PRs at
https://github.com/apache/metron-bro-plugin-kafka/pulls.  If we do decide
to release, we should get anything we need closed out.

There have been 2 commits since the 0.2.0 release (listed at the end of the
message).  This assumes a release from master.

*Metron changelog*
METRON-1801 Allow Customization of Elasticsearch Document ID
(nickwallen) closes apache/metron#1218
METRON-1799 Remove outdated bylaws from site. (justinleet) closes
apache/metron#1216
METRON-1769 Script creation of a release candidate (justinleet) closes
apache/metron#1188
METRON-1761 Allow a grok statement to be applied to each line in a
file. (ottobackwards) closes apache/metron#1184
METRON-1813 Stellar REPL Not Initialized with Client JAAS (nickwallen)
closes apache/metron#1232
METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc)
closes apache/metron#1230
METRON-1811 Alert Search Fails When Sorting by Alert Status (merrimanr)
closes apache/metron#1231
METRON-1809 Support Column Oriented Input with Batch Profiler
(nickwallen) closes apache/metron#1229
METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via mmiklavc)
closes apache/metron#1224
METRON-1792 Simplify Profile Definitions in Integration Tests
(nickwallen) closes apache/metron#1211
METRON-1807 Auto populate the recommended values to some of the metron
config parameters  (MohanDV via merrimanr) closes apache/metron#1227
METRON-1808 Add Ansible created pyc to gitignore (justinleet) closes
apache/metron#1228
METRON-1695 Expose pcap properties through Ambari (anandsubbu) closes
apache/metron#1207
METRON-1771 Update REST endpoints to support eventually consistent UI
updates (merrimanr) closes apache/metron#1190
METRON-1791 Add GUID to Messages Produced by Profiler (nickwallen)
closes apache/metron#1210
METRON-1804 Update version to 0.6.1 (justinleet) closes
apache/metron#1220
METRON-1798 Add mpack support for parser aggregation (anandsubbu)
closes apache/metron#1215
METRON-1750 Create Parser for Syslog RFC 5424 Messages (ottobackwards)
closes apache/metron#1175
METRON-1794 Include User Details When Escalating Alerts (nickwallen)
closes apache/metron#1212
METRON-1782 Add Kafka Partition and Offset to Profiler Debug Logs
(nickwallen) closes apache/metron#1202
METRON-1758 Add support for Ansible 2.6 in dev (JonZeolla via
jonzeolla) closes apache/metron#1179
METRON-1784: Re-allow remote ssh and scp in Centos full dev (mmiklavc
via mmiklavc) closes apache/metron#1204
METRON-1787 Input Time Constraints for Batch Profiler (nickwallen)
closes apache/metron#1209
METRON-1508 In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
(nickwallen) closes apache/metron#1185
METRON-1786 Pcap Topology Status Incorrect (MohanDV via nickwallen)
closes apache/metron#1206
METRON-1709 Add controls to start / stop the PCAP topology from Ambari.
(MohanDV via nickwallen) closes apache/metron#1201
METRON-1759 PCAP UI: Removing wrong Input annotations from pcap panel
component (tiborm via nickwallen) closes apache/metron#1180
METRON-1772 Support alternative input formats in the Batch Profiler
(nickwallen) closes apache/metron#1191
METRON-1770 Add Docs for Running the Profiler with Spark on YARN
(nickwallen) closes apache/metron#1189
METRON-1774 Allow user to configure JAAS client in 

Re: Bro plugin unit tests failing

[Justin Leet]

Should we just put out a discuss for releasing in general, with this
discussion noted?  The outcome of the release cadence discussion was to put
out a thread around each board report. I believe that's a few days from
now, so it's good timing. As an aside, part of the post release activities
for the plugin should probably be upping the plugin version in the main
repo to use the new version.

While we're on the topic, is there anything else we should do to improve
the plugin repo?  Off the top of my head, setting up a version of the PR
template would be helpful.  Maybe adding a section to the README.md linking
to the CONTRIBUTING.md of the main repo?

On Sun, Oct 14, 2018 at 11:14 AM Otto Fowler 
wrote:

> It is INFRA, see INFRA-17091 for example.
>
>
> On October 12, 2018 at 20:47:24, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> So it seems that the last commit before the 0.2 release of
> metron-bro-plugin-kafka broke the one basic unit test that we had. Since
> metron 0.6.0 pins to 0.1
> <
>
> https://github.com/apache/metron/blob/Metron_0.6.0/metron-deployment/ansible/roles/bro/vars/main.yml#L30
> >
>
> this wouldn't cause an obvious issue if you spun up the sensors in
> full-dev, and even if we were to point metron to 0.2 of the plugin we
> wouldn't have necessarily seen the issue because we do a --force
> <
>
> https://github.com/apache/metron/blob/Metron_0.6.0/metron-deployment/ansible/roles/bro/tasks/metron-bro-plugin-kafka.yml#L35
> >
>
> to remove interaction.
>
> After we get a chance to review/merge a recent PR
> , we should have
> a few more 'actual' unit tests, and I would like to get travis setup for
> the repo  to
> actually
> check those (PR incoming) so it's less likely this sort of thing would
> happen again. However, it doesn't appear I have the right permissions to
> be able to get travis moving on my own. Does anybody else have the right
> access, or is this an asf infrastructure ticket?
>
> I'd also like us to consider doing a 0.3 release for the plugin so we
> aren't distributing something that has broken tests. Thoughts on that?
>
> Jon
> --
>
> Jon
>

Re: Bro plugin release process docs?

[Justin Leet]

That first half of that page will be rewritten in the next couple days to
reflect the script to prepare release candidate; I didn't want to change
that page until the PR actually went in.

The release process, barring the KEYS issue (and a patch to the script to
handle the bro plugin tag), is pretty much set to be managed separately at
this point.  The docs just need an overhaul, so someone who's not me knows
what to do.

On Wed, Oct 10, 2018 at 7:01 PM zeo...@gmail.com  wrote:

> Yeah you're right when I looked closer to make the change it was step 10.
> I pushed a manual 0.2 tag to metron-bro-plugin-kafka and did a quick update
> to the cwiki, but it could use some additional love, especially if we want
> to split out the release processes generally.
>
> Jon
>
> On Wed, Oct 10, 2018 at 5:32 PM Justin Leet  wrote:
>
> > Yeah, we need to update the release process instructions. It should
> > actually be step 10 shouldn't it?  We want to keep the rc tag as-is
> because
> > it may get rejected/cancelled, and update the final tag as the release. I
> > already need to overhaul that page to reflect the addition of the RC
> > creation script, so I can also update it.
> >
> > I didn't remember there was a specific reason for the git tagging, so I
> > just followed the convention we used for the main repo, which is why we
> > have this problem.
> >
> > The release candidate script also needs to be updated (because it needs
> to
> > know the proper prior tag to pull). It's not hard to do, but we'll need
> to
> > open a Jira and update that portion of the script. It's pretty easy to
> do,
> > it's just the portion here
> > <
> >
> https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245
> > >
> > .
> >
> > On Wed, Oct 10, 2018 at 5:09 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > +1 to all of that from me, Jon. Thanks for taking care of this.
> > >
> > > On Wed, Oct 10, 2018 at 2:34 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > I wonder if we should also update the
> > > > https://cwiki.apache.org/confluence/display/METRON/Release+Process
> > > > instructions
> > > > to include tagging for the bro plugin, or if we were going to split
> out
> > > the
> > > > release processes?  I'd be happy to update the instructions (Step 8)
> if
> > > > that's the right place for now and I didn't miss a new place for the
> > > plugin
> > > > release instructions.
> > > >
> > > > Jon
> > > >
> > > > On Wed, Oct 10, 2018 at 4:31 PM zeo...@gmail.com 
> > > wrote:
> > > >
> > > > > So I was poking around on the plugin today and noticed that we have
> > > > > a apache-metron-bro-plugin-kafka_0.2.0-release and
> > > > apache-metron-bro-plugin-kafka_0.2.0-rc1
> > > > > tag, but no 0.2 (which is what bro-pkg would point to).  Anybody
> have
> > > any
> > > > > concerns if I push the 0.2 tag as discussed above?  Then we could
> > > update
> > > > > the bro package manager, and finally update what the apache/metron
> > > > full-dev
> > > > > environment(s) point to (0.2 as opposed to 0.1).  Thanks,
> > > > >
> > > > > Jon
> > > > >
> > > > > On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com  >
> > > > wrote:
> > > > >
> > > > >> I did a bit of poking around and I don't believe we ever formally
> > > wrote
> > > > >> that down.  The last release happened as a combination of actions
> > from
> > > > >> mattf and myself (mostly mattf).
> > > > >>
> > > > >> The plugin has two new commits since the last release (1 bugfix 1
> > > > >> feature) - if we want to couple version 0.2 of the plugin with a
> > > metron
> > > > >> 0.5.0 release we would need to make a 0.2 tag against HEAD of the
> > > plugin
> > > > >> repo, then increment the version in the ansible playbooks here
> > > > >> <
> > > >
> > >
> >
> https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30
> > > > >
> > > > >> .
> > > > >>
> > > > >> Or, if we just want to make a plugin release without changing
> > > > >> apache/metron, we could 

Re: Bro plugin release process docs?

[Justin Leet]

Yeah, we need to update the release process instructions. It should
actually be step 10 shouldn't it?  We want to keep the rc tag as-is because
it may get rejected/cancelled, and update the final tag as the release. I
already need to overhaul that page to reflect the addition of the RC
creation script, so I can also update it.

I didn't remember there was a specific reason for the git tagging, so I
just followed the convention we used for the main repo, which is why we
have this problem.

The release candidate script also needs to be updated (because it needs to
know the proper prior tag to pull). It's not hard to do, but we'll need to
open a Jira and update that portion of the script. It's pretty easy to do,
it's just the portion here
<https://github.com/apache/metron/blob/master/dev-utilities/release-utils/prepare-release-candidate#L245>
.

On Wed, Oct 10, 2018 at 5:09 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 to all of that from me, Jon. Thanks for taking care of this.
>
> On Wed, Oct 10, 2018 at 2:34 PM zeo...@gmail.com  wrote:
>
> > I wonder if we should also update the
> > https://cwiki.apache.org/confluence/display/METRON/Release+Process
> > instructions
> > to include tagging for the bro plugin, or if we were going to split out
> the
> > release processes?  I'd be happy to update the instructions (Step 8) if
> > that's the right place for now and I didn't miss a new place for the
> plugin
> > release instructions.
> >
> > Jon
> >
> > On Wed, Oct 10, 2018 at 4:31 PM zeo...@gmail.com 
> wrote:
> >
> > > So I was poking around on the plugin today and noticed that we have
> > > a apache-metron-bro-plugin-kafka_0.2.0-release and
> > apache-metron-bro-plugin-kafka_0.2.0-rc1
> > > tag, but no 0.2 (which is what bro-pkg would point to).  Anybody have
> any
> > > concerns if I push the 0.2 tag as discussed above?  Then we could
> update
> > > the bro package manager, and finally update what the apache/metron
> > full-dev
> > > environment(s) point to (0.2 as opposed to 0.1).  Thanks,
> > >
> > > Jon
> > >
> > > On Mon, May 28, 2018 at 8:41 AM zeo...@gmail.com 
> > wrote:
> > >
> > >> I did a bit of poking around and I don't believe we ever formally
> wrote
> > >> that down.  The last release happened as a combination of actions from
> > >> mattf and myself (mostly mattf).
> > >>
> > >> The plugin has two new commits since the last release (1 bugfix 1
> > >> feature) - if we want to couple version 0.2 of the plugin with a
> metron
> > >> 0.5.0 release we would need to make a 0.2 tag against HEAD of the
> plugin
> > >> repo, then increment the version in the ansible playbooks here
> > >> <
> >
> https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml#L30
> > >
> > >> .
> > >>
> > >> Or, if we just want to make a plugin release without changing
> > >> apache/metron, we could just make a 0.2 tag in the plugin repo, and
> then
> > >> release it in a more disjointed way.  I know that's not super helpful
> > since
> > >> I don't have documentation for doing an apache release of the plugin
> > other
> > >> than hacking something together based on what's out there for
> > apache/metron.
> > >>
> > >> Reference conversations:
> > >>  -
> > >>
> >
> https://lists.apache.org/thread.html/2606166bd5e864f1b56db302099c9a8042cdadec8fa2692fef49493f@%3Cdev.metron.apache.org%3E
> > >>  -
> > >>
> >
> https://lists.apache.org/thread.html/3aecbadbf3353e98c03ca4b680fcd998d0cd2bf5a4319238dd85ae75@%3Cdev.metron.apache.org%3E
> > >>
> > >> Jon
> > >>
> > >> On Fri, May 25, 2018 at 2:00 PM Justin Leet 
> > >> wrote:
> > >>
> > >>> At the risk of exposing my ignorance, do we have the bro plugin
> release
> > >>> process documented anywhere?  We have a doc for the main release (
> > >>> https://cwiki.apache.org/confluence/display/METRON/Release+Process),
> > >>> but I
> > >>> haven't noticed one for the bro plugin.
> > >>>
> > >>> For the current RC, it's not included and it wasn't pushed for (it
> has
> > >>> less
> > >>> changes for obvious reasons).  However, we should be making sure to
> > >>> validate if its necessary to release and having the process
> documented.
> > >>>
> > >>> Justin
> > >>>
> > >> --
> > >>
> > >> Jon
> > >>
> > > --
> > >
> > > Jon
> > >
> > --
> >
> > Jon
> >
>

Re: [DISCUSS] Split apart releases for core Metron and the Bro plugin

[Justin Leet]

https://github.com/apache/metron/pull/1188 scripts the creation of RC's.
It does this split, and allows for independent releases. The main catch is
a slight hangup with the KEYS file.

The metron-bro-kafka-plugin repo doesn't contain a KEYS file, it currently
has just been aligned with the main repo's KEYS file.The KEYS file doesn't
need to live in each submodule's directory, just in the main folder (
https://dist.apache.org/repos/dist/release/metron/).  This means it's not a
problem for the plugin repo to be missing it.

The main catch is that right now, we only publish the KEYS file when we
release the main repo. In the (admittedly rare) situation where we have a
new release manager who has to update the KEYS file, we'd currently need to
do a metron release before we could do a plugin release.

Should we split out and document a KEYS update from the main repo?
Essentially, just run through the normal PR process and then have a post
merge step of kicking up the updated KEYS file.  Then Release Process
<https://cwiki.apache.org/confluence/display/METRON/Release+Process> would
just get updated to reflect this process.


On Fri, Sep 7, 2018 at 3:15 PM Casey Stella  wrote:

> +1 to defer for this release and complete separation.  Good fences make
> good submodules. ;)
>
> On Fri, Sep 7, 2018 at 2:33 PM zeo...@gmail.com  wrote:
>
> > +1 to defer for this release and +1 to Justin's suggested release/dist
> > directory breakout and complete separation.
> >
> > Jon
> >
> > On Fri, Sep 7, 2018 at 1:43 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > +1 to deferring for this release and having the separation like NiFi.
> > Since
> > > we're bootstrapping from their process, what are they doing? I would
> > assume
> > > we'd want some sort of vote for the plugin version change as well.
> > >
> > > On Fri, Sep 7, 2018 at 10:15 AM Nick Allen  wrote:
> > >
> > > > +1 for complete separation as you've described.
> > > >
> > > > On Fri, Sep 7, 2018 at 11:31 AM Justin Leet 
> > > wrote:
> > > >
> > > > > I would like this to be a complete separation.  Complete with
> > separate
> > > > RCs,
> > > > > separate call to vote, etc. There's a bit more overhead, but plugin
> > > > > releases should be rarer and as the release infra gets improved and
> > > > > scripted out more, I don't think it'll end up being much more than
> > > > bundling
> > > > > it together.
> > > > >
> > > > > On Fri, Sep 7, 2018 at 11:27 AM Nick Allen 
> > wrote:
> > > > >
> > > > > > > Other projects, e.g. NiFi <http://apache.org/dist/nifi/>,
> split
> > > > apart
> > > > > > these releases within their dist directories.
> > > > > >
> > > > > > I prefer the way Nifi organizes it.  Definitely seems more
> > logically
> > > > > > organized.
> > > > > >
> > > > > >
> > > > > > > If we split them apart, we can make the releases independently.
> > > This
> > > > > > fixes the problem of aligning the versions (simply release the
> > plugin
> > > > > > first, update full-dev, release core Metron).
> > > > > >
> > > > > > Does this entail a complete separation; including a separate
> > > > > call-to-vote?
> > > > > > One vote for core Metron and a separate vote for plugin?
> > > > > >
> > > > > >
> > > > > > > Do we want try to get this separation done after the current
> > > release
> > > > > > cycle is over?
> > > > > >
> > > > > > +1 Let's wait for the next release to hash this out.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 7, 2018 at 10:27 AM Justin Leet <
> justinjl...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Right now, we tie together our main release and the Bro plugin,
> > as
> > > > seen
> > > > > > in
> > > > > > > our 0.4.2 release
> > > > > > > https://archive.apache.org/dist/metron/0.4.2/ and the current
> > RC.
> > > > > > >
> > > > > > > Other projects, e.g. NiFi <http://apache.org/dist/nifi/>,
> split
> > > 

Re: [DISCUSS] Feature Branch guidance

[Justin Leet]

+1 to both points.

I'm in favor of keeping architectural changes limited in a feature branch.
Architectural challenges beyond the scope of the branch should be brought
back to the community for any necessary discussion.

I don't think we've ever formalized what exactly closes out a feature
branch in terms of consensus. Typically we've been getting a few +1's and
calling it a day. It's probably worth it to formalize it while we're
already in the bylaws, assuming there's consensus on changing them.

In terms of wording, maybe something like the following (but better):

Feature Branch
Large feature changes may be made in a speculative feature branch. A
DISCUSS thread should be started on the primary project development mailing
list (dev@metron.apache.org) to propose the feature and outline minimum
architectural changes. Architectural changes are limited based on the
discuss thread unless further discussion occurs. To close the feature
branch, start a DISCUSS thread to outline branch state and solicit overall
feedback and requests. The branch can be committed after .

On Fri, Sep 28, 2018 at 2:26 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 to those 2 bullet points Casey. And thanks Justin for adding the Jira
> for fixing the website.
>
> I can think of 2 good examples to borrow from recently that were submitted
> by community contributors. Shane Ardell brought up a discussion about
> migrating from Protractor to Cypress, and Tamas Fodor brought up a
> discussion about migrating from momentjs to date-fns. This greases the
> skids by engaging community members and explaining the scope of proposed
> changes. As always, committers are able to -1 something at any time, so I
> would imagine that any contributor would be well advised to get as much buy
> in as possible prior to any large undertaking. And I would expect those
> PR's to reference the original DISCUSS threads when they come to fruition.
>
> Another example comes to mind from Ryan M with his PCAP feature branch. It
> was a lot of work, but Ryan put out a DISCUSS thread back in, I think it
> was May, outlining the intent for the FB. Subsequently, he followed up at
> the end with an accounting of all the requests from the original DISCUSS
> and any deviations from that along the way, and provided a clear
> explanation of what was in, what wasn't, what should be followed up with,
> and why. In fact, at one point I think there were some library changes that
> we saw as orthogonal to the intent of the FB and suggested they be made
> outside of the FB. Imho, this FB worked out well, though take this with a
> grain of salt as I may be biased because I was also involved with a number
> of PRs.
>
> I think Nick Allen also put together a good archetype for a FB with his
> recent work on the batch profiler. I see a couple introductory DISCUSS
> threads about the FB along with some back and forth around introducing
> Spark into the stack. He also followed up at the end to make sure there
> wasn't anything further the community wanted before he pushed to have the
> branch merged into master.
>
> *TL;DR* - We've learned a lot since the earlier days of the project, and
> our subsequent feature branches have gotten much better. We should take the
> lessons learned along the way and formalize them as Casey is recommending
> in our bylaws. I'll be following up with more specific thoughts on
> language.
>
> Best,
> Mike
>
>
> On Fri, Sep 28, 2018 at 10:13 AM Justin Leet 
> wrote:
>
> > Ticket created: https://issues.apache.org/jira/browse/METRON-1799
> >
> > I think that whole '/develop' is orphaned and can be dropped.
> >
> > On Fri, Sep 28, 2018 at 12:12 PM Casey Stella 
> wrote:
> >
> > > I just noticed this, but googling "metron bylaws" yields
> > > http://metron.apache.org/develop/bylaws.html which is not our bylaws.
> > Our
> > > bylaws are on
> > >
> https://cwiki.apache.org/confluence/display/METRON/Apache+Metron+Bylaws
> > >
> > > We should fix that.
> > >
> > > On Fri, Sep 28, 2018 at 12:02 PM Casey Stella 
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Given discussions about the current high-profile feature branch (Knox
> > > > SSO), I thought it might be appropriate to have a conversation about
> > what
> > > > constitutes a feature branch and get some of this encoded in the
> > > community
> > > > guidelines.
> > > >
> > > > Specifically, there was the request made that we split up the Knox
> SSO
> > > > feature branch due to the current implementation including a
> distinct,
> > > new
> > > > a

Re: [DISCUSS] Feature Branch guidance

[Justin Leet]

Ticket created: https://issues.apache.org/jira/browse/METRON-1799

I think that whole '/develop' is orphaned and can be dropped.

On Fri, Sep 28, 2018 at 12:12 PM Casey Stella  wrote:

> I just noticed this, but googling "metron bylaws" yields
> http://metron.apache.org/develop/bylaws.html which is not our bylaws.  Our
> bylaws are on
> https://cwiki.apache.org/confluence/display/METRON/Apache+Metron+Bylaws
>
> We should fix that.
>
> On Fri, Sep 28, 2018 at 12:02 PM Casey Stella  wrote:
>
> > Hi All,
> >
> > Given discussions about the current high-profile feature branch (Knox
> > SSO), I thought it might be appropriate to have a conversation about what
> > constitutes a feature branch and get some of this encoded in the
> community
> > guidelines.
> >
> > Specifically, there was the request made that we split up the Knox SSO
> > feature branch due to the current implementation including a distinct,
> new
> > architectural change (specifically, in my mind, the migration from
> > expressjs to Spring Boot + Zuul).  In that discussion, I made the
> assertion
> > that "for my mind a feature branch should involve the minimum
> > architectural change to accomplish a given feature." (I apologize for
> > quoting myself here ;) . I realize, however, that we have not encoded
> > this in our bylaws and I might not be speaking with the authority of the
> > community at my back.
> >
> > Ultimately, I feel very sad that we didn't get around to clarifying this
> > and we're in a situation where, had we made this clarification earlier,
> > we'd not be in a situation where a contributor has to make a major
> > refactoring to a substantial feature.  To that end, I think we should
> > hammer this out once and for all so it's clear.
> >
> > So, I'd like to separate out this discussion here.  My justification for
> > my belief (beyond that I think it's the correct thing to do) was the
> > precedent set with 777, admittedly pre-feature branches, wherein we
> > requested a series of splits to get to smaller units of functionality and
> > isolate large architectural changes.  I think that it is good practice to
> > isolate major feature changes to separate feature branches to ensure that
> > we get sufficient discussion around each of those changes.  To that end,
> > I'd like some language encoded into the bylaws describing this.
> >
> > So, I'm opening up the discussion.  Most specifically what I'd like to
> > know is:
> >
> >- What do you think about the idea that feature branches should
> >contain the minimal architectural changes to accomplish a feature
> (unless
> >the full scope of architectural changes are mentioned in the discuss
> thread
> >about the feature)?
> >- If you agree (or disagree), do you think a change to the bylaws is
> >merited to encode this policy?  If you do think so, then what wording
> would
> >you suggest?
> >
> >
> > Hopefully that all makes sense.
> >
> > Casey
> >
>

Re: Metron dev environments moving to require Ansible 2.4+

[Justin Leet]

I'm +1 on getting the PR merged in.

I'd just follow up on this thread post merge to let everyone know they have
to switch if they haven't.

On Fri, Sep 28, 2018 at 9:32 AM zeo...@gmail.com  wrote:

> Hi All,
>
> As it currently sits, once METRON-1758
>  is merged into the code
> base, Ansible 2.4 or later will be required to use any of the Metron
> ansible playbooks.  This is in contrast to the prior version requirements
> outlined in Metron documentation which specifically point to 2.0.0.2 and
> 2.2.0.0 as supported/recommended Ansible versions.  If you install Ansible
> 2.5.0 exactly you should not experience any issues spinning up pre- and post-
> merge versions of Metron.
>
> I am broadcasting this to both the user and dev communities in advance of
> any changes to provide an opportunity to voice any concerns.  Thanks,
>
> Jon
> --
>
> Jon
>

Re: [DISCUSS] Batch Profiler Feature Branch

[Justin Leet]

gt; > >> > > form the batch job runs over all 9 months?
> >> > >> > >
> >> > >> > > On Thu, Sep 20, 2018 at 11:13 AM Nick Allen <
> n...@nickallen.org>
> >> > >> wrote:
> >> > >> > >
> >> > >> > > > > How do we establish "tm" from 1.1 above? Any concerns about
> >> > >> overlap
> >> > >> > or
> >> > >> > > > gaps after the seeding is performed?
> >> > >> > > >
> >> > >> > > > Good point.  Right now, if the Streaming and Batch Profiler
> >> > overlap
> >> > >> the
> >> > >> > > > last write wins.  And presumably the output of the Streaming
> >> and
> >> > >> Batch
> >> > >> > > > Profiler are the same, so no worries, right? :)
> >> > >> > > >
> >> > >> > > > So it kind of works, but it is definitely not ideal for use
> >> case
> >> > >> 3.  I
> >> > >> > > > could add --begin and --end args to constrain the time frame
> >> over
> >> > >> which
> >> > >> > > the
> >> > >> > > > Batch Profiler runs.  I do not have that in the feature
> branch.
> >> > It
> >> > >> > would
> >> > >> > > > be easy enough to add though.
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > On Thu, Sep 20, 2018 at 12:41 PM Michael Miklavcic <
> >> > >> > > > michael.miklav...@gmail.com> wrote:
> >> > >> > > >
> >> > >> > > > > Ok, makes sense. That's sort of what I was thinking as
> well,
> >> > Nick.
> >> > >> > > > Pulling
> >> > >> > > > > at this thread just a bit more...
> >> > >> > > > >
> >> > >> > > > >1. I have an existing system that's been up a while,
> and I
> >> > have
> >> > >> > > added
> >> > >> > > > k
> >> > >> > > > >profiles - assume these are the first profiles I've
> >> created.
> >> > >> > > > >   1. I would have t0 - tm (where m is the time when the
> >> > >> profiles
> >> > >> > > were
> >> > >> > > > >   first installed) worth of data that has not been
> >> profiled
> >> > >> yet.
> >> > >> > > > >   2. The batch profiler process would be to take that
> >> exact
> >> > >> > profile
> >> > >> > > > >   definition from ZK and run the batch loader with that
> >> from
> >> > >> the
> >> > >> > > CLI.
> >> > >> > > > >   3. Profiles are now up to date from t0 - tCurrent
> >> > >> > > > >2. I've already done #1 above. Time goes by and now I
> >> want to
> >> > >> add
> >> > >> > a
> >> > >> > > > new
> >> > >> > > > >profile.
> >> > >> > > > >   1. Same first step above
> >> > >> > > > >   2. I would run the batch loader with *only* that new
> >> > profile
> >> > >> > > > >   definition to seed?
> >> > >> > > > >
> >> > >> > > > > Forgive me if I missed this in PR's and discussion in the
> FB,
> >> > but
> >> > >> how
> >> > >> > > do
> >> > >> > > > we
> >> > >> > > > > establish "tm" from 1.1 above? Any concerns about overlap
> or
> >> > gaps
> >> > >> > after
> >> > >> > > > the
> >> > >> > > > > seeding is performed?
> >> > >> > > > >
> >> > >> > > > > On Thu, Sep 20, 2018 at 10:26 AM Nick Allen <
> >> n...@nickallen.org
> >> > >
> >> > >> > > wrote:
> >> > >> > > > 

Re: [DISCUSS] Batch Profiler Feature Branch

[Justin Leet]

I think the main difference between this and the flatfile loader is that we
actively maintain our profiles in ZK for streaming.  Doing this from files
is likely going to be the main usage, particularly for speculative usage.

For me, the main use case for ZK is definitely use case 3.

I can definitely be persuaded that this isn't a blocker for right now, but
I think there will be problems in practice from not having the
functionality. E.g. "We want to refresh everything because of mistake X,
and nobody refreshed the file/ZK and they've diverged".  While nobody likes
to refresh prod data (or some subset), I have seen it happen in literally
every single project I've worked on.  On dev/integration environments this
is even more likely.  Most people probably aren't going to store these
files in their version control (even though they probably should) and these
sort of divergences will happen.

 It's just cleaner from a usage/management perspective to say "I want to
put a profile in prod, just use streaming profiler and the batch profiler
with the same setup and they're good to go."

On Thu, Sep 20, 2018 at 12:41 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Ok, makes sense. That's sort of what I was thinking as well, Nick. Pulling
> at this thread just a bit more...
>
>1. I have an existing system that's been up a while, and I have added k
>profiles - assume these are the first profiles I've created.
>   1. I would have t0 - tm (where m is the time when the profiles were
>   first installed) worth of data that has not been profiled yet.
>   2. The batch profiler process would be to take that exact profile
>   definition from ZK and run the batch loader with that from the CLI.
>   3. Profiles are now up to date from t0 - tCurrent
>2. I've already done #1 above. Time goes by and now I want to add a new
>profile.
>   1. Same first step above
>   2. I would run the batch loader with *only* that new profile
>   definition to seed?
>
> Forgive me if I missed this in PR's and discussion in the FB, but how do we
> establish "tm" from 1.1 above? Any concerns about overlap or gaps after the
> seeding is performed?
>
> On Thu, Sep 20, 2018 at 10:26 AM Nick Allen  wrote:
>
> > I think more often than not, you would want to load your profile
> definition
> > from a file.  This is why I considered the 'load from Zk' more of a
> > nice-to-have.
> >
> >- In use case 1 and 2, this would definitely be the case.  The
> profiles
> >I am working with are speculative and I am using the batch profiler to
> >determine if they are worth keeping.  In this case, my speculative
> > profiles
> >would not be in Zk (yet).
> >- In use case 3, I could see it go either way.  It might be useful to
> >load from Zk, but it certainly isn't a blocker.
> >
> >
> > > So if the config does not correctly match the profiler config held in
> ZK
> > and
> > the user runs the batch seeding job, what happens?
> >
> > You would just get a profile that is slightly different over the entire
> > time span.  This is not a new risk.  If the user changes their Profile
> > definitions in Zk, the same thing would happen.
> >
> >
> > On Thu, Sep 20, 2018 at 12:15 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > I think I'm torn on this, specifically because it's batch and would
> > > generally be run as-needed. Justin, can you elaborate on your concerns
> > > there? This feels functionally very similar to our flat file loaders,
> > which
> > > all have inputs for config from the CLI only. On the other hand, our
> flat
> > > file loaders are not typically seeding an existing structure. My
> concern
> > of
> > > a local file profiler config stems from this stated goal:
> > > > The goal would be to enable “profile seeding” which allows profiles
> to
> > be
> > > populated from a time before the profile was created.
> > > So if the config does not correctly match the profiler config held in
> ZK
> > > and the user runs the batch seeding job, what happens?
> > >
> > > On Thu, Sep 20, 2018 at 10:06 AM Justin Leet 
> > > wrote:
> > >
> > > > The profile not being able to read from ZK feels like a fairly
> > > substantial,
> > > > if subtle, set of potential problems.  I'd like to see that in either
> > > > before merging or at least pretty soon after merging.  Is it a lot of
> > > work
> > > > to add that functionality based on where things are right now?
> > > >

Re: [DISCUSS] Batch Profiler Feature Branch

[Justin Leet]

The profile not being able to read from ZK feels like a fairly substantial,
if subtle, set of potential problems.  I'd like to see that in either
before merging or at least pretty soon after merging.  Is it a lot of work
to add that functionality based on where things are right now?

On Thu, Sep 20, 2018 at 9:59 AM Nick Allen  wrote:

> Here is another limitation that I just thought. It can only read a profile
> definition from a file.  It probably also makes sense to add an option that
> allows it to read the current Profiler configuration from Zookeeper.
>
>
> > Is it worth setting up a default config that pulls from the main indexing
> output?
>
> Yes, I think that makes sense.  We want the Batch Profiler to point to the
> right HDFS URL, no matter where/how Metron is deployed.  When Metron gets
> spun-up on a cluster, I should be able to just run the Batch Profiler
> without having to fuss with the input path.
>
>
>
>
>
> On Thu, Sep 20, 2018 at 9:46 AM Justin Leet  wrote:
>
> > Re:
> >
> > >  * You do not configure the Batch Profiler in Ambari.  It is configured
> > > and executed completely from the command-line.
> > >
> >
> > Is it worth setting up a default config that pulls from the main indexing
> > output?  I'm a little on the fence about it, but it seems like making the
> > most common case more or less built-in would be nice.
> >
> > Having said that, I do not consider that a requirement for merging the
> > feature branch.
> >
> > On Wed, Sep 19, 2018 at 11:23 AM James Sirota 
> wrote:
> >
> > > I think what you have outlined above is a good initial stab at the
> > > feature.  Manual install of spark is not a big deal.  Configuring via
> > > command line while we mature this feature is ok as well.  Doesn't look
> > like
> > > configuration steps are too hard.  I think you should merge.
> > >
> > > James
> > >
> > > 19.09.2018, 08:15, "Nick Allen" :
> > > > I would like to open a discussion to get the Batch Profiler feature
> > > branch
> > > > merged into master as part of METRON-1699 [1] Create Batch Profiler.
> > All
> > > > of the work that I had in mind for our first draft of the Batch
> > Profiler
> > > > has been completed. Please take a look through what I have and let me
> > > know
> > > > if there are other features that you think are required *before* we
> > > merge.
> > > >
> > > > Previous list discussions on this topic include [2] and [3].
> > > >
> > > > (Q) What can I do with the feature branch?
> > > >
> > > >   * With the Batch Profiler, you can backfill/seed profiles using
> > > archived
> > > > telemetry. This enables the following types of use cases.
> > > >
> > > >   1. As a Security Data Scientist, I want to understand the
> > > historical
> > > > behaviors and trends of a profile that I have created so that I can
> > > > determine if I have created a feature set that has predictive value
> for
> > > > model building.
> > > >
> > > >   2. As a Security Data Scientist, I want to understand the
> > > historical
> > > > behaviors and trends of a profile that I have created so that I can
> > > > determine if I have defined the profile correctly and created a
> feature
> > > set
> > > > that matches reality.
> > > >
> > > >   3. As a Security Platform Engineer, I want to generate a
> profile
> > > > using archived telemetry when I deploy a new model to production so
> > that
> > > > models depending on that profile can function on day 1.
> > > >
> > > >   * METRON-1699 [1] includes a more detailed description of the
> > feature.
> > > >
> > > > (Q) What work was completed?
> > > >
> > > >   * The Batch Profiler runs on Spark and was implemented in Java to
> > > remain
> > > > consistent with our current Java-heavy code base.
> > > >
> > > >   * The Batch Profiler is executed from the command-line. It can be
> > > > launched using a script or by calling `spark-submit`, which may be
> > useful
> > > > for advanced users.
> > > >
> > > >   * Input telemetry can be consumed from multiple sources; for
> example
> > > HDFS
> > > > or the local file system.
> > > >
> > > >   * Input telemetry can be consu

[ANNOUNCE] Apache Metron release 0.6.0

[Justin Leet]

Hi All,

I’m happy to announce the release of Metron 0.6.0! There's a been a lot of
great work everywhere on the project, and thanks to both everyone who
contributed and our users.

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/0.6.0

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/0.6.0

The release branches in github is
https://github.com/apache/metron/tree/Metron_0.6.0 (tag
apache-metron-0.6.0-release)

The release doc book is at http://metron.apache.org/current-book/index.html
The Apache Metron web site at http://metron.apache.org/ has been updated;
please refresh your web browser cache if the new links do not immediately
appear.

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

Metron CHANGES (in reverse chronological order):

METRON-1764 Update version to 0.6.0 (justinleet) closes apache/metron#1183
METRON-1751 Storm Profiler dies when consuming null message
(nickwallen) closes apache/metron#1176
METRON-1757 Storm Profiler Serialization Exception (nickwallen)
closes apache/metron#1178
METRON-1743 CEF testPaloAltoCEF test using a confusing variable
name (JonZeolla via justinleet) closes apache/metron#1173
METRON-1752 Prevent package.lock from changing during build
(sardell via merrimanr) closes apache/metron#1177
METRON-1724 Date/time validation missing in PCAP query (tiborm via
nickwallen) closes apache/metron#1172
METRON-1739 UDP packets are not handled (merrimanr) closes
apache/metron#1168
METRON-1727: Alerts are not populated on the alerts UI after
enabling X-pack for Elastic search (MohanDV via mmiklavc) closes
apache/metron#1141
METRON-1738: Pcap directories should have correct permissions
(merrimanr via mmiklavc) closes apache/metron#1166
METRON-1737: Document Job cleanup (merrimanr via mmiklavc) closes
apache/metron#1164
METRON-1732: Fix job status liveness bug and parallelize finalizer
file writing (mmiklavc via mmiklavc) closes apache/metron#1157
METRON-1735 Empty print status option causes NPE (merrimanr)
closes apache/metron#1160
METRON-1733 PCAP UI - PCAP queries dont work on Safari
(sardell via merrimanr) closes apache/metron#1158
METRON-1734 Src and Dst port filters are incorrect after changing
to empty (merrimanr) closes apache/metron#1159
METRON-1725 Add ability to specify YARN queue for pcap jobs
(merrimanr) closes apache/metron#1153
METRON-1731: PCAP - Escape colons in output dir names (mmiklavc
via mmiklavc) closes apache/metron#1155
METRON-1702 Reload a running job in the UI (merrimanr) closes
apache/metron#1156
METRON-1722 PcapCLI should print progress to stdout (merrimanr)
closes apache/metron#1138
METRON-1728: Handle null values in config in Pcap backend more
gracefully (mmiklavc via mmiklavc) closes apache/metron#1151
METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc via
mmiklavc) closes apache/metron#1152
METRON-1713 PCAP UI - Add a way to kill a pcap job (tiborm via
merrimanr) closes apache/metron#1143
METRON-1723 PCAP UI - Unable to select/copy from packets details
in PCAP query panel (sardell via merrimanr) closes apache/metron#1139
METRON-1712 PCAP UI - Input validation (tiborm via merrimanr)
closes apache/metron#1142
METRON-1720 Better error messages when there are no results or
wireshark is not installed (merrimanr) closes apache/metron#1154
METRON-1726: Refactor PcapTopologyIntegrationTest (mmiklavc via
mmiklavc) closes apache/metron#1140
METRON-1683 PCAP UI - Fix the download progress bar (sardell via
merrimanr) closes apache/metron#1122
METRON-1675 PCAP UI - Introduce the paging capability (sardell via
merrimanr) closes apache/metron#1121
METRON-1721 New default input path is wrong in pcap CLI
(merrimanr) closes apache/metron#1137
METRON-1676 PCAP UI - Add data range selector to the filter bar
(tiborm via merrimanr) closes apache/metron#1119
METRON-1662 PCAP UI - Downloading PCAP page files (tiborm via
merrimanr) closes apache/metron#1118
METRON-1700 Create REST endpoint to get job configuration
(merrimanr) closes apache/metron#1135
METRON-1671 Create PCAP UI (tiborm via merrimanr) closes apache/metron#1103
METRON-1701 Update General notes on the installation of Pycapa on
Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
METRON-1650 Packaging docker containers are too large (jameslamb
via merrimanr) closes apache/metron#1091
METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
management pack repo info closes apache/incubator-metron#1052
METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
apache/incubator-metron#1126
METRON-1694: Clean up Metron REST docs closes apache/incubator-metron#1131

Re: [MENTORS][DISCUSS] LICENSE and NOTICE likely outdated

[Justin Leet]

There is a distinction. The dependencies_with_url.csv does manage to make
sure our dependencies (and transitive dependencies) are appropriately
accounted for.  What we also need to do is make that any changes (if
necessary) to the LICENSE and NOTICE files also make it in there.  For
example, certain attributions may be necessary in the NOTICE file.  Similar
things can happen with the LICENSE file (e.g. including licenses from
dependencies from bundled code).  It's possible that we don't have any
problems, but I also expect that since we haven't been actively maintaining
it that there might be issues.  E.g. if the UI has pulled in anything
bundled in our source, modifications to the L may need to be made.

As far as uberjars go, I believe we're fine. Like you said, we aren't
distributing them, so they aren't bundled.

Otto, you mentioned on Slack that NiFi requires some checking in having PRs
reviewed and in reviewing PRs. Could you share your experience there?

On Wed, Sep 12, 2018 at 1:36 PM Otto Fowler  wrote:

> Are you referring to the dependencies check against the csv?
>
>
> On September 12, 2018 at 13:09:48, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> I'm not sure I fully understand what is out of date. I know I have
> personally modified our licenses a couple times in the past and used an
> automated script that, I believe, Casey Stella had created for doing the
> check. I even made some improvements to it a long ways back. It rips
> through the maven dependency tree and tells you what isn't in the licenses
> file and fails with a non-zero return code. I thought that was part of our
> Travis build, or at the very least, the release lifecycle. Is that not the
> case, or is there a different context we're talking about here?
>
> I understand that convenience binaries might some issues with uberjars when
> we go that route for 1.0. But is there any issue with the uberjars as
> things currently stand? I was under the impression we are OK because we
> don't distribute them. It's part of the build, just like tools such as
> JUnit, that we don't actually ship.
>
> Justin - These are the links for guidance that I've found. Is anything else
> you've found that we should peruse while figuring this out?
>
> - https://www.apache.org/dev/licensing-howto.html
> - http://www.apache.org/legal/release-policy.html#artifacts
>
> Mike
>
>
> On Wed, Sep 12, 2018 at 10:29 AM Justin Leet 
> wrote:
>
> > Hi all,
> >
> > As mentioned on the release voting thread, there was a Slack discussion
> > around our LICENSE and NOTICE file likely being outdated because they
> > haven't been actively kept up to date since graduation. I suggested on
> the
> > vote thread that we proceed with the current release, but consider it a
> > blocker for the next release.
> >
> > Mentor input on this (and how other projects handle it), would be greatly
> > appreciated.
> >
> > This discussion should result in JIRAs that are brought back to the
> thread,
> > so we can make sure to track this.
> >
> > For context, in addition to the standard L management, when we build
> > artifacts we shade a lot of jars into a uberjars, thus bundling
> > dependencies. However, our current releases are source only, but
> > publishing convenience binaries came up in the 1.0 roadmap thread.
> >
> > I think there are a few things that need to happen to correct our current
> > issue and make this easier in the future.
> > 1) Get the LICENSE and NOTICE files up to date
> > 2) Document the process we went through getting things up to date and
> (just
> > as importantly) the reasoning behind it.
> > 3) Update the PR checklist to include LICENSE and NOTICE files for new
> (and
> > transitive) dependencies.
> > 4) Update or add any processes we need to maintain this properly (e.g.
> > release auditing)
> > 5) Possibly build tooling for making some of this auditing easier (or use
> > existing tool if anyone has suggestions)?
> >
> > Are there any other steps I'm missing that need to go into JIRAs?
> > Any other concerns regarding these files that need to be addressed?
> > Any other context I'm missing and that belongs in this discussion?
> >
>

[MENTORS][DISCUSS] LICENSE and NOTICE likely outdated

[Justin Leet]

Hi all,

As mentioned on the release voting thread, there was a Slack discussion
around our LICENSE and NOTICE file likely being outdated because they
haven't been actively kept up to date since graduation.  I suggested on the
vote thread that we proceed with the current release, but consider it a
blocker for the next release.

Mentor input on this (and how other projects handle it), would be greatly
appreciated.

This discussion should result in JIRAs that are brought back to the thread,
so we can make sure to track this.

For context, in addition to the standard L management, when we build
artifacts we shade a lot of jars into a uberjars, thus bundling
dependencies.  However, our current releases are source only, but
publishing convenience binaries came up in the 1.0 roadmap thread.

I think there are a few things that need to happen to correct our current
issue and make this easier in the future.
1) Get the LICENSE and NOTICE files up to date
2) Document the process we went through getting things up to date and (just
as importantly) the reasoning behind it.
3) Update the PR checklist to include LICENSE and NOTICE files for new (and
transitive) dependencies.
4) Update or add any processes we need to maintain this properly (e.g.
release auditing)
5) Possibly build tooling for making some of this auditing easier (or use
existing tool if anyone has suggestions)?

Are there any other steps I'm missing that need to go into JIRAs?
Any other concerns regarding these files that need to be addressed?
Any other context I'm missing and that belongs in this discussion?

[RESULT][VOTE] Metron Release Candidate 0.6.0-RC1

[Justin Leet]

The vote has passed.  Including my +1, the voting was:
3 binding +1’s
1 non-binding +1’s
no 0’s
no -1’s.

Re: [VOTE] Metron Release Candidate 0.6.0-RC1

[Justin Leet]

Hi all,

There was a discussion on Slack that I wanted to bring back to the list
regarding our LICENSE and NOTICE files.

These files appear to be stale since our graduation to TLP, however they
likely should have been updated throughout their lifetime.  I'll be
starting a discuss thread shortly regarding exactly how we go about getting
things into current state, documenting the process, etc.  The result of
this discuss thread will be a set of JIRAs for this issue.

Given that these issues have existed since graduation, I'm inclined to
continue with the release as-is and consider it a blocker for the next
release.

On Tue, Sep 11, 2018 at 6:09 PM James Sirota  wrote:

> +1 (binding) ran it up in full dev
>
> 07.09.2018, 06:30, "Justin Leet" :
> > This is a call to vote on releasing Apache Metron 0.6.0 and the Bro-Kafka
> > plugin 0.2.0
> >
> > Full list of changes in this release:
> > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES
> >
> https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES.bro-plugin
> >
> > The tags to be voted upon are:
> > (apache/metron) apache-metron-0.6.0-rc1
> > (apache/metron-bro-plugin-kafka) apache-metron-bro-plugin-kafka_0.2.0-rc1
> >
> > The source archives being voted upon can be found here:
> >
> https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-0.6.0-rc1.tar.gz
> >
> https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-bro-plugin-kafka_0.2.0-rc1.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/
> >
> > The release artifacts are signed with the following key:
> > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/KEYS
> >
> > Please vote on releasing this package as Apache Metron 0.6.0-RC1 and the
> > Bro-Kafka plugin as 0.2.0-RC1
> >
> > When voting, please list the actions taken to verify the release.
> >
> > Recommended build validation and verification instructions are posted
> > here:
> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >
> > This vote will be open for until 10pm EDT on Wednesday September 12 2018,
> > to account for the weekend.
> >
> > [ ] +1 Release this package as Apache Metron 0.3.0-RC1
> >
> > [ ] 0 No opinion
> >
> > [ ] -1 Do not release this package because...
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Re: [DISCUSS] Split apart releases for core Metron and the Bro plugin

[Justin Leet]

I would like this to be a complete separation.  Complete with separate RCs,
separate call to vote, etc. There's a bit more overhead, but plugin
releases should be rarer and as the release infra gets improved and
scripted out more, I don't think it'll end up being much more than bundling
it together.

On Fri, Sep 7, 2018 at 11:27 AM Nick Allen  wrote:

> > Other projects, e.g. NiFi <http://apache.org/dist/nifi/>, split apart
> these releases within their dist directories.
>
> I prefer the way Nifi organizes it.  Definitely seems more logically
> organized.
>
>
> > If we split them apart, we can make the releases independently.  This
> fixes the problem of aligning the versions (simply release the plugin
> first, update full-dev, release core Metron).
>
> Does this entail a complete separation; including a separate call-to-vote?
> One vote for core Metron and a separate vote for plugin?
>
>
> > Do we want try to get this separation done after the current release
> cycle is over?
>
> +1 Let's wait for the next release to hash this out.
>
>
>
>
> On Fri, Sep 7, 2018 at 10:27 AM Justin Leet  wrote:
>
> > Right now, we tie together our main release and the Bro plugin, as seen
> in
> > our 0.4.2 release
> > https://archive.apache.org/dist/metron/0.4.2/ and the current RC.
> >
> > Other projects, e.g. NiFi <http://apache.org/dist/nifi/>, split apart
> > these
> > releases within their dist directories.
> >
> > In our case this might look something like
> > 0.5.0/
> > metron-bro-plugin-kafka
> > - 0.2.0/
> >
> > Right now, with the releases tied together, we aren't upgrading full-dev
> > with the version of the plugin (because we're releasing simultaneously
> and
> > can't update the version number).
> >
> > If we split them apart, we can make the releases independently.  This
> fixes
> > the problem of aligning the versions (simply release the plugin first,
> > update full-dev, release core Metron).  The plugin also updates
> > substantially less often and we can just do those releases at a cadence
> we
> > choose.
> >
> > Any thoughts on doing this?
> > Do we want try to get this separation done after the current release
> cycle
> > is over?
> > If we do, do we have a preferred layout? I didn't see anything Apache
> > preferred in a quick search, but I definitely could have missed something
> > (and https://checker.apache.org/projs/nifi.html looks clean for NiFi,
> so I
> > assume it's fine.)
> >
>

[DISCUSS] Split apart releases for core Metron and the Bro plugin

[Justin Leet]

Right now, we tie together our main release and the Bro plugin, as seen in
our 0.4.2 release
https://archive.apache.org/dist/metron/0.4.2/ and the current RC.

Other projects, e.g. NiFi , split apart these
releases within their dist directories.

In our case this might look something like
0.5.0/
metron-bro-plugin-kafka
- 0.2.0/

Right now, with the releases tied together, we aren't upgrading full-dev
with the version of the plugin (because we're releasing simultaneously and
can't update the version number).

If we split them apart, we can make the releases independently.  This fixes
the problem of aligning the versions (simply release the plugin first,
update full-dev, release core Metron).  The plugin also updates
substantially less often and we can just do those releases at a cadence we
choose.

Any thoughts on doing this?
Do we want try to get this separation done after the current release cycle
is over?
If we do, do we have a preferred layout? I didn't see anything Apache
preferred in a quick search, but I definitely could have missed something
(and https://checker.apache.org/projs/nifi.html looks clean for NiFi, so I
assume it's fine.)

Re: [VOTE] Metron Release Candidate 0.6.0-RC1

[Justin Leet]

As full disclosure, there's a slight difference in the naming of the of the
RC artifacts for the plugin. Namely, that it actually the RC number.  This
breaks the verification script as-is.  I have an updated version added here
https://github.com/apache/metron/pull/1188/files.  If we feel it's
necessary to update that file for this release, I can quick put out a PR,
cancel the vote, and pretty easily put out a new RC with the script.

Justin

On Fri, Sep 7, 2018 at 9:30 AM Justin Leet  wrote:

> This is a call to vote on releasing Apache Metron 0.6.0 and the Bro-Kafka
> plugin 0.2.0
>
> Full list of changes in this release:
> https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES
> https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES.bro-plugin
>
> The tags to be voted upon are:
> (apache/metron) apache-metron-0.6.0-rc1
> (apache/metron-bro-plugin-kafka) apache-metron-bro-plugin-kafka_0.2.0-rc1
>
> The source archives being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-0.6.0-rc1.tar.gz
>
> https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-bro-plugin-kafka_0.2.0-rc1.tar.gz
>
> Other release files, signatures and digests can be found here:
> https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/
>
> The release artifacts are signed with the following key:
> https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/KEYS
>
> Please vote on releasing this package as Apache Metron 0.6.0-RC1 and the
> Bro-Kafka plugin as 0.2.0-RC1
>
> When voting, please list the actions taken to verify the release.
>
> Recommended build validation and verification instructions are posted
> here:
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
> This vote will be open for until 10pm EDT on Wednesday September 12 2018,
> to account for the weekend.
>
> [ ] +1 Release this package as Apache Metron 0.3.0-RC1
>
> [ ] 0 No opinion
>
> [ ] -1 Do not release this package because...
>

[VOTE] Metron Release Candidate 0.6.0-RC1

[Justin Leet]

This is a call to vote on releasing Apache Metron 0.6.0 and the Bro-Kafka
plugin 0.2.0

Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES
https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES.bro-plugin

The tags to be voted upon are:
(apache/metron) apache-metron-0.6.0-rc1
(apache/metron-bro-plugin-kafka) apache-metron-bro-plugin-kafka_0.2.0-rc1

The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-0.6.0-rc1.tar.gz
https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-bro-plugin-kafka_0.2.0-rc1.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/

The release artifacts are signed with the following key:
https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/KEYS

Please vote on releasing this package as Apache Metron 0.6.0-RC1 and the
Bro-Kafka plugin as 0.2.0-RC1

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted
here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This vote will be open for until 10pm EDT on Wednesday September 12 2018,
to account for the weekend.

[ ] +1 Release this package as Apache Metron 0.3.0-RC1

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [DISCUSS] Feature branches post-merge

[Justin Leet]

I'm also inclined to drop them. They're in master and only exist to
facilitate dev. It seems unlikely to be a common case where someone wants
to really dig through the actual feature branch history rather than what
ended up in master.

On Thu, Sep 6, 2018 at 1:49 PM Casey Stella  wrote:

> I’d get rid of them.
> On Thu, Sep 6, 2018 at 13:42 Michael Miklavcic <
> michael.miklav...@gmail.com>
> wrote:
>
> > What are we doing with feature branches once they're complete and merged
> > into master? Is our expectation that we'll keep feature branches in
> > perpetuity, or should we plan to do some house cleaning once they've been
> > merged? I did a quick check of NiFi and Kafka and don't see much by way
> of
> > feature branches in their repos. I see plenty of RC's in both the
> branches
> > and tags listings, but nothing FB related. In previous discussions, we
> > talked quite a bit about us "trailblazing here," so it may be that this
> is
> > simply without much precedent and entirely for us to decide. I can
> > definitely see value in maintaining them for future reference, as it does
> > offer a nice bucket in which to collect the commits and discussion
> nicely,
> > but I wanted to get others' thoughts.
> >
> > Best,
> > Mike
> >
>

Re: [DISCUSS] Metron Release 0.6.0?

[Justin Leet]

Do we use the artifacts directly at all? Or is it through bro-pkg only?

Also, It's very possible I'm making a mountain out of a molehill here, and
if it's something that's not particularly important, it might be worthwhile
to just stick with what we did last time, and just table this discussion
until post release.  It feels pretty nitpicky at this point, and if the
practical implications are pretty minor, I'd rather just get an RC out.

On Thu, Sep 6, 2018 at 10:02 AM zeo...@gmail.com  wrote:

> Either is fine with me.  If it's x.y in some parts of the app I prefer to
> keep it consistent throughout, but I'm also fine with lining up with
> Apache/Metron where we can.
>
> I also refreshed myself on why we avoided x.y.z initially and it was
> actually for this exact reason, we wanted consistent versioning throughout
> a repo.  This issue is with the bro plugins themselves, not bro-pkg, so I
> submit a JIRA <https://bro-tracker.atlassian.net/browse/BIT-1985>.
>
> Jon
>
>
> On Wed, Sep 5, 2018, 21:51 Justin Leet  wrote:
>
> > Makes sense. Do we have any objection to just going to the artifact being
> > 0.2?  Or do we want to keep the mixed versioning and just live with it,
> at
> > least for now?
> >
> > On Wed, Sep 5, 2018 at 8:58 PM zeo...@gmail.com 
> wrote:
> >
> > > I think mattf-horton just did that as a part of convention.  He handled
> > > that part, and I did the 0.1 tagging (as a prereq to this
> > > <
> > >
> >
> https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30
> > > >)
> > > last time the package was released.
> > >
> > > Jon
> > >
> > > On Wed, Sep 5, 2018 at 8:28 PM Justin Leet 
> > wrote:
> > >
> > > > Any idea why we released it as 0.1.0 in the artifacts version?  I'm
> > fine
> > > > with doing x.y if we need to, but I would like the artifact
> versioning
> > to
> > > > be consistent if possible.
> > > >
> > > > On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com 
> > > wrote:
> > > >
> > > > > I lied, we didn't need to update our btests because it's limited
> to a
> > > > major
> > > > > and minor version.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34
> > > > >
> > > > > Jon
> > > > >
> > > > > On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com 
> > > > wrote:
> > > > >
> > > > > > I looked into x.y.z back when we released 0.1 and it was not
> > possible
> > > > in
> > > > > > bro-pkg at the time but now it is
> > > > > > <https://github.com/bro/package-manager/issues/32>.  In order to
> > do
> > > > > this,
> > > > > > we'll also need to configure bro-pkg.meta to require the proper
> > > version
> > > > > of
> > > > > > bro-pkg, as well as update the btests for the new version string.
> > I
> > > > will
> > > > > > throw together a JIRA and PR to do all this in case we decide to
> > > align
> > > > > with
> > > > > > x.y.z; we can trash it if we decide to stay with x.y.
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Wed, Sep 5, 2018 at 7:35 PM Justin Leet <
> justinjl...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Long story short, while preparing the release candidate, I
> > > discovered
> > > > > our
> > > > > >> metron-bro-plugin-kafka is inconsistently versioned.
> > > > > >>
> > > > > >> In the repo, it's x.y (e.g. 0.2)
> > > > > >> See:
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18
> > > > > >>
> > > > > >> In our released artifact, it's x.y.z (e.g. 0.1.0)
> > > > > >> See http://archive.apache.org/dist/metron/0.4.2/
> > > > > >>
> > > > > >> Going forward from this release, I'd like the artifact to be
> > > > consistent
> > > > > >> with the repo.  I'd personally pre

Re: [DISCUSS] Metron Release 0.6.0?

[Justin Leet]

Makes sense. Do we have any objection to just going to the artifact being
0.2?  Or do we want to keep the mixed versioning and just live with it, at
least for now?

On Wed, Sep 5, 2018 at 8:58 PM zeo...@gmail.com  wrote:

> I think mattf-horton just did that as a part of convention.  He handled
> that part, and I did the 0.1 tagging (as a prereq to this
> <
> https://github.com/apache/metron/commit/2e78df67c12a6fcad726551128e9753ad36d5ee9#diff-8e3bdd364219306b1fad91047208e6e4R30
> >)
> last time the package was released.
>
> Jon
>
> On Wed, Sep 5, 2018 at 8:28 PM Justin Leet  wrote:
>
> > Any idea why we released it as 0.1.0 in the artifacts version?  I'm fine
> > with doing x.y if we need to, but I would like the artifact versioning to
> > be consistent if possible.
> >
> > On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com 
> wrote:
> >
> > > I lied, we didn't need to update our btests because it's limited to a
> > major
> > > and minor version.
> > >
> > >
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34
> > >
> > > Jon
> > >
> > > On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > I looked into x.y.z back when we released 0.1 and it was not possible
> > in
> > > > bro-pkg at the time but now it is
> > > > <https://github.com/bro/package-manager/issues/32>.  In order to do
> > > this,
> > > > we'll also need to configure bro-pkg.meta to require the proper
> version
> > > of
> > > > bro-pkg, as well as update the btests for the new version string.  I
> > will
> > > > throw together a JIRA and PR to do all this in case we decide to
> align
> > > with
> > > > x.y.z; we can trash it if we decide to stay with x.y.
> > > >
> > > > Jon
> > > >
> > > > On Wed, Sep 5, 2018 at 7:35 PM Justin Leet 
> > > wrote:
> > > >
> > > >> Long story short, while preparing the release candidate, I
> discovered
> > > our
> > > >> metron-bro-plugin-kafka is inconsistently versioned.
> > > >>
> > > >> In the repo, it's x.y (e.g. 0.2)
> > > >> See:
> > > >>
> > >
> >
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18
> > > >>
> > > >> In our released artifact, it's x.y.z (e.g. 0.1.0)
> > > >> See http://archive.apache.org/dist/metron/0.4.2/
> > > >>
> > > >> Going forward from this release, I'd like the artifact to be
> > consistent
> > > >> with the repo.  I'd personally prefer x.y.z to be entirely
> consistent
> > > >> throughout Metron, but if there's a particular reason why it was x.y
> > I'm
> > > >> happy to entertain it.
> > > >>
> > > >> If we choose to move to x.y.z, I can provide a PR to update the
> > version
> > > to
> > > >> 0.2.0 unless someone else wants to volunteer. Otherwise, I'd like to
> > > >> release the artifact as apache-metron-bro-plugin-kafka_0.2.tar.gz
> > > >>
> > > >> Justin
> > > >>
> > > >> On Tue, Sep 4, 2018 at 12:15 PM Justin Leet 
> > > >> wrote:
> > > >>
> > > >> > As an update, I'll be working on starting the release process
> > rolling
> > > >> > today. The PCAP Query panel feature branch is in master and I
> > haven't
> > > >> heard
> > > >> > of any other potential blockers. I'd appreciate everyone updating
> > > their
> > > >> > JIRAs with their state (complete, etc.) and version (0.6.0) as
> > needed.
> > > >> >
> > > >> > Do we have anything that needs to go into the UPGRADING.md file?
> > > >> >
> > > >> > A PR is out for updating the version from 0.5.1 to 0.6.0. Please
> see
> > > >> > https://github.com/apache/metron/pull/1183. I still need to spin
> up
> > > >> full
> > > >> > dev etc. before this is ready to be merged.
> > > >> >
> > > >> > Updated list of PRs that have made it into master as of Sept 4,
> 2018
> > > >> >
> > > >> > 6 days ago METRON-1751 Storm Profiler dies when consuming null
> > message
> > > >> > (nickwallen) closes apache/metron#1176
>

Re: [DISCUSS] Metron Release 0.6.0?

[Justin Leet]

Any idea why we released it as 0.1.0 in the artifacts version?  I'm fine
with doing x.y if we need to, but I would like the artifact versioning to
be consistent if possible.

On Wed, Sep 5, 2018 at 8:26 PM zeo...@gmail.com  wrote:

> I lied, we didn't need to update our btests because it's limited to a major
> and minor version.
>
>
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/src/Plugin.cc#L33-L34
>
> Jon
>
> On Wed, Sep 5, 2018 at 8:10 PM zeo...@gmail.com  wrote:
>
> > I looked into x.y.z back when we released 0.1 and it was not possible in
> > bro-pkg at the time but now it is
> > <https://github.com/bro/package-manager/issues/32>.  In order to do
> this,
> > we'll also need to configure bro-pkg.meta to require the proper version
> of
> > bro-pkg, as well as update the btests for the new version string.  I will
> > throw together a JIRA and PR to do all this in case we decide to align
> with
> > x.y.z; we can trash it if we decide to stay with x.y.
> >
> > Jon
> >
> > On Wed, Sep 5, 2018 at 7:35 PM Justin Leet 
> wrote:
> >
> >> Long story short, while preparing the release candidate, I discovered
> our
> >> metron-bro-plugin-kafka is inconsistently versioned.
> >>
> >> In the repo, it's x.y (e.g. 0.2)
> >> See:
> >>
> https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18
> >>
> >> In our released artifact, it's x.y.z (e.g. 0.1.0)
> >> See http://archive.apache.org/dist/metron/0.4.2/
> >>
> >> Going forward from this release, I'd like the artifact to be consistent
> >> with the repo.  I'd personally prefer x.y.z to be entirely consistent
> >> throughout Metron, but if there's a particular reason why it was x.y I'm
> >> happy to entertain it.
> >>
> >> If we choose to move to x.y.z, I can provide a PR to update the version
> to
> >> 0.2.0 unless someone else wants to volunteer. Otherwise, I'd like to
> >> release the artifact as apache-metron-bro-plugin-kafka_0.2.tar.gz
> >>
> >> Justin
> >>
> >> On Tue, Sep 4, 2018 at 12:15 PM Justin Leet 
> >> wrote:
> >>
> >> > As an update, I'll be working on starting the release process rolling
> >> > today. The PCAP Query panel feature branch is in master and I haven't
> >> heard
> >> > of any other potential blockers. I'd appreciate everyone updating
> their
> >> > JIRAs with their state (complete, etc.) and version (0.6.0) as needed.
> >> >
> >> > Do we have anything that needs to go into the UPGRADING.md file?
> >> >
> >> > A PR is out for updating the version from 0.5.1 to 0.6.0. Please see
> >> > https://github.com/apache/metron/pull/1183. I still need to spin up
> >> full
> >> > dev etc. before this is ready to be merged.
> >> >
> >> > Updated list of PRs that have made it into master as of Sept 4, 2018
> >> >
> >> > 6 days ago METRON-1751 Storm Profiler dies when consuming null message
> >> > (nickwallen) closes apache/metron#1176
> >> > 6 days ago METRON-1757 Storm Profiler Serialization Exception
> >> (nickwallen)
> >> > closes apache/metron#1178
> >> > 6 days ago METRON-1743 CEF testPaloAltoCEF test using a confusing
> >> variable
> >> > name (JonZeolla via justinleet) closes apache/metron#1173
> >> > 8 days ago METRON-1752 Prevent package.lock from changing during build
> >> > (sardell via merrimanr) closes apache/metron#1177
> >> > 8 days ago METRON-1724 Date/time validation missing in PCAP query
> >> (tiborm
> >> > via nickwallen) closes apache/metron#1172
> >> > 3 weeks ago METRON-1554 Pcap Query Panel (merrimanr) closes
> >> > apache/metron#1169
> >> > 3 weeks ago METRON-1739 UDP packets are not handled (merrimanr) closes
> >> > apache/metron#1168
> >> > 3 weeks ago METRON-1727: Alerts are not populated on the alerts UI
> after
> >> > enabling X-pack for Elastic search (MohanDV via mmiklavc) closes
> >> > apache/metron#1141
> >> > 3 weeks ago METRON-1738: Pcap directories should have correct
> >> permissions
> >> > (merrimanr via mmiklavc) closes apache/metron#1166
> >> > 3 weeks ago METRON-1737: Document Job cleanup (merrimanr via mmiklavc)
> >> > closes apache/metron#1164
> >> > 3 weeks ago METRON-1732: Fix job status liveness bug and parallelize
> >> > finalizer file writing (mmiklav

Re: [DISCUSS] Metron Release 0.6.0?

[Justin Leet]

Long story short, while preparing the release candidate, I discovered our
metron-bro-plugin-kafka is inconsistently versioned.

In the repo, it's x.y (e.g. 0.2)
See:
https://github.com/apache/metron-bro-plugin-kafka/blob/master/VERSION#L18

In our released artifact, it's x.y.z (e.g. 0.1.0)
See http://archive.apache.org/dist/metron/0.4.2/

Going forward from this release, I'd like the artifact to be consistent
with the repo.  I'd personally prefer x.y.z to be entirely consistent
throughout Metron, but if there's a particular reason why it was x.y I'm
happy to entertain it.

If we choose to move to x.y.z, I can provide a PR to update the version to
0.2.0 unless someone else wants to volunteer. Otherwise, I'd like to
release the artifact as apache-metron-bro-plugin-kafka_0.2.tar.gz

Justin

On Tue, Sep 4, 2018 at 12:15 PM Justin Leet  wrote:

> As an update, I'll be working on starting the release process rolling
> today. The PCAP Query panel feature branch is in master and I haven't heard
> of any other potential blockers. I'd appreciate everyone updating their
> JIRAs with their state (complete, etc.) and version (0.6.0) as needed.
>
> Do we have anything that needs to go into the UPGRADING.md file?
>
> A PR is out for updating the version from 0.5.1 to 0.6.0. Please see
> https://github.com/apache/metron/pull/1183. I still need to spin up full
> dev etc. before this is ready to be merged.
>
> Updated list of PRs that have made it into master as of Sept 4, 2018
>
> 6 days ago METRON-1751 Storm Profiler dies when consuming null message
> (nickwallen) closes apache/metron#1176
> 6 days ago METRON-1757 Storm Profiler Serialization Exception (nickwallen)
> closes apache/metron#1178
> 6 days ago METRON-1743 CEF testPaloAltoCEF test using a confusing variable
> name (JonZeolla via justinleet) closes apache/metron#1173
> 8 days ago METRON-1752 Prevent package.lock from changing during build
> (sardell via merrimanr) closes apache/metron#1177
> 8 days ago METRON-1724 Date/time validation missing in PCAP query (tiborm
> via nickwallen) closes apache/metron#1172
> 3 weeks ago METRON-1554 Pcap Query Panel (merrimanr) closes
> apache/metron#1169
> 3 weeks ago METRON-1739 UDP packets are not handled (merrimanr) closes
> apache/metron#1168
> 3 weeks ago METRON-1727: Alerts are not populated on the alerts UI after
> enabling X-pack for Elastic search (MohanDV via mmiklavc) closes
> apache/metron#1141
> 3 weeks ago METRON-1738: Pcap directories should have correct permissions
> (merrimanr via mmiklavc) closes apache/metron#1166
> 3 weeks ago METRON-1737: Document Job cleanup (merrimanr via mmiklavc)
> closes apache/metron#1164
> 3 weeks ago METRON-1732: Fix job status liveness bug and parallelize
> finalizer file writing (mmiklavc via mmiklavc) closes apache/metron#1157
> 3 weeks ago METRON-1735 Empty print status option causes NPE (merrimanr)
> closes apache/metron#1160
> 3 weeks ago METRON-1733 PCAP UI - PCAP queries dont work on Safari
> (sardell via merrimanr) closes apache/metron#1158
> 3 weeks ago METRON-1734 Src and Dst port filters are incorrect after
> changing to empty (merrimanr) closes apache/metron#1159
> 4 weeks ago METRON-1725 Add ability to specify YARN queue for pcap jobs
> (merrimanr) closes apache/metron#1153
> 4 weeks ago METRON-1731: PCAP - Escape colons in output dir names
> (mmiklavc via mmiklavc) closes apache/metron#1155
> 4 weeks ago METRON-1702 Reload a running job in the UI (merrimanr) closes
> apache/metron#1156
> 4 weeks ago METRON-1722 PcapCLI should print progress to stdout
> (merrimanr) closes apache/metron#1138
> 4 weeks ago Merge branch 'master' into feature/METRON-1554-pcap-query-panel
> 4 weeks ago METRON-1728: Handle null values in config in Pcap backend more
> gracefully (mmiklavc via mmiklavc) closes apache/metron#1151
> 4 weeks ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
> via mmiklavc) closes apache/metron#1152
> 4 weeks ago METRON-1713 PCAP UI - Add a way to kill a pcap job (tiborm via
> merrimanr) closes apache/metron#1143
> 4 weeks ago METRON-1723 PCAP UI - Unable to select/copy from packets
> details in PCAP query panel (sardell via merrimanr) closes
> apache/metron#1139
> 4 weeks ago METRON-1712 PCAP UI - Input validation (tiborm via merrimanr)
> closes apache/metron#1142
> 4 weeks ago METRON-1720 Better error messages when there are no results or
> wireshark is not installed (merrimanr) closes apache/metron#1154
> 4 weeks ago METRON-1726: Refactor PcapTopologyIntegrationTest (mmiklavc
> via mmiklavc) closes apache/metron#1140
> 4 weeks ago METRON-1683 PCAP UI - Fix the download progress bar (sardell
> via merrimanr) closes apache/metron#1122
> 4 weeks ago METRON-1675 PCAP UI - Introduce the paging capability (sardell
> via merrimanr) closes apache