Re: [VOTE] Move Apache Metron to the Apache Attic and Dissolve PMC

[Otto Fowler]

+1




From: Justin Leet  
Reply: dev@metron.apache.org  
Date: November 16, 2020 at 09:01:36
To: dev@metron.apache.org  
Subject:  [VOTE] Move Apache Metron to the Apache Attic and Dissolve PMC

Hi all,

This is a vote thread to retire Metron to the Attic, and dissolve the PMC.
This follows a discussion thread on the dev list ([DISCUSS] Retire Metron
to the Attic
<
https://lists.apache.org/thread.html/reb31f643fac20d3ad09521fd702b19922412b7a4e8e08062968268c5%40%3Cdev.metron.apache.org%3E>).

More details can be found in that discussion, but the most relevant link is
the specific process at Moving a project to the Attic
.

As noted in the process page, this is a PMC vote. As usual, feel encouraged
to contribute non-binding votes.

The vote will run 72 hours, until Nov 19th at 9:00 am EST.

Thank you,
Justin

Re: [DISCUSS] Retire Metron to the Attic

[Otto Fowler]

 I am in support of this as well,

We have substantial work to do to get metron off of the HDP version it is
on, which I think is the minimal work to keep it viable.  We do not however
have the people to finish the work that was started.

Jon Zeolla and myself will be forking the metron-bro-kafka-plugin out, as
it _has_ continued to be active.



From: Casey Stella  
Reply: dev@metron.apache.org  
Date: November 9, 2020 at 10:48:22
To: dev@metron.apache.org  
Subject:  Re: [DISCUSS] Retire Metron to the Attic

Hi all,
I'm in complete support of this. Given the current level of interest, I
believe that we should move this project to the attic.

Best,

Casey Stella

On Mon, Nov 2, 2020 at 7:28 PM Justin Leet  wrote:

> Hi all,
>
> I want to start a discussion in the community to consider retiring Metron
> to the Apache Attic and dissolve the project management committee. For
> anyone unaware, an overview is available at Apache Attic
> . In short, it's a way to provide a wind-down
> process for projects reaching their end of life. This process, and
> potential next steps, can be found here:
> http://attic.apache.org/process.html.
>
> We've seen a substantial decrease in contributions (both code and mailing
> lists) over the past year or so (see:
> https://github.com/apache/metron/commits/master). There's been some
> interest in the project, but in my personal opinion, not enough to build
or
> sustain real momentum. Additionally, there's been an inability to close
the
> loop on paring down the project's scope and building a solid foundation
for
> future work (see: Development Activity has dropped to effectively 0, what
> should we do?
> <
>
https://lists.apache.org/thread.html/rf7ea1c1afb347e352efff50f58fbd58779f71e6c814d2a5563e381d7%40%3Cdev.metron.apache.org%3E
> >).
> Finally, the last release was over a year ago and I don't see a release
> naturally building anytime soon. We haven't seen much active development
or
> participation for a substantial period of time, which to me implies we've
> reached a natural end of life.
>
> The most obvious practical effects would be no new releases, setting
source
> read-only, mailing lists closed down, automated processes ended, etc. In
> addition, the PMC would also be dissolved as part of this process.
> Community members may still fork all or part of Metron, but should keep
in
> mind that the Metron trademark remains with the ASF.
>
> What opinions are there either in favor of or against moving the project
to
> the Attic? Any questions about the process (or comments/corrections
anyone
> would like to add)?
>
> Thanks,
> Justin
>

Re: Metron-2340 - Geo Database

[Otto Fowler]

 I “believe” that those files apply to the database that they ship, looking
at the perl It doesn’t seem to mention any claim or restrictions on the
databases produced.

I don’t think you are talking about distributing the perl module, but
rather making it a build dependency,  Is that right?

On May 17, 2020 at 10:49:11, Yerex, Tom (tom.ye...@ubc.ca) wrote:

Good morning,

I have been working on a solution for Metron-2340 (
hxxps://issues.apache.org/jira/projects/METRON/issues/METRON-2340), and
need some direction. br/> <
Using Maxmind's database writer (
hxxps://github.com/maxmind/MaxMind-DB-Writer-perl), I have been
creating the mmdb files required for the build tests to succeed. The
Perl code I was going to place in dev-utilities/build-utils with the
name generate_geoip.pl, and I noticed there are some additional files
to consider.

As an example, in GeoLite2-ASN.tar.gz there is the following:

GeoLite2-ASN_20181120/
GeoLite2-ASN_20181120/COPYRIGHT.txt
GeoLite2-ASN_20181120/LICENSE.txt
GeoLite2-ASN_20181120/GeoLite2-ASN.mmdb

I am not using the original mmdb data from Maxmind, instead generating
data using the Perl library. Are the copyright and license files
required? Is there a copyright and license document from the Apache
foundation I should be using here?

Thank you,

Tom.

Re: Development Activity has dropped to effectively 0, what should we do?

[Otto Fowler]

 I think the difference is the maintenance of the core of metron that *has*
to be, and other things that may still be done, but will be worked on for
their merits or by community need and not be required for everything

On April 21, 2020 at 10:29:24, Justin Leet (justinjl...@gmail.com) wrote:

How we install depends on what we're choosing to keep around. My concern is
getting core Metron's scope down to a supportable level. This entire
conversation is probably just a thought experiment until we properly limit
the rest of our scope. It's putting the cart before the horse. I want to
emphasize this, because we're having a discussion about how to install
something that in many ways doesn't actually exist yet.

A lot of the install complexity comes from managing so many moving parts at
once (ES/Solr, the UI, Kerberos, etc.). If we cut that down, I'm not sure
we need a big installer to manage everything. Plenty of projects trust
people to be able to run convenience scripts and shell commands. Again, I
think this is an academic discussion until we figure out our overall
project direction.

On Tue, Apr 21, 2020 at 10:02 AM Nick Allen  wrote:

> Hi Tom -
>
> > Do you or anyone have enough experience to judge if it is possible to
> leverage Ansible as a replacement to deploy a working cluster?
>
> Yes, I worked a lot on the Ansible mechanism in the early days of Metron.
> This was the primary deployment mechanism before we had the Ambari MPack.
>
> We found it very difficult to use Ansible to create a one-size-fits-all
> deployment solution. It's possible, but very difficult to get a solution
> that doesn't take close monitoring and manual work arounds when
attempting
> to use it across environments of different sizes and shapes. In terms of
> usability, the Ambari MPack was a big step-up in my opinion.
>
>
> > perhaps a dedicated docker image that is designed to connect with other
> dockerized applications such as Storm, Kafka, etc..?
>
> Yes, I think that would be the way to go for a dev environment. We would
be
> able to use community supported containers for most of our underlying
> platform needs. Unfortunately, this alone would not help anyone deploy
> Metron on a cluster.
>
>
>
>
> On Tue, Apr 21, 2020 at 9:08 AM Yerex, Tom  wrote:
>
> > Hi Nick,
> >
> > I see there is a lot of work done using Ansible in the repository. Do
you
> > or anyone have enough experience to judge if it is possible to leverage
> > Ansible as a replacement to deploy a working cluster?
> >
> > Now that I am typing this out, I wonder if docker might be a solution
> that
> > would work? I don't have much experience with docker, perhaps a
dedicated
> > docker image that is designed to connect with other dockerized
> applications
> > such as Storm, Kafka, etc..?
> >
> > --Tom.
> >
> > On 2020-04-17, 11:27 AM, "Nick Allen"  wrote:
> >
> > This is a good discussion and one that I haven't fully grappled with
> > in my
> > own mind yet. I'll have more to add, but I just want to chime in on
> the
> > topic of Ambari at this point.
> >
> > ### Ambari and the Paywall
> >
> > The problem with Ambari is that its installation mechanism requires a
> > repository of compiled packages (RPMs, DEBs, etc.) To install the
> > underlying platform dependencies (like Kafka, HBase, Storm, Zk, etc)
> we
> > relied on binary packages that were made freely available by
> > Cloudera/Hortonworks. As of this past January, those packages are now
> > behind a paywall.
> >
> > Due to the paywall, installing your own HDP cluster with Ambari is
> now
> > effectively dead. I am not sure if legacy versions of Kafka, HBase,
> > Storm,
> > etc will continue to be freely available, but even if so, we cannot
> > continue to rely on this mechanism if new versions and security
> updates
> > will not be made available.
> >
> > The Apache Metron project does not publish compiled binaries or
> > packages
> > either. We do make the code freely available to allow users to build
> > and
> > publish their own Metron packages. But even with this capability,
> > unless
> > you have a means to install the underlying platform dependencies via
> > Ambari, installing Metron with Ambari has little value.
> >
> > Unfortunately, I don't see a feasible path forward for Metron's
> Ambari
> > MPack.
> >
> > ### Dev Environment
> >
> > This not only impacts the users of Apache Metron, this impacts
> > contributors
> > also. Our primary development environment relies on that Ambari
> > MPack. To
> > continue development on any of the components of Apache Metron, we
> > would
> > need to build an alternative development environment that can
> function
> > despite the paywall. That could take many shapes, but in my opinion
> it
> > would be a blocker for continuing any development on Apache Metron,
> > unfortunately.
> >
> > Please do let me know if anyone disagrees or can think of an
> > alternative
> > approach that would allow the current Ambari MPack to remain viable.
> >
> >
> >
> >
> >
> >
> >

Re: Failing to Setup Apache Metron Dev Machine using Vagrant (Require Assistance and Guidance)

[Otto Fowler]

I have tried this on master, and progressed all the way to the failure to
setup enrichment because of the geomind change.

I believe that this issue is only with the 3.1 feature branch.




On March 3, 2020 at 09:49:39, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

It looks like the binaries are no longer available? I've not encountered
this before. Not sure if this has anything to do with it (from Cloudera's
publications -
https://www.cloudera.com/content/dam/www/marketing/documents/campaign/451-cloudera-opens-up-about-opensource.pdf
)

On Mon, Mar 2, 2020 at 10:28 PM Abid Zaidi  wrote:

> Thanks Michael,
> I am using Ubuntu 19.10, 32GB RAM, i7 (12 cores),
> Vagrant machine is obviously on CentOS 7, I've given it 16GB RAM.
>
> On investigation the issue was with a Yum repo , the error is following
>
> 
> RuntimeError: Failed to execute command '/usr/bin/yum -y install unzip',
> exited with code '1', message: '
>
https://archive.cloudera.com/p/HDP/centos7/3.x/updates/3.1.5.0/repodata/repomd.xml:

> [Errno 14] HTTPS Error 401 - Unauthorized
>
> Trying other mirror.
>
> One of the configured repositories failed (*HDP-3.1-repo-1*),
> 
>
>
> I have searched the available repos but apparently for HDP 3.1.5 (
> reference
> <
https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/ch03s02s01.html>)

> it requires credentials. (e.g *
https://username:passw...@archive.cloudera.com/p/HDP/centos7/3.x/updates/3.1.5.0
> <
https://username:passw...@archive.cloudera.com/p/HDP/centos7/3.x/updates/3.1.5.0>)*

>
> For acquiring these I'd have to file a case on cloudera support, but when
> i reach there it says: *'Access to page is restricted'. *Later I got to
> know that I have to purchase their product since it is behind a payment
> wall (reference
> <
https://community.cloudera.com/t5/Support-Questions/Unable-to-access-Ambari-repo/td-p/286813>

> ).
>
> If you have any information of a work around do let me know..
>
> --
> *From:* Michael Miklavcic 
> *Sent:* Monday, March 2, 2020 10:07 AM
> *To:* dev@metron.apache.org 
> *Subject:* Re: Failing to Setup Apache Metron Dev Machine using Vagrant
> (Require Assistance and Guidance)
>
> Head over to Ambari http://node1:8080 and click the activity icon (should
> be somewhere along the top menu bar). You should be able to see more
> details about what Ambari was doing at the time of failure. It's possible
> there were just some minor issues in starting up the services. If that's
> the case, then I think all you'll need to do is start them up manually.
>
> FYI, HDP 3.1 should be deployed on CentOS 7 only. Also, the Vagrant
> environment should have at least 8 GB of mem allocated for it (that's our
> default).
>
> On Sun, Mar 1, 2020 at 5:39 AM Abid Zaidi  wrote:
>
> > I am setting up centos7 single node VM through vagrant for Dev
> Environment,
> >
> >
> > I already tried centos6 from other branches and got no luck in that,
> >
> > So I check out branch of METRON-2088-SUPPORT-HDP-3.1 and followed the
> > ReadMe, before starting I ran platform.sh and everything looked good..
> >
> > On Vagrant Up it just fails on Metron_config with message
> >
> > [node1]: FAILED! => {"changed": false, "msg": "Request failed with
status
> > FAILED"}
> >
> > At this point I could open ambari but all services were down and each
of
> > them said the same thing, Failed to install service, Below is the link
to
> > my logs:
> >
> >
> >
>
https://community.cloudera.com/t5/Support-Questions/Failing-to-Setup-Apache-Metron-Dev-Machine-using-Vagrant/m-p/290740/highlight/true#M215064
> >
> >
> > This is the furthest I've come to deploying, even with the CentOS 6 I
> came
> > this far,
> >
> > I then also tried provision with following command,
> >
> > env ANSIBLE_ARGS=' -' vagrant --ansible-tags="hdp-deploy" provision
> >
> >
> > I've done this 2-3 times and I reach to this same point,
> >
> > I need to setup a Dev Environment where I could add a few more features
> in
> > Apache Metron's REST API with UI pages on the front-end. I seem to be
far
> > away from my targets as I am unable to start up the vagrant machine, It
> > would be good if a Metron Developer could guide me with what I'm trying
> to
> > achieve.
> >
> >
> > Regards,
> >
> > Abid Zaidi.
> >
> > abidzai...@hotmail.com
> >
> >
> >
>

Re: [DISCUSS] GeoLite database licensing change, master build broken…..

[Otto Fowler]

So, what I think we need, is for someone who has some idea of how to
proceed to fill out the jira with either a list or actual tasks, that
someone else could follow to accomplish this.




On January 13, 2020 at 19:08:20, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Yes, understood. We do not provide a mechanism for cleaning individual
records from the DB file. Just a thought, but a workaround could be to
download the latest file from maxmind and simply delete the old file.
That's not ideal, but it might be something that could still be automated
depending on the maxmind implementation. Definitely open to other options,
but I wanted to at least come to the table with a simple workable option.

On Mon, Jan 13, 2020 at 4:36 PM Justin Leet  wrote:

> To clarify a bit, on Otto's comment
>
> > I have not read the new license, but the idea I believe is yes,
whomever
> > downloads and accepts the new license is then responsible for adherence
> to
> > the applicable law,
> > Which is why the Apache Foundation cannot be that entity I would think
>
>
> My concern is less the legal concern about anything regarding Metron or
> ASF, I 100% agree the ASF isn't responsible for users fulfilling their
> legal obligations. My concern there about the practical ability of users
to
> even be able to do the sort of recalculation necessary to properly act on
> such a requirement. The minimum is probably to call out they need to
clean
> it up, but that may cause a lot of friction for our users.
>
> On Mon, Jan 13, 2020 at 5:52 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > +1 to Apache Legal advice
> >
> > On Mon, Jan 13, 2020 at 3:30 PM Otto Fowler 
> > wrote:
> >
> > > Justin,
> > >
> > > I have not read the new license, but the idea I believe is yes,
> whomever
> > > downloads and accepts the new license is then responsible for
adherence
> > to
> > > the applicable law,
> > > Which is why the Apache Foundation cannot be that entity I would
think.
> > >
> > > We may want to send this past Apache Legal?
> > >
> > >
> > >
> > > On January 13, 2020 at 17:22:21, Justin Leet (justinjl...@gmail.com)
> > > wrote:
> > >
> > > On the whole, I agree. I think the immediate focus should be to rip
out
> > > maxmind from default usage in master, even if it's done a bit
roughly.
> > > Getting master at least building for people would probably be a good
> > first
> > > step.
> > >
> > > Couple further thoughts
> > >
> > > - JUnit 5 supports a variety of conditional testing conditions. It
> might
> > > be possible to do "Test is enabled if property
> > > maxmind.geo.database.location is set", and just provide instructions
to
> > > users on actually doing it instead of having to manually
enable/disable
> > the
> > > test via code.
> > > - Remove DB download/install from Ambari. I agree with removing the
> > > actual dl/load, but still keeping the default location setup.
> > > - I'm inclined to think it should be removed from the default demo
> > > enrichments. We might be able to replace the GEO_GET with some dummy
> > > stellar and just produce similar outputs, since our demo data has a
> > limited
> > > set of IPs, iirc. Obviously, we'd want to document that fairly well
and
> > > also let users manually set it up for non-demo data.
> > > - CLI tool should be fine for letting users load, it's more or less
why
> > > it existed in the first place. Maybe we also add a check if someone
> tries
> > > using the old URL and let them know.
> > > - We need to document that these changes occurred for CCPA reasons,
so
> > > that users can evaluate the consequences.
> > > - Given that I am not even slightly a lawyer, my understanding here
is
> > > likely wrong, but are there further implications here? Say a user
runs
> a
> > > bunch of data through Metron (both enrichments and profiler) using
> these
> > > IPs. A subset of those IPs is removed as it shouldn't be used. Would
a
> > user
> > > then have to remove all data derived from these IPs (e.g. anything in
> > > ES/Solr, anything in HDFS, any profiles using the data?). If they do
> have
> > > to remove it, I assume that's not directly out problem (since they
have
> > to
> > > do cleanup), but we're probably not making it easy on them in terms
of
> > > providing the ability to clean up that sort of information and rerun
> > > profiles and such.
> > >
> > >
> &

Re: [DISCUSS] Next Release - Life After 0.7.1

[Otto Fowler]

yes




On January 14, 2020 at 13:05:17, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

We should probably also get this resolved for this release.
https://issues.apache.org/jira/browse/METRON-2340. Thoughts?

On Mon, Dec 16, 2019 at 2:19 PM Shane Ardell 
wrote:

> Both PR #1527  and #1533
>  are now merged into master.
>
> On Fri, Dec 13, 2019 at 3:57 PM Justin Leet 
wrote:
>
> > I also brought up https://github.com/apache/metron/pull/1282 and
> > https://github.com/apache/metron/pull/1552 if anyone has any thoughts
> on
> > them.
> >
> > On Fri, Dec 13, 2019 at 11:58 AM Shane Ardell 

> > wrote:
> >
> > > Quick update from my end: I just left a +1 on
> > > https://github.com/apache/metron/pull/1527 and will merge it into
> master
> > > shortly. We are actively looking into the cause of the bug I
> encountered
> > in
> > > https://github.com/apache/metron/pull/1533. It would be nice to have
> > this
> > > in the release, but I would not categorize it as critical like #1527.
> I'm
> > > optimistic we will have this resolved and merged into master by this
> > > weekend, but I'm fine reducing scope to not include it.
> > >
> > > On Fri, Dec 13, 2019 at 11:24 AM Nick Allen 
> wrote:
> > >
> > > > Are we just waiting on the following PRs as release blockers? Any
> > > others?
> > > >
> > > > - https://github.com/apache/metron/pull/1533
> > > > - https://github.com/apache/metron/pull/1527
> > > >
> > > > Being towards the end of the year, people are going to be on
holiday.
> > It
> > > > would be great if we could focus on reducing scope and getting a
> > release
> > > > cut.
> > > >
> > > >
> > > > On Sat, Dec 7, 2019 at 10:04 AM Justin Leet 
> > > wrote:
> > > >
> > > > > https://github.com/apache/metron/pull/1568 and
> > > > > https://github.com/apache/metron/pull/1554 are in master now.
> > > > >
> > > > > On Fri, Dec 6, 2019 at 7:16 PM Justin Leet 

> > > > wrote:
> > > > >
> > > > > > I'd like to throw https://github.com/apache/metron/pull/1552 on
> > the
> > > > > pile.
> > > > > > Per https://issues.apache.org/jira/browse/LEGAL-491, we should
> > just
> > > > note
> > > > > > the contribution comes from dependabot. Would someone more
> familiar
> > > > with
> > > > > > the implications of upgrading that be able to review it, or
give
> > some
> > > > > > advice on what we should be looking for in the review?
> > > > > >
> > > > > > On Thu, Dec 5, 2019 at 12:06 PM Shane Ardell <
> > > shane.m.ard...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Speaking on the UI-related PRs that Justin mentioned, I also
> would
> > > > like
> > > > > to
> > > > > >> see both of them merged before a release. At the moment, #1527
> > does
> > > > not
> > > > > >> address a few "stale data state" message inconsistencies that
> > become
> > > > > >> apparent as a result of that PR's work (you can read more
about
> it
> > > in
> > > > > the
> > > > > >> PR comments). That said, I think those inconsistencies can be
> > > tracked
> > > > > and
> > > > > >> addressed separately from the current PR.
> > > > > >>
> > > > > >> On Thu, Dec 5, 2019 at 11:51 AM Michael Miklavcic <
> > > > > >> michael.miklav...@gmail.com> wrote:
> > > > > >>
> > > > > >> > I think the junit upgrade should go in also. I'm almost
> finished
> > > > > >> reviewing
> > > > > >> > that.
> > > > > >> >
> > > > > >> > On Thu, Dec 5, 2019, 8:50 AM Justin Leet <
> justinjl...@gmail.com
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > If we're going to do a bug fix release, I'd like to see
> some
> > of
> > > > the
> > > > > >> low
> > > > > >> > > hanging fix PRs get finished and merged prior to the
> release.
> > > > We've
> > > > > >> been
> > > > > >> > > lax about getting them cleaned up, so I'd like to use a
> > release
> > > as
> > > > > an
> > > > > >> > > opportunity to whittle the PRs down and put out a really
> solid
> > > > > >> release.
> > > > > >> > >
> > > > > >> > > https://github.com/apache/metron/pull/1568 should be in
> > before
> > > > > >> release.
> > > > > >> > It
> > > > > >> > > addresses an issue with our validation of
> > > > dependencies_with_url.csv
> > > > > >> and
> > > > > >> > > it's validation.
> > > > > >> > > Should https://github.com/apache/metron/pull/1282 be in?
> > Seems
> > > > like
> > > > > >> that
> > > > > >> > > should have been merged awhile ago.
> > > > > >> > >
> > > > > >> > > There's also a couple UI performance / bug fixes PRs (e.g.
> > > > > >> > > https://github.com/apache/metron/pull/1533 and
> > > > > >> > > https://github.com/apache/metron/pull/1527) that have been
> > > > sitting
> > > > > >> > awhile.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Thu, Dec 5, 2019 at 10:32 AM Nick Allen <
> > n...@nickallen.org>
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > > Hello Metron'ers -
> > > > > >> > > >
> > > > > >> > > > I would like to make the case that 

Re: [DISCUSS] GeoLite database licensing change, master build broken…..

[Otto Fowler]

Justin,

I have not read the new license, but the idea I believe is yes, whomever
downloads and accepts the new license is then responsible for adherence to
the applicable law,
Which is why the Apache Foundation cannot be that entity I would think.

We may want to send this past Apache Legal?



On January 13, 2020 at 17:22:21, Justin Leet (justinjl...@gmail.com) wrote:

On the whole, I agree. I think the immediate focus should be to rip out
maxmind from default usage in master, even if it's done a bit roughly.
Getting master at least building for people would probably be a good first
step.

Couple further thoughts

- JUnit 5 supports a variety of conditional testing conditions. It might
be possible to do "Test is enabled if property
maxmind.geo.database.location is set", and just provide instructions to
users on actually doing it instead of having to manually enable/disable the
test via code.
- Remove DB download/install from Ambari. I agree with removing the
actual dl/load, but still keeping the default location setup.
- I'm inclined to think it should be removed from the default demo
enrichments. We might be able to replace the GEO_GET with some dummy
stellar and just produce similar outputs, since our demo data has a limited
set of IPs, iirc. Obviously, we'd want to document that fairly well and
also let users manually set it up for non-demo data.
- CLI tool should be fine for letting users load, it's more or less why
it existed in the first place. Maybe we also add a check if someone tries
using the old URL and let them know.
- We need to document that these changes occurred for CCPA reasons, so
that users can evaluate the consequences.
- Given that I am not even slightly a lawyer, my understanding here is
likely wrong, but are there further implications here? Say a user runs a
bunch of data through Metron (both enrichments and profiler) using these
IPs. A subset of those IPs is removed as it shouldn't be used. Would a user
then have to remove all data derived from these IPs (e.g. anything in
ES/Solr, anything in HDFS, any profiles using the data?). If they do have
to remove it, I assume that's not directly out problem (since they have to
do cleanup), but we're probably not making it easy on them in terms of
providing the ability to clean up that sort of information and rerun
profiles and such.


On Mon, Jan 13, 2020 at 5:20 PM Otto Fowler 
wrote:

> I agree with all of that, my only thinking on contrib is, if that
component
> is not tested and has no coverage, and needs manual steps, then we may
want
> to separate it.
> We would have to have some handle on full dev as well right?
>
> On January 13, 2020 at 16:57:13, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> As I mentioned above, we have had this issue with other components
before,
> e.g. mysql. I don't see a compelling reason to discontinue or push this
> component to contrib just yet - it's a type of enrichment that happens to
> require an additional manual step. Per the article (
>
>
https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
> ),
>
> Maxmind is not requiring purchase, merely a user to register due to GDPR
> and CCPA requirements. That being said, it does cause issues for the
> integration tests. I think we should minimally add an @Ignore annotation
to
> the testLoadGeoIpDatabase test in MaxmindDbEnrichmentLoaderTest and
> provide manual instructions for running the integration test if/when
> someone submits a PR affecting this code.
>
> Users can still use the geolite DB by uploading it to HDFS.
(Incidentally,
> this even provides a crude mechanism for versioning, should a user choose
> to use the config path that way). We separated the deployment side of the
> geolite DB from the consumption side
>
>
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabase.java#L135
> .
>
> All that needs to happen is the user would drop the file in HDFS. This
had
> previously happened via Ambari - see here -
>
>
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py#L95
> .
>
> We should probably do a couple things:
>
> 1. Minimally, remove the DB download and install from Ambari - we might
> choose to keep around the HDFS path creation so that we have a reasonable
> default prepped and ready.
> 2. Add documentation to manually install the geolite DB (register with
> maxmind) OR remove it from our default demo enrichments shipped with the
> platform.
>
> Manual DB file loading would be managed using the CLI tool. You should be
> able to load it via local file U

Re: [DISCUSS] GeoLite database licensing change, master build broken…..

[Otto Fowler]

Maybe we should have a stellar command for this


On January 13, 2020 at 17:22:21, Justin Leet (justinjl...@gmail.com) wrote:

On the whole, I agree. I think the immediate focus should be to rip out
maxmind from default usage in master, even if it's done a bit roughly.
Getting master at least building for people would probably be a good first
step.

Couple further thoughts

- JUnit 5 supports a variety of conditional testing conditions. It might
be possible to do "Test is enabled if property
maxmind.geo.database.location is set", and just provide instructions to
users on actually doing it instead of having to manually enable/disable the
test via code.
- Remove DB download/install from Ambari. I agree with removing the
actual dl/load, but still keeping the default location setup.
- I'm inclined to think it should be removed from the default demo
enrichments. We might be able to replace the GEO_GET with some dummy
stellar and just produce similar outputs, since our demo data has a limited
set of IPs, iirc. Obviously, we'd want to document that fairly well and
also let users manually set it up for non-demo data.
- CLI tool should be fine for letting users load, it's more or less why
it existed in the first place. Maybe we also add a check if someone tries
using the old URL and let them know.
- We need to document that these changes occurred for CCPA reasons, so
that users can evaluate the consequences.
- Given that I am not even slightly a lawyer, my understanding here is
likely wrong, but are there further implications here? Say a user runs a
bunch of data through Metron (both enrichments and profiler) using these
IPs. A subset of those IPs is removed as it shouldn't be used. Would a user
then have to remove all data derived from these IPs (e.g. anything in
ES/Solr, anything in HDFS, any profiles using the data?). If they do have
to remove it, I assume that's not directly out problem (since they have to
do cleanup), but we're probably not making it easy on them in terms of
providing the ability to clean up that sort of information and rerun
profiles and such.


On Mon, Jan 13, 2020 at 5:20 PM Otto Fowler 
wrote:

> I agree with all of that, my only thinking on contrib is, if that
component
> is not tested and has no coverage, and needs manual steps, then we may
want
> to separate it.
> We would have to have some handle on full dev as well right?
>
> On January 13, 2020 at 16:57:13, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> As I mentioned above, we have had this issue with other components
before,
> e.g. mysql. I don't see a compelling reason to discontinue or push this
> component to contrib just yet - it's a type of enrichment that happens to
> require an additional manual step. Per the article (
>
>
https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
> ),
>
> Maxmind is not requiring purchase, merely a user to register due to GDPR
> and CCPA requirements. That being said, it does cause issues for the
> integration tests. I think we should minimally add an @Ignore annotation
to
> the testLoadGeoIpDatabase test in MaxmindDbEnrichmentLoaderTest and
> provide manual instructions for running the integration test if/when
> someone submits a PR affecting this code.
>
> Users can still use the geolite DB by uploading it to HDFS.
(Incidentally,
> this even provides a crude mechanism for versioning, should a user choose
> to use the config path that way). We separated the deployment side of the
> geolite DB from the consumption side
>
>
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabase.java#L135
> .
>
> All that needs to happen is the user would drop the file in HDFS. This
had
> previously happened via Ambari - see here -
>
>
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py#L95
> .
>
> We should probably do a couple things:
>
> 1. Minimally, remove the DB download and install from Ambari - we might
> choose to keep around the HDFS path creation so that we have a reasonable
> default prepped and ready.
> 2. Add documentation to manually install the geolite DB (register with
> maxmind) OR remove it from our default demo enrichments shipped with the
> platform.
>
> Manual DB file loading would be managed using the CLI tool. You should be
> able to load it via local file URL (
> https://en.wikipedia.org/wiki/File_URI_scheme) or via a custom hosted
> solution via the --geo_url option. See more details here -
>
>
https://github.com/apache/metron/tree/master/metron-platform/metron-data-management#geolit

Re: [DISCUSS] GeoLite database licensing change, master build broken…..

[Otto Fowler]

I agree with all of that, my only thinking on contrib is, if that component
is not tested and has no coverage, and needs manual steps, then we may want
to separate it.
We would have to have some handle on full dev as well right?

On January 13, 2020 at 16:57:13, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Hey Otto,

As I mentioned above, we have had this issue with other components before,
e.g. mysql. I don't see a compelling reason to discontinue or push this
component to contrib just yet - it's a type of enrichment that happens to
require an additional manual step. Per the article (
https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/),

Maxmind is not requiring purchase, merely a user to register due to GDPR
and CCPA requirements. That being said, it does cause issues for the
integration tests. I think we should minimally add an @Ignore annotation to
the testLoadGeoIpDatabase test in MaxmindDbEnrichmentLoaderTest and
provide manual instructions for running the integration test if/when
someone submits a PR affecting this code.

Users can still use the geolite DB by uploading it to HDFS. (Incidentally,
this even provides a crude mechanism for versioning, should a user choose
to use the config path that way). We separated the deployment side of the
geolite DB from the consumption side
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabase.java#L135.

All that needs to happen is the user would drop the file in HDFS. This had
previously happened via Ambari - see here -
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py#L95.

We should probably do a couple things:

1. Minimally, remove the DB download and install from Ambari - we might
choose to keep around the HDFS path creation so that we have a reasonable
default prepped and ready.
2. Add documentation to manually install the geolite DB (register with
maxmind) OR remove it from our default demo enrichments shipped with the
platform.

Manual DB file loading would be managed using the CLI tool. You should be
able to load it via local file URL (
https://en.wikipedia.org/wiki/File_URI_scheme) or via a custom hosted
solution via the --geo_url option. See more details here -
https://github.com/apache/metron/tree/master/metron-platform/metron-data-management#geolite2-loader

Anything else I'm missing? Probably worth some feedback from Justin Leet on
this as well.

Thanks,
Mike

On Mon, Jan 13, 2020 at 1:59 PM Otto Fowler 
wrote:

> Hi Tom, that is true, and I think that is the only viable approach. We do
> however use the database for testing during build, and we do however
setup
> the components that use the data base in the ’sample’ flow with the
> simulated sensors for our vagrant deploy… and our contrib/docker deploy
> etc.
>
> So, having the user download their own properly licensed version ( and
> having the user responsible for the privacy law issues ) is fine, but I
> think we need to talk through all the ways we are going to change the
> build, what it means for testing that component ( does it move to contrib
?
> ), and the default deployment to vagrant/topology.
>
>
>
> On January 13, 2020 at 13:41:01, Yerex, Tom (tom.ye...@ubc.ca) wrote:
>
> Hi Otto,
>
> Thank you for raising this in the discussion.
>
> It seems to me that Maxmind is proactive about providing instructions and
> code to deliver updates to the local system. I can recall being surprised
> that the current Metron solution seemed to do more than I expected, i.e.,
I
> thought I would need to get Maxmind files into the local file system
where
> Metron would pick those up and load them into HDFS and instead Metron did
> it all.
>
> Perhaps the approach to simplify Metron and have it load files from the
> local file system into HDFS, how you get the files to the local file
system
> is up to you?
>
>
> On 2020-01-13, 3:52 AM, "Otto Fowler"  wrote:
>
> https://issues.apache.org/jira/browse/METRON–2340
>
>
>
https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
>
> Maxmind has changed the way the distribute and license the geolite2
> database that we use in our builds and distribution.
>
> Master build is broken, and users are having issues setting up metron (
> https://the-asf.slack.com/archives/CB7Q6AN3T/p1578556024012200)
>
>
> We need to fix the build and figure out how we are going to move on from
> this.
>

Re: [DISCUSS] GeoLite database licensing change, master build broken…..

[Otto Fowler]

Hi Tom, that is true, and I think that is the only viable approach.  We do
however use the database for testing during build, and we do however setup
the components that use the data base in the ’sample’ flow with the
simulated sensors  for our vagrant deploy… and our contrib/docker deploy
etc.

So, having the user download their own properly licensed version ( and
having the user responsible for the privacy law issues ) is fine, but I
think we need to talk through all the ways we are going to change the
build, what it means for testing that component ( does it move to contrib ?
), and the default deployment to vagrant/topology.



On January 13, 2020 at 13:41:01, Yerex, Tom (tom.ye...@ubc.ca) wrote:

Hi Otto,

Thank you for raising this in the discussion.

It seems to me that Maxmind is proactive about providing instructions and
code to deliver updates to the local system. I can recall being surprised
that the current Metron solution seemed to do more than I expected, i.e., I
thought I would need to get Maxmind files into the local file system where
Metron would pick those up and load them into HDFS and instead Metron did
it all.

Perhaps the approach to simplify Metron and have it load files from the
local file system into HDFS, how you get the files to the local file system
is up to you?


On 2020-01-13, 3:52 AM, "Otto Fowler"  wrote:

https://issues.apache.org/jira/browse/METRON–2340

https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

Maxmind has changed the way the distribute and license the geolite2
database that we use in our builds and distribution.

Master build is broken, and users are having issues setting up metron (
https://the-asf.slack.com/archives/CB7Q6AN3T/p1578556024012200)


We need to fix the build and figure out how we are going to move on from
this.

[DISCUSS] GeoLite database licensing change, master build broken…..

[Otto Fowler]

https://issues.apache.org/jira/browse/METRON–2340

https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

Maxmind has changed the way the distribute and license the geolite2
database that we use in our builds and distribution.

Master build is broken, and users are having issues setting up metron (
https://the-asf.slack.com/archives/CB7Q6AN3T/p1578556024012200)


We need to fix the build and figure out how we are going to move on from
this.

Re: [ANNOUNCE] Apache Metron-bro-plugin-kafka release 0.3.0

[Otto Fowler]

Just a reminder, if you used my script to verify the RC, please comment :
https://github.com/apache/metron-bro-plugin-kafka/pull/38




On October 16, 2019 at 17:19:24, Justin Leet (l...@apache.org) wrote:

Hi all,

I’m pleased to announce the release of Metron 0.3.0! It's been a little
while coming, but there's a good number of improvements and fixes, both
around functionality and testing. Thanks to everyone who's contributing to
and using the plugin!

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/metron-bro-plugin-kafka/0.3.0/

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/metron-bro-plugin-kafka/0.3.0/

The release branch in GitHub is
https://github.com/apache/metron-bro-plugin-kafka/tree/Metron-bro-plugin-kafka_0.3.0
(tag
apache-metron-bro-plugin-kafka_0.3.0-release)

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

CHANGES (in reverse chronological order):

METRON-2269 Cannot run Docker tests if src is not a git repo
(ottobackwards) closes apache/metron-bro-plugin-kafka#37
METRON-2069 Add btests for bro plugin topic_name selection
(ottobackwards) closes apache/metron-bro-plugin-kafka#36
METRON-2045 Pass a version argument to the bro plugin docker
scripts (JonZeolla) closes apache/metron-bro-plugin-kafka#35
METRON-2003 Bro plugin topic should fall back to the log writer's
path (JonZeolla) closes apache/metron-bro-plugin-kafka#26
METRON-2025 Bro Kafka Plugin Docker should yum clean
(ottobackwards) closes apache/metron-bro-plugin-kafka#33
METRON-2021 Add screen to bro docker image (ottobackwards) closes
apache/metron-bro-plugin-kafka#32
METRON-2013 The bro plugin docker scripts topic name should
be configurable (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#27
METRON-2020 Running run_end_to_end.sh with docker give warning if
bash  4.0 (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#31
METRON-1991 Bro plugin docker scripts should exit nonzero when bro
and kafka counts differ (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#29
METRON-2017 The Bro plugin docker data processing script
incorrectly runs bro (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#30
METRON-1990 Bro plugin docker should exit nonzero if it encounters
issues (JonZeolla) closes apache/metron-bro-plugin-kafka#28
METRON-2004 Bro plugin kafka docker_execute_shell.sh workdir
should be unspecified (JonZeolla via ottobackwards) closes
apache/metron-bro-plugin-kafka#25
METRON-2000 Fix bro plugin docker line counting for BRO_COUNT
(JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#24
METRON-1992 Support sending a log to multiple topics (JonZeolla)
closes apache/metron-bro-plugin-kafka#23
METRON-1910 bro plugin segfaults on src/KafkaWriter.cc:72
(JonZeolla) closes apache/metron-bro-plugin-kafka#20
METRON-1911 Create Docker based test environment for Bro Kafka
Plugin (ottobackwards) closes apache/metron-bro-plugin-kafka#21
METRON-1885 Remove version from bro plugin btest (JonZeolla)
closes apache/metron-bro-plugin-kafka#19
METRON-1827 Update librdkafka in metron-bro-plugin-kafka
(JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#13
METRON-1866 Improve metron-bro-plugin-kafka documentation
(JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#17
METRON-1304 Allow metron-bro-plugin-kafka to include or exclude
logs (JonZeolla via nickwallen) closes
apache/metron-bro-plugin-kafka#2
METRON-1865 Fix metron-bro-plugin-kafka tests (JonZeolla via
jonzeolla) closes apache/metron-bro-plugin-kafka#16
METRON-1828 Improve bro plugin contributing documentation
(JonZeolla) closes apache/metron-bro-plugin-kafka#14
METRON-1818 Remove config_files from bro-pkg.meta (JonZeolla)
closes apache/metron-bro-plugin-kafka#11
METRON-1800 Increment metron-bro-plugin-kafka version (JonZeolla
via jonzeolla) closes apache/metron-bro-plugin-kafka#10
METRON-1773 Bro plugin docs should refer to Apache Metron project
(nickwallen) closes apache/metron-bro-plugin-kafka#9

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC3

[Otto Fowler]

+1 binding Ran RC script including the docker end to end testing




On October 10, 2019 at 14:38:45, Otto Fowler (ottobackwa...@gmail.com)
wrote:

I have written a new script for validation of bro-kafka RC’s

https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/bro-kafka/metron-bro-kafka-rc-check

Please give it a shot. If there are no issues with it, it will go into the
plugin project after the rc is done ( can’t test the rc without the rc )



On October 10, 2019 at 14:31:44, Justin Leet (l...@apache.org) wrote:

This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
The release candidate is available at:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/
Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/CHANGES

The tag to be voted upon is:
apache-metron-bro-plugin-kafka_0.3.0-rc2
<
https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc3
>

The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/apache-metron-bro-plugin-kafka_0.3.0-rc3.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/

The release artifacts are signed with the following key:
*https://dist.apache.org/repos/dist/release/metron/KEYS*
<https://dist.apache.org/repos/dist/release/metron/KEYS>
Please vote on releasing this package as Apache Metron-bro-plugin-kafka
0.3.0

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This is only for the plugin, so the VM validation is not required.

This vote will be open until 3pm EDT on Tuesday October 15 2019, to account
for the weekend.

[ ] +1 Release this package as Apache Metron 0.3.0-RC3

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC3

[Otto Fowler]

I have written a new script for validation of bro-kafka RC’s

https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/bro-kafka/metron-bro-kafka-rc-check

Please give it a shot. If there are no issues with it, it will go into the
plugin project after the rc is done ( can’t test the rc without the rc )




On October 10, 2019 at 14:31:44, Justin Leet (l...@apache.org) wrote:

This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
The release candidate is available at:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/
Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/CHANGES

The tag to be voted upon is:
apache-metron-bro-plugin-kafka_0.3.0-rc2
<
https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc3>


The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/apache-metron-bro-plugin-kafka_0.3.0-rc3.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC3/

The release artifacts are signed with the following key:
*https://dist.apache.org/repos/dist/release/metron/KEYS*

Please vote on releasing this package as Apache Metron-bro-plugin-kafka
0.3.0

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This is only for the plugin, so the VM validation is not required.

This vote will be open until 3pm EDT on Tuesday October 15 2019, to account
for the weekend.

[ ] +1 Release this package as Apache Metron 0.3.0-RC3

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC2

[Otto Fowler]

The fix has landed.

Also, here is a quick script to try for validation

https://github.com/ottobackwards/Metron-and-Nifi-Scripts/blob/master/bro-kafka/metron-bro-kafka-rc-check




On October 1, 2019 at 07:41:02, Otto Fowler (ottobackwa...@gmail.com) wrote:

https://github.com/apache/metron-bro-plugin-kafka/pull/37



On September 30, 2019 at 18:44:29, Otto Fowler (ottobackwa...@gmail.com)
wrote:

–1 binding.

A new feature to our docker testing to add support for the git version /
branch of the plugin to test was added during development.

This feature however makes it impossible to run the docker testing suite
against the signed src tarball for verification.

I have created METRON–2269 for this issue.



On September 30, 2019 at 09:56:54, Justin Leet (l...@apache.org) wrote:

This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
The release candidate is available at:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/CHANGES

The tag to be voted upon is:
apache-metron-bro-plugin-kafka_0.3.0-rc2
<
https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc2
>

The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/apache-metron-bro-plugin-kafka_0.3.0-rc2.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/

The release artifacts are signed with the following key:
*https://dist.apache.org/repos/dist/release/metron/KEYS*
<https://dist.apache.org/repos/dist/release/metron/KEYS>
Please vote on releasing this package as Apache Metron-bro-plugin-kafka
0.3.0

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This is only for the plugin, so the VM validation is not required.

This vote will be open for until 10am EDT on Thursday September 3 2019.

[ ] +1 Release this package as Apache Metron 0.3.0-RC2

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC2

[Otto Fowler]

https://github.com/apache/metron-bro-plugin-kafka/pull/37




On September 30, 2019 at 18:44:29, Otto Fowler (ottobackwa...@gmail.com)
wrote:

–1 binding.

A new feature to our docker testing to add support for the git version /
branch of the plugin to test was added during development.

This feature however makes it impossible to run the docker testing suite
against the signed src tarball for verification.

I have created METRON–2269 for this issue.



On September 30, 2019 at 09:56:54, Justin Leet (l...@apache.org) wrote:

This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
The release candidate is available at:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/CHANGES

The tag to be voted upon is:
apache-metron-bro-plugin-kafka_0.3.0-rc2
<
https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc2
>

The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/apache-metron-bro-plugin-kafka_0.3.0-rc2.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/

The release artifacts are signed with the following key:
*https://dist.apache.org/repos/dist/release/metron/KEYS*
<https://dist.apache.org/repos/dist/release/metron/KEYS>
Please vote on releasing this package as Apache Metron-bro-plugin-kafka
0.3.0

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This is only for the plugin, so the VM validation is not required.

This vote will be open for until 10am EDT on Thursday September 3 2019.

[ ] +1 Release this package as Apache Metron 0.3.0-RC2

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC2

[Otto Fowler]

–1 binding.

A new feature to our docker testing to add support for the git version /
branch of the plugin to test was added during development.

This feature however makes it impossible to run the docker testing suite
against the signed src tarball for verification.

I have created METRON–2269 for this issue.




On September 30, 2019 at 09:56:54, Justin Leet (l...@apache.org) wrote:

This is a call to vote on releasing Apache Metron-bro-plugin-kafka 0.3.0
The release candidate is available at:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/
Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/CHANGES

The tag to be voted upon is:
apache-metron-bro-plugin-kafka_0.3.0-rc2
<
https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc2>


The source archives being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/apache-metron-bro-plugin-kafka_0.3.0-rc2.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC2/

The release artifacts are signed with the following key:
*https://dist.apache.org/repos/dist/release/metron/KEYS*

Please vote on releasing this package as Apache Metron-bro-plugin-kafka
0.3.0

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This is only for the plugin, so the VM validation is not required.

This vote will be open for until 10am EDT on Thursday September 3 2019.

[ ] +1 Release this package as Apache Metron 0.3.0-RC2

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [DISCUSS] Completing HDP 3.1 upgrade feature branch and next steps for Metron 1.x release

[Otto Fowler]

I think we also need:

   -

   Draft of upgrade guide, reviewed and tried. What is missing? What is
   hard? Possible refactoring from there.
   -

   Draft of the new supported versions of XX for Metron
   -

   How are we going to tell current users the change in support? I am not
   sure burying in the readme is good enough. George isn’t doing blogs
   anymore, but something like that would be cool, we should also be ready
   with twitter and other things we can think of. Maybe James can do something
   like that?
   -

   Should the docker images we use for building etc be centos 7 too?
   -

   Have we removed the centos6 vagrant completely? updated the docs? Is
   that in the upgrade notes?
   -

   We have to make sure the full-dev stuff is all set and solid
   -

   What about the Amazon script and stuff, the documentation for that?

I see if I can think of more, hopefully you all will too.




On September 30, 2019 at 17:14:09, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Hi all,

We're getting close to wrapping up the feature branch work for upgrading to
HDP 3.1. As previously discussed, this is a backwards incompatible upgrade
and is going to require a 1.x Metron release. There has already been some
discussion on this here -
https://lists.apache.org/thread.html/21e9caf80c0ec4d9db4cb44423befa87cd1e42d327860369a6a13273@%3Cdev.metron.apache.org%3E
.

I want to hash out any remaining tasks we think would be gating factors for
getting the upgrade feature branch merged into master, and subsequently a
1.x Metron release. Based on feedback in the previous dev list discussion
from above, I see a utility for helping automate the upgrade process (i.e.
config backup and restore) being the last major task. Including the upgrade
script, these are the remaining tasks I've identified:

1. Finish Metron upgrade script
2. Upgrade hadoop - PR to follow from
https://github.com/mmiklavc/metron/tree/METRON-2232 (I'm currently
running this up in Vagrant)
3. Cleanup Centos 7 dev environment documentation
4. Finish Kerberos testing with all components now upgraded (depends on
hadoop upgrade being finished) https://github.com/apache/metron/pull/1519.
This should serve as our final acceptance testing for the upgrade - we've
heavily tested the FB on each PR, so this would be a final e2e check of
Metron with Kerberos enabled.
5. Final DISCUSS to merge the FB into master (previous example of what
we did with PCAP here -
https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E
)

*Note*: there are a couple tasks in the FB epic Jira around deprecating our
custom code with Kafka spouts. As it is, changing/upgrading this code is
not required for the upgrade. To reduce overall upgrade risk, we want to
consider this work as a follow on in a later point release.

Here are the implications of this upgrade work from a supported components
perspective:

- HDP 2.x is no longer supported - must upgrade
- Centos 6 is no longer supported - must upgrade to Centos 7 (Ambari
relies on binaries shipped by the HDP 3.1 distro, and they do not have any
binaries for Centos 6)
- Solr has also been upgraded from 6.6.2 to 7.4.0 -
https://issues.apache.org/jira/browse/METRON-2225 (No changes to the
Elasticsearch version have been made)

I think that covers everything. Once we get the FB merged, I would expect
we would want to move towards a discussion and vote on releasing Metron
1.0. Does this sound reasonable to everyone?

Best,
Mike

Re: [VOTE] Metron-bro-plugin-kafka Release Candidate 0.3.0-RC1

[Otto Fowler]

If you are doing the RM duties, just go a head and cut the RC.




On September 29, 2019 at 11:35:10, zeo...@gmail.com (zeo...@gmail.com)
wrote:

Hi everyone,

This took much longer than expected, but we recently cleaned up the last
blocker to getting a 0.3 release out. As a reminder, currently the latest
apache/metron-bro-plugin-kafka release has a bug causing the tests to
fail. Master no longer has this, and in fact has a fair amount of
improvements both in features and automated testing which went in since
this rc. I would like to see them all included in the new 0.3 rc (i.e.
align the release with master).

We may also want to consider a fast following release that includes the two
current open PRs (after review) that were waiting on the bro/zeek 3.0
release (which recently happened). Because these cause breaking changes I
do not think they should be in the 0.3 release, but rather are good
candidates for a 1.0.0 release (moving from x.y to x.y.z versioning as
well). At that time we may also want to consider a project rename due to
the bro project rename to zeek, but I'll leave that for a separate thread
for after we get 0.3 out.

Jon Zeolla

On Fri, Nov 30, 2018, 6:14 PM Justin Leet  wrote:

> The vote has failed. The voting was:
> 1 -1’s (binding)
>
> A new RC will be created once this issue is resolved.
>
> On Wed, Nov 28, 2018 at 10:49 AM zeo...@gmail.com 
> wrote:
>
> > -1
> >
> > In my testing it appears that an issue was introduced in 0.2 which is
> > causing a segfault on the destructor (
> >
> >
>
https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57
> > ).
> > I've opened METRON-1910 and am testing a fix now.
> >
> > On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:
> >
> > > This is a call to vote on releasing Apache Metron-bro-plugin-kafka
> 0.3.0
> > > The release candidate is available at:
> > >
> > >
> >
>
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> > >
> > > Full list of changes in this release:
> > >
> > >
> >
>
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES
> > >
> > > The tag to be voted upon is:
> > > apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > <
> > >
> >
>
https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > >
> > >
> > > The source archives being voted upon can be found here:
> > >
> > >
> >
>
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/apache-metron-bro-plugin-kafka_0.3.0-rc1.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > >
> > >
> >
>
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> > >
> > > The release artifacts are signed with the following key:
> > > *https://dist.apache.org/repos/dist/release/metron/KEYS*
> > > 
> > > Please vote on releasing this package as Apache
Metron-bro-plugin-kafka
> > > 0.3.0
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > Recommended build validation and verification instructions are posted
> > here:
> > > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > >
> > > This is only for the plugin, so the VM validation is not required.
> > >
> > > This vote will be open for until 3pm EDT on Friday November 27 2018.
> > >
> > > [ ] +1 Release this package as Apache Metron 0.3.0-RC1
> > >
> > > [ ] 0 No opinion
> > >
> > > [ ] -1 Do not release this package because...
> > >
> > --
> >
> > Jon Zeolla
> >
>

On Fri, Nov 30, 2018, 6:14 PM Justin Leet  wrote:

> The vote has failed. The voting was:
> 1 -1’s (binding)
>
> A new RC will be created once this issue is resolved.
>
> On Wed, Nov 28, 2018 at 10:49 AM zeo...@gmail.com 
> wrote:
>
> > -1
> >
> > In my testing it appears that an issue was introduced in 0.2 which is
> > causing a segfault on the destructor (
> >
> >
>
https://github.com/apache/metron-bro-plugin-kafka/commit/1dfc5239fae31a64026188109d1e346ce93d5c02#diff-361be0491d615952129ed5c8f39c9683L57
> > ).
> > I've opened METRON-1910 and am testing a fix now.
> >
> > On Tue, Nov 27, 2018 at 2:36 PM Justin Leet  wrote:
> >
> > > This is a call to vote on releasing Apache Metron-bro-plugin-kafka
> 0.3.0
> > > The release candidate is available at:
> > >
> > >
> >
>
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/
> > >
> > > Full list of changes in this release:
> > >
> > >
> >
>
https://dist.apache.org/repos/dist/dev/metron/metron-bro-plugin-kafka/0.3.0-RC1/CHANGES
> > >
> > > The tag to be voted upon is:
> > > apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > <
> > >
> >
>
https://github.com/apache/metron-bro-plugin-kafka/tree/apache-metron-bro-plugin-kafka_0.3.0-rc1
> > > >
> > >
> > > The source archives being voted upon can be found here:
> > >
> > >
> >
>

Re: [DISCUSS] Curator client upgrade

[Otto Fowler]

+1




On September 17, 2019 at 14:53:13, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Also, caught this come through the HBase dev list -
https://issues.apache.org/jira/browse/HBASE-23032. Might be worth a look at
4.2.0 while we're at it and possibly also avoid this issue that I was
running into before when trying to use curator-tests 4.x
https://issues.apache.org/jira/browse/CURATOR-446. Fixed in 4.0.1, which
would naturally be included with 4.2.0 assuming we don't run into any other
classpath or API compatibility issues.

On Tue, Sep 17, 2019 at 11:20 AM Nick Allen  wrote:

> +1 to making the change on the feature branch.
>
> We don't really know how this might affect master which is still building
> against HDP 2.6, nor is it strictly needed there. Going to Curator 4.0.0
> is only needed due to the HDP 3.1 upgrade. This is also likely to get
> more focused testing cycles in the feature branch before it has a chance
to
> break anything in master.
>
> On Tue, Sep 17, 2019 at 1:13 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey all,
> >
> > While working through the feature branch upgrade for HDP 3.1, we came
> > across some classpath related issues conflicting with Storm and Guava
> while
> > testing out Kerberos. Initially, this seemed reasonable to roll into
the
> > Kerberos fix PR, but the scope has expanded a bit because we found
> > additional impacts while working through the Hadoop upgrade branch as
> well.
> > We're now looking at splitting this out into a separate PR in the
feature
> > branch. Curator 4.0.0 is backwards compatible and will work with
> Zookeeper
> > 3.4.6. Strictly speaking, this *could* be done against master first,
but
> > there may be some duplicate testing effort there, and I'm really not
it's
> > worth it as there is absolutely no issue currently with Curator 2.12.0
in
> > master - I call this out purely for transparency/information share with
> the
> > community. I'm leaning towards us making this change against the
feature
> > branch directly as that is the only place we currently have
compatibility
> > issues.
> >
> > Best,
> > Mike
> >
>

Re: [DISCUSS] Metron installation and management - deprecating Ambari support

[Otto Fowler]

I need a little while to think about this, but I would just like to say
that NiFi is a really bad example to compare Metron too.

Nifi can be deployed as a single zip file and run from wherever it is
unzipped. It is more of an application that can be clustered with other
instances of that application. Metron is pretty different from that.

There is no equivalent usable ‘Metron’ application.

Can we think of a better example for inspiration?




On September 12, 2019 at 16:15:06, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

I'd like to discuss Metron's installation and management. We have used
Ambari for some time now, with and without an MPack for Metron (and
Elasticsearch). While this mechanism has proved useful to the project, it
is not without cost. This makes us an outlier among Apache projects in
terms of what we support as part of our installation and management
mechanism. It greatly expands the scope of responsibility for the Metron
community in terms of feature coverage that is not core to the
cybersecurity platform.

We are currently undertaking an upgrade (
https://issues.apache.org/jira/browse/METRON-2088) to beat some product EOL
deadlines for Storm and the HDP binaries we depend on for deployment with
Ambari. This upgrade is critical for the Metron community. Ambari does not
prepackage any Hadoop binaries (see for example
https://cwiki.apache.org/confluence/display/AMBARI/Quick+Start+Guide), so
for convenience we had been leveraging a commercial vendor's (Hortonworks)
binaries for HDP to fill this gap. There are a couple issues with this.

First, and most dire, is that in the recent Cloudera earnings call
(Hortonworks and Cloudera have merged, if you're unaware), they have
publicly stated that *binaries will no longer be freely available to the
public *(see footnote 1). This means that our development environment is
tightly coupled to deployment artifacts that may cease to exist, and
subsequently break over night. We need to act on this immediately.

Second, there is currently a bug in the latest versions of Ambari that
supports HDP 3.1 and our MPack and we are still working out if there is a
fix we can implement on our end to get around it - see
https://issues.apache.org/jira/browse/AMBARI-25375. This is very likely a
blocker for an upgrade if we cannot find a workaround (there has been one
failed attempt, another is under way so fingers crossed).

Many other projects simply provide binaries and basic instructions for
deploying and using their products. See NiFi, for example -
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi.

The product is developed in an a flexible and extensible way, and
individuals and vendors are free to expand on that basic functionality.

I propose we take a similar approach and move to deprecate and replace
Ambari in one fell swoop as part of our 1.0 Metron release initiative. In
addition to the reasons listed above, this would also simplify our
development process, ease onboarding of new contributors and committers in
the community, and decrease the overall burden to the community for new
features and releases. The replacement for the development environment
could be a combination of Ansible and shell/Python scripts for managing
laying down the Metron bits. One option that has been discussed in the past
is to leverage open source versions of Docker containers for our service
dependencies. Recent conversations I've had with community devs suggests
this is still a popular solution. I agree. Quite a bit of Metron system
management is currently duplicated between Ambari and the management UI,
and we could maintain or expand on the current feature set in our UI as
desired. For general Hadoop/cluster management, we would no longer provide
a prepackaged option - BYOD as it were. There are numerous OSS and
commercial solutions available today, and users and vendors alike would be
free to choose one that works best for their specific use case(s) and
customer(s).

Thanks everyone. This is not a vote, but opening the floor for discussion.

Best,
Mike Miklavcic
Metron PMC

*Notes:*

(1) From the Cloudera earnings call -
https://www.fool.com/earnings/call-transcripts/2019/09/04/cloudera-inc-cldr-q2-2020-earnings-call-transcript.aspx

"Secondly, the distribution of our compiled software, the binaries, will be
> limited. Specifically, access to these binaries, as well as support
> services and technical expertise will be available only with the current
> subscription agreement. The binaries contain Cloudera-specific
intellectual
> property, the testing, securing, and integration of various Open Source
> projects into an enterprise grade system that meets the requirements of
our
> customers.
>


Consistent with the Red Hat model, our binaries will no longer be freely
> available to non-paying customers. These changes to licensing and
> distribution will affect all subsequent versions of our current products
as
> well 

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler]

If anyone can think of the things that need to be backed up, please comment
the jira.




On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwa...@gmail.com) wrote:

Good idea METRON–2239 [blocker].



On August 27, 2019 at 16:30:13, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

You could always submit a Jira :)

On Tue, 27 Aug 2019 at 21:27, Otto Fowler  wrote:

> You are right, that is much better than backup_metron_configs.sh.
>
>
>
> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> kinda already do.
>
> Simon
>
> On Tue, 27 Aug 2019 at 20:19, Otto Fowler  wrote:
>
>> Maybe we need some automated method to backup configurations and restore
>> them.
>>
>>
>>
>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>> michael.miklav...@gmail.com) wrote:
>>
>> > Once you back up your metron configs, the same configs that worked on
>> the previous version will continue to work on the version running on HDP
>> 3.x.  If there is any discrepancy between the two or additional settings
>> will be required, those will be documented in the release notes.  From the
>> Metron perspective, this upgrade would be no different than simply
>> upgrading to the new Metron version.
>>
>> This upgrade cannot be performed the same way we've done it in the past.
>> A number of platform upgrades, including OS, are required:
>>
>>1. Requires the OS to be updated on all nodes because there are no
>>Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>2. The final new HBase code will not run on HDP 2.6
>>3. The MPack changes for new Ambari are not backwards compatible
>>4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>
>>
>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> Adding the dev list back into the thread (a reply-all was missed).
>>>
>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota 
>>> wrote:
>>>
>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>> looks like so they have time to plan and upgrade at their own pace.
>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>> merely the platform underneath that is changing.  HDP itself can be
>>>> upgraded per instructions here:
>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>
>>>> Once you back up your metron configs, the same configs that worked on
>>>> the previous version will continue to work on the version running on HDP
>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>> will be required, those will be documented in the release notes.  From the
>>>> Metron perspective, this upgrade would be no different than simply
>>>> upgrading to the new Metron version.
>>>>
>>>> James
>>>>
>>>>
>>>> 27.08.2019, 07:09, "Simon Elliston Ball" :
>>>>
>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>> of zero users who wish to remain on HDP 2.
>>>>
>>>> Perhaps those users who want to stay on the old platform can stick
>>>> their hands up and raise concerns, but this move will likely have to happen
>>>> very soon.
>>>>
>>>> Simon
>>>>
>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler 
>>>> wrote:
>>>>
>>>> Although we had the discussion, and some great ideas where passed
>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>> should look like. So that discussion would have to be picked up again so
>>>> that we could know where we are at, and make it an actual thing if we were
>>>> going to make this a 1.0 release.
>>>>
>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>> still largely applicable, including upgrade.
>>>>
>>>> Right now we hav

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler]

Good idea METRON–2239 [blocker].




On August 27, 2019 at 16:30:13, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

You could always submit a Jira :)

On Tue, 27 Aug 2019 at 21:27, Otto Fowler  wrote:

> You are right, that is much better than backup_metron_configs.sh.
>
>
>
> On August 27, 2019 at 16:05:38, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> You can do this with zk_load_configs and Ambari’s blueprint api, so we
> kinda already do.
>
> Simon
>
> On Tue, 27 Aug 2019 at 20:19, Otto Fowler  wrote:
>
>> Maybe we need some automated method to backup configurations and restore
>> them.
>>
>>
>>
>> On August 27, 2019 at 14:46:58, Michael Miklavcic (
>> michael.miklav...@gmail.com) wrote:
>>
>> > Once you back up your metron configs, the same configs that worked on
>> the previous version will continue to work on the version running on HDP
>> 3.x.  If there is any discrepancy between the two or additional settings
>> will be required, those will be documented in the release notes.  From the
>> Metron perspective, this upgrade would be no different than simply
>> upgrading to the new Metron version.
>>
>> This upgrade cannot be performed the same way we've done it in the past.
>> A number of platform upgrades, including OS, are required:
>>
>>1. Requires the OS to be updated on all nodes because there are no
>>Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>>2. The final new HBase code will not run on HDP 2.6
>>3. The MPack changes for new Ambari are not backwards compatible
>>4. YARN and HDFS/MR are also at risk to be backwards incompatible
>>
>>
>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> Adding the dev list back into the thread (a reply-all was missed).
>>>
>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota 
>>> wrote:
>>>
>>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>>> everyone will likely need to migrate by end of year.  Doing this platform
>>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>>> looks like so they have time to plan and upgrade at their own pace.
>>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>>> merely the platform underneath that is changing.  HDP itself can be
>>>> upgraded per instructions here:
>>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>>
>>>> Once you back up your metron configs, the same configs that worked on
>>>> the previous version will continue to work on the version running on HDP
>>>> 3.x.  If there is any discrepancy between the two or additional settings
>>>> will be required, those will be documented in the release notes.  From the
>>>> Metron perspective, this upgrade would be no different than simply
>>>> upgrading to the new Metron version.
>>>>
>>>> James
>>>>
>>>>
>>>> 27.08.2019, 07:09, "Simon Elliston Ball" :
>>>>
>>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>>> large number of users who require this upgrade ASAP, and in fact an aware
>>>> of zero users who wish to remain on HDP 2.
>>>>
>>>> Perhaps those users who want to stay on the old platform can stick
>>>> their hands up and raise concerns, but this move will likely have to happen
>>>> very soon.
>>>>
>>>> Simon
>>>>
>>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler 
>>>> wrote:
>>>>
>>>> Although we had the discussion, and some great ideas where passed
>>>> around, I do not believe we came to some kind of consensus on what 1.0
>>>> should look like. So that discussion would have to be picked up again so
>>>> that we could know where we are at, and make it an actual thing if we were
>>>> going to make this a 1.0 release.
>>>>
>>>> I believe that the issues raised in that discussion gating 1.0 are
>>>> still largely applicable, including upgrade.
>>>>
>>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>>> supporting 3.1 work and releases. So every user and deployment we currently
>>>> hav

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler]

You are right, that is much better than backup_metron_configs.sh.




On August 27, 2019 at 16:05:38, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

You can do this with zk_load_configs and Ambari’s blueprint api, so we
kinda already do.

Simon

On Tue, 27 Aug 2019 at 20:19, Otto Fowler  wrote:

> Maybe we need some automated method to backup configurations and restore
> them.
>
>
>
> On August 27, 2019 at 14:46:58, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> > Once you back up your metron configs, the same configs that worked on
> the previous version will continue to work on the version running on HDP
> 3.x.  If there is any discrepancy between the two or additional settings
> will be required, those will be documented in the release notes.  From the
> Metron perspective, this upgrade would be no different than simply
> upgrading to the new Metron version.
>
> This upgrade cannot be performed the same way we've done it in the past. A
> number of platform upgrades, including OS, are required:
>
>1. Requires the OS to be updated on all nodes because there are no
>Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
>2. The final new HBase code will not run on HDP 2.6
>3. The MPack changes for new Ambari are not backwards compatible
>4. YARN and HDFS/MR are also at risk to be backwards incompatible
>
>
> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Adding the dev list back into the thread (a reply-all was missed).
>>
>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota  wrote:
>>
>>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>>> everyone will likely need to migrate by end of year.  Doing this platform
>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>>> looks like so they have time to plan and upgrade at their own pace.
>>> Feature-wise, the Metron application itself will be unchanged.  It is
>>> merely the platform underneath that is changing.  HDP itself can be
>>> upgraded per instructions here:
>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>>
>>> Once you back up your metron configs, the same configs that worked on
>>> the previous version will continue to work on the version running on HDP
>>> 3.x.  If there is any discrepancy between the two or additional settings
>>> will be required, those will be documented in the release notes.  From the
>>> Metron perspective, this upgrade would be no different than simply
>>> upgrading to the new Metron version.
>>>
>>> James
>>>
>>>
>>> 27.08.2019, 07:09, "Simon Elliston Ball" :
>>>
>>> Something worth noting here is that HDP 2.6.5 is quite old and
>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>>> large number of users who require this upgrade ASAP, and in fact an aware
>>> of zero users who wish to remain on HDP 2.
>>>
>>> Perhaps those users who want to stay on the old platform can stick their
>>> hands up and raise concerns, but this move will likely have to happen very
>>> soon.
>>>
>>> Simon
>>>
>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler 
>>> wrote:
>>>
>>> Although we had the discussion, and some great ideas where passed
>>> around, I do not believe we came to some kind of consensus on what 1.0
>>> should look like. So that discussion would have to be picked up again so
>>> that we could know where we are at, and make it an actual thing if we were
>>> going to make this a 1.0 release.
>>>
>>> I believe that the issues raised in that discussion gating 1.0 are still
>>> largely applicable, including upgrade.
>>>
>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>>> supporting 3.1 work and releases. So every user and deployment we currently
>>> have will feel real pain, have to slay real dragons to move forward with
>>> metron.
>>>
>>> With regards to support for older versions, the “backward capability”
>>> that has been mentioned, I would not say that I have any specific plan for
>>> that in mind. What I would say rather, is that I believe that we must be
>>> explicit, setting expectations correctly and clearly with regards to our
>>> intent while demonstrating that we have thought through the situation. That
>>> discussion has not happened, at least I do not believe t

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler]

Maybe we need some automated method to backup configurations and restore
them.




On August 27, 2019 at 14:46:58, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

> Once you back up your metron configs, the same configs that worked on the
previous version will continue to work on the version running on HDP 3.x.
If there is any discrepancy between the two or additional settings will be
required, those will be documented in the release notes.  From the Metron
perspective, this upgrade would be no different than simply upgrading to
the new Metron version.

This upgrade cannot be performed the same way we've done it in the past. A
number of platform upgrades, including OS, are required:

   1. Requires the OS to be updated on all nodes because there are no
   Centos6 RPMs provided in HDP 3.1. Must bump to Centos7.
   2. The final new HBase code will not run on HDP 2.6
   3. The MPack changes for new Ambari are not backwards compatible
   4. YARN and HDFS/MR are also at risk to be backwards incompatible


On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Adding the dev list back into the thread (a reply-all was missed).
>
> On Tue, Aug 27, 2019 at 10:49 AM James Sirota  wrote:
>
>> I agree with Simon.  HDP 2.x platform is rapidly approaching EOL and
>> everyone will likely need to migrate by end of year.  Doing this platform
>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x
>> looks like so they have time to plan and upgrade at their own pace.
>> Feature-wise, the Metron application itself will be unchanged.  It is
>> merely the platform underneath that is changing.  HDP itself can be
>> upgraded per instructions here:
>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html
>>
>> Once you back up your metron configs, the same configs that worked on the
>> previous version will continue to work on the version running on HDP 3.x.
>> If there is any discrepancy between the two or additional settings will be
>> required, those will be documented in the release notes.  From the Metron
>> perspective, this upgrade would be no different than simply upgrading to
>> the new Metron version.
>>
>> James
>>
>>
>> 27.08.2019, 07:09, "Simon Elliston Ball" :
>>
>> Something worth noting here is that HDP 2.6.5 is quite old and
>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a
>> large number of users who require this upgrade ASAP, and in fact an aware
>> of zero users who wish to remain on HDP 2.
>>
>> Perhaps those users who want to stay on the old platform can stick their
>> hands up and raise concerns, but this move will likely have to happen very
>> soon.
>>
>> Simon
>>
>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler 
>> wrote:
>>
>> Although we had the discussion, and some great ideas where passed around,
>> I do not believe we came to some kind of consensus on what 1.0 should look
>> like. So that discussion would have to be picked up again so that we could
>> know where we are at, and make it an actual thing if we were going to make
>> this a 1.0 release.
>>
>> I believe that the issues raised in that discussion gating 1.0 are still
>> largely applicable, including upgrade.
>>
>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
>> supporting 3.1 work and releases. So every user and deployment we currently
>> have will feel real pain, have to slay real dragons to move forward with
>> metron.
>>
>> With regards to support for older versions, the “backward capability”
>> that has been mentioned, I would not say that I have any specific plan for
>> that in mind. What I would say rather, is that I believe that we must be
>> explicit, setting expectations correctly and clearly with regards to our
>> intent while demonstrating that we have thought through the situation. That
>> discussion has not happened, at least I do not believe that the prior dev
>> thread really handles it in context.
>>
>> Depending on the upgrade situation for going to 3.1, it may be that a
>> dual stream of releases or fixes or new features to the extent that we can
>> do it will greatly reduce the pain for folks, or make it viable to stick
>> with metron until they can upgrade.
>>
>> The issue of what metron *is* features wise may be another one we want
>> to take up at some point. The idea being can we separate the metron
>> _integration parts from the metron core functionality such that we can work
>> on them separately and thus support multiple platforms thro

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

[Otto Fowler]

Although we had the discussion, and some great ideas where passed around, I
do not believe we came to some kind of consensus on what 1.0 should look
like. So that discussion would have to be picked up again so that we could
know where we are at, and make it an actual thing if we were going to make
this a 1.0 release.

I believe that the issues raised in that discussion gating 1.0 are still
largely applicable, including upgrade.

Right now we have *ZERO* HDP 3.1 users. We will go from that to *only*
supporting 3.1 work and releases. So every user and deployment we currently
have will feel real pain, have to slay real dragons to move forward with
metron.

With regards to support for older versions, the “backward capability” that
has been mentioned, I would not say that I have any specific plan for that
in mind. What I would say rather, is that I believe that we must be
explicit, setting expectations correctly and clearly with regards to our
intent while demonstrating that we have thought through the situation. That
discussion has not happened, at least I do not believe that the prior dev
thread really handles it in context.

Depending on the upgrade situation for going to 3.1, it may be that a dual
stream of releases or fixes or new features to the extent that we can do it
will greatly reduce the pain for folks, or make it viable to stick with
metron until they can upgrade.

The issue of what metron *is* features wise may be another one we want to
take up at some point. The idea being can we separate the metron
_integration parts from the metron core functionality such that we can work
on them separately and thus support multiple platforms through
integrations/applications. Of course definition of metron’s value beyond
integration, and what those features and application boundaries are would
be necessary.




On August 26, 2019 at 18:52:57, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Hi devs and users,

Some questions were asked in the Slack channel about our ongoing HDP/Hadoop
upgrade and I'd like to get a discussion rolling. The original Hadoop
upgrade discuss thread can be found here
https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E

The major issue and impact with upgrading the Hadoop platform is that there
are breaking changes. Code that runs on HDP 3.1 will not run on 2.x. Here
is a sampling of core components we depend on that we know of so far that
are not backwards compatible:

   1. The Core OS - we currently base our builds and test deployment off of
   artifacts pulled from HDP. The latest rev of HDP no longer ships RPMs for
   Centos 6, which means we need to upgrade to Centos 7
   2. Ambari
   3. HBase

This differs from individual components we've upgraded in the past in that
our code could still be deployed on the old as well as new version of the
component in a backwards compatible way. Based on semantic versioning, I
don't know if we can introduce this level of change in a point release,
which is the reason for kicking off this discussion. In the past, users and
developers in the community have suggested that they are -1 on a 1.x
release that does not provide an upgrade
https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E
.

Is there a way we can avoid a 1.x release? If we do need 1.x, do we still
see upgrades as a gating function? The main issue is that this has the
potential to drag out the upgrade and further couple it with other
features. And with Storm 1.x being eol'ed, I'm not sure this is something
we can wait much longer for. I'll think on this and send out my own
thoughts once folks have had a chance to review.

Best,
Mike Miklavcic
Apache Metron, PMC, committer

Re: [DISCUSS] Deprecate Least Recently Used Pruner

[Otto Fowler]

Can you summarize what it does? Is it from OpenSOC?




On August 13, 2019 at 17:53:52, Nick Allen (n...@nickallen.org) wrote:

As part of https://github.com/apache/metron/pull/1470, I found it difficult
to update the "Least Recently Used Pruner" to work with HBase 2.0.2. I am
sure that given more time and effort, I could make it work, but is it worth
it?

This is a feature that I myself am not familiar with. I do not know of
anyone using this. I also did not find much documentation on how to use
this feature. I certainly don't know the entire user community, so please
let me know if anyone is using this functionality or believes that it
should be maintained going forward.

Would you support deprecating this feature?

Thanks

Jira issue creation

[Otto Fowler]

I know that there are many times where the PR work precedes the jira
creation, so that you are creating the jira to match the work you are
doing, and that is fine as there are all sorts of workflows for
contributors.

That being said, I don’t think it is great to have empty jiras just to
placeholder for PR’s. It would be nice if we tried to have some information
in there.

Have a good one

O

Re: Contributing to Metron

[Otto Fowler]

What is your jira id?




On August 13, 2019 at 13:06:36, Yerex, Tom (tom.ye...@ubc.ca) wrote:

I would like to get access to JIRA (or be assigned a ticket), to add
documentation to the Centos7 project folder under
metron/metron-deployment/development/*centos7*/, similar to the README.md
file that is under the centos6 folder.



I am new to open source development, hopefully this is a simple enough
change that I will not cause too much of a distraction and contribute to
the project.



Thank you,



Tom.



--

*Tom Yerex*

Cybersecurity Analyst

Cybersecurity | CISO Office

The University of British Columbia | Musqueam Traditional Territory

Ponderosa Office Annex A | Vancouver BC | V6T1Z2 Canada

Phone 604 822 6531

Privacy Matters @ UBC

Re: Hello Metron Dev Community

[Otto Fowler]

Welcome Dale!

Btw: Dale is Tigerquoll on slack :)




On July 30, 2019 at 03:09:22, Dale Richardson (tigerqu...@outlook.com)
wrote:

Hi folks,
Greetings from down under! I just thought I'd introduce myself now that I
am starting to spend some time hanging around the list. I'm Dale, I've been
involved in consulting in Data Warehousing and big data development for
more than 15 years. I've started to get involved with Metron over the past
couple of months.

I was active in Spark development during its early days and have spent some
time poking around the SOLR code base as well. I've also done a fair bit of
Kafka integration work and I've just starting to play around with Flink.

Given the timezone difference, I am guessing most people would be active
when I'm not and vice-a-versa, so I'm guessing most interactions with
people are likely to be via Jira or email.

Dale.

Re: Metron/Docker failing - should I file a Jira?

[Otto Fowler]

Hi Phillip,

The docker compose stuff is in contrib because it uses a different method
to deploy the cluster than the official builds ( which use Ambari ), and
thus has to be maintained
separately, and will always lag behind.

I don’t believe it is up to date with all the changes that have gone into
the build recently.  You can log a jira, and someone may be able to look at
it.

In the future, I would like to have a docker compose system that sets up
the ssh keys and ambari clients/server such that we can deploy using the
regular ambari
install, but that hasn’t been done.



On June 20, 2019 at 09:21:54, Phillip Henry (londonjava...@gmail.com) wrote:

Oops! I think I had forgotten to execute:

eval "$(docker-machine env metron-machine)"

Having said that, when I now run docker-compose up, I still fail with:

Step 16/23 : RUN chmod 755 /opt/kafka/bin/wait-for-it.sh
---> Running in 768c0e9ac680
chmod: cannot access '/opt/kafka/bin/wait-for-it.sh': No such file or
directory
ERROR: Service 'kafkazk' failed to build: The command '/bin/sh -c chmod 755
/opt/kafka/bin/wait-for-it.sh' returned a non-zero code: 1

I've since updated docker-compose to 1.25.0-rc1, build 8552e8e2 from 1.8
and some other errors seemed to go away.

Can anybody please confirm that Metron's Docker config is working for them?

Thanks,

Phillip





On Mon, Jun 17, 2019 at 10:26 AM Phillip Henry 
wrote:

> I was following the instructions at:
>
>
>
https://metron.apache.org/current-book/metron-contrib/metron-docker/index.html
>
> and when executing:
>
> $ docker-compose up -d
>
>
> it failed with:
>
> .
> .
> .
> Building elasticsearch
> Step 1/7 : FROM elasticsearch:2.3
> 2.3: Pulling from library/elasticsearch
> 386a066cd84a: Pull complete
> 75ea84187083: Pull complete
> 3e2e387eb26a: Pull complete
> eef540699244: Pull complete
> 1624a2f8d114: Pull complete
> 7018f4ec6e0a: Pull complete
> 6ca3bc2ad3b3: Pull complete
> 424638b495a6: Pull complete
> 2ff72d0b7bea: Pull complete
> 9d25542ccc02: Pull complete
> 35456bfab3fd: Pull complete
> c206de1a2db8: Pull complete
> 3fd5839fafc9: Pull complete
> e11632209e5b: Pull complete
> Digest:
> sha256:336e82bf4a8edee630efcd112ee388fd52b1dd04b0c47300f3efa60ed67a266e
> Status: Downloaded newer image for elasticsearch:2.3
> ---> 1c3e7681c53c
> Step 2/7 : RUN /usr/share/elasticsearch/bin/plugin install
> mobz/elasticsearch-head
> ---> Running in e95862076175
> -> Installing mobz/elasticsearch-head...
> Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
> Downloading
>
DONE

> Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip
> checksums if available ...
> NOTE: Unable to verify checksum for downloaded plugin (unable to find
> .sha1 or .md5 file to verify)
> Installed head into /usr/share/elasticsearch/plugins/head
> Removing intermediate container e95862076175
> ---> 6e6cdead0273
> Step 3/7 : ADD ./es_templates /es_templates
> ERROR: Service 'elasticsearch' failed to build: ADD failed: stat
> /var/lib/docker/tmp/docker-builder838578399/es_templates: no such file or
> directory
>
> I'm using:
>
> $ docker --version
> Docker version 18.09.2, build 6247962
>
> on
>
> $ uname -a
> Linux corsair 4.4.0-141-generic #167-Ubuntu SMP Wed Dec 5 10:40:15 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> Should I file a JIRA?
>
> Thanks,
>
> Phillip
>
>
>

Re: What's the status of Metron

[Otto Fowler]

Hi Phillip and welcome!

You certainly can help.

As far as the status, there are several ways to look at it aren’t there?

- we have active committers and reviewers, but not as many as we would like
- we have an active users like, but not insanely so
- metron _is_ part of a large bistro package HCP / cloudera ? and deployed,
supported and used there


A lot of work is going on right now in the following areas

- refactoring the UI technologies and testability
- making the build better, speed, shading etc
- de-coupling storm from our core technologies so we can introduce or
support alternative processing ( spark for example )
- work on our deployment / dev experience ( vagrant/docker )

and other things I’ve not thought of, and have PR’s etc.

we can always use help with documentation, reviews, testing, how-to’s or
development :)



On June 7, 2019 at 12:40:38, Phillip Henry (londonjava...@gmail.com) wrote:

Hello, all.

The Metron docs mentioned something about introducing oneself upon joining
the mailing list so here goes. I'm a Scala/Java/Data Engineer/Scientist
working in a SOC at a leading telco.

I've only recently come across Metron and was wondering what the community
was like, how healthy the project is (I see commits in the last few days so
that's good) and wondering if I can help.

Regards,

Phillip

Storm 1.0.x eol

[Otto Fowler]

https://storm.apache.org/2019/05/30/storm200-released.html

Re: [DISCUSS] Travis CI Build Time Limits

[Otto Fowler]

Yes, that is what I’m proposing, but if we want to try to break things down
to speed them up, we can still do that ;)


On May 23, 2019 at 11:28:51, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

I think we're talking about killing the cache just 1 time for the purpose
of verifying that the https:// repo endpoints will work. It's possible I'm
misunderstanding how we've configured our caches, but it seems like we'd
just need to kick travis 1x after the first failed build that did all the
downloading, and then the cache will have been populated.

On Wed, May 22, 2019 at 7:21 AM Nick Allen  wrote:

> > Justin Leet said
> 
> [1]: Looking
> at our build times, I'm actually concerned that if we kill the caches our
> builds won't complete. The integration tests take >45 minutes and it's
very
> possible redownloading everything goes over our remaining time.
>
> To recap, our current Travis CI build is composed of multiple jobs.
Travis'
> time limit of 50 minutes
> 
> [2]
> is per job, rather than the total build time. While our total build time
> is on the order of 2.5 hours, it is only our integration-test job which
is
> coming close to that 50 minute limit.
>
> We could try splitting our integration tests into multiple jobs. Each of
> these would have the opportunity to run in parallel (given whatever
> resources Travis can allocate to us), but more importantly each one has
its
> own 50 minute limit. For example...
>
> - Parser Integration Tests:
> - `time mvn surefire:test@integration-tests -pl
> "metron-platform/metron-parsing/metron-parsing-storm,
> metron-platform/metron-parsing/metron-parsers,
> metron-platform/metron-parsing/metron-parsers-common/"`
> - Enrichment Integration Tests
> - `time mvn surefire:test@integration-tests -pl
>
>
"metron-platform/metron-enrichment/metron-enrichment-common/,metron-platform/metron-enrichment/metron-enrichment-storm"`

> - etc, etc
>
>
> We would just need to determine how to logically split the tests up. If
> this sounds reasonable, I've already got a start on a POC.
>
> ---
> [1] https://github.com/apache/metron/pull/1417#issuecomment-494464795
> [2] https://docs.travis-ci.com/user/customizing-the-build/#build-timeouts
>

Re: [DISCUSS] Build RPM/DEBs in Travis?

[Otto Fowler]

https://issues.apache.org/jira/browse/METRON-885


On May 22, 2019 at 10:43:22, Justin Leet (justinjl...@gmail.com) wrote:

Yep, that's all I was doing. I'm in favor of adding it to Travis.

On Wed, May 22, 2019 at 10:28 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I think Justin was just giving rationale, not suggesting we shouldn't
make
> the improvements. I think at one point we also had some ambiguity about
> whether or how Docker was going to run in Travis, but I believe some
folks
> have found a path through that? I agree with you both - let's get this
> added to Travis. Good suggestion, Nick. +1.
>
> On Wed, May 22, 2019 at 7:53 AM Nick Allen  wrote:
>
> > Yes, running up Full Dev is a manual verification that is required. And
> as
> > a manual verification sometimes that will get missed. And in this
> specific
> > case, tt does seem a bit silly that the addition of a parser should
> require
> > the contributor to run up Full Dev.
> >
> > That being said, anything we can do to reduce the amount of manual
> > verification that is required is a good thing. We should be pushing
> > ourselves to an end state where no manual verification is required for
> any
> > Metron PR. I think building the RPMs/DEBs as part of the Travis build
is
> > at least a small step in the right direction.
> >
> >
> > On Wed, May 22, 2019 at 9:34 AM Justin Leet 
> wrote:
> >
> > > Theoretically, we didn't need to before there were both RPMs and DEBs
> > since
> > > running dev up (which necessitates building those) is part of the
build
> > > process. Since they've been split apart, I agree we probably should
be
> > > building them, because nobody is going to run both unless they
> > specifically
> > > done something they'd expect to affect both.
> > >
> > > On Wed, May 22, 2019 at 9:30 AM Nick Allen 
wrote:
> > >
> > > > In light of issues like this
> > https://github.com/apache/metron/pull/1419,
> > > > has anyone looked into building our RPMs and DEBs in Travis? This
> is a
> > > > very common and easy mistake to make and our CI builds really
should
> be
> > > > able to catch this.
> > > >
> > >
> >
>

Re: [RESULT][VOTE] Metron Release Candidate 0.7.1-RC2

[Otto Fowler]

I would like the announcement to state to look into the new deployment
readme for issues with the configuration ui.

I was holding my +1 for discussion on this point, and the release notes.


On May 13, 2019 at 22:34:58, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:



On Mon, May 13, 2019, 8:28 PM Justin Leet  wrote:

> The vote has passed. Including my +1, the voting was:
> 3 binding +1’s
> no 0’s
> no -1’s.
>
> I'll work on finishing out the release process tomorrow and will notify
the
> dev and user lists.
>

Re: [VOTE] Metron Release Candidate 0.7.1-RC2

[Otto Fowler]

We don’t have release notes do we?  Is our only recourse for pointing out
things to look out for the release announcement?


On May 10, 2019 at 20:58:40, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

+1 binding

Validated same as Nick.

Mike

On Thu, May 9, 2019 at 5:54 PM Nick Allen  wrote:

> +1 binding
>
> I validated the release tarball, ran the full test suite and validated
the
> CentOS 6 development environment. Everything looks solid. Let's ship it.
>
> On Wed, May 8, 2019 at 6:50 PM Justin Leet  wrote:
>
> > This is a call to vote on releasing Apache Metron 0.7.1
> >
> > Full list of changes in this release:
> > https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/CHANGES
> > The tag to be voted upon is:
> > apache-metron_0.7.1-rc2
> >
> > The source archives being voted upon can be found here:
> >
> >
>
https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/apache-metron_0.7.1-rc2.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> > https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/
> >
> > The release artifacts are signed with the following key:
> > https://dist.apache.org/repos/dist/release/metron/KEYS
> > Please vote on releasing this package as Apache Metron 0.7.1-RC2
> >
> > When voting, please list the actions taken to verify the release.
> >
> > Recommended build validation and verification instructions are posted
> > here:
> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >
> > This vote will be open for until 7pm EDT on Monday May 13 2019, to
> account
> > for the weekend.
> >
> > [ ] +1 Release this package as Apache Metron 0.7.1-RC2
> >
> > [ ] 0 No opinion
> >
> > [ ] -1 Do not release this package because...
> >
>

Re: [DISCUSS] JsonMapParser original string functionality

[Otto Fowler]

The original string would be the string specified as the message body, thus
each message in the chain produced would just be the bytes passed in, from
a specific field in the incoming message.



On May 10, 2019 at 19:55:28, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

My understanding is that chaining preserves (correctly to my mind) the
original original string.

In other words: unless the message strategy is raw message, the original
string is just passed through. Original string therefore comes from outside
Metron, and is preserved throughout Metron processes, allowing for
recreation of original form for forensics and evidentiary purposes.

Simon

> On 11 May 2019, at 00:10, Otto Fowler  wrote:
>
> What about parser chaining? Should the original string be from kafka, or
> the last parsed?
>
>
> On May 10, 2019 at 19:03:39, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> The only scenario I can think of where a parser might treat original
string
> differently, or even need to know about it would be different encoding
> locales. For example, if the string were to be encoded in a locale
specific
> to the device and choose the encoding based on metadata or parsed
content,
> then that could merit pushing it down. The other edge might be when you
> have binary data that does not go down to an original string well (eg a
> netflow parser).
>
> That said, that’s a highly unlikely edge case that could be handled by
> workarounds.
>
> I’m a definitely +1 on Nick’s idea of pulling original string up to the
> runner. Right now we’re pretty inconsistent in how it’s done, so that
would
> help.
>
> Simon
>
> Sent from my iPhone
>
> On 10 May 2019, at 23:10, Nick Allen  wrote:
>
>>> I suppose we could always allow this to be overridden, also.
>>
>> I like an on/off switch for the "original string" functionality. If on,
>> you get the original string in pristine condition. If off, no original
>> string is appended for those who care more about storage space.
>>
>> I can't think of a reason where one kind of parser would have a
different
>> original string mechanism than the others. If something like that does
>> come up, the parser can create its own original string by just naming it
>> something different and then turning "off" the switch that you
described.
>>
>>
>>
>> On Fri, May 10, 2019 at 5:53 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> I think that's an excellent idea. Can anyone think of a situation where
> we
>>> wouldn't want to add this the same way for all parsers? I suppose we
> could
>>> always allow this to be overridden, also.
>>>
>>>> On Fri, May 10, 2019 at 3:43 PM Nick Allen  wrote:
>>>>
>>>> I think maintaining the integrity of the original data makes a lot of
>>> sense
>>>> for any parser. And ideally the original string should be what came
out
>>> of
>>>> Kafka with only the minimally necessary processing.
>>>>
>>>> With that in mind, we could solve this one level up. Instead of
relying
>>> on
>>>> each parser to do this right, we could have the ParserRunner and
>>>> specifically the ParserRunnerImpl [1] handle this round-abouts here
>>>> <
>>>>
>>>
>
https://github.com/apache/metron/blob/1b6ef88c79d60022542cda7e9abbea7e720773cc/metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java#L149-L158
>>>>>
>>>> [1].
>>>> It has the raw message data and can append the original string to each
>>>> message it gets back from the parsers.
>>>>
>>>> Just another approach to consider.
>>>>
>>>> --
>>>> [1]
>>>>
>>>>
>>>
>
https://github.com/apache/metron/blob/1b6ef88c79d60022542cda7e9abbea7e720773cc/metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java#L149-L158
>>>>
>>>> On Fri, May 10, 2019 at 4:11 PM Otto Fowler 
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>>
>>>>> On May 10, 2019 at 13:57:55, Michael Miklavcic (
>>>>> michael.miklav...@gmail.com)
>>>>> wrote:
>>>>>
>>>>> When adding the capability for parsing messages in the JsonMapParser
>>>> using
>>>>> JSON Path expressions the original behavior for managing original
>>> strings
>>>>> was changed.
>

Re: [DISCUSS] JsonMapParser original string functionality

[Otto Fowler]

What about parser chaining?   Should the original string be from kafka, or
the last parsed?


On May 10, 2019 at 19:03:39, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

The only scenario I can think of where a parser might treat original string
differently, or even need to know about it would be different encoding
locales. For example, if the string were to be encoded in a locale specific
to the device and choose the encoding based on metadata or parsed content,
then that could merit pushing it down. The other edge might be when you
have binary data that does not go down to an original string well (eg a
netflow parser).

That said, that’s a highly unlikely edge case that could be handled by
workarounds.

I’m a definitely +1 on Nick’s idea of pulling original string up to the
runner. Right now we’re pretty inconsistent in how it’s done, so that would
help.

Simon

Sent from my iPhone

On 10 May 2019, at 23:10, Nick Allen  wrote:

>> I suppose we could always allow this to be overridden, also.
>
> I like an on/off switch for the "original string" functionality. If on,
> you get the original string in pristine condition. If off, no original
> string is appended for those who care more about storage space.
>
> I can't think of a reason where one kind of parser would have a different
> original string mechanism than the others. If something like that does
> come up, the parser can create its own original string by just naming it
> something different and then turning "off" the switch that you described.
>
>
>
> On Fri, May 10, 2019 at 5:53 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> I think that's an excellent idea. Can anyone think of a situation where
we
>> wouldn't want to add this the same way for all parsers? I suppose we
could
>> always allow this to be overridden, also.
>>
>>> On Fri, May 10, 2019 at 3:43 PM Nick Allen  wrote:
>>>
>>> I think maintaining the integrity of the original data makes a lot of
>> sense
>>> for any parser. And ideally the original string should be what came out
>> of
>>> Kafka with only the minimally necessary processing.
>>>
>>> With that in mind, we could solve this one level up. Instead of relying
>> on
>>> each parser to do this right, we could have the ParserRunner and
>>> specifically the ParserRunnerImpl [1] handle this round-abouts here
>>> <
>>>
>>
https://github.com/apache/metron/blob/1b6ef88c79d60022542cda7e9abbea7e720773cc/metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java#L149-L158
>>>>
>>> [1].
>>> It has the raw message data and can append the original string to each
>>> message it gets back from the parsers.
>>>
>>> Just another approach to consider.
>>>
>>> --
>>> [1]
>>>
>>>
>>
https://github.com/apache/metron/blob/1b6ef88c79d60022542cda7e9abbea7e720773cc/metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java#L149-L158
>>>
>>> On Fri, May 10, 2019 at 4:11 PM Otto Fowler 
>>> wrote:
>>>
>>>> +1
>>>>
>>>>
>>>> On May 10, 2019 at 13:57:55, Michael Miklavcic (
>>>> michael.miklav...@gmail.com)
>>>> wrote:
>>>>
>>>> When adding the capability for parsing messages in the JsonMapParser
>>> using
>>>> JSON Path expressions the original behavior for managing original
>> strings
>>>> was changed.
>>>>
>>>>
>>>>
>>>
>>
https://github.com/apache/metron/blob/master/metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/json/JSONMapParser.java#L192
>>>>
>>>> A couple issues have been reported recently regarding this change:
>>>>
>>>> 1. We're losing the actual original string, which is a legal issue for
>>>> data lineage for some customers
>>>> 2. Even for the degenerate case with no sub-messages created, the
>>>> original sub-message string is modified because of the
>>>> serialization/deserialization process with Jackson/JsonSimple. The
>> fields
>>>> are reordered bc the content is normalized.
>>>>
>>>> I looked at options for preserving formatting, but am unable to find a
>>>> method that allows you to both parse, then query the original message
>> and
>>>> then also obtain the raw string matches without the normalizing from
>>>> ser/deserialization.
>>>>
>>>> I'd like to propose that we add a configuration option for this parser
>>> that
>>>> allows the user to toggle which approach they'd like to use. My
>> personal
>>>> preference based on feedback I've gotten from multiple customers is
>> that
>>>> the default should be the older approach which takes the raw original
>>>> string. It's arguable that this change in contract is a regression, so
>>> the
>>>> default should be the earlier behavior. Any sub-messages would then
>> have
>>> a
>>>> copy of that raw original string, not just the sub-message original
>>> string.
>>>> Enabling the flag would enable the current sub-message original string
>>>> functionality.
>>>>
>>>> Mike
>>>>
>>>
>>

Re: [DISCUSS] JsonMapParser original string functionality

[Otto Fowler]

+1


On May 10, 2019 at 13:57:55, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

When adding the capability for parsing messages in the JsonMapParser using
JSON Path expressions the original behavior for managing original strings
was changed.

https://github.com/apache/metron/blob/master/metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/json/JSONMapParser.java#L192

A couple issues have been reported recently regarding this change:

1. We're losing the actual original string, which is a legal issue for
data lineage for some customers
2. Even for the degenerate case with no sub-messages created, the
original sub-message string is modified because of the
serialization/deserialization process with Jackson/JsonSimple. The fields
are reordered bc the content is normalized.

I looked at options for preserving formatting, but am unable to find a
method that allows you to both parse, then query the original message and
then also obtain the raw string matches without the normalizing from
ser/deserialization.

I'd like to propose that we add a configuration option for this parser that
allows the user to toggle which approach they'd like to use. My personal
preference based on feedback I've gotten from multiple customers is that
the default should be the older approach which takes the raw original
string. It's arguable that this change in contract is a regression, so the
default should be the earlier behavior. Any sub-messages would then have a
copy of that raw original string, not just the sub-message original string.
Enabling the flag would enable the current sub-message original string
functionality.

Mike

Re: [DISCUSS] Parser Aggregation in Management UI

[Otto Fowler]

You need to be a committer, that is all I think.
I would not use the github UI for it though, I do it through the cli



On May 8, 2019 at 09:45:24, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

Not that I'm aware of. Nick and Otto, you've created them before, did you
need any special perms?

On Wed, May 8, 2019 at 3:57 AM Shane Ardell 
wrote:

> This morning, we started to break down our work as Michael suggested in
> this thread. However, it looks like I don't have permission to create a
new
> branch in the GitHub UI or push a new branch to the apache/metron repo.
Is
> this action restricted to PMC members only?
>
> Shane
>
> On Wed, May 8, 2019 at 9:06 AM Tamás Fodor  wrote:
>
> > Here's the process we've gone through in order to implement the
feature.
> >
> > At the beginning we had some bootstrap work like creating a mock API
> > (written in NodeJS) because we were a few steps ahead the backend part.
> But
> > this is in a totally different repository so it doesn't really count.
We
> > also had to wire NgRX, our chosen 3rd party that supports the flux flow
> to
> > get a better state management. When we were ready to kick off
> implementing
> > the business logic in, we splited up the work by subfeatures like drag
> and
> > dropping table rows. At this point, we created a POC without NgRX just
to
> > let you have the feeling of how it works in real life. Later on, after
> > introducing NgRX, we had to refactor it a little bit obviously to make
it
> > compatible with NgRX. There were other subfeatures like creating and
> > editing groups in a floating pane on the right side of the window.
> > When the real backend API was ready we made the necessary changes and
> > tested whether it worked how it was expected. There were a few
difference
> > between how we originally planned the API and the current
implementation
> so
> > we had to adapt it accordingly. While we were implementing the
features,
> we
> > wrote the unit tests simultaneously. The latest task on the feature was
> > restricting the user from aggregating parsers together.
> >
> > As a first iteration, we've decided to put the restriction in because
it
> > requires a bigger effort on the backend to deal with that. In my
opinion,
> > we should get rid of the restriction because it's not intuitive and
very
> > inconvenient. In my opinion, we should let the users to aggregate the
> > running parsers together and do the job to handle this edge case on the
> > backend accordingly.
> >
> > What do you think, guys?
> >
> > Hope this helps.
> >
> > Tamas
> >
> > On Tue, May 7, 2019 at 4:34 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > This was my expectation as well.
> > >
> > > Shane, Tibor, Tamas - how did you go about breaking this down into
> chunks
> > > and/or microchunks when you collaborated offline? As Nick mentioned,
> you
> > > obviously split up work and shared it amongst yourselves. Some
> > explanation
> > > around this process would be helpful for reviewers as well. We might
be
> > > able to provide better guidance and examples to future contributors
as
> > > well.
> > >
> > > I talked a little bit with Shane about this offline last week. It
looks
> > > like you guys effectively ran a local feature branch. Replicating
that
> > > process in a feature branch in Apache is probably what you guys
should
> > > be doing for a change this size. We don't have hard limits on line
> change
> > > size, but in the past it's been somewhere around 2k-3k lines and
above
> > > being the tipping point for discussing a feature branch. Strictly
> > speaking,
> > > line quantity alone is not the only metric, but it's relevant here.
If
> > you
> > > want to make smaller incremental changes locally, there's nothing to
> keep
> > > you from doing that - I would only advise that you consider squashing
> > those
> > > commits (just ask if you're unclear about how to handle that) into a
> > single
> > > larger commit/chunk when you're ready to publish them as a chunk to
the
> > > public feature branch. So it would look something like this:
> > >
> > > Commits by person locally
> > > Shane: 1,2,3 -> squash as A
> > > Tibor: 4,5,6 -> squash as B
> > > Tamas: 7,8,9 -> squash as C
> > >
> > > Commits by person in Apache
> > > Shane: A
> > > Tibor: B
> > > Tamas: C
> > >
> > > We need to maintain a record of attribution. Your real workflow may
not
> > be
> > > that cleanly delineated, but you can choose how you want to squash in
> > those
> > > cases. Even in public collaboration, there are plenty of cases where
> > folks
> > > submit PRs against PRs, abstain from accepting attribution, and it
all
> > gets
> > > squashed down into one person's final PR commit. There are many
> options.
> > >
> > > Hope this helps.
> > >
> > > Best,
> > > Mike
> > >
> > > On Mon, May 6, 2019 at 8:19 AM Nick Allen  wrote:
> > >
> > > > Have you considered creating a feature branch for the effort? This
> > would
> > > > allow you to break the effort 

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Otto Fowler]

Despite the name, we *have* been using it as both for quite some amount of
time.  It *is* both dev and demo, and we recommend it as such on the list
all the time.

So there isn’t a decision to be made here as far as the status quo -> we
use full dev as both dev and demo.




On May 2, 2019 at 18:53:37, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

Whether or not full dev is, first and foremost, "dev" I think your
questions being up a good point. If not full_dev for introducing new users,
then what? If we want to provide a different env for letting people tinker
and try it out than we do for development, that's completely fine. But we
don't have that right now. So we can treat full_dev as multipurpose, or we
can stop directing non-devs to it, or we can add something new. I honestly
don't have any recommendations here. We've talked about docker instances
for replacing in-memory components, but I'm still not sure that solves this
problem, or adds more complexity. Given the current options on the table,
I'm inclined to go with "full_dev" serves both dev and demo purposes. Otto,
what do you think?

On Thu, May 2, 2019, 4:32 PM Otto Fowler  wrote:

> I’ve commented on the PR, and I won’t repeat it here as well, I will
> however ask again if we know and can list all of the usability issues
that
> surround this problem. IE. All the things that can happen or may happen
> for people who are not Metron developers and committers who are using
> full dev, because we keep recommending it.
>
>
>
> On May 2, 2019 at 17:38:30, Michael Miklavcic (michael.miklav...@gmail.com
> )
> wrote:
>
> PR is up. I added the doc change to the metron-deployment README since
this
> serves as the gateway doc for all the VM instances. All of which would be
> affected by the feature gap.
>
> https://github.com/apache/metron/pull/1398
>
> On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Here's the ticket I created to track it, which also references the Jira
> > for the new UI feature.
> > https://issues.apache.org/jira/browse/METRON-2100
> >
> > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> :-)
> >>
> >> I expect to have #2 out sometime today.
> >>
> >> On Thu, May 2, 2019, 12:11 PM Justin Leet 
> wrote:
> >>
> >>> >
> >>> > I personally
> >>> > don't like this feature gap in full dev. It seems Otto agrees, and
> >>> Casey at
> >>> > the very least sees it as enough of an issue to gate us from 0.8.
> >>> >
> >>>
> >>> +1 on all of this. I don't like it either.
> >>>
> >>>
> >>> > Our vote landed 2-2. We are having a discussion about what to do
with
> >>> the
> >>> > release. This is that discussion.
> >>>
> >>>
> >>> I'm going to be honest, my response was a combination of misreading
> what
> >>> you said and thinking you were proposing delaying the release more
> >>> seriously and feeling a bit blindsided by a perceived move from the
> >>> initial
> >>> "take more time than originally anticipated" (which in my head I took
> as
> >>> a
> >>> couple days) to "versus next week, or the week after" (where delaying
> >>> things weeks is something I personally would like not buried so far
> down
> >>> in
> >>> the thread). Totally my bad, sorry about that.
> >>>
> >>> Other than that, it sounds like we're pretty much in agreement.
> >>>
> >>> Here's my current understanding of the state and consensus as of
right
> >>> now
> >>> (which is subject to change as more discussion happens):
> >>>
> >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and
#3
> >>> for 0.8.0.
> >>> - I don't believe I've seen an explicit response from Otto on what
> >>> he
> >>> thinks about doing this, and from a personal perspective like to
> >>> see what
> >>> his opinion is as the person who originally brought it up.
> >>> - Mike said he's going to kick out a PR that addresses #2
> >>> - After that undergoes the normal review process and is merged, we
> >>> proceed normally and cut RC2.
> >>>
> >>>
> >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic <
> >>> michael.miklav...@gmail.com> wrote:
> >>>
> >>> > I think your later p

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

[Otto Fowler]

+1


On May 2, 2019 at 21:18:21, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

Here's the latest discussion on the subject:
https://lists.apache.org/thread.html/0aa2b0b9ed4a0f0b0d8bb018c618e62de196565f9af71f347e504076@%3Cdev.metron.apache.org%3E

I'd like to propose a vote to change our dev guidelines which will clarify
the tooling we use to produce diagrams and share the source files for those
diagrams. I propose the dev guidelines
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
and
PR checklist
https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
be
changed in the following ways:

1. Under "1.1 Contributing A Code Change"
1. Change <<"New features and significant bug fixes should be
documented in the JIRA and appropriate architecture diagrams should be
attached. Major features may require a vote.">> to <<"New features
and significant bug fixes should be documented in the JIRA. Appropriate
architecture diagrams should be created in https://www.draw.io/
and committed
to source control as per section 2.4. Diagrams may be requested of PR
submitters during review either as documentation or as an aid to the
reviewer. Major features may also require a vote.">>
2. Under "2.4 Documentation"
1. New line item <<"Diagrams - We save architecture diagram source
files in an xml format rendered by draw.io (instructions below). This
is the free tool of choice that we've agreed to use for exchanging
diagrams and their source files in Metron.">>
2. New line item >. This section
would provide basic instructions for downloading source files from
draw.io.
3. Add a new checkbox item under PR checklist heading "For documentation
related changes" with the following text
1. Have you ensured that any documentation diagrams have been
updated, along with their source files, using draw.io? See
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
for
instructions.
4. Here is the Jira for migrating/redoing existing diagrams
1. https://issues.apache.org/jira/browse/METRON-2099


We require a minimum of 72 hours for a vote, not typically including
weekend days. I'd like to leave this vote open until Wednesday 5/8, 12PM
EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your vote is
binding or non-binding.

Re: [DISCUSS] Parser Aggregation in Management UI

[Otto Fowler]

I have commented the jira.




On May 2, 2019 at 14:22:41, Shane Ardell (shane.m.ard...@gmail.com) wrote:

Hello everyone,

In response to discussions in the 0.7.1 release thread, I wanted to start a
thread regarding the parser aggregation work for the Management UI. For
anyone who has not already read and tested the PR locally, I've added a
detailed description of what we did and why to the JIRA ticket here:
https://issues.apache.org/jira/browse/METRON-1856

I'm wondering what the community thinks about what we've built thus far. Do
you see anything missing that must be part of this new feature in the UI?
Are there any strong objections to how we implemented it?

I’m also looking to see if anyone has any thoughts on how we can possibly
simplify this PR. Right now it's pretty big, and there are a lot of commits
to parse through, but I'm not sure how we could break this work out into
separate, smaller PRs opened against master. We could try to cherry-pick
the commits into smaller PRs and then merge them into a feature branch, but
I'm not sure if that's worth the effort since that will only reduce the
number commits to review, not the lines changed.

As an aside, I also want to give a little background into the introduction
of NgRx in this PR. To give a little background on why we chose to do this,
you can refer to the discussion thread here:
https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E

We previously discussed introducing a better way to manage application
state in both UIs in that thread. It was decided that NgRx was a great tool
for many reasons, one of them being that we can piecemeal it into the
application rather than doing a huge rewrite of all the application state
at once. The contributors in this PR (myself included) decided this would
be a perfect opportunity to introduce NgRx into the Management UI since we
need to manage the previous and current state with the grouping feature so
that users can undo the changes they've made (we used it for more than just
that in this feature, but that was the initial reasoning). In addition, we
greatly benefited from this when it came time to debug our work in the UI
(the discussion in the above thread link goes a little more into the
advantages of debugging with NgRx and DevTools). Removing NgRx from this
work would reduce the numbers of lines changed slightly, but it would still
be a big PR and a lot of that code would just move to the component or
service level in the Angular application.

Shane

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Otto Fowler]

to do with
>>> the
>>> > release. This is that discussion.
>>> >
>>> > On Thu, May 2, 2019, 10:52 AM Justin Leet 
>>> wrote:
>>> >
>>> > > @Mike
>>> > > I have a different question: Why is this enough to consider
delaying
>>> a
>>> > > release in the first place for a fairly involved fix?
>>> > >
>>> > > There was a discuss thread, where the general agreement was that we
>>> had
>>> > > enough value to do a release (Over a month ago. And more things
have
>>> gone
>>> > > into master since then). There's a good number of fixes, and not
just
>>> > > trivial ones either. The general consensus here seems to be that
the
>>> > > management UI issue is fairly minor for a point release (after all,
>>> > there's
>>> > > been multiple people who think option 2 is sufficient), but becomes
>>> > > important if we want to release a minor version. The question I
asked
>>> > > myself about this was ""Does this issue detract enough value that a
>>> > release
>>> > > isn't worthwhile?" and my answer was, and still is, "No, we have
>>> enough
>>> > > value to do a meaningful release".
>>> > >
>>> > > I'm fine with delaying or cancelling a release because we find
issues
>>> > that
>>> > > are severe enough or we don't think there's enough value anymore,
>>> but to
>>> > be
>>> > > entirely honest, I'm absolutely shocked this issue has blown up so
>>> much.
>>> > > However, if you want to have a discuss thread to reevaluate if it's
>>> > > worthwhile to do a release, go for it. The communities' calculus on
>>> the
>>> > > "Does this issue detract enough value that a release isn't
>>> worthwhile?"
>>> > may
>>> > > be different than mine.
>>> > >
>>> > > Having said all that, to a large extent, I think you're right. It
>>> really
>>> > > doesn't matter* that much* if we release next week or the week
after
>>> or
>>> > > whenever. But at the same time I personally get super frustrated
>>> when I
>>> > go
>>> > > to use a project, find a bug, it's already known and fixed, but it
>>> just
>>> > > never puts out a released version. Every cutoff is largely
>>> arbitrary,
>>> > but
>>> > > I think getting our improvements and fixes out there is important.
>>> One of
>>> > > the things we've done fairly well is put out releases at a fairly
>>> decent
>>> > > cadence for a project this large. I really don't want to set the
>>> > precedent
>>> > > of just increasingly pushing out point releases for stuff like
this.
>>> > >
>>> > > On Thu, May 2, 2019 at 12:52 PM Nick Allen 
>>> wrote:
>>> > >
>>> > > > I think any open source project needs to strive to cut releases
>>> > > regularly.
>>> > > > This is healthy for the project and community. It gets new
>>> features
>>> > and
>>> > > > functionality out to the community so we can get feedback, find
>>> what is
>>> > > > working and what is not, iterate and improve. You probably agree
>>> with
>>> > > > this.
>>> > > >
>>> > > > While releasing this week or next may not matter in the grand
>>> scheme,
>>> > if
>>> > > we
>>> > > > want to cut releases regularly, then we need to bear down and
just
>>> do
>>> > it.
>>> > > > Case in point, I opened the initial discussion for this release
on
>>> > March
>>> > > > 13th [1] and it is now May 2nd and we have yet to release 7 weeks
>>> > later.
>>> > > >
>>> > > > --
>>> > > > [1]
>>> > > >
>>> > > >
>>> > >
>>> >
>>>
https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E
>>> > > >
>>> > > >
>>> > > > On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic <
>>> > > > michael.miklav...@gmail.com> wrote:
>

Re: [DISCUSS] Dev guideline changes for architecture diagrams

[Otto Fowler]

+1, great job Mike


On May 2, 2019 at 18:06:28, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

I thought it might be useful as a more general purpose bucket, and it's how
we refer to the image files in the site-book generation. Also, there may be
things that might not qualify as a diagram, but have some value. A few
README's use screenshots. Here's an example:

1.
https://github.com/apache/metron/blob/master/metron-deployment/Kerberos-ambari-setup.md
2.
https://github.com/apache/metron/tree/master/metron-deployment/readme-images
.


What do you think?

On Thu, May 2, 2019 at 5:19 AM Otto Fowler  wrote:

> I’m all set with this, but for one question, why image and image-source
as
> opposed to diagram and diagram source? Isn’t it more descriptive? Is
> there another reason ( like for doc gen )?
>
>
>
> On May 1, 2019 at 20:12:47, Michael Miklavcic (michael.miklav...@gmail.com
> )
> wrote:
>
> Picking up where things left off in the VOTE thread on the subject,
> I'm presenting a revision to my original proposal below. I'd like to get
> this signed off on before submitting it for another vote. Otto, picking
up
> where we left off, let me know if this looks good to you.
>
> I'd like to propose a vote to change our dev guidelines which will
clarify
> the tooling we use to produce diagrams and share the source files for
those
> diagrams. The original discuss thread is noted at the end of this email.
I
> propose the dev guidelines
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> and
> PR checklist
>
>
https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
> be
> changed in the following ways:
>
> 1. Under "1.1 Contributing A Code Change"
> 1. Change <<"New features and significant bug fixes should be
> documented in the JIRA and appropriate architecture diagrams should be
> attached. Major features may require a vote.">> to <<"New features
> and significant bug fixes should be documented in the JIRA. Appropriate
> architecture diagrams should be created in https://www.draw.io/
> and committed
> to source control as per section 2.4. Diagrams may be requested of PR
> submitters during review either as documentation or as an aid to the
> reviewer. Major features may also require a vote.">>
> 2. Under "2.4 Documentation"
> 1. New line item <<"Diagrams - We save architecture diagram source
> files in an xml format rendered by draw.io (instructions below). This
> is the free tool of choice that we've agreed to use for exchanging
> diagrams and their source files in Metron.">>
> 2. New line item < "/images-source" and rendered diagrams and images belong in
> "/images."
> 3. New subsection <<"Creating and Modifying Diagrams">>. This section
> would provide basic instructions for downloading source files from
> draw.io.
> 3. Add a new checkbox item under PR checklist heading "For documentation
> related changes" with the following text
> 1. Have you ensured that any documentation diagrams have been
> updated, along with their source files, using draw.io? See
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> for
> instructions.
> 4. Here is the Jira for migrating/redoing existing diagrams
> 1. https://issues.apache.org/jira/browse/METRON-2099
>
>
>
> - Original DISCUSS thread -
>
>
https://lists.apache.org/thread.html/3ae02f1e32044b1a7648899700d44611aefdab6caa09fb3196292425@%3Cdev.metron.apache.org%3E
> - Original VOTE thread -
>
>
https://lists.apache.org/thread.html/c41dca65a46354c161d58caa7c79cfac6758f437b6638d12bd5c5622@%3Cdev.metron.apache.org%3E
>

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Otto Fowler]

I remember this now, but I’m not sure how I would have related this to a
parser aggregation pr honestly.


On May 2, 2019 at 07:54:13, Shane Ardell (shane.m.ard...@gmail.com) wrote:

Here's a link to the ngrx discussion thread from a few months back:
https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E

On Thu, May 2, 2019 at 1:17 PM Otto Fowler  wrote:

> If you can find a link in the archives for that thread, it would really
> help.
>
> I don’t think sending them up as one sensor would work…. as something
> quick. I think it is an interesting idea from a higher level that would
> need some more thought though ( IE: what if every sensor in the ui was a
> sensor group, and the existing entries where just groups of 1 ).
>
> As far as I can see, we have brought up the idea of a release ourselves,
I
> don’t see why we don’t just swarm this issue and get it right then
release.
>
>
>
> On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote:
>
> In PR#1360 we introduced a new state management strategy involving a new
> module called Ngrx. We had a discussion thread on this a few months ago
and
> we successfully convinced you about the benefits. This is one of the
> reasons why this PR is going to be still huge after cleaning up the
commit
> history. After you having a look at the changes and the feature itself,
> there's likely have questions about why certain parts work as they do.
The
> thing what I'd like to point out is that, yes, it probably takes more
time
> to get it in.
>
> In order to being able to release the RC, wouldn't it be an easy and
quick
> fix on the backend if it sent the aggregated parsers to the client as
they
> were one sensor? It's just an idea, it might be wrong, but at least we
> shouldn't have to wait until the aforementioned PR gets ready to be
merged
> to the master.
>
> On Wed, May 1, 2019 at 4:16 PM Justin Leet  wrote:
>
> > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for
0.8.0.
> > #3 seems like a total waste of time and effort.
> >
> > The wall of text version:
> > I agree this isn't "just the wrong thing shown", but for completely
> > different reasons.
> >
> > To be extremely clear about what the problem is: Our "dev" environment
> > (whose very name implies the audience is develops) uses a
> performance-based
> > advanced feature to ensure that all our of sample flows are regularly
run
> > and produce data. This feature has a bare minimal implementation to be
> > enabled via Ambari, which it currently is by default. This is because
of
> > the limited resources available that previously resulted in us turning
> off
> > Yaf, and therefore testing it during regular full dev runs. Right now
> > however, this feature is not exposed through the management UI, and
> > therefore it isn't obvious what the implications are. Am I missing
> anything
> > here?
> >
> > For users actually choosing to use the parser aggregation feature in a
> > non-full-dev environment, I'd expect substantially more care to be
> involved
> > given the lack of easy configuration for it (after all, why would you
> > bother running the aggregated parser alongside the regular parser? This
> > could be more explicitly stated, but again that feels like a doc
problem.
> > Right now I could essentially provide two of the same parser and create
> the
> > same problem, so right now aggregation is only special because it runs
on
> > dev by default). This is, in my opinion, primarily a first impression
> > problem and likely one of many areas that could use improved
> documentation.
> >
> > Quite frankly, I think the issue pointed out here could mostly be
> resolved
> > by documenting how the current aggregation is done in dev, and telling
> how
> > to change it. Especially for a 0.x.1 release, which is primarily bug
> > fixes. As can be inferred from my vote, I don't think this problem is a
> > problem that needs solving in a point release. I would support
improving
> > the documentation, both full-dev and for aggregation in general for the
> > 0.7.1 point release, while making a 0.8.0 release contingent upon the
> > outstanding PRs to enable it in the management UI.
> >
> > There are a couple deeper issues, imo, that I care substantially more
> about
> > than this in particular
> > * The dev environment is being used as our intro for users, because
it's
> > convenient for us to not maintain more environments (which has been a
> major
> > pain point in the past). Worse, the dev environment strongly implies
it's
> > for M

Re: [DISCUSS] Dev guideline changes for architecture diagrams

[Otto Fowler]

I’m all set with this, but for one question,  why image and image-source as
opposed to diagram and diagram source?  Isn’t it more descriptive?  Is
there another reason ( like for doc gen )?



On May 1, 2019 at 20:12:47, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

Picking up where things left off in the VOTE thread on the subject,
I'm presenting a revision to my original proposal below. I'd like to get
this signed off on before submitting it for another vote. Otto, picking up
where we left off, let me know if this looks good to you.

I'd like to propose a vote to change our dev guidelines which will clarify
the tooling we use to produce diagrams and share the source files for those
diagrams. The original discuss thread is noted at the end of this email. I
propose the dev guidelines
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
and
PR checklist
https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
be
changed in the following ways:

1. Under "1.1 Contributing A Code Change"
1. Change <<"New features and significant bug fixes should be
documented in the JIRA and appropriate architecture diagrams should be
attached. Major features may require a vote.">> to <<"New features
and significant bug fixes should be documented in the JIRA. Appropriate
architecture diagrams should be created in https://www.draw.io/
and committed
to source control as per section 2.4. Diagrams may be requested of PR
submitters during review either as documentation or as an aid to the
reviewer. Major features may also require a vote.">>
2. Under "2.4 Documentation"
1. New line item <<"Diagrams - We save architecture diagram source
files in an xml format rendered by draw.io (instructions below). This
is the free tool of choice that we've agreed to use for exchanging
diagrams and their source files in Metron.">>
2. New line item >. This section
would provide basic instructions for downloading source files from
draw.io.
3. Add a new checkbox item under PR checklist heading "For documentation
related changes" with the following text
1. Have you ensured that any documentation diagrams have been
updated, along with their source files, using draw.io? See
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
for
instructions.
4. Here is the Jira for migrating/redoing existing diagrams
1. https://issues.apache.org/jira/browse/METRON-2099



- Original DISCUSS thread -
https://lists.apache.org/thread.html/3ae02f1e32044b1a7648899700d44611aefdab6caa09fb3196292425@%3Cdev.metron.apache.org%3E
- Original VOTE thread -
https://lists.apache.org/thread.html/c41dca65a46354c161d58caa7c79cfac6758f437b6638d12bd5c5622@%3Cdev.metron.apache.org%3E

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Otto Fowler]

ent UI, we should be able to
> entirely divorce these two overlapping domains. I'd love to see parsers
> ripped out of Ambari, then full-dev manages all the setup via REST. At
that
> point, we can easily tell everyone to just use the management UI.
>
> On Wed, May 1, 2019 at 7:23 AM Otto Fowler 
> wrote:
>
> > I think it would help if the full consequences of having the UI show
the
> > wrong status where listed.
> >
> > Someone trying metron, will, by default , see the wrong thing in the UI
> for
> > the ONLY sensors they have that are running and doing data.
> >
> > What happens when they try to start them to make them work? One, two or
> > all?
> > What happens when he edits them or try to add transformations? One, two
> or
> > all?
> > What other things can you do with the sensors in the ui? What happens?
> >
> > Are we recommending aggregation on the list and elsewhere for users?
Are
> > we recommending something that is going to ensure they get into this
> > situation?
> >
> > I think this is more than ‘just the wrong thing shown’ in the ui.
> >
> >
> >
> >
> > On April 30, 2019 at 20:48:10, Michael Miklavcic (
> > michael.miklav...@gmail.com) wrote:
> >
> > The vote for RC1 did not pass and I'd like to kickstart some discussion
> > about what we should do.
> >
> > I started taking a look at PR#1360 and it looks like this isn't quite
as
> > close to being able go in as I had originally expected. I want to talk
> > about options here. It seems to me that we can:
> >
> > 1. Wait for PR#1360 to go in, but this is likely going to take more
time
> > than originally anticipated
> > 2. Accept the issue in full dev, but add some notes in the developer
> > docs about the current feature gap and why sensors aren't showing
status
> in
> > the management UI when aggregation is enabled.
> > 3. Find some other workable UI solution.
> > 4. Other option?
> >
> > All things considered, I'm personally leaning towards #2 in the
> short-term,
> > but I think we should probably talk about this a bit before deciding
what
> > RC2 should be.
> >
> > Best,
> > Mike
> >
>

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Otto Fowler]

I think it would help if the full consequences of having the UI show the
wrong status where listed.

Someone trying metron, will, by default , see the wrong thing in the UI for
the ONLY sensors they have that are running and doing data.

What happens when they try to start them to make them work? One, two or all?
What happens when he edits them or try to add transformations? One, two or
all?
What other things can you do with the sensors in the ui?  What happens?

Are we recommending aggregation on the list and elsewhere for users?  Are
we recommending something that is going to ensure they get into this
situation?

I think this is more than ‘just the wrong thing shown’ in the ui.




On April 30, 2019 at 20:48:10, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

The vote for RC1 did not pass and I'd like to kickstart some discussion
about what we should do.

I started taking a look at PR#1360 and it looks like this isn't quite as
close to being able go in as I had originally expected. I want to talk
about options here. It seems to me that we can:

1. Wait for PR#1360 to go in, but this is likely going to take more time
than originally anticipated
2. Accept the issue in full dev, but add some notes in the developer
docs about the current feature gap and why sensors aren't showing status in
the management UI when aggregation is enabled.
3. Find some other workable UI solution.
4. Other option?

All things considered, I'm personally leaning towards #2 in the short-term,
but I think we should probably talk about this a bit before deciding what
RC2 should be.

Best,
Mike

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

[Otto Fowler]

while we can land a pr and accept the regressions, I do not think we should
do a release with our default sample environment broken.

-1

On April 27, 2019 at 18:11:42, Justin Leet (justinjl...@gmail.com) wrote:

> Mike is correct, that is because of the combination of full dev
> restrictions and the lack of support in the configuration UI for parser
> aggregation. This was introduced in
> https://github.com/apache/metron/pull/1207 and also was true of the last
> release. Currently, parser aggregation is an advanced/manual feature whose
> (bare minimum) configuration can be done via Ambari, out of convenience.
>
> I haven't looked into it, but https://github.com/apache/metron/pull/1360
> is
> likely the work for this (and need additional work before merging).
>
> I'm personally letting my binding +1 stand, although I would support
> either
> ensuring we get that PR cleaned up and in and/or additional documentation
> regarding the current limitations of this feature.
>
>
> On Sat, Apr 27, 2019 at 2:38 PM Anand Subramanian <
> asubraman...@hortonworks.com> wrote:
>
> I can confirm that I've seen the Mgmt UI shows the sensor status correctly
> when they run as single topologies.
>
> -Anand
>
> On 4/27/19, 11:37 PM, "Michael Miklavcic" 
> wrote:
>
> I believe that is bc of parser aggregation. The UI does not support it
> currently. IIRC there was a PR to change the bro, snort, and yaf
> sensors to
> aggregated bc full dev didn't have enough resources. The upshot is
> that the
> UI still works for single sensors, but the feature for enabling
> aggregated
> sensors has not yet been completed.
>
> On Sat, Apr 27, 2019, 11:33 AM Otto Fowler 
> wrote:
>
> -1
>
> Ran the script and ran full dev, all good.
> In the configuration ui, the status of the sensors is not correct.
>
> It
>
> does not show any running, but they are running in storm and the
>
> data was
>
> moved correctly.
>
>
> On April 26, 2019 at 09:58:02, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Curious Anand,
> are your steps for bringing up an open stack cluster something we
>
> could
>
> script like the AWS stuff?
>
>
> On April 26, 2019 at 09:35:29, Anand Subramanian (
> asubraman...@hortonworks.com) wrote:
>
> +1 (non-binding)
>
> * Built RPMs and mpacks.
> * Brought up Metron stack on 12-node CentOS 7 openstack cluster.
> * Ran sensor-stubs and validated events in the Alerts UI for the
>
> default
>
> sensors.
> * Management UI, Alerts UI and Swagger UI sanity check
>
> Regards,
> Anand
>
> On 4/26/19, 5:18 AM, "Nick Allen"  wrote:
>
> +1 Verified release with all documented steps and ran up Full Dev.
>
> On Thu, Apr 25, 2019 at 6:10 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> Ok cool, just finished the validation and updated the steps in the
>
> doc to
>
> reflect the current code base.
>
> On Thu, Apr 25, 2019 at 3:45 PM Nick Allen 
>
> wrote:
>
>
> No voting required. Those are just docs. Whoever is willing to
>
> correct
>
> and has access, should be able to. Good catch.
>
> On Thu, Apr 25, 2019 at 4:32 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> We're also not "incubator-metron" any longer. Do we require
>
> any kind
>
> of
>
> voting or +1 on that verification page to make corrections to
>
> it?
>
>
> On Thu, Apr 25, 2019 at 2:29 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> fyi, the steps in this doc have changed slightly per this
>
> naming
>
> convention change as well -
>
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds.
>
>
>
>
> On Thu, Apr 25, 2019 at 1:25 PM Justin Leet <
>
> justinjl...@gmail.com
>
>
> wrote:
>
>
> For everyone taking the time to validate and vote on the
>
> RC, there
>
> is
>
> a
>
> caveat. The naming conventions for the two repos are now
>
> aligned
>
> (_, instead of being '-' in the main
>
> repo and
>
> '_'
>
> in
>
> the plugin repo) along with the location of the KEYS file,
>
> I have
>
> a
>
> PR
>
> out
>
> to update the metron-rc-check script (
> https://github.com/apache/metron/pull/1394).
>
> This accounts for both of these changes, and should allow
>
> the
>
> script
>
> to
>
> be
>
> run normally.
>
> On Thu, Apr 25, 2019 at 3:22 PM Justin Leet <
>
> justinjl...@gmail.com>
>
> wrote:
>
> This is a call to vote on releasing Apache Metron 0.7.1
>
> Full list of change

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

[Otto Fowler]

-1

Ran the script and ran full dev, all good.
In the configuration ui, the status of the sensors is not correct.  It does
not show any running, but they are running in storm and the data was moved
correctly.


On April 26, 2019 at 09:58:02, Otto Fowler (ottobackwa...@gmail.com) wrote:

Curious Anand,
are your steps for bringing up an open stack cluster something we could
script like the AWS stuff?


On April 26, 2019 at 09:35:29, Anand Subramanian (
asubraman...@hortonworks.com) wrote:

+1 (non-binding)

* Built RPMs and mpacks.
* Brought up Metron stack on 12-node CentOS 7 openstack cluster.
* Ran sensor-stubs and validated events in the Alerts UI for the default
sensors.
* Management UI, Alerts UI and Swagger UI sanity check

Regards,
Anand

On 4/26/19, 5:18 AM, "Nick Allen"  wrote:

+1 Verified release with all documented steps and ran up Full Dev.

On Thu, Apr 25, 2019 at 6:10 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Ok cool, just finished the validation and updated the steps in the doc to
> reflect the current code base.
>
> On Thu, Apr 25, 2019 at 3:45 PM Nick Allen  wrote:
>
> > No voting required. Those are just docs. Whoever is willing to correct
> > and has access, should be able to. Good catch.
> >
> > On Thu, Apr 25, 2019 at 4:32 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > We're also not "incubator-metron" any longer. Do we require any kind
of
> > > voting or +1 on that verification page to make corrections to it?
> > >
> > > On Thu, Apr 25, 2019 at 2:29 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > fyi, the steps in this doc have changed slightly per this naming
> > > > convention change as well -
> > > > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds.
> > > >
> > > >
> > > >
> > > > On Thu, Apr 25, 2019 at 1:25 PM Justin Leet 
> > > wrote:
> > > >
> > > >> For everyone taking the time to validate and vote on the RC, there
> is
> > a
> > > >> caveat. The naming conventions for the two repos are now aligned
> > > >> (_, instead of being '-' in the main repo and
> '_'
> > in
> > > >> the plugin repo) along with the location of the KEYS file, I have a
> PR
> > > out
> > > >> to update the metron-rc-check script (
> > > >> https://github.com/apache/metron/pull/1394).
> > > >>
> > > >> This accounts for both of these changes, and should allow the
script
> > to
> > > be
> > > >> run normally.
> > > >>
> > > >> On Thu, Apr 25, 2019 at 3:22 PM Justin Leet 
> > > >> wrote:
> > > >>
> > > >> > This is a call to vote on releasing Apache Metron 0.7.1
> > > >> >
> > > >> > Full list of changes in this release:
> > > >> > https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/CHANGES
> > > >> > The tag to be voted upon is:
> > > >> > apache-metron_0.7.1-rc1
> > > >> >
> > > >> > The source archives being voted upon can be found here:
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/apache-metron_0.7.1-rc1.tar.gz
> > > >> >
> > > >> > Other release files, signatures and digests can be found here:
> > > >> > https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/
> > > >> >
> > > >> > The release artifacts are signed with the following key:
> > > >> > https://dist.apache.org/repos/dist/release/metron/KEYS
> > > >> > Please vote on releasing this package as Apache Metron 0.7.1-RC1
> > > >> >
> > > >> > When voting, please list the actions taken to verify the release.
> > > >> >
> > > >> > Recommended build validation and verification instructions are
> > posted
> > > >> > here:
> > > >> >
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > > >> >
> > > >> > This vote will be open for until 4pm EDT on Tuesday April 30
2019,
> > to
> > > >> > account for the weekend..
> > > >> >
> > > >> > [ ] +1 Release this package as Apache Metron 0.7.1-RC1
> > > >> >
> > > >> > [ ] 0 No opinion
> > > >> >
> > > >> > [ ] -1 Do not release this package because...
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

[Otto Fowler]

When I say module, I mean at the root of any module, so each module could
have it’s own diagrams.
And the project root wold have diagrams for the ‘overall'


On April 26, 2019 at 13:44:30, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

The convention that seems to have been followed thus far has been to plop
the images in the root of the module they're relevant to. Maybe relocating
them to a central place would make it easier. The site-book image link
rewriting might be simpler then as well. The only downside to this approach
would be that the artifacts are split from their respective modules, but I
honestly don't see that as a problem.

On Fri, Apr 26, 2019, 11:40 AM Otto Fowler  wrote:

> On April 26, 2019 at 13:19:05, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> @otto when I get your responses to my Q's inline below I can post another
> revision.
>
> On Thu, Apr 25, 2019 at 11:52 AM Otto Fowler 
> wrote:
>
> > - We need to specify the format I think, and then say that draw io is
the
> > tool for the format and not just specify the tool.
> >
>
> Format for the source files, rendered files, or both? I believe their
> source file format is a proprietary XML format. For rendered images, I
> don't have a strong opinion and am happy to leave that up to the
> implementer. If we want to be more opinionated, i.e. specify png, svg,
> jpeg, etc. I could probably be persuaded. For the source file comment,
> maybe it would help if I did the full write-up for 3.1 wrt instructions
for
> how to produce the diagrams and source files from draw.io.
>
>
> I think what you say would be ok then, if draw io only has one source
> format.
>
> I don’t care about the image format either, I’m surprised nobody has a
> strong opinion about it.
>
> Do we want a standard place to put the diagrams?
>
> module/
> - diagrams/
> - foo.xml
> - foo.png
> - pr1234.xml
> - pr1234.jpeg
> - METRON-13244.xml
> - METRON-13244.png
> - EnrichementArchitecture.xml
> - EnrichementArchitecture.png
>
>
>
>
>
>
> > - Existing diagrams, in order to be modified, will have to be converted
> to
> > this format, there should be jiras for that
> >
> > Makes sense - I think I'll create those Jiras in lockstep with this
vote
> getting approval
>
>
> > 2.1 "New features and significant bug fixes should be documented in the
> > JIRA. Appropriate architecture diagrams should be created in
> > https://www.draw.io/ and committed “
> > "New features and significant bug fixes should be documented in the
JIRA.
> > Appropriate architecture diagrams should be created in
> > https://www.draw.io/ and
> > committed. Diagrams may be requested of PR submitters during review
> either
> > as documentation or as an aid to the reviewer “
> >
> > We could/should/can use the diagrams as
> >
> > - documentation
> > - simple aids for understanding PRs and communication ( Nick and I used
> > them for such yesterday to great effect to make sure we were on the
same
> > page ).
> >
> > I’m not sure we don’t want to have that blurb in there
> >
> > I'm happy to add this as well, +1 to that.
>
>
> >
> > On April 25, 2019 at 12:57:47, Michael Miklavcic (
> > michael.miklav...@gmail.com) wrote:
> >
> > I'd like to propose a vote to change our dev guidelines which will
> clarify
> > the tooling we use to produce diagrams and share the source files for
> those
> > diagrams. The original discuss thread is noted at the end of this
email.
> I
> > propose the dev guidelines
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> > and
> > PR checklist
> >
> >
>
>
https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
> > be
> > changed in the following ways:
> >
> > 1. We specify that draw.io is the free tool of choice for sharing
> > diagrams in Metron and that the source files will be maintained/shared
in
> > source control.
> > 2. Under "1.1 Contributing A Code Change"
> > 1. Change "New features and significant bug fixes should be
> > documented in the JIRA and appropriate architecture diagrams should be
> > attached. Major features may require a vote." to "New features and
> > significant bug fixes should be documented in the JIRA. Appropriate
> > architecture diagrams should be created in https://www.draw.io/
> > and committed
> > to source control with their XML source files and final rendered image.
> > Major features may require a vote."
> > 3. Un

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

[Otto Fowler]

Curious Anand,
are your steps for bringing up an open stack cluster something we could
script like the AWS stuff?


On April 26, 2019 at 09:35:29, Anand Subramanian (
asubraman...@hortonworks.com) wrote:

+1 (non-binding)

* Built RPMs and mpacks.
* Brought up Metron stack on 12-node CentOS 7 openstack cluster.
* Ran sensor-stubs and validated events in the Alerts UI for the default
sensors.
* Management UI, Alerts UI and Swagger UI sanity check

Regards,
Anand

On 4/26/19, 5:18 AM, "Nick Allen"  wrote:

+1 Verified release with all documented steps and ran up Full Dev.

On Thu, Apr 25, 2019 at 6:10 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Ok cool, just finished the validation and updated the steps in the doc to
> reflect the current code base.
>
> On Thu, Apr 25, 2019 at 3:45 PM Nick Allen  wrote:
>
> > No voting required. Those are just docs. Whoever is willing to correct
> > and has access, should be able to. Good catch.
> >
> > On Thu, Apr 25, 2019 at 4:32 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > We're also not "incubator-metron" any longer. Do we require any kind
of
> > > voting or +1 on that verification page to make corrections to it?
> > >
> > > On Thu, Apr 25, 2019 at 2:29 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > fyi, the steps in this doc have changed slightly per this naming
> > > > convention change as well -
> > > > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds.

> > > >
> > > >
> > > >
> > > > On Thu, Apr 25, 2019 at 1:25 PM Justin Leet 
> > > wrote:
> > > >
> > > >> For everyone taking the time to validate and vote on the RC, there
> is
> > a
> > > >> caveat. The naming conventions for the two repos are now aligned
> > > >> (_, instead of being '-' in the main repo and
> '_'
> > in
> > > >> the plugin repo) along with the location of the KEYS file, I have
a
> PR
> > > out
> > > >> to update the metron-rc-check script (
> > > >> https://github.com/apache/metron/pull/1394).
> > > >>
> > > >> This accounts for both of these changes, and should allow the
script
> > to
> > > be
> > > >> run normally.
> > > >>
> > > >> On Thu, Apr 25, 2019 at 3:22 PM Justin Leet 

> > > >> wrote:
> > > >>
> > > >> > This is a call to vote on releasing Apache Metron 0.7.1
> > > >> >
> > > >> > Full list of changes in this release:
> > > >> > https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/CHANGES
> > > >> > The tag to be voted upon is:
> > > >> > apache-metron_0.7.1-rc1
> > > >> >
> > > >> > The source archives being voted upon can be found here:
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/apache-metron_0.7.1-rc1.tar.gz
> > > >> >
> > > >> > Other release files, signatures and digests can be found here:
> > > >> > https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC1/
> > > >> >
> > > >> > The release artifacts are signed with the following key:
> > > >> > https://dist.apache.org/repos/dist/release/metron/KEYS
> > > >> > Please vote on releasing this package as Apache Metron 0.7.1-RC1
> > > >> >
> > > >> > When voting, please list the actions taken to verify the
release.
> > > >> >
> > > >> > Recommended build validation and verification instructions are
> > posted
> > > >> > here:
> > > >> >
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > > >> >
> > > >> > This vote will be open for until 4pm EDT on Tuesday April 30
2019,
> > to
> > > >> > account for the weekend..
> > > >> >
> > > >> > [ ] +1 Release this package as Apache Metron 0.7.1-RC1
> > > >> >
> > > >> > [ ] 0 No opinion
> > > >> >
> > > >> > [ ] -1 Do not release this package because...
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

[Otto Fowler]

- We need to specify the format I think, and then say that draw io is the
tool for the format and not just specify the tool.
- Existing diagrams, in order to be modified, will have to be converted to
this format, there should be jiras for that

2.1 "New features and significant bug fixes should be documented in the
JIRA. Appropriate architecture diagrams should be created in
https://www.draw.io/ and committed “
"New features and significant bug fixes should be documented in the JIRA.
Appropriate architecture diagrams should be created in https://www.draw.io/ and
committed.  Diagrams may be requested of PR submitters during review either
as documentation or as an aid to the reviewer “

We could/should/can use the diagrams as

- documentation
- simple aids for understanding PRs and communication ( Nick and I used
them for such yesterday to great effect to make sure we were on the same
page ).

I’m not sure we don’t want to have that blurb in there



On April 25, 2019 at 12:57:47, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

I'd like to propose a vote to change our dev guidelines which will clarify
the tooling we use to produce diagrams and share the source files for those
diagrams. The original discuss thread is noted at the end of this email. I
propose the dev guidelines
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
and
PR checklist
https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
be
changed in the following ways:

1. We specify that draw.io is the free tool of choice for sharing
diagrams in Metron and that the source files will be maintained/shared in
source control.
2. Under "1.1 Contributing A Code Change"
1. Change "New features and significant bug fixes should be
documented in the JIRA and appropriate architecture diagrams should be
attached. Major features may require a vote." to "New features and
significant bug fixes should be documented in the JIRA. Appropriate
architecture diagrams should be created in https://www.draw.io/
and committed
to source control with their XML source files and final rendered image.
Major features may require a vote."
3. Under "2.4 Documentation"
1. Add a new section with instructions entitled "Creating and Modifying
Diagrams". This section would provide basic instructions for downloading
source files from draw.io.
4. Add a new checkbox item under PR checklist heading "For documentation
related changes" with the following text
1. Have you ensured that any documentation diagrams have been
updated, along with their source files, using draw.io? See
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
for
instructions.

We require a minimum of 72 hours for a vote, not typically including
weekend days. I'd like to leave this vote open until Tuesday, 12PM
EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your vote is
binding or non-binding.

https://lists.apache.org/thread.html/3ae02f1e32044b1a7648899700d44611aefdab6caa09fb3196292425@%3Cdev.metron.apache.org%3E

Cheers,
Mike

Re: [DISCUSS] Format for sharing architecture source files and rendered images

[Otto Fowler]

Also, the section should either have a blurb and like for draw.io or a
reference footnote etc.


On April 17, 2019 at 21:36:03, Otto Fowler (ottobackwa...@gmail.com) wrote:

I think we should try draw.io, and should add ( with vote i think ) a new
section to the developer documention like:
Architecture Diagrams

The project supports the XXX format along with accompanying image
version in  format for architectural diagrams.

Architectural diagrams are important because reasons, and while they may
not be required for a PR with architectural implications, they may be
requested as part of the review process when deemed prudent.

—

Along with that, we can discuss how to store them in git and have them
integrated with the documentation *and* the website.




On April 16, 2019 at 11:57:46, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Hi devs,

We've talked about this fairly ad hoc on PRs before, and I thought it worth
bringing up more formally. Architecture diagrams are a strategically useful
addition to the core developer documentation. Examples include:

1.
https://github.com/apache/metron/tree/master/metron-platform/metron-parsing#parser-architecture
2.
https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment/metron-enrichment-storm#enrichment-architecture
3.
https://github.com/apache/metron/tree/master/metron-platform/metron-job#job-state-statechart
4.
https://github.com/apache/metron/tree/master/metron-interface#knox-requestresponse-flow

We currently have a couple formats for maintaining the source diagram files
across the project as well as a few diagrams with no source files. I've
personally used draw.io in the past because it's publicly available and
free to use. I also see Powerpoint.

1. No source file -
https://github.com/apache/metron/blob/master/metron-analytics/metron-maas-service/maas_arch.png
2. draw.io source -
https://github.com/apache/metron/blob/master/metron-platform/metron-job/metron-job_state_statechart_diagram.xml
3. powerpoint -
https://github.com/apache/metron/blob/master/metron-interface/flow_diagrams.pptx

I'm interested in what the developer community has access to and would
prefer. As I mentioned before, I'm partial to draw.io if only because it
does not require a fee to use. PowerPoint is probably easier to use, but it
does limit the contributors who can edit the images. Are there other
options that people have used they would prefer over any of the above? Are
we still interested in maintaining the source files (my personal vote is
yes)?

Best,
Mike

Re: [DISCUSS] Format for sharing architecture source files and rendered images

[Otto Fowler]

I think we should try draw.io, and should add ( with vote i think ) a new
section to the developer documention like:
Architecture Diagrams

The project supports the XXX format along with accompanying image
version in  format for architectural diagrams.

Architectural diagrams are important because reasons, and while they may
not be required for a PR with architectural implications, they may be
requested as part of the review process when deemed prudent.

—

Along with that, we can discuss how to store them in git and have them
integrated with the documentation *and* the website.





On April 16, 2019 at 11:57:46, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Hi devs,

We've talked about this fairly ad hoc on PRs before, and I thought it worth
bringing up more formally. Architecture diagrams are a strategically useful
addition to the core developer documentation. Examples include:

1.
https://github.com/apache/metron/tree/master/metron-platform/metron-parsing#parser-architecture
2.
https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment/metron-enrichment-storm#enrichment-architecture
3.
https://github.com/apache/metron/tree/master/metron-platform/metron-job#job-state-statechart
4.
https://github.com/apache/metron/tree/master/metron-interface#knox-requestresponse-flow

We currently have a couple formats for maintaining the source diagram files
across the project as well as a few diagrams with no source files. I've
personally used draw.io in the past because it's publicly available and
free to use. I also see Powerpoint.

1. No source file -
https://github.com/apache/metron/blob/master/metron-analytics/metron-maas-service/maas_arch.png
2. draw.io source -
https://github.com/apache/metron/blob/master/metron-platform/metron-job/metron-job_state_statechart_diagram.xml
3. powerpoint -
https://github.com/apache/metron/blob/master/metron-interface/flow_diagrams.pptx

I'm interested in what the developer community has access to and would
prefer. As I mentioned before, I'm partial to draw.io if only because it
does not require a fee to use. PowerPoint is probably easier to use, but it
does limit the contributors who can edit the images. Are there other
options that people have used they would prefer over any of the above? Are
we still interested in maintaining the source files (my personal vote is
yes)?

Best,
Mike

Re: Problems with Dev deployment.

[Otto Fowler]

Sorry, fixed it


On April 10, 2019 at 11:34:33, zeo...@gmail.com (zeo...@gmail.com) wrote:

Wow, I didn't realize that never got finalized/merged. Looks like there is
a failure in travis on that PR, if you get that wrapped up I would think we
should take another look at that and maybe get it merged. It has been a
while but I recall I was pretty happy with it after my review cycle with
you.

- Jon Zeolla
zeo...@gmail.com


On Wed, Apr 10, 2019 at 9:17 AM Otto Fowler 
wrote:

> These issues are the reason https://github.com/apache/metron/pull/1261
was
> done. It would be nice if we could get by them.
>
>
> On April 10, 2019 at 08:13:04, Dale Richardson (tigerqu...@outlook.com)
> wrote:
>
> Older pre-req versions are mentioned at:
>
>
>
https://metron.apache.org/current-book/metron-deployment/vagrant/codelab-platform/index.html
> Metron – Developer Image for Apache Metron on Virtualbox<
>
>
https://metron.apache.org/current-book/metron-deployment/vagrant/codelab-platform/index.html
> >
>
> Developer Image for Apache Metron on Virtualbox. This image is a fully
> functional Metron installation that has been pre-loaded with Ambari, HDP
> and Metron.
> metron.apache.org
>
>
>
https://metron.apache.org/current-book/metron-deployment/vagrant/full-dev-platform/index.html
> Metron – Full Development Platform<
>
>
https://metron.apache.org/current-book/metron-deployment/vagrant/full-dev-platform/index.html
> >
>
> Full Development Platform. This project fully automates the provisioning
> and deployment of Apache Metron and all necessary prerequisites on a
> single, virtualized host running on Virtualbox.
> metron.apache.org
>
>
>
>
https://metron.apache.org/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
> Metron – Quick Development Platform<
>
>
https://metron.apache.org/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
> >
>
> Quick Development Platform. This project fully automates the provisioning
> and deployment of Apache Metron and all necessary prerequisites on a
> single, virtualized host running on Virtualbox.
> metron.apache.org
>
>
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
>
> Metron Install on Ubuntu/Debian single-node VM with Vagrant and Ambari -
> Metron - Apache Software Foundation - cwiki.apache.org<
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548>

> Contributed by Umesh Kaushik < umesh.kaus...@bhujang.net > Introduction.
> These instructions are for an Ubuntu host, with occasional comments about
> how to do similar tasks in CentOS.
> cwiki.apache.org
>
> Which links to
> https://gist.github.com/dpalomar/96b826dac5c2e8b62cbf4c86dbd1c9df
>
>
>
> 
> From: Michael Miklavcic 
> Sent: Wednesday, 10 April 2019 4:15 AM
> To: dev@metron.apache.org
> Subject: Re: Problems with Dev deployment.
>
> Where are you seeing 2.0.0.2 for Ansible? Should be 2.4.0+ now.
>
>
https://github.com/apache/metron/blob/master/metron-deployment/development/centos6/README.md
>
>
>
> On Tue, Apr 9, 2019, 6:59 PM Michael Miklavcic <
> michael.miklav...@gmail.com>
>
> wrote:
>
> > That would be awesome man! Yes, Jira tickets for every change. I don't
> > recall off the top of my head the version - I'll have to look when I
get
> > back in front of a computer.
> >
> > On Tue, Apr 9, 2019, 6:56 PM Dale Richardson 
> > wrote:
> >
> >> Hi Michael,
> >> Yep that was the issue, looks like "brew install ansible" configures
> >> ansible for python 3. Happy to patch the documentation with updated
> >> install instructions (I will add the MacOS Mojave Drive permission
issue
> >> as well). Do you know if the minimum ansible version requirement has
> >> changed at all? (some documentation lists it as 2.0.0.2 or 2.2.2.0).
Do
> >> you guys usually put in a Jira to cover small documentation changes at
> all?
> >>
> >> Regards,
> >> Dale.
> >> 
> >> From: Otto Fowler 
> >> Sent: Sunday, 7 April 2019 1:40 PM
> >> To: dev@metron.apache.org
> >> Subject: Re: Problems with Dev deployment.
> >>
> >> Can you pull down
> >>
>
>
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fmetron%2Fpull%2F1261data=02%7C01%7C%7Cc00c98c5300d45adf8c508d6bbab3180%7C84df9e7fe9f640afb435%7C1%7C0%7C636902741276312736sdata=oylL4njlCkTYaLsyLYl44Zy8%2BoZoB4HVBCb5F2vgxng%3Dreserved=0
> >> and try?
> >> That should eliminate any env. issues.
> >

Re: Problems with Dev deployment.

[Otto Fowler]

These issues are the reason https://github.com/apache/metron/pull/1261 was
done.  It would be nice if we could get by them.


On April 10, 2019 at 08:13:04, Dale Richardson (tigerqu...@outlook.com)
wrote:

Older pre-req versions are mentioned at:

https://metron.apache.org/current-book/metron-deployment/vagrant/codelab-platform/index.html
Metron – Developer Image for Apache Metron on Virtualbox<
https://metron.apache.org/current-book/metron-deployment/vagrant/codelab-platform/index.html>

Developer Image for Apache Metron on Virtualbox. This image is a fully
functional Metron installation that has been pre-loaded with Ambari, HDP
and Metron.
metron.apache.org

https://metron.apache.org/current-book/metron-deployment/vagrant/full-dev-platform/index.html
Metron – Full Development Platform<
https://metron.apache.org/current-book/metron-deployment/vagrant/full-dev-platform/index.html>

Full Development Platform. This project fully automates the provisioning
and deployment of Apache Metron and all necessary prerequisites on a
single, virtualized host running on Virtualbox.
metron.apache.org


https://metron.apache.org/current-book/metron-deployment/vagrant/quick-dev-platform/index.html
Metron – Quick Development Platform<
https://metron.apache.org/current-book/metron-deployment/vagrant/quick-dev-platform/index.html>

Quick Development Platform. This project fully automates the provisioning
and deployment of Apache Metron and all necessary prerequisites on a
single, virtualized host running on Virtualbox.
metron.apache.org



https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548

Metron Install on Ubuntu/Debian single-node VM with Vagrant and Ambari -
Metron - Apache Software Foundation - cwiki.apache.org<
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548>
Contributed by Umesh Kaushik < umesh.kaus...@bhujang.net > Introduction.
These instructions are for an Ubuntu host, with occasional comments about
how to do similar tasks in CentOS.
cwiki.apache.org

Which links to
https://gist.github.com/dpalomar/96b826dac5c2e8b62cbf4c86dbd1c9df




From: Michael Miklavcic 
Sent: Wednesday, 10 April 2019 4:15 AM
To: dev@metron.apache.org
Subject: Re: Problems with Dev deployment.

Where are you seeing 2.0.0.2 for Ansible? Should be 2.4.0+ now.
https://github.com/apache/metron/blob/master/metron-deployment/development/centos6/README.md



On Tue, Apr 9, 2019, 6:59 PM Michael Miklavcic 

wrote:

> That would be awesome man! Yes, Jira tickets for every change. I don't
> recall off the top of my head the version - I'll have to look when I get
> back in front of a computer.
>
> On Tue, Apr 9, 2019, 6:56 PM Dale Richardson 
> wrote:
>
>> Hi Michael,
>> Yep that was the issue, looks like "brew install ansible" configures
>> ansible for python 3. Happy to patch the documentation with updated
>> install instructions (I will add the MacOS Mojave Drive permission issue
>> as well). Do you know if the minimum ansible version requirement has
>> changed at all? (some documentation lists it as 2.0.0.2 or 2.2.2.0). Do
>> you guys usually put in a Jira to cover small documentation changes at
all?
>>
>> Regards,
>> Dale.
>> 
>> From: Otto Fowler 
>> Sent: Sunday, 7 April 2019 1:40 PM
>> To: dev@metron.apache.org
>> Subject: Re: Problems with Dev deployment.
>>
>> Can you pull down
>>
https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fmetron%2Fpull%2F1261data=02%7C01%7C%7Cc00c98c5300d45adf8c508d6bbab3180%7C84df9e7fe9f640afb435%7C1%7C0%7C636902741276312736sdata=oylL4njlCkTYaLsyLYl44Zy8%2BoZoB4HVBCb5F2vgxng%3Dreserved=0
>> and try?
>> That should eliminate any env. issues.
>>
>> I’ll update it to latest master now.
>>
>>
>>
>> On April 7, 2019 at 08:33:52, Dale Richardson (tigerqu...@outlook.com)
>> wrote:
>>
>> Hi Folks,
>> I've been walking through some of the documentations on building and
>> deploying the Metron development image.
>> I am building Metron version 0.7. The deployment appears to go OK up
until
>> the Ambari deployment stage, and then I get the following error:
>>
>>
>> SyntaxError: invalid syntax
>>
>> INFO interface: detail: The full traceback is:
>>
>> Traceback (most recent call last):
>>
>> File
>>
>>
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/task_executor.py",

>> line 140, in run
>>
>> res = self._execute()
>>
>> File
>>
>>
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/task_executor.py",

>> lin

Re: Problems with Dev deployment.

[Otto Fowler]

Can you pull down https://github.com/apache/metron/pull/1261 and try?
That should eliminate any env. issues.

I’ll update it to latest master now.



On April 7, 2019 at 08:33:52, Dale Richardson (tigerqu...@outlook.com)
wrote:

Hi Folks,
I've been walking through some of the documentations on building and
deploying the Metron development image.
I am building Metron version 0.7. The deployment appears to go OK up until
the Ambari deployment stage, and then I get the following error:


SyntaxError: invalid syntax

INFO interface: detail: The full traceback is:

Traceback (most recent call last):

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/task_executor.py",
line 140, in run

res = self._execute()

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/task_executor.py",
line 612, in _execute

result = self._handler.run(task_vars=variables)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/plugins/action/normal.py",
line 46, in run

result = merge_hash(result, self._execute_module(task_vars=task_vars,
wrap_async=wrap_async))

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/plugins/action/__init__.py",
line 742, in _execute_module

(module_style, shebang, module_data, module_path) =
self._configure_module(module_name=module_name, module_args=module_args,
task_vars=task_vars)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/plugins/action/__init__.py",
line 178, in _configure_module

environment=final_environment)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/module_common.py",
line 973, in modify_module

environment=environment)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/module_common.py",
line 791, in _find_module_utils

recursive_finder(module_name, b_module_data, py_module_names,
py_module_cache, zf)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/module_common.py",
line 538, in recursive_finder

tree = ast.parse(data)

File
"/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py",
line 35, in parse

return compile(source, filename, mode, PyCF_ONLY_AST)

File "", line 230

except requests.ConnectionError, e:

^

MacOS Mojava (Version 10.14.3)
VirtualBox version 5.2.26
Vagrant version 2.2.4

ansible 2.7.9
config file =
/Users/user/work/metron/metron-deployment/development/centos6/ansible.cfg
ansible python module location =
/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.7.2 (default, Feb 12 2019, 08:15:36) [Clang 10.0.0
(clang-1000.11.45.5)]



Python2 version is 2.7.16

commit 2263983761e77b7eec52f70f5e8f8001bac83125 (HEAD, tag:
apache-metron_0.7.0-release, tag: apache-metron-0.7.0-rc1,
origin/Metron_0.7.0)

Does anybody have any suggestions on how to proceed?

Thanks,
Dale.

Re: Problems with Dev deployment.

[Otto Fowler]

Check the readme for instructions


On April 7, 2019 at 09:40:34, Otto Fowler (ottobackwa...@gmail.com) wrote:

Can you pull down https://github.com/apache/metron/pull/1261 and try?
That should eliminate any env. issues.

I’ll update it to latest master now.



On April 7, 2019 at 08:33:52, Dale Richardson (tigerqu...@outlook.com)
wrote:

Hi Folks,
I've been walking through some of the documentations on building and
deploying the Metron development image.
I am building Metron version 0.7. The deployment appears to go OK up until
the Ambari deployment stage, and then I get the following error:


SyntaxError: invalid syntax

INFO interface: detail: The full traceback is:

Traceback (most recent call last):

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/task_executor.py",
line 140, in run

res = self._execute()

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/task_executor.py",
line 612, in _execute

result = self._handler.run(task_vars=variables)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/plugins/action/normal.py",
line 46, in run

result = merge_hash(result, self._execute_module(task_vars=task_vars,
wrap_async=wrap_async))

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/plugins/action/__init__.py",
line 742, in _execute_module

(module_style, shebang, module_data, module_path) =
self._configure_module(module_name=module_name, module_args=module_args,
task_vars=task_vars)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/plugins/action/__init__.py",
line 178, in _configure_module

environment=final_environment)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/module_common.py",
line 973, in modify_module

environment=environment)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/module_common.py",
line 791, in _find_module_utils

recursive_finder(module_name, b_module_data, py_module_names,
py_module_cache, zf)

File
"/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible/executor/module_common.py",
line 538, in recursive_finder

tree = ast.parse(data)

File
"/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py",
line 35, in parse

return compile(source, filename, mode, PyCF_ONLY_AST)

File "", line 230

except requests.ConnectionError, e:

^

MacOS Mojava (Version 10.14.3)
VirtualBox version 5.2.26
Vagrant version 2.2.4

ansible 2.7.9
config file =
/Users/user/work/metron/metron-deployment/development/centos6/ansible.cfg
ansible python module location =
/usr/local/Cellar/ansible/2.7.9/libexec/lib/python3.7/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.7.2 (default, Feb 12 2019, 08:15:36) [Clang 10.0.0
(clang-1000.11.45.5)]



Python2 version is 2.7.16

commit 2263983761e77b7eec52f70f5e8f8001bac83125 (HEAD, tag:
apache-metron_0.7.0-release, tag: apache-metron-0.7.0-rc1,
origin/Metron_0.7.0)

Does anybody have any suggestions on how to proceed?

Thanks,
Dale.

Master is failing RAT check

[Otto Fowler]

https://travis-ci.org/apache/metron

Looks like an issue with node.

Re: [DISCUSS] Ambari MPacks, Metron upgrades

[Otto Fowler]

The long and the short of it is that I don't think there is likely
to be much more time invested in enhancing, upgrading, or expanding the
Ambari MPack features for Metron from this point on. We have many users
that currently depend on the installation process, and I think the option
should continue to exist, but we should start the process of decoupling our
installation from Ambari.


That seems like just the short of it.  Why don’t you think it is likely
more time will be used on mpack?

Can you elaborate?



On March 29, 2019 at 13:12:53, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Hi devs,

I wanted to kickstart a high level discussion on what the future of
development on Ambari MPacks and upgrades will/should look like ongoing for
Metron. The long and the short of it is that I don't think there is likely
to be much more time invested in enhancing, upgrading, or expanding the
Ambari MPack features for Metron from this point on. We have many users
that currently depend on the installation process, and I think the option
should continue to exist, but we should start the process of decoupling our
installation from Ambari.

I propose that we extract and decouple our install python scripts that are
in the MPack. We're already executing on decoupling our core platform code
wrt Storm. The scripts are already pretty well modularized, but we'll want
to replace some of the core Ambari features we currently depend on with our
own implementations. I'll emphasize that this refactoring should not and
cannot break the Ambari install, at least for now - my thought is that
validation would include a successful deployment with fulldev and the
Amazon EC2 cloud deployment script. We can re-evaluate what long term
support for Ambari will look like at a later time. For now, I want to
emphasize that this is just to enable flexibility.

The other nut to crack here is upgrades, and more to the point of why I'm
starting this thread in the first place. We've talked about it a while now,
and I think it's about time we got serious about paving the way to making
upgrades easier. I think we should make this process a pluggable set of
scripts as well. And from there, the community can plug that into whatever
cluster management software they wish.

Does this sound reasonable? I'm eager to hear everyone's thoughts.

Best,
Mike Miklavcic

Re: [DISCUSS] Central Navigation for Alerts and Management UI

[Otto Fowler]

Maybe you should post to the users@ list


On March 11, 2019 at 06:25:48, Shane Ardell (shane.m.ard...@gmail.com)
wrote:

Thank you both for explaining the original design choice. That does make
sense.

However, like both of you point out, I am also curious if anyone has direct
user feedback indicating a need for either persona to switch between
screens.

Shane

On Tue, Mar 5, 2019 at 7:25 PM Rita McKissick 
wrote:

> That was my thought, too. The Management UI is meant for the Operations
> persona.
> And the Alerts UI is meant for the SOC analyst persona. If we see a need
> for either of these
> personas to use both of the UIs, then the ability to switch between the
> two UIs would
> be great. Otherwise, I'm not sure that ability is necessary.
>
> As an aside, as the tech writer I would love to be able to switch between
> the two UIs, but I'm not
> really one of our supported personas __ Darn!
>
> Rita
>
> Rita McKissick ! Sr. Technical Writer
> rmckiss...@hortonworks.com
> (mobile) 831-234-3676
>
>
> On 3/5/19, 9:50 AM, "Michael Miklavcic" 
> wrote:
>
> The original design was done with the intent to keep the user profiles
> (soc
> analyst vs ops personnel) separate and enable a microservices-oriented
> architecture. I don't have a strong opinion one way or the other, but
> I'd
> be interested to hear whether others in the community find this wall
> useful, or if we should come back to a single pain of glass.
>
> Mike
>
> On Tue, Mar 5, 2019 at 9:12 AM Shane Ardell 
> wrote:
>
> > Hello everyone,
> >
> > I recently started experimenting with implementing a navigation bar
> in both
> > the Alerts and Management UI. It would allow us to navigate between
> the two
> > UIs through links instead of manually entering a url or opening
> separate
> > tabs from Ambari.
> >
> > I'm just wondering what everyone's thoughts are. Is this something
> we want
> > in Metron?
> >
>
>
>

Re: [DISCUSS] Upgrading HBase and Kafka support

[Otto Fowler]

https://github.com/apache/metron-bro-plugin-kafka/tree/master/docker


On March 8, 2019 at 14:28:20, zeo...@gmail.com (zeo...@gmail.com) wrote:

So most importantly I want to make sure to give Otto credit for being the
one who cleaned up the rudimentary testing steps we had for testing the
plugin and turned it into the docker end to end. Right now we manually run
the tests, as there were a few follow-ons we needed to work through before
it's ready for Travis. In my opinion, once METRON-2003 (PR 26) gets in
it'll be ready to have Travis. There isn't any current Maven use

Jon

On Fri, Mar 8, 2019 at 12:26 PM Otto Fowler 
wrote:

> I believe that the TestContainers allows the ide case
>
>
> On March 8, 2019 at 11:38:24, Michael Miklavcic (
> michael.miklav...@gmail.com)
> wrote:
>
> I'm -1 on #1 unless there's some desperately compelling reason to go that
> route. It would be a regression in our test coverage, and at that point
> it's really just duplicating our unit tests as opposed to checking our
> integration.
>
> I'm good with 3. Gating factors for a successful implementation would be
> that as a developer I can:
>
> 1. Run it in my IDE without having to do anything extra (the beauty of
> the in-mem component is that @BeforeClass spins it up automatically - we
> should keep doing something along those lines)
> 2. Run it via Maven cli
> 3. Run it in Travis as part of our normal build
>
> It's probably worth looking at Kafka's testing infrastructure straight
from
> the source - https://github.com/apache/kafka/blob/trunk/tests/README.md.
> They leverage Docker containers now for system tests.
>
> Best,
> Mike
>
>
> On Fri, Mar 8, 2019 at 7:47 AM Ryan Merriman  wrote:
>
> > I have been researching the effort involved to upgrade to HDP 3. Along
> the
> > way I've found a couple challenging issues that we will need to solve,
> both
> > involving our integration testing strategy.
> >
> > The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there
> > have been significant changes to the API. This creates an issue in the
> > KafkaComponent class, which we use as an in-memory Kafka server in
> > integration tests. Most of the classes that were previously used have
> gone
> > away, and to the best of my knowledge, were not supported as public
APIs.
> > I also don't see any publicly documented APIs to replace them.
> >
> > The second issue is HBase. We are moving from 1.1.2 to 2.0.2 so another
> > significant change. This creates an issue in the MockHTable class
> > becausethe HTableInterface class has changed to Table, essentially
> > requiring that MockHTable be rewritten to conform to the new interface.
> > It's my opinion that this class is complicated and difficult to
maintain
> as
> > it is anyways.
> >
> > These 2 issues have the potential to add a significant amount of work
to
> > upgrading Metron to HDP 3. I want to take a step back and review our
> > options before we move forward. Here are some initial thoughts I had on
> > how to approach this. For HBase:
> >
> > 1. Update MockHTable to work with the new HBase API. We would continue
> > using a mock server approach for HBase.
> > 2. Research replacing MockHTable with an in-memory HBase server.
> > 3. Replace MockHTable with a Docker container running HBase.
> >
> > For Kafka:
> >
> > 1. Replace KafkaComponent with a mock server implementation.
> > 2. Update KafkaComponent to work with the new API. We would probably
> > need to leverage some internal Kafka classes. I do not see a testing
> > API
> > documented publicly.
> > 3. Replace KafkaComponent with a Docker container running Kafka.
> >
> > What other options are there? Whatever we choose I think we should
follow
> > a similar approach for both (mock servers, in memory servers, Docker,
> other
> > options I'm not thinking of).
> >
> > This will not shock anyone but I would be in favor of Docker
containers.
> > They have the advantage of classpath isolation, easy upgrades, and
> accurate
> > integration testing. The downside is we will have to adjusts our tests
> and
> > travis script to incorporate these Docker containers into our build
> > process. We have discussed this at length in the past and it has
> generally
> > stalled for various reasons. Maybe if we move a few services at a time
it
> > might be more palatable? As for the other 2 approaches, I think if
either
> > worked well we wouldn't be having this discussion. Mock servers are
hard
> > to maintain and I don't see in memory testing classes documented in
> > javadocs for either service.
> >
> > Thoughts?
> >
>
-- 

Jon Zeolla

Re: [DISCUSS] Upgrading HBase and Kafka support

[Otto Fowler]

I believe that the TestContainers allows the ide case


On March 8, 2019 at 11:38:24, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

I'm -1 on #1 unless there's some desperately compelling reason to go that
route. It would be a regression in our test coverage, and at that point
it's really just duplicating our unit tests as opposed to checking our
integration.

I'm good with 3. Gating factors for a successful implementation would be
that as a developer I can:

1. Run it in my IDE without having to do anything extra (the beauty of
the in-mem component is that @BeforeClass spins it up automatically - we
should keep doing something along those lines)
2. Run it via Maven cli
3. Run it in Travis as part of our normal build

It's probably worth looking at Kafka's testing infrastructure straight from
the source - https://github.com/apache/kafka/blob/trunk/tests/README.md.
They leverage Docker containers now for system tests.

Best,
Mike


On Fri, Mar 8, 2019 at 7:47 AM Ryan Merriman  wrote:

> I have been researching the effort involved to upgrade to HDP 3. Along
the
> way I've found a couple challenging issues that we will need to solve,
both
> involving our integration testing strategy.
>
> The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there
> have been significant changes to the API. This creates an issue in the
> KafkaComponent class, which we use as an in-memory Kafka server in
> integration tests. Most of the classes that were previously used have
gone
> away, and to the best of my knowledge, were not supported as public APIs.
> I also don't see any publicly documented APIs to replace them.
>
> The second issue is HBase. We are moving from 1.1.2 to 2.0.2 so another
> significant change. This creates an issue in the MockHTable class
> becausethe HTableInterface class has changed to Table, essentially
> requiring that MockHTable be rewritten to conform to the new interface.
> It's my opinion that this class is complicated and difficult to maintain
as
> it is anyways.
>
> These 2 issues have the potential to add a significant amount of work to
> upgrading Metron to HDP 3. I want to take a step back and review our
> options before we move forward. Here are some initial thoughts I had on
> how to approach this. For HBase:
>
> 1. Update MockHTable to work with the new HBase API. We would continue
> using a mock server approach for HBase.
> 2. Research replacing MockHTable with an in-memory HBase server.
> 3. Replace MockHTable with a Docker container running HBase.
>
> For Kafka:
>
> 1. Replace KafkaComponent with a mock server implementation.
> 2. Update KafkaComponent to work with the new API. We would probably
> need to leverage some internal Kafka classes. I do not see a testing
> API
> documented publicly.
> 3. Replace KafkaComponent with a Docker container running Kafka.
>
> What other options are there? Whatever we choose I think we should follow
> a similar approach for both (mock servers, in memory servers, Docker,
other
> options I'm not thinking of).
>
> This will not shock anyone but I would be in favor of Docker containers.
> They have the advantage of classpath isolation, easy upgrades, and
accurate
> integration testing. The downside is we will have to adjusts our tests
and
> travis script to incorporate these Docker containers into our build
> process. We have discussed this at length in the past and it has
generally
> stalled for various reasons. Maybe if we move a few services at a time it
> might be more palatable? As for the other 2 approaches, I think if either
> worked well we wouldn't be having this discussion. Mock servers are hard
> to maintain and I don't see in memory testing classes documented in
> javadocs for either service.
>
> Thoughts?
>

Re: [DISCUSS] Upgrading HBase and Kafka support

[Otto Fowler]

I think I have mentioned it before, but https://www.testcontainers.org could
be a viable approach for this methodology (3).
I would think it would be worth looking at.



On March 8, 2019 at 09:47:54, Ryan Merriman (merrim...@gmail.com) wrote:

I have been researching the effort involved to upgrade to HDP 3. Along the
way I've found a couple challenging issues that we will need to solve, both
involving our integration testing strategy.

The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there
have been significant changes to the API. This creates an issue in the
KafkaComponent class, which we use as an in-memory Kafka server in
integration tests. Most of the classes that were previously used have gone
away, and to the best of my knowledge, were not supported as public APIs.
I also don't see any publicly documented APIs to replace them.

The second issue is HBase. We are moving from 1.1.2 to 2.0.2 so another
significant change. This creates an issue in the MockHTable class
becausethe HTableInterface class has changed to Table, essentially
requiring that MockHTable be rewritten to conform to the new interface.
It's my opinion that this class is complicated and difficult to maintain as
it is anyways.

These 2 issues have the potential to add a significant amount of work to
upgrading Metron to HDP 3. I want to take a step back and review our
options before we move forward. Here are some initial thoughts I had on
how to approach this. For HBase:

1. Update MockHTable to work with the new HBase API. We would continue
using a mock server approach for HBase.
2. Research replacing MockHTable with an in-memory HBase server.
3. Replace MockHTable with a Docker container running HBase.

For Kafka:

1. Replace KafkaComponent with a mock server implementation.
2. Update KafkaComponent to work with the new API. We would probably
need to leverage some internal Kafka classes. I do not see a testing API
documented publicly.
3. Replace KafkaComponent with a Docker container running Kafka.

What other options are there? Whatever we choose I think we should follow
a similar approach for both (mock servers, in memory servers, Docker, other
options I'm not thinking of).

This will not shock anyone but I would be in favor of Docker containers.
They have the advantage of classpath isolation, easy upgrades, and accurate
integration testing. The downside is we will have to adjusts our tests and
travis script to incorporate these Docker containers into our build
process. We have discussed this at length in the past and it has generally
stalled for various reasons. Maybe if we move a few services at a time it
might be more palatable? As for the other 2 approaches, I think if either
worked well we wouldn't be having this discussion. Mock servers are hard
to maintain and I don't see in memory testing classes documented in
javadocs for either service.

Thoughts?

Re: [DISCUSS] Architecture documentation

[Otto Fowler]

I really like the idea of architecture.md -> **/architecture.md.

We overall do not have javadoc in a lot of areas, and could maybe start
working on it as we go and think about asking for it in reviews.
We are also missing the Parser Programmer’s Guide, how to add a parser to
the metron system/install etc and other things.



On February 25, 2019 at 15:22:47, Ryan Merriman (merrim...@gmail.com) wrote:

I feel like the code itself is pretty well documented. I updated existing
javadocs and added javadocs to classes that didn't have them before this
PR. In my opinion the level of documentation for these classes has
increased significantly.

On Mon, Feb 25, 2019 at 1:52 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Tentatively agreed on further clarification of what we consider in/out of
> scope for documentation re: document something that wasn't documented
> before. Ryan, can you give a quick summary of what you *have*
added/updated
> in documentation on this PR vs what you want to leave out?
>
> My initial concern in punting on docs right now is that part of what made
> this PR/task more challenging in the first place was not having
> documentation. We risk losing context and detail again if we don't do
this
> immediately. Would it be reasonable to split it up as follows?:
>
> 1. Additional overarching documentation feels out of scope - make it a
> follow on (see comments below).
> 2. Adding documentation to our existing README's and java code comments
> that describe the new/modified functionality should be in scope because
> it's part of the unit of work. I expect that a developer should be able
> to
> look at the code, tests, comments, and README's and understand how this
> code functions without having to start from scratch.
>
> The way we've handled follow-on work before, at least as far as feature
> branches are concerned, was to create Jiras and link them to the
> appropriate discussions for context. Maybe we can take that one step
> further and do the release manager a favor by also labeling the
> required/requested release on the Jira as a gating factor. This follows
our
> pattern for intermittent test failure reporting, e.g.
>
>
https://issues.apache.org/jira/browse/METRON-1946?jql=project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-failure%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> .
>
> I'm also in favor of continuing to document architecture and technical
> details as part of the code base as Ryan and Jon have suggested. I think
we
> should have an "architecture.md" in metron root that replaces this -
>
>
https://github.com/apache/metron/blob/d7d4fd9afb19e2bd2e66babb7e1514a19eae07d0/README.md#navigating-the-architecture
> and covers the broad architecture with links to the appropriate modules
for
> detail. Minimally, it would be nice if we had a simple diagram showing
the
> basic flow of data in Metron. I think we probably want an updated version
> of this wiki entry from back in the day -
> https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture
>
> Best,
> Mike
>
>
> On Mon, Feb 25, 2019 at 7:18 AM Nick Allen  wrote:
>
> > I don't think we should hold up this work to document something that
> wasn't
> > previously documented. A follow-on is sufficient.
> >
> > On Mon, Feb 25, 2019 at 8:50 AM Ryan Merriman 
> wrote:
> >
> > > Recently I submitted a PR 

> > > that
> > > introduces a large number of changes to a critical part of our code
> base.
> > > Reviewers feel like it is significant enough to document at an
> > > architectural level (and I agree). There are a couple points I would
> > like
> > > to clarify.
> > >
> > > Generally architectural documentation lives in the README of the
> > > appropriate module. Do we want to continue documenting architecture
> > here?
> > > I think it makes sense because it will be versioned along with the
> code.
> > > Just wanted to confirm there are no objections to continuing this
> > practice.
> > >
> > > A reviewer suggested we could accept the PR as is and leave the
> > > architectural documentation as a follow on. I think this makes sense
> > > because it can be tedious to maintain a large PR as other smaller
> commits
> > > are accepted into master. An important requirement is the
> documentation
> > > follow on must be completed in a timely manner, before the next
> release.
> > > Are there any objections to doing it this way?
> > >
> >
>

RE: How to provide hbase-site.xml to Stellar Processor Java API

[Otto Fowler]

Please link it to https://issues.apache.org/jira/browse/METRON-1409.
This is a jira around hosting stellar that I created a while ago.


On January 25, 2019 at 07:02:30, Anil Donthireddy (
anil.donthire...@sstech.us) wrote:

Hi Mohan DV,



I appreciate your time for sharing the much useful information.



As per my understanding there seems to be no scope to pass my own build
Hbase configuration object to execute Stellar queries in my extended API. I
may need to re-write lot of things in my extended API in the way Stellar
processor works to override the Hbase configuration.



@Otto Fowler: I will raise a Jira case with the usecase and the issue.



Thanks,

Anil.



*From:* Otto Fowler [mailto:ottobackwa...@gmail.com]
*Sent:* Thursday, January 24, 2019 11:15 PM
*To:* Anil Donthireddy ; u...@metron.apache.org;
dev@metron.apache.org; Mohan Venkateshaiah 
*Cc:* Maxim Dashenko ; Satish Abburi <
satish.abb...@sstech.us>; James Sirota ;
Christopher Berry 
*Subject:* Re: How to provide hbase-site.xml to Stellar Processor Java API



But still file a jira ;)





On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (
mvenkatesha...@hortonworks.com) wrote:

Hi Anil,

I had done similar to this in the past . In the stellar to get the hbase
configuration we call HBaseConfiguration.create() , in that call hbase adds
the hbase-site and core-site as resources to the config we probably SHOULD
let people specify a base config.

What I had done was in the global config, I set a property called
hbase.provider.impl, it's the fully qualified class name for a class that
implements the TableProvider interface which has one method:

public HTableInterface getTable(Configuration config, String tableName)
throws IOException

if you implement your own where you ignore the config argument and resolve
the hbase table with your own injected config that will work

Thanks
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler"  wrote:

Hi Anil,
Can you create a jira on this with these details and a general overview of
your use case?
It looks like the HbaseConfiguration we use in the HTableConnector
is done using the create() method, which creates from resources.

I think we would need to do some work to support the external file.



On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthire...@sstech.us) wrote:

Hi,



I have written a java application which uses Stellar processor and execute
the stellar expressions. The issue I am facing is I am unable to connect
Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
code. As it is not the proper way of packaging the hbase-site.xml with Jar,
I would like to understand how the hbase-site.xml is being set to classpath
while starting profiler topology.



The ways I tried are

1) Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf




One more step I would like to try to fix the issue is to set property “
*zookeeper.znode.parent*” for configuration object while instantiating
HbaseConnector. But it is in the scope of metron code to try this fix.



I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.



Thanking you.



Thanks,

Anil.

Re: How to provide hbase-site.xml to Stellar Processor Java API

[Otto Fowler]

But still file a jira ;)


On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (
mvenkatesha...@hortonworks.com) wrote:

Hi Anil,

I had done similar to this in the past . In the stellar to get the hbase
configuration we call HBaseConfiguration.create() , in that call hbase adds
the hbase-site and core-site as resources to the config we probably SHOULD
let people specify a base config.

What I had done was in the global config, I set a property called
hbase.provider.impl, it's the fully qualified class name for a class that
implements the TableProvider interface which has one method:

public HTableInterface getTable(Configuration config, String tableName)
throws IOException

if you implement your own where you ignore the config argument and resolve
the hbase table with your own injected config that will work

Thanks
Mohan DV

On 1/24/19, 8:56 PM, "Otto Fowler"  wrote:

Hi Anil,
Can you create a jira on this with these details and a general overview of
your use case?
It looks like the HbaseConfiguration we use in the HTableConnector
is done using the create() method, which creates from resources.

I think we would need to do some work to support the external file.



On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthire...@sstech.us) wrote:

Hi,



I have written a java application which uses Stellar processor and execute
the stellar expressions. The issue I am facing is I am unable to connect
Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
code. As it is not the proper way of packaging the hbase-site.xml with Jar,
I would like to understand how the hbase-site.xml is being set to classpath
while starting profiler topology.



The ways I tried are

1) Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf




One more step I would like to try to fix the issue is to set property “
*zookeeper.znode.parent*” for configuration object while instantiating
HbaseConnector. But it is in the scope of metron code to try this fix.



I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.



Thanking you.



Thanks,

Anil.

Re: How to provide hbase-site.xml to Stellar Processor Java API

[Otto Fowler]

Hi Anil,
Can you create a jira on this with these details and a general overview of
your use case?
It looks like the HbaseConfiguration we use in the HTableConnector
is done using the create() method, which creates from resources.

I think we would need to do some work to support the external file.



On January 24, 2019 at 10:14:46, Anil Donthireddy (
anil.donthire...@sstech.us) wrote:

Hi,



I have written a java application which uses Stellar processor and execute
the stellar expressions. The issue I am facing is I am unable to connect
Hbase unless I place hbase-site.xml in src/main/resources/ folder of the
code. As it is not the proper way of packaging the hbase-site.xml with Jar,
I would like to understand how the hbase-site.xml is being set to classpath
while starting profiler topology.



The ways I tried are

1)  Setting the classpath to hbase conf folder using command “java -cp
$CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar”

2)  Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop
classpath

/usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf



One more step I would like to try to fix the issue is to set property “
*zookeeper.znode.parent*” for configuration object while instantiating
HbaseConnector. But it is in the scope of metron code to try this fix.



I would like to know if any one able to provide hbase-site.xml to any Java
Application or anyone able to extend Metron Stellar Processor and execute
profile definitions successfully.

Please provide any inputs to resolve the issue.



Thanking you.



Thanks,

Anil.

Re: [DISCUSS] Writer class refactor

[Otto Fowler]

Agreed


On January 18, 2019 at 14:52:32, Ryan Merriman (merrim...@gmail.com) wrote:

I am on board with that. In that case, I think it's even more important
that we get the Writer interfaces right.

On Fri, Jan 18, 2019 at 1:34 PM Otto Fowler 
wrote:

> I think that the writers should be loaded as, and act as extension
points,
> such that it is possible to have 3rd party writers, and would structure
> them as such.
>
>
>
> On January 18, 2019 at 13:55:00, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> Recently there was a bug reported by a user where a parser that emits
> multiple messages from a single tuple doesn't work correctly:
> https://issues.apache.org/jira/browse/METRON-1968. This has exposed a
> problem with how the writer classes work.
>
> The fundamental issue is this: the writer classes operate under the
> assumption that there is a 1 to 1 mapping between tuples and messages to
> be
> written. A couple of examples:
>
> KafkaWriter
> <
>
https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java#L236>

>
> -
> This class writes messages by iterating through the list of tuples and
> fetching the message with the same index. This is the cause of the Jira
> above. We could iterate through the message list instead but then we
don't
> know which tuples have been fully processed. It would be possible for a
> batch to be flushed before all messages from a tuple are passed to the
> writer.
>
> BulkWriterComponent
> <
>
https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java#L250>

>
> - The tuple list size is used to determine when a batch should be
flushed.
> While inherently incorrect in my opinion (should be message list size),
> this also causes an issue where only the first message from the last
tuple
> in a batch is written.
>
> I do not believe there are easy fixes to these problems. There is no way
> to properly store the relationship between tuples and messages to be
> written with the current BulkMessageWriter interface and
> BulkWriterResponse
> class. If we did have a way, how should we handle partial failures? If
> multiple messages are parsed from a tuple but only half of them are
> written
> successfully, what should happen? Should we replay the tuple? Should we
> just report the failed messages and continue on? I think it may be a good
> time to review our writer classes and consider a refactor. Do others
> agree? Are there easy fixes I'm missing?
>
> Assuming there is interest in refactoring, I will throw out some ideas
for
> consideration. For those not as familiar with the writer classes, they
are
> organized as follows (in order from lowest to highest level):
>
> Writers - These classes do the actual writing and implement the
> BulkMessageWriter or MessageWriter interfaces. There are 6
implementations
> I can see including KafkaWriter, SolrWriter, ElasticsearchWriter,
> HdfsWriter, etc. There is also an implementation that adapts a
> MessageWriter to a BulkMessageWriter (WriterToBulkWriter). The result of
a
> writing operation is a BulkWriterResponse containing a list of either
> successful or failed tuples.
>
> Writer Containers - This includes the BulkWriterComponent and
> WriterHandler
> classes. These are responsible for batching and flushing messages,
> handling errors and acking tuples.
>
> Bolts - This includes ParserBolt, WriterBolt and BulkMessageWriterBolt.
> These classes implement the Storm Bolt interfaces, setup
> writers/components
> and execute tuples.
>
> I think the first step is to reevaluate the separation of concerns for
> these classes. Here is how I would change from what we currently have:
>
> Writers - These classes should only be concerned with writing messages
and
> reporting what happened. They would also manage the lifecycle and
> configuration of the underlying client libraries as they do now. Instead
> of accepting 2 separate lists, they should accept a data structure that
> accurately represents the relationship between tuples and messages.
>
> Writer Containers - These classes would continue to handling batching and
> flushing but would only report the results of a flush rather than
actually
> doing the acking or error handling.
>
> Bolts - These would now be responsible for acking and error reporting on
> tuples. They would transform a tuple into something the Writer Containers
> can accept as input.
>
> I think working through this and adjusting the contracts between the
> different layers will be necessary to fix the bugs described above. While
> we're at it I think there are other improvem

Re: [DISCUSS] Writer class refactor

[Otto Fowler]

I think that the writers should be loaded as, and act as extension points,
such that it is possible to have 3rd party writers, and would structure
them as such.



On January 18, 2019 at 13:55:00, Ryan Merriman (merrim...@gmail.com) wrote:

Recently there was a bug reported by a user where a parser that emits
multiple messages from a single tuple doesn't work correctly:
https://issues.apache.org/jira/browse/METRON-1968. This has exposed a
problem with how the writer classes work.

The fundamental issue is this: the writer classes operate under the
assumption that there is a 1 to 1 mapping between tuples and messages to be
written. A couple of examples:

KafkaWriter
<
https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java#L236>

-
This class writes messages by iterating through the list of tuples and
fetching the message with the same index. This is the cause of the Jira
above. We could iterate through the message list instead but then we don't
know which tuples have been fully processed. It would be possible for a
batch to be flushed before all messages from a tuple are passed to the
writer.

BulkWriterComponent
<
https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java#L250>

- The tuple list size is used to determine when a batch should be flushed.
While inherently incorrect in my opinion (should be message list size),
this also causes an issue where only the first message from the last tuple
in a batch is written.

I do not believe there are easy fixes to these problems. There is no way
to properly store the relationship between tuples and messages to be
written with the current BulkMessageWriter interface and BulkWriterResponse
class. If we did have a way, how should we handle partial failures? If
multiple messages are parsed from a tuple but only half of them are written
successfully, what should happen? Should we replay the tuple? Should we
just report the failed messages and continue on? I think it may be a good
time to review our writer classes and consider a refactor. Do others
agree? Are there easy fixes I'm missing?

Assuming there is interest in refactoring, I will throw out some ideas for
consideration. For those not as familiar with the writer classes, they are
organized as follows (in order from lowest to highest level):

Writers - These classes do the actual writing and implement the
BulkMessageWriter or MessageWriter interfaces. There are 6 implementations
I can see including KafkaWriter, SolrWriter, ElasticsearchWriter,
HdfsWriter, etc. There is also an implementation that adapts a
MessageWriter to a BulkMessageWriter (WriterToBulkWriter). The result of a
writing operation is a BulkWriterResponse containing a list of either
successful or failed tuples.

Writer Containers - This includes the BulkWriterComponent and WriterHandler
classes. These are responsible for batching and flushing messages,
handling errors and acking tuples.

Bolts - This includes ParserBolt, WriterBolt and BulkMessageWriterBolt.
These classes implement the Storm Bolt interfaces, setup writers/components
and execute tuples.

I think the first step is to reevaluate the separation of concerns for
these classes. Here is how I would change from what we currently have:

Writers - These classes should only be concerned with writing messages and
reporting what happened. They would also manage the lifecycle and
configuration of the underlying client libraries as they do now. Instead
of accepting 2 separate lists, they should accept a data structure that
accurately represents the relationship between tuples and messages.

Writer Containers - These classes would continue to handling batching and
flushing but would only report the results of a flush rather than actually
doing the acking or error handling.

Bolts - These would now be responsible for acking and error reporting on
tuples. They would transform a tuple into something the Writer Containers
can accept as input.

I think working through this and adjusting the contracts between the
different layers will be necessary to fix the bugs described above. While
we're at it I think there are other improvements we could also make:

Decouple Storm - It would be beneficial to remove the dependency on tuples
in our writers and writer containers. We could replace this with a simple
abstraction (an id would probably work fine). This will allow us to more
easily port Metron to other streaming platforms.

Remove MessageWriter Interface - This is not being actively used as far as
I can tell. Is that true? Removing this will make our code simpler and
easier to follow (WriterHandler and WriterToBulkWriter classes can probably
go away). I don't see any reason future writers, even those without bulk
writing capabilities, could not fit into the BulkMessageWriter interface.
A writer could either iterate through messages and write 

Re: Site-book broken in master

[Otto Fowler]

https://github.com/oasp/asciidoc-link-checker


On December 20, 2018 at 12:59:39, Otto Fowler (ottobackwa...@gmail.com)
wrote:

+1 for merging a fix.



On December 20, 2018 at 12:43:57, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Thanks for the feedback Otto. I'm not too familiar with our infrastructure
afa impact of switching to ascii doc, but I think it's worth considering if
it allows us to have a consistent rendering markdown that's easy to use.
Automated link testing is a good idea. I wonder if Cypress could do that
well, or if we should just write our own link tester that spins up embedded
Jetty, or other option.

*Anyone opposed to me merging this PR in sooner than the 24 hour period?*
This is a broken master as far as I'm concerned, despite what Travis says.

In a separate PR, I'd also like to add a simple mvn clean site run to our
Travis builds, which should catch the bigger issues.

On Wed, Dec 19, 2018 at 8:51 PM Otto Fowler  wrote:

> Maybe we should consider ascii doc, it shows in github, and can generate
> multiple formats.
>
>
>
> On December 19, 2018 at 22:49:26, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> What would help is an automated test the tests all the links as well.
>
>
>
> On December 19, 2018 at 19:55:03, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> I'd also like to submit a call for suggestions for making this process
> simpler. I think we've made some excellent progress towards getting an
> html-generated version of our documentation, but it's also proving to be
> somewhat error-prone. Github and Doxia have different expectations as to
> formatting, and most devs are generally looking at Github rendering only
> because we get that for free as part of the Github PR lifecycle. In
> addition, images are managed independently. Missing a reference, getting
> the relative path wrong, or getting the rewrite rules incorrect will result
> in a bad time. I've seen at least a few PR's in recent memory unwittingly
> fall into this trap.
>
> On Wed, Dec 19, 2018 at 5:47 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Fix is out in this PR: https://github.com/apache/metron/pull/1309
> > Jira: https://issues.apache.org/jira/browse/METRON-1950
> >
> > It ended up being a bit bigger than I hoped because we had some remaining
> > incompatibilities between our Github markdown and the Doxia-expected
> > formatting. This caused a cascade of inconsistent results for anyone
> > attempting to generate a page that works in both Github markdown and
> Doxia.
> > The latest fixes here should address all of this in the parsers
> > documentation. I suspect we may have other markdown pages out there that
> > will have similar issues, so this PR/Jira/notice should serve as a recipe
> > for remedying them in the future.
> >
> > Best,
> > Mike
> >
> >
> > On Wed, Dec 19, 2018 at 3:46 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> FYI, site-book is currently broken in master. I think I've found the
> >> source of the issue and will be issuing a fix shortly.
> >>
> >> Best,
> >> Mike
> >>
> >
>
>

Re: Site-book broken in master

[Otto Fowler]

+1 for merging a fix.



On December 20, 2018 at 12:43:57, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Thanks for the feedback Otto. I'm not too familiar with our infrastructure
afa impact of switching to ascii doc, but I think it's worth considering if
it allows us to have a consistent rendering markdown that's easy to use.
Automated link testing is a good idea. I wonder if Cypress could do that
well, or if we should just write our own link tester that spins up embedded
Jetty, or other option.

*Anyone opposed to me merging this PR in sooner than the 24 hour period?*
This is a broken master as far as I'm concerned, despite what Travis says.

In a separate PR, I'd also like to add a simple mvn clean site run to our
Travis builds, which should catch the bigger issues.

On Wed, Dec 19, 2018 at 8:51 PM Otto Fowler  wrote:

> Maybe we should consider ascii doc, it shows in github, and can generate
> multiple formats.
>
>
>
> On December 19, 2018 at 22:49:26, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> What would help is an automated test the tests all the links as well.
>
>
>
> On December 19, 2018 at 19:55:03, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> I'd also like to submit a call for suggestions for making this process
> simpler. I think we've made some excellent progress towards getting an
> html-generated version of our documentation, but it's also proving to be
> somewhat error-prone. Github and Doxia have different expectations as to
> formatting, and most devs are generally looking at Github rendering only
> because we get that for free as part of the Github PR lifecycle. In
> addition, images are managed independently. Missing a reference, getting
> the relative path wrong, or getting the rewrite rules incorrect will result
> in a bad time. I've seen at least a few PR's in recent memory unwittingly
> fall into this trap.
>
> On Wed, Dec 19, 2018 at 5:47 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Fix is out in this PR: https://github.com/apache/metron/pull/1309
> > Jira: https://issues.apache.org/jira/browse/METRON-1950
> >
> > It ended up being a bit bigger than I hoped because we had some remaining
> > incompatibilities between our Github markdown and the Doxia-expected
> > formatting. This caused a cascade of inconsistent results for anyone
> > attempting to generate a page that works in both Github markdown and
> Doxia.
> > The latest fixes here should address all of this in the parsers
> > documentation. I suspect we may have other markdown pages out there that
> > will have similar issues, so this PR/Jira/notice should serve as a recipe
> > for remedying them in the future.
> >
> > Best,
> > Mike
> >
> >
> > On Wed, Dec 19, 2018 at 3:46 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> FYI, site-book is currently broken in master. I think I've found the
> >> source of the issue and will be issuing a fix shortly.
> >>
> >> Best,
> >> Mike
> >>
> >
>
>

Re: Site-book broken in master

[Otto Fowler]

What would help is an automated test the tests all the links as well.



On December 19, 2018 at 19:55:03, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

I'd also like to submit a call for suggestions for making this process
simpler. I think we've made some excellent progress towards getting an
html-generated version of our documentation, but it's also proving to be
somewhat error-prone. Github and Doxia have different expectations as to
formatting, and most devs are generally looking at Github rendering only
because we get that for free as part of the Github PR lifecycle. In
addition, images are managed independently. Missing a reference, getting
the relative path wrong, or getting the rewrite rules incorrect will result
in a bad time. I've seen at least a few PR's in recent memory unwittingly
fall into this trap.

On Wed, Dec 19, 2018 at 5:47 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Fix is out in this PR: https://github.com/apache/metron/pull/1309
> Jira: https://issues.apache.org/jira/browse/METRON-1950
>
> It ended up being a bit bigger than I hoped because we had some remaining
> incompatibilities between our Github markdown and the Doxia-expected
> formatting. This caused a cascade of inconsistent results for anyone
> attempting to generate a page that works in both Github markdown and
Doxia.
> The latest fixes here should address all of this in the parsers
> documentation. I suspect we may have other markdown pages out there that
> will have similar issues, so this PR/Jira/notice should serve as a recipe
> for remedying them in the future.
>
> Best,
> Mike
>
>
> On Wed, Dec 19, 2018 at 3:46 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> FYI, site-book is currently broken in master. I think I've found the
>> source of the issue and will be issuing a fix shortly.
>>
>> Best,
>> Mike
>>
>

Re: Site-book broken in master

[Otto Fowler]

Maybe we should consider ascii doc, it shows in github, and can generate
multiple formats.



On December 19, 2018 at 22:49:26, Otto Fowler (ottobackwa...@gmail.com)
wrote:

What would help is an automated test the tests all the links as well.



On December 19, 2018 at 19:55:03, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

I'd also like to submit a call for suggestions for making this process
simpler. I think we've made some excellent progress towards getting an
html-generated version of our documentation, but it's also proving to be
somewhat error-prone. Github and Doxia have different expectations as to
formatting, and most devs are generally looking at Github rendering only
because we get that for free as part of the Github PR lifecycle. In
addition, images are managed independently. Missing a reference, getting
the relative path wrong, or getting the rewrite rules incorrect will result
in a bad time. I've seen at least a few PR's in recent memory unwittingly
fall into this trap.

On Wed, Dec 19, 2018 at 5:47 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Fix is out in this PR: https://github.com/apache/metron/pull/1309
> Jira: https://issues.apache.org/jira/browse/METRON-1950
>
> It ended up being a bit bigger than I hoped because we had some remaining
> incompatibilities between our Github markdown and the Doxia-expected
> formatting. This caused a cascade of inconsistent results for anyone
> attempting to generate a page that works in both Github markdown and
Doxia.
> The latest fixes here should address all of this in the parsers
> documentation. I suspect we may have other markdown pages out there that
> will have similar issues, so this PR/Jira/notice should serve as a recipe
> for remedying them in the future.
>
> Best,
> Mike
>
>
> On Wed, Dec 19, 2018 at 3:46 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> FYI, site-book is currently broken in master. I think I've found the
>> source of the issue and will be issuing a fix shortly.
>>
>> Best,
>> Mike
>>
>

Re: [VOTE] Metron Release Candidate 0.7.0-RC1

[Otto Fowler]

+1 binding
Ran through script


On December 12, 2018 at 14:10:22, Nick Allen (n...@nickallen.org) wrote:

+1 binding

- All of the tarballs, checksums, and signatures are correct
- All of the tests and integration tests ran successfully.
- The release also spun-up correctly in the development environment.


FYI - I had to slightly modify the metron-rc-check script for this to
work. See the patch below.

diff --git a/dev-utilities/release-utils/metron-rc-check
b/dev-utilities/release-utils/metron-rc-check
index 143ba85a2..4552e5568 100755
--- a/dev-utilities/release-utils/metron-rc-check
+++ b/dev-utilities/release-utils/metron-rc-check
@@ -67,8 +67,6 @@ for i in "$@"; do
# --bro=0.1.0
#
-b=*|--bro=*)
- BRO="${i#*=}"
- shift # past argument=value
;;

#
@@ -157,6 +155,7 @@ fi
echo "Working directory $WORK"

KEYS="$METRON_RC_DIST/KEYS"
+KEYS="https://dist.apache.org/repos/dist/release/metron/KEYS;
METRON_ASSEMBLY="$METRON_RC_DIST/apache-metron-$METRON_VERSION-$RC.tar.gz"
METRON_ASSEMBLY_SIG="$METRON_ASSEMBLY.asc"


On Tue, Dec 11, 2018 at 2:43 PM Justin Leet  wrote:

> This is a call to vote on releasing Apache Metron 0.7.0
>
> Full list of changes in this release:
> https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/CHANGES
> The tag to be voted upon is:
> apache-metron-0.7.0-rc1
>
> The source archives being voted upon can be found here:
>
>
https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/apache-metron-0.7.0-rc1.tar.gz
>
> Other release files, signatures and digests can be found here:
> https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/
>
> The release artifacts are signed with the following key:
> https://dist.apache.org/repos/dist/release/metron/KEYS
> Please vote on releasing this package as Apache Metron 0.7.0-RC1
>
> When voting, please list the actions taken to verify the release.
>
> Recommended build validation and verification instructions are posted
> here:
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
> This vote will be open for until 3pm EDT on Friday December 14 2018.
>
> [ ] +1 Release this package as Apache Metron 0.7.0-RC1
>
> [ ] 0 No opinion
>
> [ ] -1 Do not release this package because...
>

Re: [DISCUSS] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Otto Fowler]

+1

We will need jiras and PR’s for updating our scripts post move however.


On December 9, 2018 at 06:32:44, Roy Lenferink (rlenfer...@apache.org)
wrote:

Hi folks,

Checking the mail-archives I noticed the message below missed the dev@metron
list [1].
Does anyone have a problem with starting the process to migrate the
existing Metron
git-wip-us repos [2][3] to gitbox?

This means integrated access and easy PRs on the repos (write access
to the GitHub repos).

I can't imagine anyone will say no, but we need to "document support
for the decision" from a mailing list post, so, here it is.

If there are no objections after 72 hours, I will create a ticket with
INFRA for moving
over the metron repositories to GitBox.

- Roy

[1] http://mail-archives.apache.org/mod_mbox/metron-dev/201812.mbox/browser
[2] https://git-wip-us.apache.org/repos/asf?p=metron.git
[3] https://git-wip-us.apache.org/repos/asf?p=metron-bro-plugin-kafka.git

-- Forwarded message -
From: Daniel Gruno 
Date: vr 7 dec. 2018 om 17:53
Subject: [NOTICE] Mandatory relocation of Apache git repositories on
git-wip-us.apache.org
To: us...@infra.apache.org 


[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]

Hello Apache projects,

I am writing to you because you may have git repositories on the
git-wip-us server, which is slated to be decommissioned in the coming
months. All repositories will be moved to the new gitbox service which
includes direct write access on github as well as the standard ASF
commit access via gitbox.apache.org.

## Why this move? ##
The move comes as a result of retiring the git-wip service, as the
hardware it runs on is longing for retirement. In lieu of this, we
have decided to consolidate the two services (git-wip and gitbox), to
ease the management of our repository systems and future-proof the
underlying hardware. The move is fully automated, and ideally, nothing
will change in your workflow other than added features and access to
GitHub.

## Timeframe for relocation ##
Initially, we are asking that projects voluntarily request to move
their repositories to gitbox, hence this email. The voluntary
timeframe is between now and January 9th 2019, during which projects
are free to either move over to gitbox or stay put on git-wip. After
this phase, we will be requiring the remaining projects to move within
one month, after which we will move the remaining projects over.

To have your project moved in this initial phase, you will need:

- Consensus in the project (documented via the mailing list)
- File a JIRA ticket with INFRA to voluntarily move your project repos
over to gitbox (as stated, this is highly automated and will take
between a minute and an hour, depending on the size and number of
your repositories)

To sum up the preliminary timeline;

- December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
relocation
- January 9th -> February 6th: Mandated (coordinated) relocation
- February 7th: All remaining repositories are mass migrated.

This timeline may change to accommodate various scenarios.

## Using GitHub with ASF repositories ##
When your project has moved, you are free to use either the ASF
repository system (gitbox.apache.org) OR GitHub for your development
and code pushes. To be able to use GitHub, please follow the primer
at: https://reference.apache.org/committer/github


We appreciate your understanding of this issue, and hope that your
project can coordinate voluntarily moving your repositories in a
timely manner.

All settings, such as commit mail targets, issue linking, PR
notification schemes etc will automatically be migrated to gitbox as
well.

With regards, Daniel on behalf of ASF Infra.

PS:For inquiries, please reply to us...@infra.apache.org, not your
project's dev list :-).

Re: [DISCUSS] Recurrent Large Indexing Error Messages

[Otto Fowler]

Why not have a second indexing topology configured just for errors?
We can load the same code with two different configurations in two
topologies.



On December 5, 2018 at 03:55:59, Ali Nazemian (alinazem...@gmail.com) wrote:

I think if you look at the indexing error management, it is pretty much
similar to parser and enrichment error use cases. It is even more common to
expect something ended up in error topics. I think a wider independent job
can be used to take care of error management. It can be decided to add a
separate topology later on to manage error logs and create
alert/notifications separately.
It can be even integrated with log feeder and log search.
The scenario of sending solution operational logs to the same solution is a
bit weird and not enterprise friendly. Normally platform operation team
would be a separate team with different objectives and probably they have
got a separate monitoring/notification solution in placed already.

I don't think it is the end of the world if this part is left to be managed
by users. So I prefer option 2 as a short term. Long term solution can be
discussed separately.

Cheers,
Ali

On Sat, 20 Oct. 2018, 05:20 Nick Allen  I want to discuss solutions for the problem that I have described in
> METRON-1832; Recurrent Large Indexing Error Messages. I feel this is a
very
> easy trap to fall into when using the default settings that currently
come
> with Metron.
>
>
> ## Problem
>
>
> https://issues.apache.org/jira/browse/METRON-1832
>
>
> If any index destination like HDFS, Elasticsearch, or Solr goes down
while
> the Indexing topology is running, an error message is created and sent
back
> to the user-defined error topic. By default, this is defined to also be
> the 'indexing' topic.
>
> The Indexing topology then consumes this error message and attempts to
> write it again. If the index destination is still down, another error
> occurs and another error message is created that encapsulates the
original
> error message. That message is then sent to the 'indexing' topic, which
is
> later consumed, yet again, by the Indexing topology.
>
> These error messages will continue to be recycled and grow larger and
> larger as each new error message encapsulates all previous error messages
> in the "raw_message" field.
>
> Once the index destination recovers, one giant error message will finally
> be written that contains massively duplicated, useless information which
> can further negatively impact performance of the index destination.
>
> Also, the escape character '\' continually compounds one another leading
to
> long strings of '\' characters in the error message.
>
>
> ## Background
>
> There was some discussion on how to handle this on the original PR #453
> https://github.com/apache/metron/pull/453.
>
> ## Solutions
>
> (1) The first, easiest option is to just do nothing. There was already a
> discussion around this and this is the solution that we landed on in
#453.
>
> Pros: Really easy; do nothing.
>
> Cons: Intermittent problems with ES/Solr can easily create very large
error
> messages that can significantly impact both search and ingest
performance.
>
>
> (2) Change the default indexing error topic to 'indexing_errors' to avoid
> recycling error messages. Nothing will consume from the 'indexing_errors'
> topic, thus preventing a cycle.
>
> Pros: Simple, easy change that prevents the cycle.
>
> Cons: Recoverable indexing errors are not visible to users as they will
> never be indexed in ES/Solr.
>
> (2) Add logic to limit the number times a message can be 'recycled'
through
> the Indexing topology. This effectively sets a maximum number of retry
> attempts. If a message fails N times, then write the message to a
separate
> unrecoverable, error topic.
>
> Pros: Recoverable errors are visible to users in ES/Solr.
>
> Cons: More complex. Users still need to check the unrecoverable, error
> topic for potential problems.
>
> (4) Do not further encapsulate error messages in the 'raw_message' field.
> If an error message fails, don't encapsulate it in another error message.
> Just push it to the error topic as-is. Could add a field that indicates
> how many times the message has failed.
>
> Pros: Prevents giant error messages from being created from recoverable
> errors.
>
> Cons: Extended outages would still cause the Indexing topology to
> repeatedly recycle these error messages, which would ultimately exhaust
> resources in Storm.
>
>
>
> What other ways can we solve this?
>

Re: [DISCUSS] Managing intermittent test failures

[Otto Fowler]

+1


On November 29, 2018 at 10:26:53, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Every now and then we see intermittent test failures, and rather than
sweeping them under the rug, we should have a simple method to track and
handle them. I started creating Jiras for tests that I've seen fail, but
that don't fail consistently, or even fail more than once. For example,
https://issues.apache.org/jira/browse/METRON-1851.

I think we're all taking steps to varying degrees already, but I want to
call it out formally. I propose we create a ticket and add the label
"test-failure." It might also make sense to send a quick note to the dev
list or Slack channel, so attention can be brought to it and anyone else
that may have run into an issue with the test can chime in. We can clean
them out every few months - maybe do a review going into a release and
close any that have not been reproduced for some time. What do you all
think?

Mike

Re: Unzipping Cypress

[Otto Fowler]

OK,

I think what is happening is that in my PR, I’m building metron in Docker
and deploying to vagrant.  I have updated my PR to map the cypress cache
into the Docker container.
Thanks!


On November 28, 2018 at 10:29:25, Shane Ardell (shane.m.ard...@gmail.com)
wrote:

For me, it's at ~/Library/Caches/Cypress, but the path depends on your OS:
https://docs.cypress.io/guides/getting-started/installing-cypress.html#Binary-cache

On Wed, Nov 28, 2018 at 4:19 PM Otto Fowler 
wrote:

> Where is the cache path?
>
>
> On November 28, 2018 at 09:34:18, Shane Ardell (shane.m.ard...@gmail.com)
> wrote:
>
> https://github.com/cypress-io/cypress/issues/1813
>

Re: Unzipping Cypress

[Otto Fowler]

Where is the cache path?


On November 28, 2018 at 09:34:18, Shane Ardell (shane.m.ard...@gmail.com)
wrote:

https://github.com/cypress-io/cypress/issues/1813

Re: Unzipping Cypress

[Otto Fowler]

I’m sorry for the confusion, but I see this locally.  I’m not looking at
travis.
I have been doing a lot of full dev deployments.



On November 27, 2018 at 11:40:42, Shane Ardell (shane.m.ard...@gmail.com)
wrote:

Otto,

Do you have a Travis log you can share that shows Cypress downloaded vs.
using a cached version? It looks like my latest merge to master uses a
cached version: https://travis-ci.org/apache/metron/jobs/458440883#L7292.

Thanks in advance.

On Mon, Nov 26, 2018 at 6:14 PM Shane Ardell 
wrote:

> It seems we can pretty easily configure the .travis.yml config file to
> cache our npm modules:
>
https://docs.cypress.io/guides/guides/continuous-integration.html#Caching-the-Cypress-binary
>
> It also looks like we are already trying to cache our npm modules in the
> Travis config, but, obviously, it's not working as intended. I can take a
> look into why tomorrow.
>
> On Mon, Nov 26, 2018 at 5:33 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Shane, Tibor - Can you guys chime in on this?
>>
>> On Mon, Nov 26, 2018 at 9:13 AM Otto Fowler 
>> wrote:
>>
>> > Isn’t there a way we can cache it?
>> >
>> >
>> > On November 26, 2018 at 10:59:20, Nick Allen (n...@nickallen.org)
>> wrote:
>> >
>> > Yes, I have noticed that too. If not a way to reduce the time, we
should
>> > not be logging the unzipping process percentile-by-percentile in the
>> Travis
>> > CI builds.
>> >
>> > On Sat, Nov 24, 2018 at 9:49 AM Otto Fowler 
>> > wrote:
>> >
>> > > Anyone else seeing a lot of time taken downloading and unzipping
>> Cypress
>> > on
>> > > builds?
>> > > What is up with that?™
>> > >
>> > > ottO
>> > >
>> >
>>
>

Re: Unzipping Cypress

[Otto Fowler]

Isn’t there a way we can cache it?


On November 26, 2018 at 10:59:20, Nick Allen (n...@nickallen.org) wrote:

Yes, I have noticed that too. If not a way to reduce the time, we should
not be logging the unzipping process percentile-by-percentile in the Travis
CI builds.

On Sat, Nov 24, 2018 at 9:49 AM Otto Fowler 
wrote:

> Anyone else seeing a lot of time taken downloading and unzipping Cypress
on
> builds?
> What is up with that?™
>
> ottO
>

Unzipping Cypress

[Otto Fowler]

Anyone else seeing a lot of time taken downloading and unzipping Cypress on
builds?
What is up with that?™

ottO

Re: [DISCUSS] Add ngrx to handle state management in Angular

[Otto Fowler]

Can you describe what you mean by “state” in a little more detail?  Not a
complete description, maybe just a crib list.


On November 22, 2018 at 07:21:43, Shane Ardell (shane.m.ard...@gmail.com)
wrote:

As both the Management and Alerts UI grow in size, managing application
state continues to become more and more complex. To help us deal with
managing all of this state and ensuring our application derives state from
a single source of truth, I suggest we start using NgRx, a state management
library based on the Redux pattern but built for Angular. It is by far the
most popular library of this type for Angular. As you can see in the
project's GitHub Insights tab ,
it's quite actively worked on and releases are pretty frequent. The project
is licensed under MIT.

As far as an approach to integration, I don't think we necessarily need a
big refactoring right off the bat. I feel something like this can be done
in a piecemeal approach over time. I think we can start by introducing it
into the project the next time we have a new application feature.

What are everyone's thoughts around this?

Cheers,
Shane

Re: [DISCUSS] Deprecating MySQL

[Otto Fowler]

I would like to understand the work required to move our JDBC support ( or
adapt the current support to the abstraction ) to /contrib.
We could default and only officially support LDAP, but have the /contrib (
or /extension_examples ) have a “this is how you would support jdbc for
auth “ project.



On November 15, 2018 at 15:01:10, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Yes, makes sense. +1 to that.

On Thu, Nov 15, 2018 at 12:54 PM James Sirota  wrote:

> To clarify my position, I don't have a problem with mySql or any other
> projects relying on it. mySql in itself is not an issue. What I don't
> want is for a customer to be presented with an option to chose and
> configure two options for authenticating the UI, which I think is
> needless. It adds complexity for not much value. Since LDAP is clearly
> the better way to go that should be what we support without explicitly
> giving a user an option to switch to JDBC. A user can still do so by
> extending our abstractions if that is what they chose to do, but this
would
> not be officially supported by us. We would not be providing a config or
> an mPack to do this. A user would have to do it on their own.
>
> James
>
>
>
> 15.11.2018, 12:15, "Michael Miklavcic" :
> > Incidentally, even without the Metron piece in the picture, what is the
> > answer for Ambari's database dependency? Which uses a SQL data store.
> Does
> > this actually solve the problem of "customers won't install Metron bc
SQL
> > store?" or are there other issues we need to address?
> >
> > On Thu, Nov 15, 2018 at 9:30 AM James Sirota 
wrote:
> >
> >> Hi Guys,
> >>
> >> My opinion on this, as is with Knox SSO, is that the code should be
> >> pluggable to support JDBC, but we should not continue to support the
> >> concrete implementation and expose it to users via a setting. This is
a
> >> fairly minor feature and the added complexity of supporting switching
> >> between JDBC and LDAP is simply not worth it. We need to strike a
> balance
> >> between ease of use and capabilities/extensibility. For features that
> are
> >> worth it such as with analytics and stream processing, the extra
> capability
> >> is worth the added complexity in configuration. But for this, it is
> not.
> >> So let's keep JDBC around for a release to allow users to migrate to
> LDAP,
> >> deprecate it, and move on.
> >>
> >> Thanks,
> >> James
> >>
> >> 13.11.2018, 16:03, "Simon Elliston Ball"  >:
> >> > We went over the hbase user settings thing on extensive discussions
> at
> >> the time. Storing an arbitrary blob of JSON which is only ever
> accessed by
> >> a single key (username) was concluded to be a key value problem, not a
> >> relational problem. Hbase was concluded to be massive overkill as a
key
> >> value store in this usecase, unless it was already there and ready to
> go,
> >> which in the case of Metron, it is, for enrichments, threat intel and
> >> profiles. Hence it ended up in Hbase, as a conveniently present data
> store
> >> that matched the usage patterns. See
> >>
>
https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
> >> and METRON-1337 for discussion.
> >> >
> >> > Simon
> >> >
> >> >> On 13 Nov 2018, at 18:50, Michael Miklavcic <
> >> michael.miklav...@gmail.com> wrote:
> >> >>
> >> >> Thanks for the write up Simon. I don't think I see any major
> problems
> >> with
> >> >> deprecating the general sql store. However, just to clarify, Metron
> >> does
> >> >> NOT require any specific backing store. It's 100% JPA, which means
> >> anything
> >> >> that can be configured with the Spring properties we expose. I
think
> >> the
> >> >> most opinionated thing we do there is ship an extremely basic table
> >> >> creation script for h2 and mysql as a simple example for schema. As
> an
> >> >> example, we simply use H2 in full dev, which is entirely in-memory
> and
> >> spun
> >> >> up automatically from configuration. The recent work by Justin Leet
> >> removes
> >> >> the need to use a SQL store at all if you choose LDAP -
> >> >> https://github.com/apache/metron/pull/1246. I'll let him comment
> >> further on
> >> >> this, but I think there is one small change that could be made via
a
> >> toggle
> >> >> in Ambari that would even eliminate the user from seeing JDBC
> settings
> >> >> altogether during install if they choose LDAP. Again, I think I'm
on
> >> board
> >> >> with deprecating the SQL backing store as I pointed this out on the
> >> Knox
> >> >> thread as well, but I just wanted to make sure everyone has an
> accurate
> >> >> picture of the current state.
> >> >>
> >> >> I had to double check on the HBase config you mentioned, but it
does
> >> appear
> >> >> that we use it for the Alerts UI. I don't think I realized we were
> >> storing
> >> >> config there instead of the Zookeeper store we use for other system
> >> >> configuration. Ironically enough, I think that it probably makes
> more
> >> 

Re: [DISCUSS] Knox SSO feature branch review and features

[Otto Fowler]

That does sound good Simon, I think I miss understood that the default LDAP
was standard with KNOX/ambari and not something we would be doing ourselves.


On November 16, 2018 at 10:54:48, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

I think there is a lot to be said for defaulting to Knox on... we also that
way get some 'secure by default' - at least ssl by default. The do-nothing
I think you're proposing would be around the authentication, right? Knox
does ship with a demo LDAP server we could have some defaults (kinda like
we do with the dev spring profile today) which might be a good way of
achieving a similar effect, and that's configured by default by Ambari.
Would that meet the need, or do you think we should provide a "yeah I'm
sure we don't need to authenticate that connection, let them in" identity
provider for Knox? That way we would have to have them impersonate a known
user for the REST api to work, but you would get the seemless, no auth
access.

To be honest, I'm a fan of the first option where we give people a nice
simple, no external config and just use some sensible default users in the
demo LDAP instance that Knox owns, and then default to Knox on to give us
our nice single access point.

Simon


On Fri, 16 Nov 2018 at 15:35, Otto Fowler  wrote:

> Those are all valid points. I think it is ( was ) worth discussion at
> lease a little.
>
> WRT Knox and defaults:
>
> I have in the past used “do-nothing” implementations as default
> placeholders for functionality
> that needed extensive per customer configuration, or configuration
outside
> the responsibility of the product.
>
> Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX
> configuration with “do-nothing” providers
> for auth etc. The users would then configure the providers ( based on the
> provider(s) we support ) at a later time.
>
> We could write the providers, as everyone has pointed out how extensible
> KNOX is ;)
>
> Would that be a valid way to simplify the issue?
> What would the fallout of that be?
>
>
>
> On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> Most of the research I've done around adding Metron as a Knox service is
> based on how other projects do it. The documentation is not easy to
follow
> so I learned by reading other service definition files. The assumption
> that we are doing things drastically different is false.
>
> I completely agree with Simon. Why would we want to be dependent on
Knox's
> release cycle? How does that benefit us? It may reduce some operational
> complexity but it makes our install process more complicated because we
> require a certain version of Knox (who knows when that gets released).
> What do we do in the meantime? I would also like to point out that Metron
> is inherently different than other Hadoop stack services. We are a
> full-blown application with multiple UIs so the way we expose services
> through Knox may be a little different.
>
> I think this will be easier to discuss when we can all see what is
actually
> involved. I am working on a PR that adds Metron as a Knox service and
will
> have that out soon. That should give everyone more context.
>
> On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > You could say the same thing about Ambari, but that provides mpacks.
Knox
> > is also designed to be extensible through Knox service stacks since
they
> > realized they can’t support every project. The challenge is that the
docs
> > have not made it as easy as they could for the ecosystem to plug into
> Knox,
> > which has led to some confusion around this being a recommended pattern
> > (which it is).
> >
> > The danger of trying to get your bits into Knox is that that ties you
to
> > their release cycle (a problem Ambari has felt hard, hence their
> community
> > is moving away from the everything inside model towards everything is
an
> > mpack).
> >
> > A number of implementations of Knox also use the approach Ryan is
> > suggesting for their own organization specific end points, so it’s not
> like
> > this is an uncommon, or anti-pattern, it’s more the way Knox is
designed
> to
> > work in the future, than the legacy of it only being able to handle a
> > subset of Hadoop projects.
> >
> > Knox remains optional In our scenario, but we keep control over the
> > shipping of things like rewrite rules, which allows Metron to control
its
> > release destiny should things like url patterns in the ui need to
change
> > (with a new release of angular / new module / new rest endpoint etc)
> > instead of making a Metron release dependent on a Knox release.
> >
>

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

[Otto Fowler]

Maybe if we can’t document a concrete solution, we can document or codify a
procedure.   Should for example the owner of the feature branch always work
with the release manager when there are issues?


On November 16, 2018 at 10:26:11, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

It's a good suggestion, and I've been thinking about how to best handle
this. Honestly, the right answer might be to do git rebase on master from
the PR branch rather than a merge. That might avoid this situation
altogether. Of course, that also comes with all the obligatory warnings
about rebasing publicly shared branches, and that anyone how has their own
copy will get conflicts. But I think that risk is probably minimal. I'll
put something together that covers both options along with the problems and
solutions for each. Hopefully that will make future collaboration that fits
this pattern easier for people.

On Fri, Nov 16, 2018, 5:42 AM Otto Fowler 
> Maybe this is worth a confluence entry, not as a guide, but just to
> document what you did.
>
> On November 15, 2018 at 19:07:40, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Ok, this is finally merged! Whew!
>
> Here's how I polished up the history at the end. I used other feature
> branch merges as a guideline around commit messaging.
>
> *   fcd644ca 2018-11-15 | METRON-1834: Migrate Elasticsearch from
> TransportClient to new Java REST API (mmiklavc via mmiklavc) (HEAD ->
> master, origin/master, origin/HEAD, master-merge) [mmiklavc]
> |\
> | * 8bf3b6ec 2018-11-15 | METRON-1834: Migrate Elasticsearch from
> TransportClient to new Java REST API (mmiklavc via mmiklavc) closes
> apache/metron#1242 (stella-es-base2) [mmiklavc]
> | * e7e19fbb 2018-10-08 | METRON-1834: Migrate Elasticsearch from
> TransportClient to new Java REST API (cstella via mmiklavc) [cstella]
> * | 0c4c622b 2018-11-14 | METRON-1749 Update Angular to latest release in
> Management UI (sardell via nickwallen) closes apache/metron#1217 [sardell]
>
> On Thu, Nov 15, 2018 at 4:29 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Absolutely, that's part of what I did to validate. This output below also
>> exactly matches the diff I get when I run it from the raw PR branch.
>>
>> git diff master --stat
>>  Upgrading.md
>>   |   7 +++
>>  dependencies_with_url.csv
>>  |   2 +
>>  metron-deployment/Kerberos-manual-setup.md
>>   | 154
>> ++---
>>  
>> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
>>   |   9 
>>  
>> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
>>  |   2 -
>>  
>> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
>> |   3 +-
>>  
>> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
>>   |  10 
>>  
>> metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/MetaAlertServiceImpl.java
>> |   2 +-
>>  metron-platform/elasticsearch-shaded/pom.xml
>>   |  47
>> +++-
>>  
>> metron-platform/elasticsearch-shaded/src/main/resources/META-INF/log4j-provider.properties
>>   |  18 ---
>>  metron-platform/metron-common/README.md
>>  |  48
>> +
>>  metron-platform/metron-common/src/main/config/zookeeper/global.json
>>  |   1 -
>>  
>> metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigOption.java
>> |   7 +++
>>  metron-platform/metron-elasticsearch/README.md
>>   |  45
>> +++-
>>  metron-platform/metron-elasticsearch/pom.xml
>>   |  32
>> +--
>>  
>> metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java
>>   

Re: [DISCUSS] Knox SSO feature branch review and features

[Otto Fowler]

Those are all valid points.  I think it is ( was ) worth discussion at
lease a little.

WRT Knox and defaults:

I have in the past used “do-nothing” implementations as default
placeholders for functionality
that needed extensive per customer configuration, or configuration outside
the responsibility of the product.

Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX
configuration with “do-nothing” providers
for auth etc.  The users would then configure the providers ( based on the
provider(s) we support ) at a later time.

We could write the providers, as everyone has pointed out how extensible
KNOX is ;)

Would that be a valid way to simplify the issue?
What would the fallout of that be?



On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com) wrote:

Most of the research I've done around adding Metron as a Knox service is
based on how other projects do it. The documentation is not easy to follow
so I learned by reading other service definition files. The assumption
that we are doing things drastically different is false.

I completely agree with Simon. Why would we want to be dependent on Knox's
release cycle? How does that benefit us? It may reduce some operational
complexity but it makes our install process more complicated because we
require a certain version of Knox (who knows when that gets released).
What do we do in the meantime? I would also like to point out that Metron
is inherently different than other Hadoop stack services. We are a
full-blown application with multiple UIs so the way we expose services
through Knox may be a little different.

I think this will be easier to discuss when we can all see what is actually
involved. I am working on a PR that adds Metron as a Knox service and will
have that out soon. That should give everyone more context.

On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> You could say the same thing about Ambari, but that provides mpacks. Knox
> is also designed to be extensible through Knox service stacks since they
> realized they can’t support every project. The challenge is that the docs
> have not made it as easy as they could for the ecosystem to plug into
Knox,
> which has led to some confusion around this being a recommended pattern
> (which it is).
>
> The danger of trying to get your bits into Knox is that that ties you to
> their release cycle (a problem Ambari has felt hard, hence their
community
> is moving away from the everything inside model towards everything is an
> mpack).
>
> A number of implementations of Knox also use the approach Ryan is
> suggesting for their own organization specific end points, so it’s not
like
> this is an uncommon, or anti-pattern, it’s more the way Knox is designed
to
> work in the future, than the legacy of it only being able to handle a
> subset of Hadoop projects.
>
> Knox remains optional In our scenario, but we keep control over the
> shipping of things like rewrite rules, which allows Metron to control its
> release destiny should things like url patterns in the ui need to change
> (with a new release of angular / new module / new rest endpoint etc)
> instead of making a Metron release dependent on a Knox release.
>
> Imagine how we would have done with the Ambari side if we’d had to wait
> for them to release every time we needed to change something in the
> mpack... we don’t want that happening with Knox.
>
> Simon
>
> > On 16 Nov 2018, at 13:22, Otto Fowler  wrote:
> >
> >
>
https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support
> >
> > Solr is angular for example.
> >
> >
> > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Ok, here is something I don’t understand, but would like to.
> >
> > Knox comes configured with build in services for a number of other
apache
> > products and UI’s.
> > It would seem to me, that the best integration with Knox would be to do
> > what these other products have done.
> >
> >
> > 1. Do whatever you have to do to make your own stuff compatible.
> > 2. Create a knox service definition and provide it or try to get it
into
> > knox itself
> >
> > This would make the knox integration with metron optional and pluggable
> > wouldn’t it?
> >
> > Then knox with metron would just be the same as knox with anything
else.
> > Please help me if I am wrong, but we seem to be going our own way here.
> > Why don’t we just do what these other products have done?
> > Why don’t we try to get apache metron services accepted to the knox
> > project? Why don’t we model our knox integration with how XYZ does it?
> > Have we looked at how others integrate? H

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Otto Fowler]

Can you generate the jiras that would be included in the release?


On November 16, 2018 at 10:05:50, Justin Leet (justinjl...@gmail.com) wrote:

Given that we've had a couple major PRs (the ES client migration along with
the Angular upgrade stuff, and I'm sure others), I'd be in favor of
releasing both the plugin and the main repo.

I'd be in favor of doing something like:
metron-bro-plugin-kafka release 0.3.0
PR to update full dev
metron release 0.7.0 (given that things changed a good amount)

I would also expect this process to most likely be kicked off post
Thanksgiving break (Thursday 22nd and Friday 23rd for anybody not in the
US).

I can kick out another summarization thread if we want, but basically:
* Nov 26 - Start the Bro plugin release process
* Once that finishes, PR to update full dev with new plugin version
* Once PR is in, start metron release process (hopefully) sometime the week
of the 3rd?

Are there any objections to staggering the releases like that? They could
also be done together, but it means that we have to update full dev to
match the plugin version post release.

On Wed, Nov 14, 2018 at 10:29 AM zeo...@gmail.com  wrote:

> In my opinion metron-bro-plugin-kafka is ready for a release. Anything
> else people would want to see? Once it gets released, I would like to
> update full dev to use the newest version prior to any future metron
> release (0.6.1 or whatever we choose).
>
> Jon
>
> On Wed, Nov 7, 2018 at 8:07 PM zeo...@gmail.com  wrote:
>
> > So, about this release, anybody have time to review
> > apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13?
> >
> > Jon
> >
> > On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> And I do think we will be ready to roll another Metron release in the
> near
> >> future as well.
> >>
> >> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet 
> >> wrote:
> >>
> >> > I tend to agree with Mike. I think we could do a release of the main
> >> > project and get benefit, but I think skipping this cycle (especially
> >> since
> >> > we had a release last month) let's us get things like the ES client
> >> > migration in and settled and just generally make recommending a
newer
> >> > version easier.
> >> >
> >> > The main reason for doing a release for the Bro plugin is to address
> >> > shortcomings that we discovered and are addressing.
> >> >
> >> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic <
> >> > michael.miklav...@gmail.com> wrote:
> >> >
> >> > > Migrating the ES client, refactoring parser bolt. There are some
> >> > interface
> >> > > changes in flight right now that I think would be beneficial to
see
> in
> >> > the
> >> > > next release.
> >> > >
> >> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen 
> >> wrote:
> >> > >
> >> > > > I am in favor of a release for both.
> >> > > >
> >> > > > There are a lot of really useful bug fixes, management of pcap
> >> through
> >> > > > Ambari, more flexibility for configuring JAAS in Ambari,
increased
> >> > > > Elasticsearch performance, the Syslog parser, and the Batch
> >> Profiler,
> >> > > among
> >> > > > others. I would be happy with calling it a 0.6.1 point release.
> >> > > >
> >> > > > Mike - What is outstanding that you would like to see in the
> >> release?
> >> > > >
> >> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic <
> >> > > > michael.miklav...@gmail.com> wrote:
> >> > > >
> >> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin
> release.
> >> It
> >> > > > seems
> >> > > > > like it's ready to go, and I think there are a few more things
> I'd
> >> > like
> >> > > > to
> >> > > > > see get into our next Metron release so I'm good with holding
> off
> >> > > there.
> >> > > > >
> >> > > > > Mike
> >> > > > >
> >> > > > > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet <
> >> justinjl...@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > > As you might recall from a prior discussion about release
> >> cadence,
> >> > we
> >> > > > > were
> >> > > > > > interested in initiating release threads near our board
> reports
> >> to
> >> > > see
> >> > > > if
> >> > > > > > we wanted to do releases or not. Additionally, the work is
> done
> >> to
> >> > do
> >> > > > two
> >> > > > > > separate releases, so our options are releasing both, a
single
> >> one,
> >> > > or
> >> > > > > > neither.
> >> > > > > >
> >> > > > > > Having said that, a metron-bro-kafka-plugin 0.3.0 release
came
> >> up
> >> > on
> >> > > > this
> >> > > > > > thread
> >> > > > > > <
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E
> >> > > > > > >.
> >> > > > > > In particular, the prospect of a release came up in the
> context
> >> of
> >> > > > > having a
> >> > > > > > version with better (and working) testing.
> >> > > > > >
> >> > > > > > Version Number
> 

Re: [DISCUSS] Knox SSO feature branch review and features

[Otto Fowler]

https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support

Solr is angular for example.


On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com)
wrote:

Ok,  here is something I don’t understand, but would like to.

Knox comes configured with build in services for a number of other apache
products and UI’s.
It would seem to me, that the best integration with Knox would be to do
what these other products have done.


1. Do whatever you have to do to make your own stuff compatible.
2. Create a knox service definition and provide it or try to get it into
knox itself

This would make the knox integration with metron optional and pluggable
wouldn’t it?

Then knox with metron would just be the same as knox with anything else.
Please help me if I am wrong, but we seem to be going our own way here.
Why don’t we just do what these other products have done?
Why don’t we try to get apache metron services accepted to the knox
project?  Why don’t we model our knox integration with how XYZ does it?
Have we looked at how others integrate?   Having all the code and being
able to track stuff is kind of the point of this whole thing isn’t it?

Maybe this is implied and I’m missing it, if so I apologize.

I think consistency with the rest of the hadoop stack with knox helps us.



On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) wrote:

1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI
as far as I know. I put up a PR with a proposed solution here:
https://github.com/apache/metron/pull/1266.
2) Yes Knox is a service you can install with Ambari, similar to Ranger or
Spark. There are some things that are specifically configured in Knox and
there are some things specific to Metron. I will put up a PR with the
changes needed so you can see exactly what is involved.
3) I don't understand what you mean here. Is this a question?
4) I think it's a little early to predict the Ambari changes required.
This will depend on how tasks 1-3 go. I imagine it's similar to other
mpack work: expose some parameters in ambari and bind those to config
files. My understanding from this thread so far is that we should focus on
a manual, documented approach to start.

On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Thanks Ryan,
>
> 1) Can you clarify "not a good way to do this?" Are you saying we don't
> have a way to set this and need to add the config option, or that a
> solution is not obvious and it's unclear what to do? It seems to me you're
> saying the former, but I'd like to be sure.
> 2) Is Knox not a service made available by Ambari similar to Ranger or
> Spark? I'm assuming that similar to Kerberos, there are some things that
> are specifically configured in Knox and others that are app-specific. Some
> explanation of what this looks like would be helpful.
> 3) Sounds like this follows pretty naturally from #1
> 4) Relates to #2. I think we need some guidance on what a manual vs
> MPack/automated install would look like.
>
> Cheers,
> Mike
>
>
> On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman  wrote:
>
> > Wanted to give an update on the context path issue. I investigated
> > rewriting url links in the outgoing static assets with Knox and it was
> not
> > trivial. Fortunately I found a simple solution that works with or
> without
> > Knox. I changed the base tag in index.html from  to  > href="./">, or in other words made the base href relative.
> >
> > I believe I am at the point where I can task this out and provide a high
> > level overview of the changes needed. I think that each task will be a
> > manageable size and can stand alone so I don't think we need a feature
> > branch.
> >
> > The first task involves a general change to the UI code. We need a way
> to
> > set the path to the REST service with a configuration setting because it
> is
> > different with and without Knox. Currently there is not a good way to do
> > this in the UI. We can use the environment files but that is a build
> time
> > setting and is not flexible. I can see this capability being useful for
> > other use cases in the future. I think we could even split this up into
> 2
> > separate tasks, one for the alerts UI and one for the management UI.
> >
> > The second task involves adding Knox to our stack either by default as a
> > dependency in the mpack or with a documented approach. We would add our
> > REST service, Alerts UI, and Management UI as services in Knox.
> Everything
> > would continue to function as it currently does but with all
> communication
> > going through Knox. LDAP authentication would be required when using
> Knox
> > and Knox wi

Re: [DISCUSS] Knox SSO feature branch review and features

[Otto Fowler]

Ok,  here is something I don’t understand, but would like to.

Knox comes configured with build in services for a number of other apache
products and UI’s.
It would seem to me, that the best integration with Knox would be to do
what these other products have done.


1. Do whatever you have to do to make your own stuff compatible.
2. Create a knox service definition and provide it or try to get it into
knox itself

This would make the knox integration with metron optional and pluggable
wouldn’t it?

Then knox with metron would just be the same as knox with anything else.
Please help me if I am wrong, but we seem to be going our own way here.
Why don’t we just do what these other products have done?
Why don’t we try to get apache metron services accepted to the knox
project?  Why don’t we model our knox integration with how XYZ does it?
Have we looked at how others integrate?   Having all the code and being
able to track stuff is kind of the point of this whole thing isn’t it?

Maybe this is implied and I’m missing it, if so I apologize.

I think consistency with the rest of the hadoop stack with knox helps us.



On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) wrote:

1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI
as far as I know. I put up a PR with a proposed solution here:
https://github.com/apache/metron/pull/1266.
2) Yes Knox is a service you can install with Ambari, similar to Ranger or
Spark. There are some things that are specifically configured in Knox and
there are some things specific to Metron. I will put up a PR with the
changes needed so you can see exactly what is involved.
3) I don't understand what you mean here. Is this a question?
4) I think it's a little early to predict the Ambari changes required.
This will depend on how tasks 1-3 go. I imagine it's similar to other
mpack work: expose some parameters in ambari and bind those to config
files. My understanding from this thread so far is that we should focus on
a manual, documented approach to start.

On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Thanks Ryan,
>
> 1) Can you clarify "not a good way to do this?" Are you saying we don't
> have a way to set this and need to add the config option, or that a
> solution is not obvious and it's unclear what to do? It seems to me
you're
> saying the former, but I'd like to be sure.
> 2) Is Knox not a service made available by Ambari similar to Ranger or
> Spark? I'm assuming that similar to Kerberos, there are some things that
> are specifically configured in Knox and others that are app-specific.
Some
> explanation of what this looks like would be helpful.
> 3) Sounds like this follows pretty naturally from #1
> 4) Relates to #2. I think we need some guidance on what a manual vs
> MPack/automated install would look like.
>
> Cheers,
> Mike
>
>
> On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman 
wrote:
>
> > Wanted to give an update on the context path issue. I investigated
> > rewriting url links in the outgoing static assets with Knox and it was
> not
> > trivial. Fortunately I found a simple solution that works with or
> without
> > Knox. I changed the base tag in index.html from  to
 > href="./">, or in other words made the base href relative.
> >
> > I believe I am at the point where I can task this out and provide a
high
> > level overview of the changes needed. I think that each task will be a
> > manageable size and can stand alone so I don't think we need a feature
> > branch.
> >
> > The first task involves a general change to the UI code. We need a way
> to
> > set the path to the REST service with a configuration setting because
it
> is
> > different with and without Knox. Currently there is not a good way to
do
> > this in the UI. We can use the environment files but that is a build
> time
> > setting and is not flexible. I can see this capability being useful for
> > other use cases in the future. I think we could even split this up into
> 2
> > separate tasks, one for the alerts UI and one for the management UI.
> >
> > The second task involves adding Knox to our stack either by default as
a
> > dependency in the mpack or with a documented approach. We would add our
> > REST service, Alerts UI, and Management UI as services in Knox.
> Everything
> > would continue to function as it currently does but with all
> communication
> > going through Knox. LDAP authentication would be required when using
> Knox
> > and Knox will authenticate with the REST service by passing along an
> > Authorization header. Enabling Knox would be a manual process that
> > involves deploying assets (Knox descriptor files) and changing
> > configuration. There would be no change to how the UI functions by
> default
> > (without Knox) and either LDAP or JDBC authentication could still be
> used..
> >
> > The third task involves enabling SSO with Knox. We would update the
REST
> > service so that it can authenticate 

Re: [DISCUSS] Knox SSO feature branch review and features

[Otto Fowler]

Welcome and thanks!


On November 12, 2018 at 10:12:44, Sandeep Moré (moresand...@gmail.com)
wrote:

Hello Ryan,

I am still catching up on the architecture so let me know if I am
misunderstanding anything.
You could have multiple serviced deployed in Knox
1. for metron (metron/api/v1)
2. for alerts-ui (metron-alerts-ui/alerts-list)
and have them run in one Knox instance and you could have one service
reference from other (not recommended but possible).

You can tailor rewrite rules to update the context path for the assets as
well, pointed out by Simon.

Knox Wiki [1] has some blogs and tutorials that you can look at. This [2]
is a good tutorial on how rewriting static assets, there is also a blog [3]
on basics of rewrite rules that should be a good reference.

I would also be glad to look at the service definitions you have and answer
any questions.

[1] https://cwiki.apache.org/confluence/display/KNOX/Index
[2]
https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox
[3]
https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox

Best,
Sandeep

P.S. I am a Knox committer and new to Metron.

On Mon, Nov 12, 2018 at 9:59 AM Ryan Merriman  wrote:

> I'm just coming up to speed on Knox so maybe rewriting assets links are
> trivial. If anyone has a good example of how to do that or can point to
> some documentation, please share.
>
> On Mon, Nov 12, 2018 at 8:54 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > Doing the Knox proxy work first certainly does make a lot of sense vs
the
> > SSO first approach, so I'm in favour of this. It bypasses all the
> anti-CORS
> > proxying stuff the other solution needed by being on the same URL
space.
> >
> > Is there are reason we're not re-writing the asset link URLs in Knox?
We
> > should have a reverse content rewrite rule to avoid that problem and
make
> > it entirely transparent whether there is Knox or not. We shouldn't be
> > changing anything about the UI services themselves. If the rewrite
> service
> > is complete, there is no change to base ref in the UI code, Knox would
> > effectively apply it by content filtering. Note also that the gateway
URL
> > is configurable and likely to vary from Knox to Knox, so baking it into
> the
> > ng build will break non-full-dev builds. (e.g. gateway/default could
well
> > be gateway/xyz).
> >
> > I would also like to discuss removing the JDBC auth, because it's a set
> of
> > plaintext passwords in a mysql DB... it introduces a problematic
> dependency
> > (mysql) a ton of java dependencies we could cut out (JPA, eclipselink)
> and
> > opens up a massive security hole. I personally know of several
> > organisations who are blocked from using Metron by the presence of the
> JDBC
> > authentication method in its current form.
> >
> > Simon
> >
> > On Mon, 12 Nov 2018 at 14:36, Ryan Merriman 
wrote:
> >
> > > Let me clarify on exposing both legacy and Knox URLs at the same
time.
> > The
> > > base urls will look something like this:
> > >
> > > Legacy REST - http://node1:8082/api/v1
> > > Legacy Alerts UI - http://node1:4201:/alerts-list
> > >
> > > Knox REST - https://node1:8443/gateway/default/metron/api/v1
> > > Knox Alerts UI -
> > > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list
> > >
> > > If Knox were turned on and the alerts UI deployed as is, it would not
> > > work. This is because static assets are referenced with
> > > http://node1:4201/assets/some-asset.js which does not include the
> > correct
> > > context path to the alerts UI in knox. To make it work, you have to
> set
> > > the base ref to "/gateway/default/metron-alerts-ui" so that static
> assets
> > > are referenced at
> > >
> https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js
> > .
> > > When you do that, the legacy alerts UI will no longer work. I guess
> the
> > > point I'm trying to make is that we would have to switch between them
> or
> > > have 2 separate application running. I imagine most users only need
> one
> > or
> > > the other running so probably not an issue.
> > >
> > > Jon, the primary upgrade consideration I see is with authentication.
> To
> > be
> > > able to use Knox, you would have to upgrade to LDAP-based
> authentication
> > if
> > > you were still using JDBC-based authentication in REST. The urls
would
> > > also change obviously.
> > >
> > > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com 
> > wrote:
> > >
> > > > Phew, that was quite the thread to catch up on.
> > > >
> > > > I agree that this should be optional/pluggable to start, and I'm
> > > interested
> > > > to hear the issues as they relate to upgrading an existing cluster
> > (given
> > > > the suggested approach) and exposing both legacy and knox URLs at
the
> > > same
> > > > time.
> > > >
> > > > Jon
> > > >
> > > > On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic <
> > > > michael.miklav...@gmail.com>
> > > > wrote:
> > > >
> > > > > A couple more things, and I think this goes 

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

[Otto Fowler]

g");
> 183 properties.put("long_field", longType);
> 184 Map floatType = new HashMap<>();
> 185 floatType.put("type", "float");
> 186 properties.put("latitude", floatType);
> 187 Map doubleType = new HashMap<>();
> 188 doubleType.put("type", "double");
> 189 properties.put("score", doubleType);
> 190   }
> 191
> 192   /**
> 193* Add test fields to a template with defined types in case they are
> not defined in the sensor template shipped with Metron.
> 194* This is useful for testing certain cases, for example faceting on
> fields of various types.
> 195* @param template
> 196* @param docType
> 197*/
> 198   private static void addTestFieldMappings(JSONObject template, String
> docType) {
> 199 Map mappings = (Map) template.get("mappings");
> 200 Map docTypeJSON = (Map) mappings.get(docType);
> 201 Map properties = (Map) docTypeJSON.get("properties");
> 202 Map longType = new HashMap<>();
> 203 longType.put("type", "long");
> 204 properties.put("long_field", longType);
> 205 Map floatType = new HashMap<>();
> 206 floatType.put("type", "float");
> 207 properties.put("latitude", floatType);
> 208 Map doubleType = new HashMap<>();
> 209 doubleType.put("type", "double");
> 210 properties.put("score", doubleType);
> 211   }
>
>
> On Thu, Nov 15, 2018 at 4:21 PM Otto Fowler 
> wrote:
>
>> Can you diff the trees to be sure?
>>
>>
>> On November 15, 2018 at 17:52:40, Michael Miklavcic (
>> michael.miklav...@gmail.com) wrote:
>>
>> So amazingly, this still has results in conflicts, but I am able to
>> resolve
>> them manually in a sensible fashion.
>> git merge -X theirs es-rebased
>> CONFLICT (rename/rename): Rename
>>
>> "metron-interface/metron-config/src/app/rxjs-operators.ts"->"metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunnerResults.java"
>> in branch "HEAD" rename
>>
>> "metron-interface/metron-config/src/app/rxjs-operators.ts"->"metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/FieldMapping.java"
>> in "stella-es-base2"
>>
>> So where I landed gives a history like the following:
>> * df1195aa 2018-11-15 | METRON-1834: Migrate Elasticsearch from
>> TransportClient to new Java REST API (mmiklavc via mmiklavc) closes
>> apache/metron#1242 (HEAD -> master-merge) [mmiklavc]
>> |\
>> | * 590b3669 2018-11-15 | METRON-1834: Migrate Elasticsearch from
>> TransportClient to new Java REST API (mmiklavc via mmiklavc) closes
>> apache/metron#1242 (stella-es-base2) [mmiklavc]
>> | * a7c7dc28 2018-10-08 | Casey Stella - elasticsearch rest client
>> migration base work (stella-es-base) [cstella]
>> * | 0c4c622b 2018-11-14 | METRON-1749 Update Angular to latest release in
>> Management UI (sardell via nickwallen) closes apache/metron#1217
>> (origin/master, origin/HEAD, master) [sardell]
>> ...
>>
>> I can modify a7c7dc28 commit message as well and hopefully this will be
>> good for everyone?
>>
>> Cheers,
>> Mike
>>
>>
>>
>>
>> On Thu, Nov 15, 2018 at 2:29 PM Justin Leet 
>> wrote:
>>
>> > I took a look at this with Mike a bit, and it seems like it's pretty
>> > painful and without a clear way to avoid remerging conflicts. If the
>> > latest attempt doesn't work, I'm in favor of getting it in and just
>> getting
>> > it down to as few commits as reasonably possible.
>> >
>> > On Thu, Nov 15, 2018 at 4:12 PM Michael Miklavcic <
>> > michael.miklav...@gmail.com> wrote:
>> >
>> > > I'm attempting 1 more option, which would be to do a "git merge
>> > > --strategy-option theirs" after having done the commit wrangling in
>> the
>> > PR
>> > > branch. Will reply back with results.
>> > >
>> > > On Thu, Nov 15, 2018 at 2:02 PM Michael Miklavcic <
>> > > michael.miklav...@gmail.com> wrote:
>> > >
>> > > > Yes, definitely.
>> > > >
>> > > > On Thu, Nov 15, 2018 at 2:01 PM Casey Stella 
>> > wrote:
>> > > >
>> > > >> Can you at least rename your commits to have METRON-1834 prefixing
>> > them?
>> > 

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

[Otto Fowler]

Can you diff the trees to be sure?


On November 15, 2018 at 17:52:40, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

So amazingly, this still has results in conflicts, but I am able to resolve
them manually in a sensible fashion.
git merge -X theirs es-rebased
CONFLICT (rename/rename): Rename
"metron-interface/metron-config/src/app/rxjs-operators.ts"->"metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/ParserRunnerResults.java"

in branch "HEAD" rename
"metron-interface/metron-config/src/app/rxjs-operators.ts"->"metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/FieldMapping.java"

in "stella-es-base2"

So where I landed gives a history like the following:
* df1195aa 2018-11-15 | METRON-1834: Migrate Elasticsearch from
TransportClient to new Java REST API (mmiklavc via mmiklavc) closes
apache/metron#1242 (HEAD -> master-merge) [mmiklavc]
|\
| * 590b3669 2018-11-15 | METRON-1834: Migrate Elasticsearch from
TransportClient to new Java REST API (mmiklavc via mmiklavc) closes
apache/metron#1242 (stella-es-base2) [mmiklavc]
| * a7c7dc28 2018-10-08 | Casey Stella - elasticsearch rest client
migration base work (stella-es-base) [cstella]
* | 0c4c622b 2018-11-14 | METRON-1749 Update Angular to latest release in
Management UI (sardell via nickwallen) closes apache/metron#1217
(origin/master, origin/HEAD, master) [sardell]
...

I can modify a7c7dc28 commit message as well and hopefully this will be
good for everyone?

Cheers,
Mike




On Thu, Nov 15, 2018 at 2:29 PM Justin Leet  wrote:

> I took a look at this with Mike a bit, and it seems like it's pretty
> painful and without a clear way to avoid remerging conflicts. If the
> latest attempt doesn't work, I'm in favor of getting it in and just
getting
> it down to as few commits as reasonably possible.
>
> On Thu, Nov 15, 2018 at 4:12 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I'm attempting 1 more option, which would be to do a "git merge
> > --strategy-option theirs" after having done the commit wrangling in the
> PR
> > branch. Will reply back with results.
> >
> > On Thu, Nov 15, 2018 at 2:02 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Yes, definitely.
> > >
> > > On Thu, Nov 15, 2018 at 2:01 PM Casey Stella 
> wrote:
> > >
> > >> Can you at least rename your commits to have METRON-1834 prefixing
> them?
> > >> On Thu, Nov 15, 2018 at 15:19 Michael Miklavcic <
> > >> michael.miklav...@gmail.com>
> > >> wrote:
> > >>
> > >> > https://github.com/apache/metron/pull/1242
> > >> >
> > >> > TL;DR
> > >> > I'd like to discuss the best option to merge METRON-1834 into
> master.
> > I
> > >> > want to propose handling this like a feature branch and merging it
> > >> as-is.
> > >> > ---
> > >> >
> > >> > I'm sure most folks' initial reaction will be some skepticism akin
> to
> > >> "have
> > >> > you tried turning it off again," as this was my initial reaction
as
> > >> well.
> > >> > It does not seem like this should be difficult. And I'm hoping
that
> > this
> > >> > may be some esoteric thing on my system, though I believe this is
a
> > real
> > >> > problem. A rather tedious explanation follows of what I've tried
and
> > the
> > >> > problems encountered along the way. What seemed like a really
simple
> > >> > problem instead appears to be a bit much for Git to handle without
> > >> > requiring redoing merges and another full round of testing. I'd
much
> > >> prefer
> > >> > to avoid that in this instance.
> > >> >
> > >> > This PR is ready to be merged into master. It's recent and very
> close
> > to
> > >> > fully up to date in the branch. Latest master merges cleanly.
There
> is
> > >> an
> > >> > attribution to Casey Stella for the base point of this PR that I
> need
> > to
> > >> > include when getting this into master. When I created my branch, I
> > >> > collapsed his initial set of commits into a single squashed commit
> on
> > >> > master at the time, and I started to work from there. Over time, I
> > made
> > >> a
> > >> > number of additional commits and merges from master. Now for the
> > issues.
> > >> >
> > >> > Originally, my expectation was that I could have 2 commits - the
> > >> original
> > >> > squashed commit from Casey along with all my additional commits
(and
> > the
> > >> > merges with master) right on top. Nice clean history on master.
> Turns
> > >> out,
> > >> > this doesn't work as cleanly as expected because a combination of
> the
> > >> > multiple merges and the need to keep the original commit with
> > >> attribution
> > >> > to Casey's work. A normal git pull --squash works fine, as
expected,
> > >> but we
> > >> > lose the base commit, and therefore the requisite attribution.
Here
> > are
> > >> > some other things I've tried, to no avail.
> > >> >
> > >> > 1. Git pull --squash after a merge with master. This will squash
> > the
> > >> > entire tree back to the branch point. No good.
> > >> > 2. Git rebase -i 

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

[Otto Fowler]

Proper attribution and the correct code are the most important things, not
the number of commits.


On November 15, 2018 at 16:29:04, Justin Leet (justinjl...@gmail.com) wrote:

I took a look at this with Mike a bit, and it seems like it's pretty
painful and without a clear way to avoid remerging conflicts. If the
latest attempt doesn't work, I'm in favor of getting it in and just getting
it down to as few commits as reasonably possible.

On Thu, Nov 15, 2018 at 4:12 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I'm attempting 1 more option, which would be to do a "git merge
> --strategy-option theirs" after having done the commit wrangling in the
PR
> branch. Will reply back with results.
>
> On Thu, Nov 15, 2018 at 2:02 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Yes, definitely.
> >
> > On Thu, Nov 15, 2018 at 2:01 PM Casey Stella 
wrote:
> >
> >> Can you at least rename your commits to have METRON-1834 prefixing
them?
> >> On Thu, Nov 15, 2018 at 15:19 Michael Miklavcic <
> >> michael.miklav...@gmail.com>
> >> wrote:
> >>
> >> > https://github.com/apache/metron/pull/1242
> >> >
> >> > TL;DR
> >> > I'd like to discuss the best option to merge METRON-1834 into
master.
> I
> >> > want to propose handling this like a feature branch and merging it
> >> as-is.
> >> > ---
> >> >
> >> > I'm sure most folks' initial reaction will be some skepticism akin
to
> >> "have
> >> > you tried turning it off again," as this was my initial reaction as
> >> well.
> >> > It does not seem like this should be difficult. And I'm hoping that
> this
> >> > may be some esoteric thing on my system, though I believe this is a
> real
> >> > problem. A rather tedious explanation follows of what I've tried and
> the
> >> > problems encountered along the way. What seemed like a really simple
> >> > problem instead appears to be a bit much for Git to handle without
> >> > requiring redoing merges and another full round of testing. I'd much
> >> prefer
> >> > to avoid that in this instance.
> >> >
> >> > This PR is ready to be merged into master. It's recent and very
close
> to
> >> > fully up to date in the branch. Latest master merges cleanly. There
is
> >> an
> >> > attribution to Casey Stella for the base point of this PR that I
need
> to
> >> > include when getting this into master. When I created my branch, I
> >> > collapsed his initial set of commits into a single squashed commit
on
> >> > master at the time, and I started to work from there. Over time, I
> made
> >> a
> >> > number of additional commits and merges from master. Now for the
> issues.
> >> >
> >> > Originally, my expectation was that I could have 2 commits - the
> >> original
> >> > squashed commit from Casey along with all my additional commits (and
> the
> >> > merges with master) right on top. Nice clean history on master.
Turns
> >> out,
> >> > this doesn't work as cleanly as expected because a combination of
the
> >> > multiple merges and the need to keep the original commit with
> >> attribution
> >> > to Casey's work. A normal git pull --squash works fine, as expected,
> >> but we
> >> > lose the base commit, and therefore the requisite attribution. Here
> are
> >> > some other things I've tried, to no avail.
> >> >
> >> > 1. Git pull --squash after a merge with master. This will squash
> the
> >> > entire tree back to the branch point. No good.
> >> > 2. Git rebase -i master. Allows you to cleanly apply changes, but
> >> then
> >> > it ends up having problems with a clean rebase and shows
> conflicts. I
> >> > expect this is because of the merge history being necessary.
> >> > 3. Checking out a branch from the base point squashed commit from
> >> Casey,
> >> > and attempt to apply my changes on top. Numerous methods for
> >> > squashing/rebasing my changes on top applies nicely in the branch.
> >> But
> >> > then
> >> > it once again causes merge conflicts when I attempt to get this
> onto
> >> > master. Things I attempted include: manually copying files,
> rebasing
> >> > all my
> >> > commits plus merges on top of the base commit, git merge --squash,
> >> > intimidation.
> >> >
> >> > For one example of the result I'm talking about, this looks "good"
but
> >> it's
> >> > missing a ton of recent commits because they get caught up in the
> rebase
> >> > and get squashed in with my commit. When you attempt to merge this
> onto
> >> > master, it is just plain wrong (see example below with merge
> conflicts).
> >> > * 22c3b3bc 2018-11-15 | METRON-1834: Migrate Elasticsearch from
> >> > TransportClient to new Java REST API (mmiklavc via mmiklavc) closes
> >> > apache/metron#1242 (HEAD -> stella-es-base2) [mmiklavc]
> >> > * 84232e90 2018-10-08 | METRON-1834: Elasticsearch rest client
> migration
> >> > base work starting point for apache/metron#1242 (cstella via
mmiklavc)
> >> > [cstella]
> >> > * 5bfc08c5 2018-10-08 | METRON-1792 Simplify Profile Definitions in
> >> > Integration Tests (nickwallen) closes 

New WIP PR, a new Full Dev option

[Otto Fowler]

I have submitted a WIP PR [#1261](https://github.com/apache/metron/pull/1261)
that makes it possible to run / try the metron full dev environment with
only Vagrant, VirtualBox and Docker installed, as opposed to having to have
all the dev tools and ansible at the right version.

I think this would be a big help for new users.

Please give it a look.

I look forward to your feedback.

O