Re: [DEV COMMUNITY MEETING] Call for Ideas and Schedule

[zeo...@gmail.com]

I like your list of potential topics.  I'm also in to attend - that time
works well for me.

I would be interested in talking about our release process, as I would like
to suggest that we formalize upgrade and installation instructions to be
included as a part of a release, and talk through any concerns/questions
with the secondary repo.  I know I have at least one discussion point on
each of those suggestions.

Also, maybe we can chat about adding security scanning to our code base,
including on PRs.

Jon

On Thu, Dec 14, 2017, 15:43 Otto Fowler  wrote:

> Dev Community Meeting Call
>
> I would like to propose a developer community meeting.
>
> I propose that we set the meeting early next week, and will throw out
> Monday, December 18th at 09:30AM PST, 12:30 on the East Coast and 5:30 in
> London Towne.
>
> This meeting will be held over a web-ex, the details of which will be
> included in the actual meeting notice.
>
> Please reply to this with scheduling concerns and topic suggestions.
> Potential Topics
>
>- Call for reviewers, ideas how to get more involvement, what people can
>do to help
>- Feature branches : we have two now, what are they and how are we going
>to work on them
>- Extension Repository: Default deployment and installation of parsers
>as it relates to ‘777’
>- General ‘777’ discussion
>
> Developer Community Meeting Disclaimers
>
>- Developer Community meetings are a means for realtime discussion of
>development issues
>- These meetings are not specifically aimed at demonstrations, unless
>one is required or requested as part of such discussion
>- These meetings are geared towards Metron development issues, not user
>issues with deployment or shipped functionality
>- There are *NO* decisions made in these meetings. The mailing list is
>the official communication record of the Apache Metron Project, and as
> such
>all public decisions are to be made on the list, as to give the greatest
>opportunity for community involvement.
>- There *ARE* proposals that can be made and discussed in these
>meetings, that will then be discussed on list for decision.
>- Notes will be taken of these meetings, and they will be posted to the
>list
>- There may also be breakout posts to the list per proposal or topic,
>for more detailed discussion
>
-- 

Jon

Re: [DISCUSS] Stellar Documentation Autogeneration

[zeo...@gmail.com]

A huge +1 from me.  This would be great

On Thu, Dec 14, 2017 at 3:39 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 from me, great idea Justin. I did a bit of digging around also and the
> Doclet approach you're already using seems the way to go. I didn't come
> across any libraries that would make this easier or better. Not sure if
> Swagger has anything along these lines?
>
> On Thu, Dec 14, 2017 at 1:00 PM, Otto Fowler 
> wrote:
>
> > I think this is a great idea, and I looked at the POC and it isn’t as bad
> > as you make it out to be;)
> >
> > What I would like to see is documentation for Stellar functions, by
> > namespace generated. I would also
> > like the capability to document at the namespace level.
> >
> > Often we have namespace level concepts that don’t fit into any given
> > function’s documentation.
> > Setting aside the how of the namespace documentation for a moment, based
> on
> > the POC I would
> > suggest that we
> >
> > * find all namespaces
> > * create a page per namespace
> > * document each function in it’s namespace’s page
> > * include the namespace doc in that page
> >
> > Each module that exports stellar function’s should have it’s own
> > documentation.  As part of breaking stellar out to it’s own module
> > we should remove stellar documentation from stellar common that applies
> to
> > functions outside that module.
> >
> >
> >
> > On December 14, 2017 at 14:32:56, Justin Leet (justinjl...@gmail.com)
> > wrote:
> >
> > I think it would be valuable to have the documentation around Stellar
> being
> > autogenerated. We have most of the info we'd want in the @Stellar
> > annotation, and ideally, we could just pull this info out and produce
> some
> > docs similar to what we already manually maintain. This came up a bit in
> > the context of https://issues.apache.org/jira/browse/METRON-1361
> >
> > I put together a super, super (super!) rough POC of using the approach of
> > Javadoc-style doclet processing that reads the annotations and kicks out
> > something pretty close to the current docs (without any fancy stuff like
> > the table of contents and so on).
> >
> > Right now, there'd be a good deal more to do that to make it usable. Off
> > the top of my head, the main things I wanted to look at before really
> even
> > taking an actual stab at it are
> >
> > 1) abstracting out the markdown formatting from the annotation parsing
> > 2) Making sure we can integrate this approach without breaking current
> > Javadocs
> > 3) Managing things across projects (since we put in Stellar functions all
> > over).
> > 4) Slightly more though about how we'd manage it.
> >
> > Otto's alluded to having a couple thoughts, and I'm more than happy to
> get
> > a better idea of what we want the end state to look like (either this or
> > something else, e.g. an annotation processor during compile phase or if
> > someone knows a tool that takes care of this sort of thing.)
> >
> > Any thoughts?
> >
>
-- 

Jon

Re: [DISCUSS] Community Meetings

[zeo...@gmail.com]

This sounds great Otto, thanks.

Jon

On Thu, Dec 14, 2017 at 11:24 AM Laurens Vets  wrote:

>
> Sounds good to me :)
>
> On 2017-12-14 05:59, Otto Fowler wrote:
> > Ok,
> >
> > So we will be concerned with two types of meetings.  I’ll take
> > responsibility for calling the meetings and ‘moderation’.
> >
> > Dev meetings
> >  - feedback on how things are going overall
> >  - discussions on specific technical problems
> >  - discussion of possible improvements
> >
> > User meetings
> >  - demos
> >  - user content ( how I’m using metron )
> >  - some unavoidable discussion on problems
> >  - some requirements gathering triage
> >
> > ALL
> >  - I will call
> >  - I will gather input for agenda
> >  - I will distribute the agenda
> >  - I will distribute the notes to the list and on confluence
> >  - No decisions will be made, only discussed and then put to list
> >  - besides general nodes, breakout messages for topical discussion or
> > decisions
> >
> >
> >
> > How does that sound?
> >
> >
> > On December 13, 2017 at 16:41:29, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > +1
> >
> >
> > On December 13, 2017 at 16:39:52, James Sirota (jsir...@apache.org)
> > wrote:
> >
> > I can set up a dedicated Zoom room with a recurrent meeting and give
> > PMC
> > members rights to the room. I think hosting these meetings should not
> > be a
> > problem. I would vote not to record them, but rather provide the notes
> > after the meeting. It's a lot easier to skim through the notes than
> > jump
> > around in a recording. As Simon mentioned, I would also make it
> > explicitly
> > clear that the meetings are dev meetings. These are not user Q and
> > are
> > not meant to be overviews of how different features of Metron work. If
> > we
> > want to do feature demos or provide user content I would want that to
> > be in
> > its own separate meeting.
> >
> > Thanks,
> > James
> >
> > 13.12.2017, 05:00, "Otto Fowler" :
> >> I am ok with just notes and no recording.
> >>
> >> On December 13, 2017 at 04:37:20, Simon Elliston Ball (
> >> si...@simonellistonball.com) wrote:
> >>
> >> Good points Larry, we would need to get consent from everyone on the
> >> call
> >> to record to properly comply with regulations in some countries. We
> >> would
> >> definitely need someone to step up as note taker.
> >>
> >> Something else to think about is intended audience. Previously we’ve
> >> had
> >> meeting like this which have been very detailed Dev@ focussed (which
> >> is a
> >> great thing) but have rather alienated participants in User@ land. We
> >> need
> >> to make it clear what level we’re talking about to be inclusive.
> >>
> >> Simon
> >>
> >>>  On 13 Dec 2017, at 00:44, larry mccay  wrote:
> >>>
> >>>  Not sure about posting the recordings - you will need to check and
> >>> make
> >>>  sure that doesn't violate anything.
> >>>
> >>>  Just a friendly reminder...
> >>>  It is important that meetings have notes and a summary that is sent
> >>> out
> >>>  describing topics to be decided on the mailing list.
> >>>  No decisions can be made in the community meeting itself - this
> >>> gives
> >>>  others in other timezones and commitments review and voice in the
> >>
> >> decisions.
> >>>  If it didn't happen on the mailing lists then it didn't happen. :)
> >>>
> >>>  On Tue, Dec 12, 2017 at 1:39 PM, Simon Elliston Ball <
> >>>  si...@simonellistonball.com> wrote:
> >>>
>   Yes, I do.
> 
>   I suspect the best bet will be to post recordings somewhere on the
>   apache.org  metron site.
> 
>   Simon
> 
> >  On 12 Dec 2017, at 18:36, Otto Fowler 
> > wrote:
> >
> >  Excellent, do you have the > 40 min + record option?
> >
> >  On December 12, 2017 at 13:19:55, Simon Elliston Ball (
> >  si...@simonellistonball.com) wrote:
> >
> >  Happy to volunteer a zoom room. That seems to have worked for most
> > in
> >>
> >> the
> >  past.
> >
> >  Simon
> >
> >>  On 12 Dec 2017, at 18:09, Otto Fowler 
> > wrote:
> >>
> >>  Thanks! I think I’d like something hosted though.
> >>
> >>  On December 12, 2017 at 11:18:52, Ahmed Shah (
>   ahmeds...@cmail.carleton.ca)
> >>  wrote:
> >>
> >>  Hello,
> >>
> >>  wrt "- How are we going to host it"...
> >>
> >>  I've used BigBlueButton as an end user at our University.
> >>
> >>  It is LGPL open source.
> >>
> >>  https://bigbluebutton.org/
> >>  https://bigbluebutton.org/developers/
> >>
> >>  -Ahmed
> >>
> >>  ___
> >>  Ahmed Shah (PMP, M. Eng.)
> >>  Cybersecurity Analyst & Developer
> >>  GCR - Cybersecurity Operations Center
> >>  Carleton University -
> >> cugcr.com
> 

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

>
> > *
> >
> > Files with unapproved licenses:
> >
> >
> >
> >
> /Users/ottofowler/tmp/release_ver/apache-metron-0.4.2-rc1/metron-interface/metron-alerts/dist/styles.f56deed131e58bd7ee04.bundle.css
> >
> > *
> >
> >
> >
> > On December 8, 2017 at 04:34:24, Matt Foley (ma...@apache.org)
> wrote:
> >
> > Colleagues,
> > I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
> >
> > Given the complexity of this RC, I’d appreciate if a couple people
> would be
> > willing to kick the tires before we put it up for a vote.
> >
> > I will myself be going thru the Verify Build process this weekend,
> as I
> > won’t be able to do it Friday.
> >
> > Thanks,
> > --Matt
> >
> >
> > On 12/4/17, 2:05 PM, "zeo...@gmail.com" <zeo...@gmail.com> wrote:
> >
> > Can we resolve the conversation regarding the second repo? I was
> waiting
> > to get more input/preferences from people There's also a
> documentation
> > update that fixes a few broken Stellar docs that already has aa +1,
> I just
> > need to merge it.
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 17:01 Casey Stella <ceste...@gmail.com> wrote:
> >
> > > I would be in favor of a release at this point.
> > >
> > > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org>
> wrote:
> > >
> > > > Hey all,
> > > > I see METRON-1252 was resolved over the weekend. Shall I go
> ahead and
> > > > start the process with 0.4.2 release?
> > > > Does anyone have any commits they feel strongly should go in
> before
> > 0.4.2
> > > > is done, or are we ready to call it good?
> > > >
> > > > I believe there is consensus the 0.4.2 release should include a
> release
> > > of
> > > > the current state of the metron-bro-plugin-kafka. I will
> continue the
> > > > discussion in that thread as to the process for accomplishing
> that, but
> > > > plan on it happening.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > > holiday.
> > > > Regarding status of the release effort, still pending
> METRON-1252, so
> > > > not making the release branch yet.
> > > >
> > > > Regards,
> > > > --Matt
> > > >
> > > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > (With release manager hat on)
> > > >
> > > > The community has proposed a release of Metron in the near
> > > future,
> > > > focusing on Meta-alerts running in Elasticsearch.
> > > > Congrats on getting so many of the below already done. At this
> > > > point, only METRON-1252, and the discussion of how to handle
> joint
> > > release
> > > > of the Metron bro plugin, remain as gating items for the
> release. I
> > > > project these will be resolved next week, so let’s propose the
> > following:
> > > >
> > > > Sometime next week, after the last bits are done, I’ll start the
> > > > release process and create the release branch.
> > > >
> > > > The proposed new version will be 0.4.2, unless there are backward
> > > > incompatible changes that support making it 0.5.0.
> > > > Currently there are NO included Jiras labeled
> > > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > > ***If anyone knows that some of the commits included since 0.4.1
> > > > introduce backward incompatibility, please say so now on this
> thread,
> > and
> > > > mark the Jira as such.***
> > > >
> > > > The 90 or so jiras/commits already in master branch

Re: [DISCUSS] Community Meetings

[zeo...@gmail.com]

I think this is a great idea.  Hangouts works well but last I checked has a
user # limitation.  I don't have any other good suggestions, sorry, but I'm
in to attend.

Jon

On Mon, Dec 11, 2017, 16:42 Otto Fowler  wrote:

> I think that we all want to have regular community meetings.  We may be
> better able to keep to a regular schedule with these meetings if we spread
> out the responsibility for them from James and Casey, both of whom have a
> lot on their plate already.
>
> I would be willing to coordinate and run the meetings, and would welcome
> anyone else who wants to help when they can.
>
> The only issue for me is I do not have a web-ex account that I can use to
> hold the meeting.  So I’ll need some recommendations for a suitable
> alternative.  I have not been able to find an Apache Friendly alternative,
> in the same way that Atlassian is apache friendly.
>
>
> So - from what I can see we need to:
>
> - Talk through who is going to do it
> - How are we going to host it
> - When are we going to do it
>
> Anything else?
>
> ottO
>
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

I poked around the RC briefly and spun up full dev with and without
sensors, no issues so far.

Jon

On Fri, Dec 8, 2017 at 4:34 AM Matt Foley <ma...@apache.org> wrote:

> Colleagues,
> I’ve posted Metron-0.4.2-RC1 and Metron-bro-plugin-kafka-0.1 to
> https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC1/
>
> Given the complexity of this RC, I’d appreciate if a couple people would
> be willing to kick the tires before we put it up for a vote.
>
> I will myself be going thru the Verify Build process this weekend, as I
> won’t be able to do it Friday.
>
> Thanks,
> --Matt
>
>
> On 12/4/17, 2:05 PM, "zeo...@gmail.com" <zeo...@gmail.com> wrote:
>
> Can we resolve the conversation regarding the second repo?  I was
> waiting
> to get more input/preferences from people  There's also a documentation
> update that fixes a few broken Stellar docs that already has aa +1, I
> just
> need to merge it.
>
> Jon
>
> On Mon, Dec 4, 2017, 17:01 Casey Stella <ceste...@gmail.com> wrote:
>
> > I would be in favor of a release at this point.
> >
> > On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <ma...@apache.org> wrote:
> >
> > > Hey all,
> > > I see METRON-1252 was resolved over the weekend.  Shall I go ahead
> and
> > > start the process with 0.4.2 release?
> > > Does anyone have any commits they feel strongly should go in
> before 0.4.2
> > > is done, or are we ready to call it good?
> > >
> > > I believe there is consensus the 0.4.2 release should include a
> release
> > of
> > > the current state of the metron-bro-plugin-kafka.  I will continue
> the
> > > discussion in that thread as to the process for accomplishing
> that, but
> > > plan on it happening.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/26/17, 6:26 PM, "Matt Foley" <ma...@apache.org> wrote:
> > >
> > > Hope everyone (at least in the U.S.) had a great Thanksgiving
> > holiday.
> > > Regarding status of the release effort, still pending
> METRON-1252, so
> > > not making the release branch yet.
> > >
> > > Regards,
> > > --Matt
> > >
> > > On 11/17/17, 1:32 PM, "Matt Foley" <ma...@apache.org> wrote:
> > >
> > > (With release manager hat on)
> > >
> > > The community has proposed a release of Metron in the near
> > future,
> > > focusing on Meta-alerts running in Elasticsearch.
> > > Congrats on getting so many of the below already done.  At
> this
> > > point, only METRON-1252, and the discussion of how to handle joint
> > release
> > > of the Metron bro plugin, remain as gating items for the release.
> I
> > > project these will be resolved next week, so let’s propose the
> following:
> > >
> > > Sometime next week, after the last bits are done, I’ll
> start the
> > > release process and create the release branch.
> > >
> > > The proposed new version will be 0.4.2, unless there are
> backward
> > > incompatible changes that support making it 0.5.0.
> > > Currently there are NO included Jiras labeled
> > > ‘backward-incompatible’, nor having Docs Text indicating so.
> > > ***If anyone knows that some of the commits included since
> 0.4.1
> > > introduce backward incompatibility, please say so now on this
> thread, and
> > > mark the Jira as such.***
> > >
> > > The 90 or so jiras/commits already in master branch since
> 0.4.1
> > > are listed below.
> > > Thanks,
> > > --Matt
> > >
> > > METRON-1301 Alerts UI - Sorting on Triage Score
> Unexpectedly
> > > Filters Some Records (nickwallen) closes apache/metron#832
> > > METRON-1294 IP addresses are not formatted correctly
> in facet
> > > and group results (merrimanr) closes apache/metron#827
> > > METRON-1291 Kafka produce REST endpoint does not work
> in a
> > > Kerberized cluster (merrimanr) closes apache/metron#826
> > > METRON-1290 Only first 10 alerts are update when a
> MetaAlert
> > > status is changed to i

Re: New PMC members

[zeo...@gmail.com]

Congratulations, guys!  Well deserved by all 3.

Jon

On Thu, Dec 7, 2017 at 10:48 AM Kyle Richardson 
wrote:

> Congratulations guys! Well deserved.
>
> -Kyle
>
> On Thu, Dec 7, 2017 at 10:18 AM, Nick Allen  wrote:
>
> > Congrats to all 3 for joining the PMC!
> >
> > On Thu, Dec 7, 2017 at 10:12 AM, Casey Stella 
> wrote:
> >
> > > Well, obviously, I meant Metron instead of Impala.  To this point, we
> > > should have a wiki page around templates for this, similar to the
> impala
> > > project. :)
> > >
> > > On Thu, Dec 7, 2017 at 10:06 AM, Casey Stella 
> > wrote:
> > >
> > > > The Project Management Committee (PMC) for Apache Impala has invited
> > Otto
> > > > Fowler, Michael Miklavcic and Justin Leet  to become a PMC member and
> > we
> > > > are pleased to announce that they have accepted.
> > > >
> > > > Congratulations and welcome!
> > > >
> > > >
> > >
> >
>
-- 

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

FYI to be uber clear about the effects of what I'm doing, spinning up
full-dev only when including the sensors will be broken on the bro plugin
install step between when I push the changes, and when mattf pushes the 0.1
tag to apache/metron-bro-plugin-kafka.

Jon

On Thu, Dec 7, 2017 at 3:05 PM zeo...@gmail.com <zeo...@gmail.com> wrote:

> Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone
>
> Jon
>
> On Thu, Dec 7, 2017 at 2:32 PM Matt Foley <ma...@apache.org> wrote:
>
>> I can start the release process tonight.
>>
>>
>>
>> Jon, you mentioned you want to commit
>>
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>>
>> before the release.  Is it convenient for you to do so today?
>>
>>
>>
>> Thanks,
>>
>> --Matt
>>
>>
>>
>> From: Nick Allen <n...@nickallen.org>
>> Date: Thursday, December 7, 2017 at 10:13 AM
>> To: "dev@metron.apache.org" <dev@metron.apache.org>
>> Cc: Matt Foley <ma...@apache.org>
>> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>>
>>
>>
>> I am more interested in getting a release cut.  If me moving to the (a)
>> camp gets us to consensus and cuts a release faster, then I'll do it.
>> Let's get this release train moving.
>>
>>
>>
>> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <justinjl...@gmail.com>
>> wrote:
>>
>> Do we have any further discussion on this?  Pardon me if I misstate
>> anyone's position, but it seems like we have a couple people (Otto and Jon
>> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably
>> a
>> section of people like myself without a particular horse in the race.
>>
>> It seems like we need to come to some sort of consensus so that we can get
>> the release bus moving again, and right now it seems like (a) is gathering
>> more explicit support.  Do we have a compelling reason to not do (a)? To
>> be
>> honest, my main worry is more "If we do (a) are we going to be miserable
>> if
>> we need to iterate or adjust?" I'm not seeing anything that suggests
>> anything too terrible, so unless we see some more discussion, I suggest we
>> move forward with (a).
>>
>>
>> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com <zeo...@gmail.com>
>> wrote:
>>
>> > I would prefer a, but I was initially thinking of doing the plugin first
>> > and then get in the two PRs out to use this new tag, which are already
>> +1'd
>> > and just waiting on this conversation.  For reference,
>> > https://github.com/apache/metron/pull/847 and
>> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>> >
>> > Jon
>> >
>> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ottobackwa...@gmail.com> wrote:
>> >
>> > > It seems to me, as I believe I have stated before that a) feels like
>> the
>> > > proper way to handle this.  It is how I have seen other projects like
>> > NiFi
>> > > handle things as well.
>> > >
>> > >
>> > >
>> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
>> > >
>> > > Okay, looking at this from the perspective of making a release:
>> > >
>> > >
>> > >
>> > > We have two choices:
>> > >
>> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
>> > > metron-bro-plugin-kafka, at the same time and using the same process
>> > > (modulo the necessary) as Metron.  This is dirt simple.
>> > >
>> > > b) I or someone needs to:
>> > >
>> > > - open a jira,
>> > >
>> > > - add the submodule to the Metron code tree,
>> > >
>> > > - possibly (optionally) add build mechanism to the maven poms, and
>> > >
>> > > - document as much as we think appropriate regarding what it is,
>> how
>> > to
>> > > build it, and how to update it,
>> > >
>> > > and commit that before the 0.4.2 release.
>> > >
>> > >
>> > >
>> > > What is the will of the community?
>> > >
>> > > Thanks,
>> > >
>> > > --Matt
>> > >
>> > >
>> > >
>> > > From: Nick Allen <n...@nickallen.org>
>> >

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone

Jon

On Thu, Dec 7, 2017 at 2:32 PM Matt Foley <ma...@apache.org> wrote:

> I can start the release process tonight.
>
>
>
> Jon, you mentioned you want to commit
>
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
>
> before the release.  Is it convenient for you to do so today?
>
>
>
> Thanks,
>
> --Matt
>
>
>
> From: Nick Allen <n...@nickallen.org>
> Date: Thursday, December 7, 2017 at 10:13 AM
> To: "dev@metron.apache.org" <dev@metron.apache.org>
> Cc: Matt Foley <ma...@apache.org>
> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>
>
>
> I am more interested in getting a release cut.  If me moving to the (a)
> camp gets us to consensus and cuts a release faster, then I'll do it.
> Let's get this release train moving.
>
>
>
> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <justinjl...@gmail.com>
> wrote:
>
> Do we have any further discussion on this?  Pardon me if I misstate
> anyone's position, but it seems like we have a couple people (Otto and Jon
> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
> section of people like myself without a particular horse in the race.
>
> It seems like we need to come to some sort of consensus so that we can get
> the release bus moving again, and right now it seems like (a) is gathering
> more explicit support.  Do we have a compelling reason to not do (a)? To be
> honest, my main worry is more "If we do (a) are we going to be miserable if
> we need to iterate or adjust?" I'm not seeing anything that suggests
> anything too terrible, so unless we see some more discussion, I suggest we
> move forward with (a).
>
>
> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com <zeo...@gmail.com> wrote:
>
> > I would prefer a, but I was initially thinking of doing the plugin first
> > and then get in the two PRs out to use this new tag, which are already
> +1'd
> > and just waiting on this conversation.  For reference,
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ottobackwa...@gmail.com> wrote:
> >
> > > It seems to me, as I believe I have stated before that a) feels like
> the
> > > proper way to handle this.  It is how I have seen other projects like
> > NiFi
> > > handle things as well.
> > >
> > >
> > >
> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
> > >
> > > Okay, looking at this from the perspective of making a release:
> > >
> > >
> > >
> > > We have two choices:
> > >
> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > > metron-bro-plugin-kafka, at the same time and using the same process
> > > (modulo the necessary) as Metron.  This is dirt simple.
> > >
> > > b) I or someone needs to:
> > >
> > > - open a jira,
> > >
> > > - add the submodule to the Metron code tree,
> > >
> > > - possibly (optionally) add build mechanism to the maven poms, and
> > >
> > > - document as much as we think appropriate regarding what it is,
> how
> > to
> > > build it, and how to update it,
> > >
> > > and commit that before the 0.4.2 release.
> > >
> > >
> > >
> > > What is the will of the community?
> > >
> > > Thanks,
> > >
> > > --Matt
> > >
> > >
> > >
> > > From: Nick Allen <n...@nickallen.org>
> > > Reply-To: "dev@metron.apache.org" <dev@metron.apache.org>
> > > Date: Tuesday, November 28, 2017 at 9:09 AM
> > > To: "dev@metron.apache.org" <dev@metron.apache.org>
> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> > Bro'
> > >
> > >
> > >
> > > I'll add a bit to Jon's technical comments.
> > >
> > >
> > >
> > > * We only created a separate repo because it was a technical
> requirement
> > to
> > > leverage the bro-pkg mechanism.
> > >
> > > * Leveraging the new bro-pkg mechanism has many advantages as outlined
> by
> > > Jon.
> > >
> > > * Enabling the bro-pkg mechanism is backwards compatibl

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

I would prefer a, but I was initially thinking of doing the plugin first
and then get in the two PRs out to use this new tag, which are already +1'd
and just waiting on this conversation.  For reference,
https://github.com/apache/metron/pull/847 and
https://github.com/apache/metron-bro-plugin-kafka/pull/4

Jon

On Mon, Dec 4, 2017, 20:54 Otto Fowler <ottobackwa...@gmail.com> wrote:

> It seems to me, as I believe I have stated before that a) feels like the
> proper way to handle this.  It is how I have seen other projects like NiFi
> handle things as well.
>
>
>
> On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
>
> Okay, looking at this from the perspective of making a release:
>
>
>
> We have two choices:
>
> a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> metron-bro-plugin-kafka, at the same time and using the same process
> (modulo the necessary) as Metron.  This is dirt simple.
>
> b) I or someone needs to:
>
> - open a jira,
>
> - add the submodule to the Metron code tree,
>
> - possibly (optionally) add build mechanism to the maven poms, and
>
> - document as much as we think appropriate regarding what it is, how to
> build it, and how to update it,
>
> and commit that before the 0.4.2 release.
>
>
>
> What is the will of the community?
>
> Thanks,
>
> --Matt
>
>
>
> From: Nick Allen <n...@nickallen.org>
> Reply-To: "dev@metron.apache.org" <dev@metron.apache.org>
> Date: Tuesday, November 28, 2017 at 9:09 AM
> To: "dev@metron.apache.org" <dev@metron.apache.org>
> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
>
>
>
> I'll add a bit to Jon's technical comments.
>
>
>
> * We only created a separate repo because it was a technical requirement to
> leverage the bro-pkg mechanism.
>
> * Leveraging the new bro-pkg mechanism has many advantages as outlined by
> Jon.
>
> * Enabling the bro-pkg mechanism is backwards compatible. I can install the
> plugin exactly how we use to.
>
>
>
> While I agree with Jon's technical comments, I disagree with the
> non-technical ones.
>
>
>
> (1) I do not want to change our release management process. While we needed
> to make a new repo (a technical change), I did not want that to change how
> we operate as a community (our procedures, policies, versioning and release
> cycles).
>
>
>
> (2) I see no value in adopting a separate release management process for
> the Bro plugin alone. Having a separate release process does not make the
> plugin *more* available to the Bro community.
>
>
>
> (3) I also see no value in positioning the plugin to be spun-out of the
> Metron project. It is a part of Metron and I want to see it benefit and
> evolve "the Apache-way".
>
>
>
> In my mind, the best way to accommodate the additional repo, while
> minimizing changes to our release management process, is to treat the new
> repo as a submodule. I fail to see significant downsides to this approach.
> A few extract command-line options do not seem overly onerous to me.
>
>
>
> Many thanks go to Jon for all the hard work he has put in enhancing the
> plugin.
>
>
>
>
>
> On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
> In an attempt to keep this from becoming unbearably long, I will try to
> keep my responses short, but I would be happy to elaborate. That's a fairly
> good timeline and summary, but here are some clarifications in
> corresponding order:
>
>
>
> - The plugin history is quite short and you can probably get a good bit of
> context just by looking at the commits.
>
> - The plugin is only useful to the bro community, but it is rather popular.
>
> - The Bro team created the idea of bro packages, which can include bro
> plugins, bro scripts, or BroControl plugins. So, instead of having a
> 'plugins' repo, they moved to have a 'packages' repo which is by default
> referenced by a bro-pkg tool they wrote for package management.
>
> - I believe I kicked this off (or at least I did in my head) when I started
> complaining about the plugin divergence that occurred due to the move to
> bro/plugins (the right move at the time), but Metron's use of a local
> directory that hadn't been kept up to date. My current efforts are an
> attempt to make sure this doesn't happen again, and to take advantage of
> the bro-pkg benefits.
>
> - The gap between ~3/31 and actual progress on 11/12 is completely on me -
> I had intentions of doing this work sooner but failed to do so.
>
> - You can most definitely still install/use the bro plugin wi

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

RON-1177 Stale running topologies seen post-kerberization
> > and cause exceptions (nickwallen) closes apache/metron#748
> > METRON-1158 Build backend for grouping alerts into meta
> alerts
> > (justinleet) closes apache/metron#734
> > METRON-1146: Add ability to parse JSON string into JSONObject
> > for stellar closes apache/incubator-metron#727
> > METRON-1176 REST: HDFS Service should support setting
> > permissions on files when writing (ottobackwards) closes
> apache/metron#749
> > METRON-1114 Add group by capabilities to search REST endpoint
> > (merrimanr) closes apache/metron#702
> > METRON-1167 Define Session Specific Global Configuration
> > Values in the REPL (nickwallen) closes apache/metron#740
> > METRON-1171: Better validation for the SUBSTRING stellar
> > function closes apache/incubator-metron#745
> >
> >
> >
> > On 11/17/17, 11:59 AM, "Nick Allen" <n...@nickallen.org> wrote:
> >
> > I just wanted to send an update on where we are at.  We've
> > gotten a lot
> > done here recently as you can see below.
> >
> >   ✓ DONE (1) First, METRON-1289 needs to go in.  This one was
> > a fairly big
> > effort and I am hearing that we are pretty close.
> >
> >   ✓ DONE (2) METRON-1294 fixes an issue in how field types
> are
> > looked-up.
> >
> >   ✓ DONE (3) METRON-1290 is next.  While this may have been
> > fixed in
> > M-1289, there may be some test cases we want from this PR.
> >
> >   ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> > logic.
> >
> >   ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> > metaalerts.
> >
> >   (6) That leads us to Raghu's UI work in METRON-1252.  This
> > introduces the
> > UI bits that depend on all the previous backend work.
> >
> >   (7) At this point, we should have our best effort at
> running
> > Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> >   (8) After we cut the release, we can introduce the work for
> > ES 5.x in
> > METRON-939.  I know we will need lots of help testing and
> > reviewing this
> > one.
> >
> >
> >
> > We also have an outstanding question that needs resolved
> > BEFORE we
> > release.  We need to come to a consensus on how to release
> > having moved our
> > Bro Plugin to a separate repo.  I don't think we've heard
> from
> > everyone on
> > this.  I'd urge everyone to chime in so we can choose a path
> > forward.
> >
> > If anyone is totally confused in regards to that discussion,
> I
> > can try and
> > send an options summary again as a separate discuss thread.
> > The original
> > chain was somewhere around here [1].
> >
> > [1]
> > https://lists.apache.org/thread.html/
> > 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> > 3Cdev.metron.apache.org%3E
> >
> >
> >
> > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> > n...@nickallen.org> wrote:
> >
> > > Hi Guys -
> > >
> > > I want to follow-up on this discussion.  It sounds like
> most
> > people are in
> > > agreement with the general approach.
> > >
> > > A lot of people have been working hard on Metaalerts and
> > Elasticsearch.  I
> > > have checked-in with those doing the heavy lifting and have
> > compiled a more
> > > detailed plan based on where we are at now.  To the best of
> > my knowledge
> > > here is the plan of attack for finishing out this effort.
> > >
> > >   (1) First, METRON-1289 needs to go in.  This one was a
> > fairly big effort
> > > and I am hearing that we are pretty close.
> > >
> > >   (2) METRON-1294 fixes an issue in how field types are
> > looked-up.
> > >
> > >   (3) METRON-1290 is next.  While this may have been fixed
> > in M-1289,
> > > th

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

In an attempt to keep this from becoming unbearably long, I will try
to keep my responses short, but I would be happy to elaborate.  That's a
fairly good timeline and summary, but here are some clarifications in
corresponding order:

- The plugin history is quite short and you can probably get a good bit of
context just by looking at the commits
.
- The plugin is only useful to the bro community, but it is rather popular.
- The Bro team created the idea of bro packages, which can include bro
plugins, bro scripts, or BroControl plugins.  So, instead of having a
'plugins' repo, they moved to have a 'packages' repo which is by default
referenced by a bro-pkg tool they wrote for package management.
- I believe I kicked this off (or at least I did in my head) when I started
complaining 
about the plugin divergence that occurred due to the move to bro/plugins
(the right move at the time), but Metron's use of a local directory that
hadn't been kept up to date.  My current efforts are an attempt to make
sure this doesn't happen again, and to take advantage of the bro-pkg
benefits.
- The gap between ~3/31 and actual progress on 11/12 is completely on me -
I had intentions of doing this work sooner but failed to do so.
- You can most definitely still install/use the bro plugin without using
bro-pkg.  In fact, the README in my PR

still has the instructions on how to do so.

Q1:  The simple explanation is that the only thing that makes a plugin a
bro package is the inclusion of a bro-pkg.meta file
,
and it includes a build_command which could easily be manually performed to
install by hand (assuming dependencies are met).

I've worked with other projects that use submodules and while I'm fine
discussing it, I suggest that we don't implement it.  I put together a
quick example of why here[1], using the bro project as an example since
it's top of mind.

Q2:   I think the answer to Q1 answers this.  There is absolutely nothing
stopping a git clone && cd $dir && configure && make && make install, but
using bro-pkg to install/load takes into account dependencies

and
unit tests

when it is loaded (and thus fails early and more intuitively).  It only
must be a separate repo (or, more technically correct, a git branch that
includes only the package) because of how bro-pkg works.  If you'd like to
get an idea of how this would work in application for Bro users, you can
see my test instructions here
 (specifically
step #3).  If a 0.1 tag gets pushed to apache/metron-bro-plugin-kafka, the
command could be `bro-pkg install metron-bro-plugin-kafka --version 0.1` or
`bro-pkg install apache/metron-bro-plugin-kafka --version 0.1` due to this
 (the
--force is just to remove user interaction, for an ansible spin-up).


1:  To clone the Bro git repo, you must run `git clone --recursive
https://github.com/bro/bro` (note the --recursive).  Not too big of a deal,
but requires that you remember it and existing instructions/blog posts may
give users inaccurate steps.  Let's make this worse and try to checkout
their latest release, v2.5.2, and automatically update the submodules
appropriately via `git checkout v2.5.2 --recurse-submodules`.  This fails
because aux/plugins (https://github.com/bro/plugins) was removed since
their latest release.  Okay, we can work around this using `git checkout
v2.5.2` and then remember to `git submodule update` every time you checkout
a release or branch.  But because they have nested submodules, we actually
need to run `git submodule update --recursive`.  I can't imagine opting
*into* a workflow anything like this.  There are other options as well,
such as git subtrees, but those I am less familiar with.

Jon

On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler  wrote:

> I am not sure that our use of the plugin necessarily equates to it being
> implicitly coupled to Metron.  It seems like the Right Thing To Do[image:
> ™], esp.
>  for an Apache project would be to make this available for use by the
> greater bro community.
> Unless we expect to do extensive iterative work on the plugin, which would
> then make the decision to spin it out now premature.
>
> Then again, I might be wrong ;)
>
>
> On November 27, 2017 at 19:58:11, Matt Foley (ma...@apache.org) wrote:
>
> [Please pardon me that the below is a little labored. I’m trying to
> understand the implications for both release and use, which 

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

The reason we decided to do that was because it is the best way for it to
be used (and thus improved on and quality tested) by the broader bro
community.  If it's any indication of it's popularity, there was just an
email on the bro mailing list about the plugin a few days ago, and I've
already received a PR against my temporary holding place for this plugin to
improve it (GitHub.com/jonzeolla/jzeolla-metron-bro-plugin-kafka), which
would never have happened if it wasn't a standalone repo.  Previously Nick
pushed the code into a bro/plugins repo, which is now deprecated in favor
of bro packages, but while it was there the community was able to identify
issues which helped improve its quality.

By putting it in its own repo we were also able to turn it into a bro
package, similar to debs or rpms but slightly different.  This has numerous
benefits, including turning setup into single command that installs and
loads the plugin, in addition to handling version management and performing
on installation unit tests to identify issues before use.

I would advocate to treat this code as a dependancy, but not necessarily a
core part of Metron, like we currently do.  This also allows us to maintain
separate versioning and release cycles (i.e. not necessarily lined up with
Metron releases), which I expect will be very simple, infrequent, and
easily tested.  Happy to discuss this further - thanks,

Jon

On Mon, Nov 27, 2017, 17:58 James Sirota  wrote:

> I agree with Nick.  Since the plugin is tightly coupled with Metron why
> not just pull it into the main repo and version it with the rest of the
> code?  Do we really need the second repo for the plug-in?
>
> Thanks,
> James
>
>
>
> 16.11.2017, 08:06, "Nick Allen" :
> >>  I would suggest that we institute a release procedure for the package
> >>  itself, but I don't think it necessarily has to line up with metron
> >>  releases (happy to be persuaded otherwise). Then we can just link
> metron
> >>  to metron-bro-plugin-kafka by pointing to specific
> >>  metron-bro-plugin-kafka releases (git tags
> >>  <
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> >>  versioning>
> >>  ).
> >>  Right now, full-dev spins up against the
> >>  apache/metron-bro-plugin-kafka master branch, which is not a good idea
> to
> >>  have in place for an upcoming release. That is the crux of why I think
> we
> >>  need to finalize the move to bro 2.5.2 and the plugin packaging before
> our
> >>  next release (working on it as we speak).
> >>  Jon
> >
> > ​I replayed Jon's comments from the other thread above.​
> >
> > My initial thought, is that I would not want to manage two separate
> release
> > processes. I don't want to have a roll call, cut release candidates and
> > test both.
> >
> > I was thinking we would just need to change some of the behind-the-scenes
> > processes handled by the release manager. This is one area where I had
> > thought using a submodule in Git would help.
> >
> > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen  wrote:
> >
> >>  + Restarting the thread to include mentors.
> >>
> >>  The code of the 'Kafka Plugin for Bro' is now maintained in the
> external
> >>  repository that we set up a while back.
> >>
> >> - Metron Core: git://git.apache.org/metron.git
> >> - Kafka Plugin for Bro: git://git.apache.org/
> >> metron-bro-plugin-kafka.git
> >>
> >>  (Q) Do we need to change anything in the release procedure to account
> for
> >>  this?
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
-- 

Jon

Re: [DISCUSS] NPM / Node Problems

[zeo...@gmail.com]

Note that I cleaned up the ansible scripts that install C++ 11 in my latest
PR <https://github.com/apache/metron/pull/847/files>, but it's not super
relevant to this conversation.

Jon

On Mon, Nov 27, 2017 at 10:42 AM zeo...@gmail.com <zeo...@gmail.com> wrote:

> That was also required for bro 2.5.2, so I did that here
> <https://github.com/apache/metron/commit/59fe1b453279bf5c7df627ea656c762b3a98e777>.
> Feel free to reuse the approach elsewhere
>
> Jon
>
> On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
>> First issue is that we need c++ 11 on centos 6.8
>>
>>
>>
>> On November 27, 2017 at 09:53:55, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Well, that’s good news on that issue. Reproducing the problem is half way
>> to solving it, right?
>>
>> I would still say there are some systemic things going on that have
>> manifested in a variety of ways on both the users and dev list, so it’s
>> worth us having a good look at a more robust approach to node dependencies
>> (both npm ones, and the native ones)
>>
>> Simon
>>
>> On 27 Nov 2017, at 13:30, Otto Fowler <ottobackwa...@gmail.com> wrote:
>>
>> I can reproduce the failure in out ansible docker build container, which
>> is
>> also centos.
>> The issue is building our node on centos in all these cases.
>>
>>
>>
>> On November 27, 2017 at 07:02:51, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Thinking about this, doesn’t our build plugin explicitly install it’s own
>> node? So actually all the node version things may be a red herring, since
>> this is under our control through the pom. Not sure if we actually
>> exercising this control. It seems that some of the errors people report
>> are
>> more to do with compilation failures for native node modules, which is
>> doesn’t pin (i.e. things like system library dependencies). I’m not sure
>> what we have in the dependency tree that requires complex native
>> dependencies, but this might just be one of those node things we could doc
>> around.
>>
>> This scenario would be fixed by standardising the build container.
>>
>> Yarn’s big thing is that it enables faster dependency resolution and local
>> caching, right? It does not seem to address any of the problems we see,
>> but
>> sure, it’s the shiny new dependency system for node modules, which might
>> make npm less horrible to deal with, so worth looking into.
>>
>> The other issue that I’ve seen people run into a lot is flat out download
>> errors. This could be helped by finding our versions, maybe with yarn, but
>> let’s face it, package-lock.json could also do that with npm, albeit with
>> a
>> slightly slower algorithm. However, short of bundling and hosting deps
>> ourselves, I suspect the download errors are beyond our control, and
>> certainly beyond the scope of this project (fix maven, fix npm, fix all
>> the
>> node hosting servers…)
>>
>> Simon
>>
>>
>> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda <
>> raghumitra@gmail.com>
>> wrote:
>> >
>> > Looking at some of the build failure emails and past experience i
>> > would suggest having a node & npm version check in our build scripts
>> > and moving dependency management to yarn.
>> >
>> > We need not restrict the build to a specific version of node & npm but
>> > we can surely suggest a min version required to build UI successfully.
>> >
>> > -Raghu
>> >
>> >
>> >
>> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
>> > <si...@simonellistonball.com> wrote:
>> >> Agreeing with Nick, it seems like the main reason people are building
>> themselves, and hitting all these environmental issues, is that we do not
>> as a project produce binary release artefacts (the rpms which users could
>> just install) and instead leave that for the commercial distributors to
>> do.
>> >>
>> >> Yarn may help with some of the dependency version issues we’re having,
>> but not afaik with the core missing library headers / build tools / node
>> and npm version issue, those would seem to fit a documentation fix and
>> improvements to platform-info to flag the problems, so this can then be a
>> pre-flight check tool as well as a diagnostic tool.
>> >>
>> >> Another option I would put on the table is to standardise our build
>> environmen

Re: [DISCUSS] NPM / Node Problems

[zeo...@gmail.com]

That was also required for bro 2.5.2, so I did that here
.
Feel free to reuse the approach elsewhere

Jon

On Mon, Nov 27, 2017 at 10:03 AM Otto Fowler 
wrote:

> First issue is that we need c++ 11 on centos 6.8
>
>
>
> On November 27, 2017 at 09:53:55, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Well, that’s good news on that issue. Reproducing the problem is half way
> to solving it, right?
>
> I would still say there are some systemic things going on that have
> manifested in a variety of ways on both the users and dev list, so it’s
> worth us having a good look at a more robust approach to node dependencies
> (both npm ones, and the native ones)
>
> Simon
>
> On 27 Nov 2017, at 13:30, Otto Fowler  wrote:
>
> I can reproduce the failure in out ansible docker build container, which is
> also centos.
> The issue is building our node on centos in all these cases.
>
>
>
> On November 27, 2017 at 07:02:51, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Thinking about this, doesn’t our build plugin explicitly install it’s own
> node? So actually all the node version things may be a red herring, since
> this is under our control through the pom. Not sure if we actually
> exercising this control. It seems that some of the errors people report are
> more to do with compilation failures for native node modules, which is
> doesn’t pin (i.e. things like system library dependencies). I’m not sure
> what we have in the dependency tree that requires complex native
> dependencies, but this might just be one of those node things we could doc
> around.
>
> This scenario would be fixed by standardising the build container.
>
> Yarn’s big thing is that it enables faster dependency resolution and local
> caching, right? It does not seem to address any of the problems we see, but
> sure, it’s the shiny new dependency system for node modules, which might
> make npm less horrible to deal with, so worth looking into.
>
> The other issue that I’ve seen people run into a lot is flat out download
> errors. This could be helped by finding our versions, maybe with yarn, but
> let’s face it, package-lock.json could also do that with npm, albeit with a
> slightly slower algorithm. However, short of bundling and hosting deps
> ourselves, I suspect the download errors are beyond our control, and
> certainly beyond the scope of this project (fix maven, fix npm, fix all the
> node hosting servers…)
>
> Simon
>
>
> > On 27 Nov 2017, at 07:28, RaghuMitra Kandikonda <
> raghumitra@gmail.com>
> wrote:
> >
> > Looking at some of the build failure emails and past experience i
> > would suggest having a node & npm version check in our build scripts
> > and moving dependency management to yarn.
> >
> > We need not restrict the build to a specific version of node & npm but
> > we can surely suggest a min version required to build UI successfully.
> >
> > -Raghu
> >
> >
> >
> > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball
> >  wrote:
> >> Agreeing with Nick, it seems like the main reason people are building
> themselves, and hitting all these environmental issues, is that we do not
> as a project produce binary release artefacts (the rpms which users could
> just install) and instead leave that for the commercial distributors to do.
> >>
> >> Yarn may help with some of the dependency version issues we’re having,
> but not afaik with the core missing library headers / build tools / node
> and npm version issue, those would seem to fit a documentation fix and
> improvements to platform-info to flag the problems, so this can then be a
> pre-flight check tool as well as a diagnostic tool.
> >>
> >> Another option I would put on the table is to standardise our build
> environment, so that the non-java bits are run in a standard docker image
> or something fo the sort, that way we can take control of all the
> environmental and OS dependent pieces, much as we do right now with the rpm
> build sections of the mpack build.
> >>
> >> The challenge here will be adding the relevant maven support. At the
> moment we’re relying on the maven npm and node build plugins, this would
> likely need replacing with something custom and a challenge to support to
> go dow this route.
> >>
> >> Perhaps the real answer here is to push people who are just kicking the
> tyres towards a binary distribution, or at least rpm artefacts as part of
> the Apache release to give them a head start for a happy path on a known
> good OS environment.
> >>
> >> Simon
> >>
> >>> On 24 Nov 2017, at 16:01, Nick Allen  wrote:
> >>>
> >>> Yes, it is a problem. I think you've identified a couple important
> things
> >>> that we could address in parallel. I see these as challenges we need to
> >>> solve for the dev community.
> >>>
> >>> (1) NPM is causing us some 

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

I propose that we coordinate the review of METRON-1329
<https://github.com/apache/metron-bro-plugin-kafka/pull/4> and METRON-1313
<https://github.com/apache/metron/pull/847>, then merge METRON-1329, pursue
a 0.1 release for apache/metron-bro-plugin-kafka, and then finalize
METRON-1313 <https://github.com/apache/metron/pull/847> prior to the
upcoming apache/metron release.  At that point the roadmap outlined here
<https://lists.apache.org/thread.html/286231d11fcb8349e0378d281a08d197876f80122056479156643104@%3Cdev.metron.apache.org%3E>
will be complete, and both of the issues you outlined below would be
addressed.

I did test making a bro package release of 0.4.2.0, and we can do that, but
the related (nested) plugin version must be major.minor, so it would need
to be something like 0.4 (or, my preference, 0.1).  I would rather keep the
package and plugin versions in line with each other than try to line the
package version up with apache/metron versions, and have the plugin version
out of whack (which is what people would see when checking to see if this
package was properly installed - see the version here
<https://github.com/apache/metron-bro-plugin-kafka/blob/master/tests/Baseline/kafka.show-plugin/output>
).

Let me know what you all think - happy to discuss further.

Jon

On Thu, Nov 16, 2017 at 1:26 PM Matt Foley <ma...@apache.org> wrote:

> There’s two issues, I think:
> (1) We’d like to be able to version and evolve the main body of Metron and
> the metron-bro-plugin-kafka separately.
> (2) We want to be able to assure that each release of Metron has a
> known-working version of metron-bro-plugin-kafka
>
> At a very simple level, we can achieve these by
> (1) Don’t put in version-checking constraints, or make those constraints
> very loose and of a form that assumes forward-compatibility, ie “version of
> metron-bro-plugin-kafka must be AT LEAST ‘x’ to be used with version ‘y’ of
> Metron.”
> (2) But we should still do an archive branch of metron-bro-plugin-kafka
> when we make a release branch of Metron, unless the metron-bro-plugin-kafka
> has zero changes since the last Metron release.
>
> I would recommend that we use a 4-element version for
> metron-bro-plugin-kafka, and at least the first 3 elements should be equal
> to the corresponding Metron release version.  This isn’t necessary, but it
> makes things easy to keep track of.  That still leaves room for necessary
> patches on a given release line.
>
> If you prefer other approaches, please propose.  When we reach consensus,
> I can edit the Release Process to document it.
> Cheers,
> --Matt
>
> On 11/16/17, 7:22 AM, "zeo...@gmail.com" <zeo...@gmail.com> wrote:
>
> I expect a few version changes up front to add some new features to the
> package (0.1 for the initial release, 0.{2..n} for some new features,
> 1.0
> when we stabilize) but after that it will probably only be updated to
> follow kafka/librdkafka updates.
>
> Jon
>
> On Thu, Nov 16, 2017 at 10:10 AM Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > How often to we expect to change this?  If it is effectively pinned
> then a
> > release process is not that bad.
> >
> >
> > On November 16, 2017 at 10:06:53, Nick Allen (n...@nickallen.org)
> wrote:
> >
> > >
> > > I would suggest that we institute a release procedure for the
> package
> > > itself, but I don't think it necessarily has to line up with metron
> > > releases (happy to be persuaded otherwise). Then we can just link
> metron
> > > to metron-bro-plugin-kafka by pointing to specific
> > > metron-bro-plugin-kafka releases (git tags
> > > <
> >
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> > > versioning>
> > > ).
> > > Right now, full-dev spins up against the
> > > apache/metron-bro-plugin-kafka master branch, which is not a good
> idea to
> > > have in place for an upcoming release. That is the crux of why I
> think we
> > > need to finalize the move to bro 2.5.2 and the plugin packaging
> before
> > our
> > > next release (working on it as we speak).
> > > Jon
> >
> >
> > ​I replayed Jon's comments from the other thread above.​
> >
> > My initial thought, is that I would not want to manage two separate
> release
> > processes. I don't want to have a roll call, cut release candidates
> and
> > test both.
> >
> > I was thinking we would just need to change some of the
> behind-the-

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

Groups Not Setup Correctly
> >> (nickwallen) closes apache/metron#759
> >>METRON-1185: Stellar REPL does not work on a kerberized cluster when
> >> calling functions interacting with HBase closes
> apache/incubator-metron#755
> >>METRON-1186: Profiler Functions use classutils from shaded storm
> >> closes apache/incubator-metron#758
> >>METRON-1173: Fix pointers to old stellar docs closes
> >> apache/incubator-metron#746
> >>METRON-1179: Make STATS_ADD to take a list closes
> >> apache/incubator-metron#750
> >>METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list
> >> and not require a port closes apache/incubator-metron#751
> >>METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
> >> apache/metron#753
> >>METRON-1177 Stale running topologies seen post-kerberization and
> cause
> >> exceptions (nickwallen) closes apache/metron#748
> >>METRON-1158 Build backend for grouping alerts into meta alerts
> >> (justinleet) closes apache/metron#734
> >>METRON-1146: Add ability to parse JSON string into JSONObject for
> >> stellar closes apache/incubator-metron#727
> >>METRON-1176 REST: HDFS Service should support setting permissions on
> >> files when writing (ottobackwards) closes apache/metron#749
> >>METRON-1114 Add group by capabilities to search REST endpoint
> >> (merrimanr) closes apache/metron#702
> >>METRON-1167 Define Session Specific Global Configuration Values in
> the
> >> REPL (nickwallen) closes apache/metron#740
> >>METRON-1171: Better validation for the SUBSTRING stellar function
> >> closes apache/incubator-metron#745
> >>
> >>
> >>
> >> On 11/17/17, 11:59 AM, "Nick Allen" <n...@nickallen.org> wrote:
> >>
> >>I just wanted to send an update on where we are at.  We've gotten a
> lot
> >>done here recently as you can see below.
> >>
> >>  ✓ DONE (1) First, METRON-1289 needs to go in.  This one was a
> fairly
> >> big
> >>effort and I am hearing that we are pretty close.
> >>
> >>  ✓ DONE (2) METRON-1294 fixes an issue in how field types are
> >> looked-up.
> >>
> >>  ✓ DONE (3) METRON-1290 is next.  While this may have been fixed in
> >>M-1289, there may be some test cases we want from this PR.
> >>
> >>  ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
> >>
> >>  ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> metaalerts.
> >>
> >>  (6) That leads us to Raghu's UI work in METRON-1252.  This
> >> introduces the
> >>UI bits that depend on all the previous backend work.
> >>
> >>  (7) At this point, we should have our best effort at running
> >> Metaalerts
> >>on Elasticsearch 2.x. I propose that we cut a release here.
> >>
> >>  (8) After we cut the release, we can introduce the work for ES 5.x
> in
> >>METRON-939.  I know we will need lots of help testing and reviewing
> >> this
> >>one.
> >>
> >>
> >>
> >>We also have an outstanding question that needs resolved BEFORE we
> >>release.  We need to come to a consensus on how to release having
> >> moved our
> >>Bro Plugin to a separate repo.  I don't think we've heard from
> >> everyone on
> >>this.  I'd urge everyone to chime in so we can choose a path forward.
> >>
> >>If anyone is totally confused in regards to that discussion, I can
> try
> >> and
> >>send an options summary again as a separate discuss thread.  The
> >> original
> >>chain was somewhere around here [1].
> >>
> >>[1]
> >>https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e9
> >> 23a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
> >>
> >>
> >>
> >>On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <n...@nickallen.org>
> >> wrote:
> >>
> >>> Hi Guys -
> >>>
> >>> I want to follow-up on this discussion.  It sounds like most people
> >> are in
> >>> agreement with the general approach.
> >>>
> >>> A lot of people have been working hard on Metaalerts and
> >> Elasticsearch.  I
> >>> have checked-in with those doing the heavy lifting and have compiled
> >> a mo

master full-dev issues?

[zeo...@gmail.com]

Anybody else having issues spinning up full-dev?  I'm consistently failing
on the Metron Alerts UI install.  Spun it up fine yesterday for my other
testing.

2017-11-16 17:57:41,772 - Execution of '/usr/bin/yum -d 0 -e 0 -y
install metron-common' returned 1. Error: Nothing to do


https://gist.github.com/JonZeolla/3923f93a3e31c8064c4d77db89431106

Jon
-- 

Jon

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

I expect a few version changes up front to add some new features to the
package (0.1 for the initial release, 0.{2..n} for some new features, 1.0
when we stabilize) but after that it will probably only be updated to
follow kafka/librdkafka updates.

Jon

On Thu, Nov 16, 2017 at 10:10 AM Otto Fowler 
wrote:

> How often to we expect to change this?  If it is effectively pinned then a
> release process is not that bad.
>
>
> On November 16, 2017 at 10:06:53, Nick Allen (n...@nickallen.org) wrote:
>
> >
> > I would suggest that we institute a release procedure for the package
> > itself, but I don't think it necessarily has to line up with metron
> > releases (happy to be persuaded otherwise). Then we can just link metron
> > to metron-bro-plugin-kafka by pointing to specific
> > metron-bro-plugin-kafka releases (git tags
> > <
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> > versioning>
> > ).
> > Right now, full-dev spins up against the
> > apache/metron-bro-plugin-kafka master branch, which is not a good idea to
> > have in place for an upcoming release. That is the crux of why I think we
> > need to finalize the move to bro 2.5.2 and the plugin packaging before
> our
> > next release (working on it as we speak).
> > Jon
>
>
> ​I replayed Jon's comments from the other thread above.​
>
> My initial thought, is that I would not want to manage two separate release
> processes. I don't want to have a roll call, cut release candidates and
> test both.
>
> I was thinking we would just need to change some of the behind-the-scenes
> processes handled by the release manager. This is one area where I had
> thought using a submodule in Git would help.
>
>
>
>
>
>
>
>
>
> On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen  wrote:
>
> > + Restarting the thread to include mentors.
> >
> > The code of the 'Kafka Plugin for Bro' is now maintained in the external
> > repository that we set up a while back.
> >
> > - Metron Core: git://git.apache.org/metron.git
> > - Kafka Plugin for Bro: git://git.apache.org/
> > metron-bro-plugin-kafka.git
> >
> > (Q) Do we need to change anything in the release procedure to account for
> > this?
> >
>
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

The problem is that we're currently pinning to master and if something in
the plugin breaks backward compatibility, our prior releases will be
perpetually broken, which is why I suggest it blocks the upcoming release.

The alternative would be to update ansible to pin to a specific commit or
to make a release in apache/metron-bro-plugin-kafka sooner rather than
later and pin to its branch/tag.  That feels like a waste of time though,
as the 2.5.2 upgrade is somewhat trivial.

Jon

On Thu, Nov 16, 2017 at 10:14 AM Nick Allen <n...@nickallen.org> wrote:

> While I think that the Bro work is super valuable, Jon, I am not sure that
> we need to block the next release for it.  In my mind, the "theme" of the
> next release is Metaalerts running on ES 2.x.  I'd prefer to just stick
> with that.
>
> I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs merged
> and start cutting release candidates ASAP.  That could be possible by end
> of week based on the "big one" (METRON-1289) getting merged in last night.
>
> Of course, if you happen to get all the Bro work done in time, then
> definitely let's include it in the next release.  But I am wary of blocking
> the release for that work.  No need for you to rush through it.
>
> Just one man's opinion.  Would like to hear feedback from more of the
> community.
>
>
>
> On Thu, Nov 16, 2017 at 8:01 AM, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
> > The way master's full-dev is set up right now is non optimal for the bro
> > plugin configuration, and I would like to complete the roadmap I outlined
> > in my discuss thread before a release.  I have a PR open against
> > Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
> > expect it will take me until end of next week at the latest to have the
> > rest of the work done to move us to 2.5.2, and to pin to a specific
> package
> > version.  At that point I'm comfortable with a release.
> >
> > Jon
> >
> > On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
> >
> > > I’ve been listening.  Looks like there are still a number of major
> issues
> > > to be committed first, right?
> > > The discussion on this thread constitutes sufficient engagement, I
> think,
> > > especially given the Subject line :-)
> > > Would the folks working on the 6 issues listed by Nick care to suggest
> a
> > > cut-off date by which they’ll probably have those fixes in?
> > > I’ll be happy to run the release process, if the community so wishes,
> as
> > > soon as those issues are committed.
> > >
> > > I guess I should, pro forma, send the list of commits already in since
> > the
> > > last release.  I’ll do that today.
> > > Also, if anyone wishes to raise a hand and propose additional commits
> are
> > > needed, please do so on this thread.
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 11/15/17, 2:09 PM, "Casey Stella" <ceste...@gmail.com> wrote:
> > >
> > > I'd say that if a release is this imminent that we had better
> notify
> > > the
> > > release manager who will make a release announcement, Nick.  Matt,
> > are
> > > you
> > > tuning in to this?
> > >
> > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <n...@nickallen.org>
> > > wrote:
> > >
> > > > Hi Guys -
> > > >
> > > > I want to follow-up on this discussion.  It sounds like most
> people
> > > are in
> > > > agreement with the general approach.
> > > >
> > > > A lot of people have been working hard on Metaalerts and
> > > Elasticsearch.  I
> > > > have checked-in with those doing the heavy lifting and have
> > compiled
> > > a more
> > > > detailed plan based on where we are at now.  To the best of my
> > > knowledge
> > > > here is the plan of attack for finishing out this effort.
> > > >
> > > >   (1) First, METRON-1289 needs to go in.  This one was a fairly
> big
> > > effort
> > > > and I am hearing that we are pretty close.
> > > >
> > > >   (2) METRON-1294 fixes an issue in how field types are
> looked-up.
> > > >
> > > >   (3) METRON-1290 is next.  While this may have been fixed in
> > > M-1289, there
> > > > may be some test cases we want from this PR.
> > > >
> > > 

Re: [DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[zeo...@gmail.com]

I would suggest that we institute a release procedure for the package
itself, but I don't think it necessarily has to line up with metron
releases (happy to be persuaded otherwise).  Then we can just link metron
to metron-bro-plugin-kafka by pointing to specific
metron-bro-plugin-kafka releases (git tags

).

Right now, full-dev spins up against the
apache/metron-bro-plugin-kafka master branch, which is not a good idea to
have in place for an upcoming release.  That is the crux of why I think we
need to finalize the move to bro 2.5.2 and the plugin packaging before our
next release (working on it as we speak).

Jon

On Thu, Nov 16, 2017 at 9:10 AM Nick Allen  wrote:

> The code of the 'Kafka Plugin for Bro' is now maintained in the external
> repository that we set up a while back.  Do we need to change anything in
> the release procedure to account for this?
>
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

My PR is to turn it into a package containing a plugin*

On Thu, Nov 16, 2017, 08:01 zeo...@gmail.com <zeo...@gmail.com> wrote:

> The way master's full-dev is set up right now is non optimal for the bro
> plugin configuration, and I would like to complete the roadmap I outlined
> in my discuss thread before a release.  I have a PR open against
> Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
> expect it will take me until end of next week at the latest to have the
> rest of the work done to move us to 2.5.2, and to pin to a specific package
> version.  At that point I'm comfortable with a release.
>
> Jon
>
> On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:
>
>> I’ve been listening.  Looks like there are still a number of major issues
>> to be committed first, right?
>> The discussion on this thread constitutes sufficient engagement, I think,
>> especially given the Subject line :-)
>> Would the folks working on the 6 issues listed by Nick care to suggest a
>> cut-off date by which they’ll probably have those fixes in?
>> I’ll be happy to run the release process, if the community so wishes, as
>> soon as those issues are committed.
>>
>> I guess I should, pro forma, send the list of commits already in since
>> the last release.  I’ll do that today.
>> Also, if anyone wishes to raise a hand and propose additional commits are
>> needed, please do so on this thread.
>> Thanks,
>> --Matt
>>
>>
>> On 11/15/17, 2:09 PM, "Casey Stella" <ceste...@gmail.com> wrote:
>>
>> I'd say that if a release is this imminent that we had better notify
>> the
>> release manager who will make a release announcement, Nick.  Matt,
>> are you
>> tuning in to this?
>>
>> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <n...@nickallen.org>
>> wrote:
>>
>> > Hi Guys -
>> >
>> > I want to follow-up on this discussion.  It sounds like most people
>> are in
>> > agreement with the general approach.
>> >
>> > A lot of people have been working hard on Metaalerts and
>> Elasticsearch.  I
>> > have checked-in with those doing the heavy lifting and have
>> compiled a more
>> > detailed plan based on where we are at now.  To the best of my
>> knowledge
>> > here is the plan of attack for finishing out this effort.
>> >
>> >   (1) First, METRON-1289 needs to go in.  This one was a fairly big
>> effort
>> > and I am hearing that we are pretty close.
>> >
>> >   (2) METRON-1294 fixes an issue in how field types are looked-up.
>> >
>> >   (3) METRON-1290 is next.  While this may have been fixed in
>> M-1289, there
>> > may be some test cases we want from this PR.
>> >
>> >   (4) METRON-1301 addresses a problem with the sorting logic.
>> >
>> >   (5) METRON-1291 fixes an issue with escalation of metaalerts.
>> >
>> >   (6) That leads us to Raghu's UI work in METRON-1252.  This
>> introduces the
>> > UI bits that depend on all the previous backend work.
>> >
>> >   (7) At this point, we should have our best effort at running
>> Metaalerts
>> > on Elasticsearch 2.x. I propose that we cut a release here.
>> >
>> >   (8) After we cut the release, we can introduce the work for ES
>> 5.x in
>> > METRON-939.  I know we will need lots of help testing and reviewing
>> this
>> > one.
>> >
>> > Please correct me if I am wrong.  I will try and send out updates
>> as we
>> > make progress.
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Nov 6, 2017 at 1:03 PM, zeo...@gmail.com <zeo...@gmail.com>
>> wrote:
>> >
>> > > I agree, I think it's very reasonable to move in line with Nick's
>> > > proposal.  I would also suggest that we outline what the target
>> versions
>> > > would be to add in the METRON-777 components, since it has been
>> > functional
>> > > for a very long time but not reviewed and has some really rockstar
>> > > improvements.
>> > >
>> > > Jon
>> > >
>> > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
>> ottobackwa...@gmail.com>
>> > > wrote:
>> > >
>>  

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

The way master's full-dev is set up right now is non optimal for the bro
plugin configuration, and I would like to complete the roadmap I outlined
in my discuss thread before a release.  I have a PR open against
Apache/metron-bro-plugin-kafka right now to turn it into a plugin, and I
expect it will take me until end of next week at the latest to have the
rest of the work done to move us to 2.5.2, and to pin to a specific package
version.  At that point I'm comfortable with a release.

Jon

On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote:

> I’ve been listening.  Looks like there are still a number of major issues
> to be committed first, right?
> The discussion on this thread constitutes sufficient engagement, I think,
> especially given the Subject line :-)
> Would the folks working on the 6 issues listed by Nick care to suggest a
> cut-off date by which they’ll probably have those fixes in?
> I’ll be happy to run the release process, if the community so wishes, as
> soon as those issues are committed.
>
> I guess I should, pro forma, send the list of commits already in since the
> last release.  I’ll do that today.
> Also, if anyone wishes to raise a hand and propose additional commits are
> needed, please do so on this thread.
> Thanks,
> --Matt
>
>
> On 11/15/17, 2:09 PM, "Casey Stella" <ceste...@gmail.com> wrote:
>
> I'd say that if a release is this imminent that we had better notify
> the
> release manager who will make a release announcement, Nick.  Matt, are
> you
> tuning in to this?
>
> On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <n...@nickallen.org>
> wrote:
>
> > Hi Guys -
> >
> > I want to follow-up on this discussion.  It sounds like most people
> are in
> > agreement with the general approach.
> >
> > A lot of people have been working hard on Metaalerts and
> Elasticsearch.  I
> > have checked-in with those doing the heavy lifting and have compiled
> a more
> > detailed plan based on where we are at now.  To the best of my
> knowledge
> > here is the plan of attack for finishing out this effort.
> >
> >   (1) First, METRON-1289 needs to go in.  This one was a fairly big
> effort
> > and I am hearing that we are pretty close.
> >
> >   (2) METRON-1294 fixes an issue in how field types are looked-up.
> >
> >   (3) METRON-1290 is next.  While this may have been fixed in
> M-1289, there
> > may be some test cases we want from this PR.
> >
> >   (4) METRON-1301 addresses a problem with the sorting logic.
> >
> >   (5) METRON-1291 fixes an issue with escalation of metaalerts.
> >
> >   (6) That leads us to Raghu's UI work in METRON-1252.  This
> introduces the
> > UI bits that depend on all the previous backend work.
> >
> >   (7) At this point, we should have our best effort at running
> Metaalerts
> > on Elasticsearch 2.x. I propose that we cut a release here.
> >
> >   (8) After we cut the release, we can introduce the work for ES 5.x
> in
> > METRON-939.  I know we will need lots of help testing and reviewing
> this
> > one.
> >
> > Please correct me if I am wrong.  I will try and send out updates as
> we
> > make progress.
> >
> >
> >
> >
> >
> > On Mon, Nov 6, 2017 at 1:03 PM, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
> >
> > > I agree, I think it's very reasonable to move in line with Nick's
> > > proposal.  I would also suggest that we outline what the target
> versions
> > > would be to add in the METRON-777 components, since it has been
> > functional
> > > for a very long time but not reviewed and has some really rockstar
> > > improvements.
> > >
> > > Jon
> > >
> > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> ottobackwa...@gmail.com>
> > > wrote:
> > >
> > > > I think the ES cutover should be the start of the 0.5.x series,
> and we
> > > > continue on with 0.4.x for the
> > > > metadata improvements etc.  We could chose to focus 0.5.x’s first
> > > releases
> > > > on not only ES but
> > > > getting a handle on kibana and the mpack situation as well.
> > > >
> > > >
> > > >
> > > >
> > > > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>

Assign METRON-1307 to Brian Hurley and close

[zeo...@gmail.com]

I'm unable to find Brian Hurley in the list of assignees, but he was the
one who contributed the fix[1].  Can someone assign and close this JIRA?
Thanks,

Jon

1:  https://github.com/apache/metron/pull/835
-- 

Jon

Re: Committing to the metron-bro-plugin-kafka repo

[zeo...@gmail.com]

Whups, here is a correctly updated roadmap.  Also, the METRON-1309 PR is
out and I've completed my testing so it's ready for review:

1.  *DONE* - Move the bro-plugin-kafka to its own repository (Prereq
to METRON-813).
2.  *PR OPEN* - Update metron-deployment to pull the plugin from
apache/metron-bro-plugin-kafka (METRON-1309).
3.  *PR OPEN* - Reorganize the plugin naming to be more appropriate for a
package (METRON-1303, Prereq to METRON-813).
4.  Transform the plugin to be a package containing a plugin (METRON-813).
5.  Add the package to bro/packages[4].
During 2-5.  Upgrade full-dev to run on CentOS 7 (METRON-559, soft prereq
to METRON-1088).
6.  Upgrade bro to 2.5.2 (METRON-1088).
7.  Update metron-deployment to install the package instead of using the
plugin in apache/metron.
8.  Add features/improvements (METRON-1304, METRON-1305, etc.) to
apache/metron-bro-plugin-kafka.

Jon

On Wed, Nov 8, 2017 at 2:57 PM zeo...@gmail.com <zeo...@gmail.com> wrote:

> I'm not strongly against it, but my biggest interest was not wasting time
> doing something that will get ripped out fairly quickly.  That said,
> discussing this is taking more time than doing the work, and I should have
> a PR out soon that does what you requested as soon as my full-dev tests
> complete.  New roadmap:
>
> 1.  *DONE* - Move the bro-plugin-kafka to its own repository (Prereq
> to METRON-813).
> 2.  Update metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (METRON-1309).
> 3.  *PR OPEN* - Reorganize the plugin naming to be more appropriate for a
> package (METRON-1303, Prereq to METRON-813).
> 4.  Transform the plugin to be a package containing a plugin (METRON-813).
> 5.  Add the package to bro/packages[4].
> During 2-5.  Upgrade full-dev to run on CentOS 7 (METRON-559, soft prereq
> to METRON-1088).
> 6.  Upgrade bro to 2.5.2 (METRON-1088).
> 7.  Update metron-deployment to install the package instead of using the
> plugin in apache/metron.
> 8.  Remove the plugin[1] from apache/metron entirely.
> 9.  Add features/improvements (METRON-1304, METRON-1305, etc.) to
> apache/metron-bro-plugin-kafka.
>
> Jon
>
> On Wed, Nov 8, 2017 at 12:59 PM Nick Allen <n...@nickallen.org> wrote:
>
>> Let me put the sub-module discussion aside for a second.  I don't really
>> have much of an opinion on that quite yet.  I need to research it a bit
>> more, but thanks for sharing the pros/cons.
>>
>> My main point is that I would like to see us update our deployment code to
>> deploy the Bro plugin from the new code repository BEFORE we make any
>> changes to the code.  So the next bit of work would do the following:
>>  * Remove the legacy code from `metron/metron-sensors/bro-kafka-plugin`
>>  * Update `metron/metron-deployment/roles/bro` so that it builds and
>> deploys the plugin to Full Dev from the new repository
>>
>> What do you think?
>>
>>
>>
>> On Wed, Nov 8, 2017 at 11:00 AM zeo...@gmail.com <zeo...@gmail.com>
>> wrote:
>>
>> > So, here's my argument against the sub-module approach:
>> > - If we add a sub-module into apache/metron then the way you clone from
>> > github changes (--recursive).  This could break any instructions to
>> spin up
>> > Metron full-dev, when they're using the bro plugin (forums, blog posts,
>> > metron READMEs, etc.).
>> > - This also requires that we commit a change to apache/metron in order
>> to
>> > make any changes to apache/metron-bro-plugin-kafka effective in the
>> > project.  Not that big of a deal, but notable.
>> > - The overhead of these two changes at once is very minimal, since
>> > packaging is effectively moving the same commands we run in ansible to
>> be
>> > run by bro-pkg.
>> >
>> > If we follow the approach I lined up before we can version the package
>> in
>> > bro-pkg.meta, and when our scripts are used spin up bro we can just pin
>> a
>> > specific version of the package, and bump it up in apache/metron as we
>> want
>> > to use updates of apache/metron-bro-plugin-kafka.  That said, this is
>> > implying that we're okay making two changes at once with my method -
>> use an
>> > external repo for the sensor, and move to a package from a plugin.
>> Since
>> > moving from a package to a plugin is literally just the following
>> commands,
>> > I'm not too concerned:
>> >  - Clone the package repo and checkout to the right branch, if
>> applicable
>> >  - Run the built-in unit tests.  If they fail, alert the user and get
>> > confirmation if they want to move forward
>> >  - Run the configure a

Re: Committing to the metron-bro-plugin-kafka repo

[zeo...@gmail.com]

I'm not strongly against it, but my biggest interest was not wasting time
doing something that will get ripped out fairly quickly.  That said,
discussing this is taking more time than doing the work, and I should have
a PR out soon that does what you requested as soon as my full-dev tests
complete.  New roadmap:

1.  *DONE* - Move the bro-plugin-kafka to its own repository (Prereq
to METRON-813).
2.  Update metron-deployment to pull the plugin from
apache/metron-bro-plugin-kafka (METRON-1309).
3.  *PR OPEN* - Reorganize the plugin naming to be more appropriate for a
package (METRON-1303, Prereq to METRON-813).
4.  Transform the plugin to be a package containing a plugin (METRON-813).
5.  Add the package to bro/packages[4].
During 2-5.  Upgrade full-dev to run on CentOS 7 (METRON-559, soft prereq
to METRON-1088).
6.  Upgrade bro to 2.5.2 (METRON-1088).
7.  Update metron-deployment to install the package instead of using the
plugin in apache/metron.
8.  Remove the plugin[1] from apache/metron entirely.
9.  Add features/improvements (METRON-1304, METRON-1305, etc.) to
apache/metron-bro-plugin-kafka.

Jon

On Wed, Nov 8, 2017 at 12:59 PM Nick Allen <n...@nickallen.org> wrote:

> Let me put the sub-module discussion aside for a second.  I don't really
> have much of an opinion on that quite yet.  I need to research it a bit
> more, but thanks for sharing the pros/cons.
>
> My main point is that I would like to see us update our deployment code to
> deploy the Bro plugin from the new code repository BEFORE we make any
> changes to the code.  So the next bit of work would do the following:
>  * Remove the legacy code from `metron/metron-sensors/bro-kafka-plugin`
>  * Update `metron/metron-deployment/roles/bro` so that it builds and
> deploys the plugin to Full Dev from the new repository
>
> What do you think?
>
>
>
> On Wed, Nov 8, 2017 at 11:00 AM zeo...@gmail.com <zeo...@gmail.com> wrote:
>
> > So, here's my argument against the sub-module approach:
> > - If we add a sub-module into apache/metron then the way you clone from
> > github changes (--recursive).  This could break any instructions to spin
> up
> > Metron full-dev, when they're using the bro plugin (forums, blog posts,
> > metron READMEs, etc.).
> > - This also requires that we commit a change to apache/metron in order to
> > make any changes to apache/metron-bro-plugin-kafka effective in the
> > project.  Not that big of a deal, but notable.
> > - The overhead of these two changes at once is very minimal, since
> > packaging is effectively moving the same commands we run in ansible to be
> > run by bro-pkg.
> >
> > If we follow the approach I lined up before we can version the package in
> > bro-pkg.meta, and when our scripts are used spin up bro we can just pin a
> > specific version of the package, and bump it up in apache/metron as we
> want
> > to use updates of apache/metron-bro-plugin-kafka.  That said, this is
> > implying that we're okay making two changes at once with my method - use
> an
> > external repo for the sensor, and move to a package from a plugin.  Since
> > moving from a package to a plugin is literally just the following
> commands,
> > I'm not too concerned:
> >  - Clone the package repo and checkout to the right branch, if applicable
> >  - Run the built-in unit tests.  If they fail, alert the user and get
> > confirmation if they want to move forward
> >  - Run the configure and make commands
> >
> > You can see an aggregate of all bro package configs here
> > <https://github.com/bro/packages/blob/master/aggregate.meta>, which show
> > different scenarios/examples of what needs to be added to make something
> > into a bro package - it is rather straightfoward.  Happy to continue
> > discussing.
> >
> > Jon
> >
> > On Wed, Nov 8, 2017 at 9:34 AM Nick Allen <n...@nickallen.org> wrote:
> >
> > > ​I would suggest that we shift our existing deployment routines to use
> > the
> > > new code repository, prior to making changes to "packag-ify" it.  That
> > way
> > > we know that our code reorganization is definitely working and has been
> > > successful.
> > >
> > > Prior to step 2, we would...
> > >* Add a sub-module pointing to the repo and ensure that the Ansible
> > > deployment to Full Dev can deploy Bro with the Kafka plugin
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Nov 7, 2017 at 9:19 AM, zeo...@gmail.com <zeo...@gmail.com>
> > wrote:
> > >
> > > > So here's an update on this, and I'm looking for any suggestions
> > > regarding
>

Re: Committing to the metron-bro-plugin-kafka repo

[zeo...@gmail.com]

So, here's my argument against the sub-module approach:
- If we add a sub-module into apache/metron then the way you clone from
github changes (--recursive).  This could break any instructions to spin up
Metron full-dev, when they're using the bro plugin (forums, blog posts,
metron READMEs, etc.).
- This also requires that we commit a change to apache/metron in order to
make any changes to apache/metron-bro-plugin-kafka effective in the
project.  Not that big of a deal, but notable.
- The overhead of these two changes at once is very minimal, since
packaging is effectively moving the same commands we run in ansible to be
run by bro-pkg.

If we follow the approach I lined up before we can version the package in
bro-pkg.meta, and when our scripts are used spin up bro we can just pin a
specific version of the package, and bump it up in apache/metron as we want
to use updates of apache/metron-bro-plugin-kafka.  That said, this is
implying that we're okay making two changes at once with my method - use an
external repo for the sensor, and move to a package from a plugin.  Since
moving from a package to a plugin is literally just the following commands,
I'm not too concerned:
 - Clone the package repo and checkout to the right branch, if applicable
 - Run the built-in unit tests.  If they fail, alert the user and get
confirmation if they want to move forward
 - Run the configure and make commands

You can see an aggregate of all bro package configs here
<https://github.com/bro/packages/blob/master/aggregate.meta>, which show
different scenarios/examples of what needs to be added to make something
into a bro package - it is rather straightfoward.  Happy to continue
discussing.

Jon

On Wed, Nov 8, 2017 at 9:34 AM Nick Allen <n...@nickallen.org> wrote:

> ​I would suggest that we shift our existing deployment routines to use the
> new code repository, prior to making changes to "packag-ify" it.  That way
> we know that our code reorganization is definitely working and has been
> successful.
>
> Prior to step 2, we would...
>* Add a sub-module pointing to the repo and ensure that the Ansible
> deployment to Full Dev can deploy Bro with the Kafka plugin
>
>
>
>
>
> On Tue, Nov 7, 2017 at 9:19 AM, zeo...@gmail.com <zeo...@gmail.com> wrote:
>
> > So here's an update on this, and I'm looking for any suggestions
> regarding
> > the roadmap.  If anybody isn't clear about the implementation specifics
> or
> > history here, I'm happy to reiterate, but it has also been discussed on
> the
> > mailing list in the past.
> >
> > Right now, we have a direct migration of metron-sensors/bro-plugin-
> > kafka[1]
> > to its own repository[2], which is required in order to turn it into a
> bro
> > package (the preferred mechanism of managing and installing bro plugins
> as
> > of 2.5).  Bro 2.5 also has an improvement[3] which I think we should
> > consider important to take advantage of in our testing environments and
> > instructions.  Here is a more well-defined roadmap suggestion:
> >
> > 1.  *DONE* - Move the bro-plugin-kafka to its own repository (Prereq
> > to METRON-813).
> > 2.  *PR OPEN* - Reorganize the plugin naming to be more appropriate for a
> > package (METRON-1303, Prereq to METRON-813).
> > 3.  Transform the plugin to be a package containing a plugin
> (METRON-813).
> > 4.  Add the package to bro/packages[4].
> > During 2-4.  Upgrade full-dev to run on CentOS 7 (METRON-559, soft prereq
> > to METRON-1088).
> > 5.  Upgrade bro to 2.5.2 (METRON-1088).
> > 6.  Update metron-deployment to install the package instead of using the
> > plugin in apache/metron.
> > 7.  Remove the plugin[1] from apache/metron entirely.
> > 8.  Add features/improvements (METRON-1304, METRON-1305, etc.) to
> > apache/metron-bro-plugin-kafka.
> >   - This is why my METRON-1304 PR has "DO NOT MERGE" - it was simply an
> > easy win to bust out this morning while I was thinking of it.
> >
> > Some alternatives we may want to consider:
> > - In the interim, we could change our ansible scripts to pull in the code
> > from the external repository, but build it the same as we currently do
> (as
> > a plugin but not a package).
> > - We could replace bro-plugin-kafka folder with a sub-module pointing to
> > the external repo, but this may be more trouble than it's worth (new
> > cloning instructions, new folder name, etc.).
> > - We can try to work around some of the issues with running Bro 2.5 on
> > CentOS 6, but it would be less preferred.
> >
> > Any comments are welcome.
> >
> > 1:
> > https://github.com/apache/metron/tree/master/metron-
> > sensors/bro-plugin-kafka
> >

Re: Committing to the metron-bro-plugin-kafka repo

[zeo...@gmail.com]

So here's an update on this, and I'm looking for any suggestions regarding
the roadmap.  If anybody isn't clear about the implementation specifics or
history here, I'm happy to reiterate, but it has also been discussed on the
mailing list in the past.

Right now, we have a direct migration of metron-sensors/bro-plugin-kafka[1]
to its own repository[2], which is required in order to turn it into a bro
package (the preferred mechanism of managing and installing bro plugins as
of 2.5).  Bro 2.5 also has an improvement[3] which I think we should
consider important to take advantage of in our testing environments and
instructions.  Here is a more well-defined roadmap suggestion:

1.  *DONE* - Move the bro-plugin-kafka to its own repository (Prereq
to METRON-813).
2.  *PR OPEN* - Reorganize the plugin naming to be more appropriate for a
package (METRON-1303, Prereq to METRON-813).
3.  Transform the plugin to be a package containing a plugin (METRON-813).
4.  Add the package to bro/packages[4].
During 2-4.  Upgrade full-dev to run on CentOS 7 (METRON-559, soft prereq
to METRON-1088).
5.  Upgrade bro to 2.5.2 (METRON-1088).
6.  Update metron-deployment to install the package instead of using the
plugin in apache/metron.
7.  Remove the plugin[1] from apache/metron entirely.
8.  Add features/improvements (METRON-1304, METRON-1305, etc.) to
apache/metron-bro-plugin-kafka.
  - This is why my METRON-1304 PR has "DO NOT MERGE" - it was simply an
easy win to bust out this morning while I was thinking of it.

Some alternatives we may want to consider:
- In the interim, we could change our ansible scripts to pull in the code
from the external repository, but build it the same as we currently do (as
a plugin but not a package).
- We could replace bro-plugin-kafka folder with a sub-module pointing to
the external repo, but this may be more trouble than it's worth (new
cloning instructions, new folder name, etc.).
- We can try to work around some of the issues with running Bro 2.5 on
CentOS 6, but it would be less preferred.

Any comments are welcome.

1:
https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka
2:  https://github.com/apache/metron-bro-plugin-kafka
3:  https://www.bro.org/sphinx/cluster/index.html#logger
4:  https://github.com/bro/packages

On Tue, Nov 7, 2017 at 8:52 AM Nick Allen <n...@nickallen.org> wrote:

> > As of bro-pkg 1.1, I need to redo my `bro-pkg.meta` work to support the
> newly-added
> `external_depends`, and also upgrade to bro 2.5.2
>
> Isn't upgrading to 2.5.2 an enhancement that we need to wait on before we
> finish some clean-up tasks?
>
> What all do you think we need to do before we start accepting enhancements?
>
> Thanks for the update and all the hard work, Jon.
>
> On Mon, Nov 6, 2017 at 10:02 PM, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
> > Sorry for the delay here - I pushed this out tonight (link
> > <https://github.com/apache/metron-bro-plugin-kafka/commits/master>, link
> > <https://git-wip-us.apache.org/repos/asf?p=metron-bro-plugin-kafka.git
> >).
> > As of bro-pkg 1.1, I need to redo my `bro-pkg.meta` work to support the
> > newly-added `external_depends`, and also upgrade to bro 2.5.2 (somewhat
> > non-trivial due to the C++11 requirement, and new bro log files/fields)
> so
> > we can use the bro package manager to install the plugin.  Hopefully I
> can
> > get this wrapped up soon so we can accept external PRs like this one
> > <https://github.com/JonZeolla/metron-bro-plugin-kafka/pull/1>.
> >
> > Jon
> >
> > On Mon, Sep 18, 2017 at 11:52 AM Nick Allen <n...@nickallen.org> wrote:
> >
> > > Nice!  Looks good to me.
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Sep 18, 2017 at 11:35 AM zeo...@gmail.com <zeo...@gmail.com>
> > > wrote:
> > >
> > > > Okay, I took a stab at it this morning, can I get a double check
> before
> > > > pushing it out?  The latest commit would be opened as a PR.
> > > >
> > > > https://github.com/JonZeolla/metron-bro-plugin-kafka/tree/dev
> > > >
> > > > Jon
> > > >
> > > > On Fri, Sep 15, 2017 at 12:54 PM zeo...@gmail.com <zeo...@gmail.com>
> > > > wrote:
> > > >
> > > > > Good point, I can take that task re migrating the revision history
> of
> > > the
> > > > > folder.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Fri, Sep 15, 2017, 12:14 Nick Allen <n...@nickallen.org> wrote:
> > > > >
> > > > >> Hi Jon -
> > > > >>
> > > > >> I agree with 

Re: Committing to the metron-bro-plugin-kafka repo

[zeo...@gmail.com]

Sorry for the delay here - I pushed this out tonight (link
<https://github.com/apache/metron-bro-plugin-kafka/commits/master>, link
<https://git-wip-us.apache.org/repos/asf?p=metron-bro-plugin-kafka.git>).
As of bro-pkg 1.1, I need to redo my `bro-pkg.meta` work to support the
newly-added `external_depends`, and also upgrade to bro 2.5.2 (somewhat
non-trivial due to the C++11 requirement, and new bro log files/fields) so
we can use the bro package manager to install the plugin.  Hopefully I can
get this wrapped up soon so we can accept external PRs like this one
<https://github.com/JonZeolla/metron-bro-plugin-kafka/pull/1>.

Jon

On Mon, Sep 18, 2017 at 11:52 AM Nick Allen <n...@nickallen.org> wrote:

> Nice!  Looks good to me.
>
>
>
>
>
>
> On Mon, Sep 18, 2017 at 11:35 AM zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
> > Okay, I took a stab at it this morning, can I get a double check before
> > pushing it out?  The latest commit would be opened as a PR.
> >
> > https://github.com/JonZeolla/metron-bro-plugin-kafka/tree/dev
> >
> > Jon
> >
> > On Fri, Sep 15, 2017 at 12:54 PM zeo...@gmail.com <zeo...@gmail.com>
> > wrote:
> >
> > > Good point, I can take that task re migrating the revision history of
> the
> > > folder.
> > >
> > > Jon
> > >
> > > On Fri, Sep 15, 2017, 12:14 Nick Allen <n...@nickallen.org> wrote:
> > >
> > >> Hi Jon -
> > >>
> > >> I agree with you on the approach.  We should first copy everything as
> it
> > >> is
> > >> to the new repo.  We should maintain the revision history too.  I'm
> sure
> > >> there is a way to do it, but would have to research a bit.  Then we
> > apply
> > >> your changes on top of that.
> > >>
> > >> Thanks
> > >>
> > >> On Thu, Sep 14, 2017 at 1:36 AM, zeo...@gmail.com <zeo...@gmail.com>
> > >> wrote:
> > >>
> > >> > So, I've been working on METRON-813
> > >> > <https://issues.apache.org/jira/browse/METRON-813> lately and I
> have
> > an
> > >> > initial run at it ready to go here
> > >> > <https://github.com/JonZeolla/metron-bro-plugin-kafka> (squashed
> > >> history,
> > >> > see a better history there
> > >> > <
> https://github.com/JonZeolla/metron-bro-plugin-kafka/commits/bro-pkg
> > >> >).
> > >> > Since the metron-bro-plugin-kafka repo is empty, I can't open a PR
> > >> against
> > >> > it on GitHub for review.  Does anybody have a suggestion regarding
> how
> > >> to
> > >> > move forward?  I see two options:
> > >> > 1. I make the initial commit a direct copy of the bro-plugin-kafka
> > >> folder
> > >> > <https://github.com/apache/metron/tree/master/metron-
> > >> > sensors/bro-plugin-kafka>
> > >> > (I believe this would require a new JIRA for a direct copy), and
> then
> > >> open
> > >> > a PR for the METRON-813 changes to get reviewed via the normal
> > process.
> > >> > 2. I make the initial commit the result of METRON-813, but review
> > occurs
> > >> > via the mailing list and using my fork.
> > >> >
> > >> > I prefer 1, but wanted to put it up for discussion.  Once we decide
> on
> > >> the
> > >> > correct approach then I would be happy to put together a testing
> plan
> > >> for
> > >> > the PR as well.
> > >> >
> > >> > Just to clarify, the general roadmap for getting this used in
> > >> apache/metron
> > >> > is:
> > >> > 1.  Create a bro package in apache/metron-bro-plugin-kafka
> > >> > 2.  Update the ansible bro setup
> > >> > <https://github.com/apache/metron/tree/master/metron-
> > >> > deployment/roles/bro/tasks>
> > >> > to install/configure bro-pkg (`pip install bro-pkg && bro-pkg
> > >> autoconfig`)
> > >> > and use it to install the apache/metron-bro-plugin-kafka package.
> > >> >
> > >> > I will also be adding this to the official bro package manager
> > >> > <https://github.com/bro/packages>, but out of an abundance of
> > caution I
> > >> > plan to setup ansible to pull the package directly from the
> > >> > apache/metron-bro-plugin-kafka using bro-pkg instead of going
> through
> > >> the
> > >> > bro/packages package source (which removes the bro/packages
> > dependency).
> > >> >
> > >> > Feedback on all of the above is welcome.
> > >> >
> > >> > Jon
> > >> > --
> > >> >
> > >> > Jon
> > >> >
> > >>
> > > --
> > >
> > > Jon
> > >
> > --
> >
> > Jon
> >
>
-- 

Jon

Re: [DISCUSS] Upcoming Release

[zeo...@gmail.com]

I agree, I think it's very reasonable to move in line with Nick's
proposal.  I would also suggest that we outline what the target versions
would be to add in the METRON-777 components, since it has been functional
for a very long time but not reviewed and has some really rockstar
improvements.

Jon

On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler  wrote:

> I think the ES cutover should be the start of the 0.5.x series, and we
> continue on with 0.4.x for the
> metadata improvements etc.  We could chose to focus 0.5.x’s first releases
> on not only ES but
> getting a handle on kibana and the mpack situation as well.
>
>
>
>
> On November 6, 2017 at 12:48:45, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> I agree with your proposal, Nick. I think having a stabilizing release
> prior to upgrading ES/Kibana makes sense.
>
> On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen  wrote:
>
> > I would like to start a discussion around upcoming releases. We have a
> > couple separate significant tracks of work that we need to reconcile in
> our
> > release schedule.
> >
> > (1) We have had (and have in review) a good number of bug fixes required
> to
> > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
> >
> >
> > (2) We also have ongoing work to upgrade our infrastructure to
> > Elasticsearch 5.x, which will not be backwards compatible.
> >
> >
> > I would like to see a release that has our best work on ES 2.x before we
> > migrate to 5.x. I would propose the following.
> >
> > Release N+1: Introduce Metaalerts running on ES 2.x
> >
> > Release N+2: Cut-over to ES 5.x
> >
> >
> > (Q) Is it worth cutting a separate release for ES 2.x? Is there a better
> > way to handle the cut-over to 5.x?
> >
>
-- 

Jon

Re: [DISCUSS] - Remove Kibana

[zeo...@gmail.com]

I'm probably okay with marking it as deprecated in two releases (after
moving to 5.x, thus not really helping with the migration), but it depends
a lot on increased functionality for the metron alerts UI IMO.

Jon

On Wed, Nov 1, 2017 at 12:51 PM Otto Fowler  wrote:

> I don’t think we should remove it until there is a viable alternative for
> the capabilities we rely on it for.
> Also, the ‘story’ around metron integration with kibana needs to be solid
> and well supported at that time.
>
>
>
> On November 1, 2017 at 12:13:18, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> As part of the ES upgrade, I got to thinking that it makes sense to remove
> Kibana and the dashboards we're currently bundling in the MPack. To be
> clear, this would not remove the ability to independently install and use
> Kibana if the user so chooses, it would only remove the dashboards, and
> potentially, the Ambari/MPack management support that we ship.
>
> *pros*
> Removes need to support tooling outside our wheelhouse
> Smaller testing effort for ongoing support and future upgrades
> Simplifies our base Metron install via Ambari. Also simplifies full dev
> setup.
>
> *cons*
> User would need to install and setup their own Kibana instance and
> dashboards.
> If any existing users are using this, they'd need to backup and manage
> their own Kibana dashboards going forward. They would also need to handle
> any upgrade issues with Kibana post-Metron ES 5.x upgrade.
>
> Any concerns?
>
> Mike
>
-- 

Jon

Re: [DISCUSS] Release Process Update

[zeo...@gmail.com]

I still kind of like the build image historically, since not everybody who
interfaces with our project will _know_ that the build must always succeed
for a release, but I agree that this is a clean approach so I rolled back
the wiki changes.  Thanks,

Jon

On Tue, Oct 24, 2017 at 2:34 PM Matt Foley <ma...@apache.org> wrote:

> The release wouldn’t have been made if the build didn’t succeed.
> And the Release Manager doesn’t need one more fiddly manual edit to do.
> I recommend the Release Process instructions be put back, and instead we
> incorporate
> https://github.com/apache/metron/pull/815
> Rational in
> https://issues.apache.org/jira/browse/METRON-1278
>
> Thanks,
> --Matt
>
> On 10/24/17, 5:37 AM, "zeo...@gmail.com" <zeo...@gmail.com> wrote:
>
> Hmm, I kind of like it as a historical validation/confirmation of build
> success, but I can see where you are coming from.  Here
> <
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=66854770=34=33
> >
> is the wiki diff, feel free to critique/alter.
>
> Jon
>
> On Tue, Oct 24, 2017 at 7:07 AM Kyle Richardson <
> kylerichards...@gmail.com>
> wrote:
>
> > +1 I agree with Matt and Justin. The Travis build widget doesn't
> make sense
> > in the published release documentation. No sense in fixing
> retrospectively.
> >
> > -Kyle
> >
> > On Mon, Oct 23, 2017 at 3:13 PM Matt Foley <mfo...@hortonworks.com>
> wrote:
> >
> > > I agree with Justin.  This micro-feature is intended as a github
> widget,
> > > which causes the top-level README to give all viewers an immediate
> flag
> > > whether the build is healthy or not.  It does not belong in a
> rendered
> > > site-book.
> > >
> > > Removing the widget during site-book build, can be done with a
> one-line
> > > addition to HREF_REWRITE_LIST in:
> > >
> > >
> >
> https://github.com/apache/metron/blob/master/site-book/bin/generate-md.sh#L75
> > >
> > > Recommend not worrying about historical site-books (they naturally
> > > obsolesce out of the “dist/release” area).
> > >
> > > Cheers,
> > > --Matt
> > >
> > > On 10/23/17, 6:38 AM, "Justin Leet" <justinjl...@gmail.com> wrote:
> > >
> > > I'd argue it shouldn't be in the site-book at all. Presumably
> we
> > aren't
> > > releasing while Travis is broken, so it's not useful
> information for
> > > anyone
> > > looking at docs. It just carries over from the main README.
> Seems
> > > like we
> > > should just scrub it when we do the other fixes to the READMEs
> to
> > make
> > > them
> > > suitable for site-book.  At that point it's just gone
> entirely. from
> > > the
> > > next release.
> > >
> > > Doesn't solve the problem of prior releases (assuming we care
> enough
> > > to do
> > > anything).
> > >
> > > On Mon, Oct 23, 2017 at 9:32 AM, zeo...@gmail.com <
> zeo...@gmail.com>
> > > wrote:
> > >
> > > > Today I was poking around the Metron site and documentation,
> and I
> > > noticed
> > > > that the site-book's travis build status image is pointing to
> > master
> > > for
> > > > all of our releases.  We should probably update the release
> process
> > > > <
> > https://cwiki.apache.org/confluence/display/METRON/Release+Process>
> > > to
> > > > pin
> > > > this to the specific release.
> > > >
> > > > I will happily handle the documentation update but I wanted
> to ask,
> > > what
> > > > should it be pointing to?  Repointing is incredibly
> straightforward
> > > > <https://travis-ci.org/apache/metron/branches>, but I'm not
> sure
> > > would be
> > > > more appropriate to use - the release branches (e.g.
> Metron_0.4.1),
> > > or
> > > > the release tags (e.g. apache-metron-0.4.1-release)?  I'm
> not clear
> > > on
> > > > their specific uses in our environment.  In reviewing our
> current
> > > process,
> > > > it appears that we _could_ use either.
> > > >
> > > > I also wanted to ask, does anybody think that this should
> get fixed
> > > > historically?  I think that this might be an excessive
> amount of
> > > hassle,
> > > > but I wanted to put it out there since I'm not intimately
> familiar
> > > with
> > > > what we'd need to do in order to clean this up.
> > > >
> > > > Jon
> > > > --
> > > >
> > > > Jon
> > > >
> > >
> > >
> > >
> >
> --
>
> Jon
>
>
>
>
> --

Jon

Re: [DISCUSS] Release Process Update

[zeo...@gmail.com]

Hmm, I kind of like it as a historical validation/confirmation of build
success, but I can see where you are coming from.  Here
<https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=66854770=34=33>
is the wiki diff, feel free to critique/alter.

Jon

On Tue, Oct 24, 2017 at 7:07 AM Kyle Richardson <kylerichards...@gmail.com>
wrote:

> +1 I agree with Matt and Justin. The Travis build widget doesn't make sense
> in the published release documentation. No sense in fixing retrospectively.
>
> -Kyle
>
> On Mon, Oct 23, 2017 at 3:13 PM Matt Foley <mfo...@hortonworks.com> wrote:
>
> > I agree with Justin.  This micro-feature is intended as a github widget,
> > which causes the top-level README to give all viewers an immediate flag
> > whether the build is healthy or not.  It does not belong in a rendered
> > site-book.
> >
> > Removing the widget during site-book build, can be done with a one-line
> > addition to HREF_REWRITE_LIST in:
> >
> >
> https://github.com/apache/metron/blob/master/site-book/bin/generate-md.sh#L75
> >
> > Recommend not worrying about historical site-books (they naturally
> > obsolesce out of the “dist/release” area).
> >
> > Cheers,
> > --Matt
> >
> > On 10/23/17, 6:38 AM, "Justin Leet" <justinjl...@gmail.com> wrote:
> >
> > I'd argue it shouldn't be in the site-book at all. Presumably we
> aren't
> > releasing while Travis is broken, so it's not useful information for
> > anyone
> > looking at docs. It just carries over from the main README.  Seems
> > like we
> > should just scrub it when we do the other fixes to the READMEs to
> make
> > them
> > suitable for site-book.  At that point it's just gone entirely. from
> > the
> > next release.
> >
> > Doesn't solve the problem of prior releases (assuming we care enough
> > to do
> > anything).
> >
> > On Mon, Oct 23, 2017 at 9:32 AM, zeo...@gmail.com <zeo...@gmail.com>
> > wrote:
> >
> > > Today I was poking around the Metron site and documentation, and I
> > noticed
> > > that the site-book's travis build status image is pointing to
> master
> > for
> > > all of our releases.  We should probably update the release process
> > > <
> https://cwiki.apache.org/confluence/display/METRON/Release+Process>
> > to
> > > pin
> > > this to the specific release.
> > >
> > > I will happily handle the documentation update but I wanted to ask,
> > what
> > > should it be pointing to?  Repointing is incredibly straightforward
> > > <https://travis-ci.org/apache/metron/branches>, but I'm not sure
> > would be
> > > more appropriate to use - the release branches (e.g. Metron_0.4.1),
> > or
> > > the release tags (e.g. apache-metron-0.4.1-release)?  I'm not clear
> > on
> > > their specific uses in our environment.  In reviewing our current
> > process,
> > > it appears that we _could_ use either.
> > >
> > > I also wanted to ask, does anybody think that this should get fixed
> > > historically?  I think that this might be an excessive amount of
> > hassle,
> > > but I wanted to put it out there since I'm not intimately familiar
> > with
> > > what we'd need to do in order to clean this up.
> > >
> > > Jon
> > > --
> > >
> > > Jon
> > >
> >
> >
> >
>
-- 

Jon

[DISCUSS] Release Process Update

[zeo...@gmail.com]

Today I was poking around the Metron site and documentation, and I noticed
that the site-book's travis build status image is pointing to master for
all of our releases.  We should probably update the release process
 to pin
this to the specific release.

I will happily handle the documentation update but I wanted to ask, what
should it be pointing to?  Repointing is incredibly straightforward
, but I'm not sure would be
more appropriate to use - the release branches (e.g. Metron_0.4.1), or
the release tags (e.g. apache-metron-0.4.1-release)?  I'm not clear on
their specific uses in our environment.  In reviewing our current process,
it appears that we _could_ use either.

I also wanted to ask, does anybody think that this should get fixed
historically?  I think that this might be an excessive amount of hassle,
but I wanted to put it out there since I'm not intimately familiar with
what we'd need to do in order to clean this up.

Jon
-- 

Jon

Re: new committer: Raghu Mitra

[zeo...@gmail.com]

Congratulations, Raghu!

Jon

On Fri, Oct 20, 2017, 12:11 Simon Elliston Ball 
wrote:

> Congratulations Raghu. Well deserved with all that awesome UI work that’s
> coming in.
>
> Simon
>
> > On 20 Oct 2017, at 17:10, James Sirota  wrote:
> >
> >
> >
> > The Project Management Committee (PMC) for Apache Metron
> > has invited Raghu Mitra to become a committer and we are pleased
> > to announce that he has accepted.
> >
> >
> > Being a committer enables easier contribution to the
> > project since there is no need to go via the patch
> > submission process. This should enable better productivity.
> > Being a PMC member enables assistance with the management
> > and to guide the direction of the project.
>
> --

Jon

Re: Suricata parser

[zeo...@gmail.com]

I would love to see one, and if it doesn't exist in the next few weeks I'm
going to take a stab at it.

Jon

On Mon, Sep 25, 2017, 09:49 Carolyn Duby  wrote:

>
> Is anyone working on a Suricata parser?
>
> https://suricata-ids.org/
>
>
> I was not able to find an enhancement request for it.
>
> Thanks
> Carolyn
>
-- 

Jon

Re: Metron 0.4.2 release date

[zeo...@gmail.com]

As of right now I'm not aware of any discussions regarding a next release,
and I believe the METRON-777 features are at least a few months out from
being reviewed and merged in (There is a fair amount of work in chunking it
up to be reviewed, then work to review and merge it in).  ES 5.x is also in
progress but not even open as a PR yet, let alone in master and ready to be
included in a release.  I'm really looking forward to those
changes/improvements as well, but I wouldn't be expecting them in the next
few months, and trying to look further into the future than that at this
point would be difficult.

That said, if anybody else has a more detailed timeline in mind, I would
love to hear more.

Jon

On Sun, Oct 8, 2017, 09:05 Ali Nazemian  wrote:

> Hi all,
>
> I was wondering when Metron 0.4.2 will be released and whether it includes
> Metron-777 and Elasticsearch 5.x or not?
>
> Cheers,
> Ali
>
-- 

Jon

Re: who is having problems installing?

[zeo...@gmail.com]

First would be to migrate docs from the wiki into the site-book so we have
a more concise place to point people to regarding documentation, because
there is some good stuff in the wiki, and some good things in the
site-books, but attempts to link them together is currently broken all over
the place (perfect example - this
<https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture> is
the architecture page on the wiki).  I am thinking about some more concise
areas.  An example would cover things like "an intro to the environment"
(Maybe an architecture sub-bullet?) and goes through Storm, Kafka, HDFS,
etc. and covers "How its used" in a more succinct way.  Ignoring that, I
was thinking about things like:

https://cwiki.apache.org/confluence/display/METRON/Data+Loads
https://cwiki.apache.org/confluence/display/METRON/UI
https://cwiki.apache.org/confluence/display/METRON/Deployment+Scripts

Like I said before, some of this is covered in one location
<https://metron.apache.org/current-book/metron-interface/metron-alerts/index.html>,
but not another <https://cwiki.apache.org/confluence/display/METRON/UI>,
which is extremely confusing.  If I found a place where documentation said
it was coming soon, I wouldn't necessarily keep digging to find
documentation for that item.

Another thing I noticed via my recent perusal was that
https://cwiki.apache.org/confluence/display/METRON/Streaming is not
accurate.  Missing indexing, errors, etc.  I'm sure there are plenty more
examples as well, and I don't think it's reasonable to point people to the
wiki almost at all any longer (the squid walk-through is a good example of
something still very valuable) because doing so is overly confusing/nuanced
- we really need to get it migrated to the site-book.

Jon

On Fri, Oct 6, 2017 at 2:22 PM James Sirota <jsir...@apache.org> wrote:

> Can you give an example?  My personal view is that our docs explain Metron
> fundamentals pretty well.  If this is not the case, then would be willing
> to take a look and see how we can make them more consumable.  The problem
> with videos is that they become out of date very quickly and it's a lot of
> effort to re-record them.
>
> Thanks,
> James
>
> 06.10.2017, 11:05, "zeo...@gmail.com" <zeo...@gmail.com>:
> > To generalize a bit, I think it would be helpful to have a single or
> series
> > of recordings, write-ups, or even just pointers to some good high-level
> > docs to introduce people to each component used in Metron, and then a
> > description of how it's used in the Metron environment. I know I spend a
> > lot of time talking with people hitting the fundamentals, which is to be
> > expected for a growing project like ours. That helps provide a super
> > high-level context for what is happening in the cluster and where to look
> > if you're seeing certain types of issues.
> >
> > Jon
> >
> > On Fri, Oct 6, 2017 at 1:56 PM James Sirota <jsir...@apache.org> wrote:
> >
> >>  Hi Guys,
> >>
> >>  How about a meeting at 11 AM PST on this? Can everyone who needs to
> make
> >>  the meeting? If you could come with a Hadoop cluster (including Kafka,
> >>  storm, HDFS, Hbase) pre-installed I can walk you through the steps
> required
> >>  to install Metron. Does that seem reasonable?
> >>
> >>  Thanks,
> >>  James
> >>
> >>  04.10.2017, 23:01, "Ronirose Caryll De Castro" <
> >>  ronirose.decas...@pointwest.com.ph>:
> >>  > That's wonderful and very generous of the team.
> >>  >
> >>  > Here's on my end:
> >>  >
> >>  > ​- What environment are you installing on (a single VM, multiple VMs,
> >>  bare
> >>  > metal, AWS, etc)
> >>  > == Single VM, but I'm trying to install on multiple VMs
> >>  > - What OS are you using
> >>  > ​ ==​ Ubuntu Xenial and Zesty, but trying to use CentOS
> >>  > - How many sensors are you going to be consuming
> >>  > ​ == Unidentified. I was doing a test install at the moment.
> >>  >
> >>  > *Thank you!*
> >>  > *Caryll*
> >>  >
> >>  > On Thu, Oct 5, 2017 at 4:01 AM, James Sirota <jsir...@apache.org>
> wrote:
> >>  >
> >>  >> Yes the intent is for everyone that has any type of metron
> installation
> >>  >> issue or question attend the meeting
> >>  >>
> >>  >> 03.10.2017, 17:35, "Ronirose Caryll De Castro" <
> >>  >> ronirose.decas...@pointwest.com.ph>:
> >>  >> > 

Re: Quick Dev

[zeo...@gmail.com]

I say we kill it and repoint the site.  That will give us one less thing to
upgrade to centos 7 as well.

Jon

On Fri, Oct 6, 2017, 08:27 Justin Leet  wrote:

> So what are we going to do with Quick Dev?  I'm pretty sure everybody's
> been using full dev for awhile now (and quick dev is probably broken since
> I'm sure we haven't been regularly updating it).
>
> I just realized our website links to a wiki page that says to use quick
> dev.  Given that quick dev is broken or behind or both, this is going to be
> incredibly confusing and misleading to anyone wandering in.
>
> A short term fix is to just point that wiki page to full dev, so we aren't
> at least broken to anyone coming in.
>
> After that we should figure out what we're doing with quick and full dev
> and take care of whatever needs to be done.
>
> Thoughts?
>
> Justin
>
-- 

Jon

Re: SUM aggregator not working?

[zeo...@gmail.com]

You're right, with ES 5 we can use periods directly instead of transforming
them in indexing to colons (actually, this feature was reintroduced sin 2.4
).  I outlined
this as a benefit in the original JIRA
, along with a
ton of other benefits including native IPv6 support 

Jon

On Wed, Oct 4, 2017 at 5:03 PM Casey Stella  wrote:

> Ok, so this is subtle.  Your rules are wrong and I totally understand why
> you thought they were right.
>
> When we index into ES, we take . and convert them to :, however PRIOR to
> indexing (when threat triage is running) those fields have .'s not :'s
> Therefore, your rules should be:
>
> userIdentity.sessionContext.attributes.mfaAuthenticated == 'False'
> and
> additionalEventData.MFAUsed == 'No'
>
> The same general argument goes for your threat triage stellar expressions.
>
>
> Sorry about the confusion, we do that mapping because ES doesn't handle
> those .'s well.  Hey, maybe ES 5 is more sane about that sort of thing and
> we can avoid doing that transformation.
>
> Casey
>
> On Wed, Oct 4, 2017 at 4:38 PM, Laurens Vets  wrote:
>
> > No idea whether it's a bug yet, I just need a 2nd set of eyes :)
> >
> > This is my event as indexed in ES (Obviously some parts have been
> > obfuscated):
> >
> > {
> >   "_index": "cloudtrail_index_2017.10.04.19",
> >   "_type": "cloudtrail_doc",
> >   "_id": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
> >   "_score": null,
> >   "_timestamp": 1507143907108,
> >   "_source": {
> > "eventID": "9e3d5468-2d97-4b9a-9821-5c61fec8c158",
> > "additionalEventData:MFAUsed": "No",
> > "adapter:stellaradapter:end:ts": "1507143907145",
> > "threatinteljoinbolt:joiner:ts": "1507143907153",
> > "eventVersion": "1.05",
> > "threat:triage:rules:0:comment": "Checks whether the field is_work is
> > true or false.",
> > "sourceIPAddress": "208.110.73.106",
> > "eventSource": "signin.amazonaws.com",
> > "enrichmentsplitterbolt:splitter:begin:ts": "1507143907143",
> > "enrichmentjoinbolt:joiner:ts": "1507143907147",
> > "additionalEventData:MobileVersion": "No",
> > "threat:triage:rules:0:name": "Not WORK",
> > "source:type": "cloudtrail",
> > "original_string": "{\"eventVersion\":\"1.05\",\"
> > userIdentity\":{\"type\":\"IAMUser\",\"principalId\":\"AIDAI
> > 5ITCMVR3BQV5DUFW\",\"arn\":\"arn:aws:iam:::user/
> > \",\"accountId\":\"\",\"userName\":\"<
> > EMAIL>\"},\"eventTime\":\"2017-10-04T18:57:31Z\",\"eventSource\":\"
> > signin.amazonaws.com\",\"eventName\":\"ConsoleLogin\"
> > ,\"awsRegion\":\"us-east-1\",\"sourceIPAddress\":\"208.110.7
> > 3.106\",\"userAgent\":\"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
> > Gecko/20100101 Firefox/56.0\",\"requestParame
> > ters\":null,\"responseElements\":{\"ConsoleLogin\":\"
> > Success\"},\"additionalEventData\":{\"LoginTo\":\"https://
> > console.aws.amazon.com/console/home?state=hashArgs%23=true\
> 
> > <
> https://console.aws.amazon.com/console/home?state=hashArgs%23=true%5C
> >
> > ",\"MobileVersion\":\"No\",\"MFAUsed\":\"No\"},\"eventID\":
> > \"9e3d5468-2d97-4b9a-9821-5c61fec8c158\",\"eventType\":\
> > "AwsConsoleSignIn\",\"recipientAccountId\":\"\"}",
> > "eventTime": "2017-10-04T18:57:31Z",
> > "eventName": "ConsoleLogin",
> > "recipientAccountId": "",
> > "userIdentity:principalId": "AIDAI5ITCMVR3BQV5DUFW",
> > "threatintelsplitterbolt:splitter:end:ts": "1507143907148",
> > "threat:triage:rules:0:score": 20,
> > "timestamp": 1507143907108,
> > "threat:triage:rules:0:reason": "208.110.73.106 is not an WORK
> > network!",
> > "awsRegion": "us-east-1",
> > "is_work": false,
> > "userIdentity:userName": "",
> > "enrichmentsplitterbolt:splitter:end:ts": "1507143907143",
> > "threat:triage:score": 20,
> > "is_alert": "true",
> > "userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
> > Gecko/20100101 Firefox/56.0",
> > "adapter:stellaradapter:begin:ts": "1507143907145",
> > "eventType": "AwsConsoleSignIn",
> > "userIdentity:arn": "arn:aws:iam:::user/",
> > "userIdentity:accountId": "",
> > "userIdentity:type": "IAMUser",
> > "threatintelsplitterbolt:splitter:begin:ts": "1507143907148",
> > "guid": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
> > "additionalEventData:LoginTo": "https://console.aws.amazon.co
> > m/console/home?state=hashArgs%23=true",
> > "responseElements:ConsoleLogin": "Success"
> >   },
> >   "fields": {
> > "adapter:stellaradapter:end:ts": [
> >   1507143907145
> > ],
> > "threatinteljoinbolt:joiner:ts": [
> >   1507143907153
> > ],
> > "enrichmentsplitterbolt:splitter:end:ts": [
> >   1507143907143
> > ],
> > "enrichmentsplitterbolt:splitter:begin:ts": [
> >   

Re: [DISCUSS] Build broken due to transitive dependencies

[zeo...@gmail.com]

Hmm, 0.4.1 built fine for me.

Jon

On Mon, Oct 2, 2017 at 10:44 AM Casey Stella  wrote:

> Ok, the build is broken in metron-config due to some transitive changes
> that happened in npm-land:
>
> [INFO]
>
> /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32
> [INFO]   throw new Error('Cyclic dependency: '+JSON.stringify(node))
> [INFO] ^
> [INFO] Error: Cyclic dependency: "[object Object]"
> [INFO] at visit
>
> (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32:13)
> [INFO] at visit
>
> (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:48:9)
>
> Evidently one of our transitive dependencies has changed and we have ended
> up with a cyclic dependency.  I'm not sure where or why yet, but I believe
> this breaks both master and our 0.4.1 release (I haven't confirmed this
> yet, but I strongly suspect).
>
> While the good work of tracking down this specific error is done, I'd like
> to bring up a broader discussion point: our practice of not fixing versions
> for our node dependencies.  This is, in effect, causing a few problems:
>
>- We do not have a consistent, repeatable build.
>- We set ourselves up for possible license violation without knowing
>about it (a transitive dependency changes its license)
>
> As we stand, we have a release which doesn't not build after we have
> released it and tested it.  It seems to me that we should at a minimum as a
> stopgap:
>
>- fix the versions of our dependencies so that they are in a working
>state
>- consider a point release to get a working build.
>
> I guess my questions to those of us with more javascript and UI experience
> is as follows:
>
>- Does fixing the version of our dependencies actually fix the problem
>transitively?
>- IF not, then how do we get a version of a build which is consistent
>and repeatable and does not expose us to downstream licensing issues?
>
> Thanks,
>
> Casey
>
-- 

Jon

Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

[zeo...@gmail.com]

When is the meeting, given Otto mentioned he can't make 10am?  Or did that
change

Jon

On Mon, Sep 25, 2017, 19:19 James Sirota <jsir...@apache.org> wrote:

> Great. Thank you, Otto.  I would encourage everyone to watch it so that we
> have constructive feedback for tomorrow and are able to arrive to a decision
>
> Thanks,
> James
>
> 25.09.2017, 08:27, "Otto Fowler" <ottobackwa...@gmail.com>:
> > https://youtu.be/-ISycoP3TVA
> >
> > The video is short and simple. Hopefully it is what you are looking for.
> >
> > On September 21, 2017 at 16:54:13, zeo...@gmail.com (zeo...@gmail.com)
> > wrote:
> >
> > I won't be able to make it and would really like to make sure there's a
> > recording for this one, if possible. I'm unavailable until Thursday of
> > next week, but not necessarily suggesting this gets moved.
> >
> > Jon
> >
> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ottobackwa...@gmail.com> wrote:
> >
> >>  I can’t make that time, can we make it later in the day?
> >>
> >>  On September 21, 2017 at 11:40:37, James Sirota (jsir...@apache.org)
> >>  wrote:
> >>
> >>  https://hortonworks.webex.com/meet/jsirota
> > --
> >
> > Jon
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
-- 

Jon

Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

[zeo...@gmail.com]

I won't be able to make it and would really like to make sure there's a
recording for this one, if possible.  I'm unavailable until Thursday of
next week, but not necessarily suggesting this gets moved.

Jon

On Thu, Sep 21, 2017, 15:04 Otto Fowler  wrote:

> I can’t make that time, can we make it later in the day?
>
>
> On September 21, 2017 at 11:40:37, James Sirota (jsir...@apache.org)
> wrote:
>
> https://hortonworks.webex.com/meet/jsirota
>
-- 

Jon

Re: feature branch bumps

[zeo...@gmail.com]

But wait, I thought we had established that this was such a fundamental
change that it was hard to chunk it out and keep master working.

Jon

On Wed, Sep 20, 2017 at 3:08 PM Nick Allen  wrote:

> > Otto: Well, if there is an alternative merge strategy, I’m all ears.
>
> Yes, the alternative strategy is what I mentioned. :)  Copied in below.
>
> > Nick: Are you going to bite-off small chunks of the feature branch and
> introduce PRs against master?
>
>
>
>
>
> On Wed, Sep 20, 2017 at 12:02 PM Otto Fowler 
> wrote:
>
> > Well, if there is an alternative merge strategy, I’m all ears.
> > As it stands, the only complete code / conceptual review has been the
> > bundles-lib by mattf.
> > So there is still a great deal of review activity to do.
> >
> > Also conceptually there is ( might not be complete )
> >
> > * the discuss thread topic from this morning ( metron parsers v.
> > extensions wrt registration and management )
> > * default configurations ( parser, enrichment, indexing, elasticsearch)
> >
> >
> >
> >
> > On September 20, 2017 at 11:37:17, Nick Allen (n...@nickallen.org)
> wrote:
> >
> > Ok, so it sounds like you are envisioning a "big bang" merge between the
> > feature branch and master at some point.  Not ideal in my mind, but maybe
> > we need to be more pragmatic with this one.
> >
> > Do we have other tasks (like these PRs) to complete before we are ready
> to
> > consider the merge?
> >
> > I am just looking to help however I can in killing this feature branch;
> > meaning I want your code in master, as soon as possible. :)
> >
> >
> >
> >
> >
> >
> > On Wed, Sep 20, 2017 at 10:27 AM Otto Fowler 
> > wrote:
> >
> >> So, were the functionality is not FB related, I have been doing PR’s
> >> against master ( such as hdfs service ability to set permissions ).
> >> I don’t think we have talked about the end game PR from feature to
> >> master, I am don’t know how we would do multiple PRs to bring it down
> >> once ‘accepted’.  I think that approach needs to to be discussed and
> >> agreed on.
> >>
> >>
> >> On September 20, 2017 at 10:23:44, Nick Allen (n...@nickallen.org)
> wrote:
> >>
> >> So it sounds like these PRs do move us closer to bringing the two
> >> branches together.  But I think I am missing your high-level approach
> >> though.
> >>
> >> How are we going to get all of the functionality from the feature branch
> >> into master?  Are you going to bite-off small chunks of the feature
> branch
> >> and introduce PRs against master?
> >>
> >>
> >>
> >> On Wed, Sep 20, 2017 at 9:23 AM Otto Fowler 
> >> wrote:
> >>
> >>> Right now, these two PR’s are stacked, the UI depends on the
> >>> BundleSystem changes.
> >>> I would rather bring them down to feature before I do master
> >>> integration, than integrate master
> >>> into feature and then bring it out to the stacked PR’s.  Simple as
> that.
> >>>
> >>>
> >>>
> >>> On September 20, 2017 at 08:35:09, Nick Allen (n...@nickallen.org)
> >>> wrote:
> >>>
> >>> Hi Otto -
> >>>
> >>> What is the plan for bringing the feature branch and master together?
> Do
> >>> these PRs move us closer to bringing the two branches together?
> >>>
> >>> Thanks
> >>>
> >>>
> >>>
> >>> On Wed, Sep 20, 2017 at 8:19 AM Otto Fowler 
> >>> wrote:
> >>>
> >>> > Can I get a bump on https://github.com/apache/metron/pull/747 and
> >>> > https://github.com/apache/metron/pull/761?
> >>> >
> >>> > The next time I take master is going to be a bit of integration work,
> >>> and I
> >>> > would like to unstack and get my stuff integrated first.
> >>> >
> >>>
> >>>
>
-- 

Jon

Re: [DISUCUSS] [CALL FOR COMMENT] Metron parsers as actual extensions

[zeo...@gmail.com]

Per our prior conversations, I prefer option 2 - treating third party and
built-in the same way.  I would love to see signing of extensions in the
future as a potential follow-on so we could verify the Metron built-ins
(and even third parties).

Jon

On Wed, Sep 20, 2017 at 10:22 AM Otto Fowler 
wrote:

> Simon, I’m sorry, I may not have answered your question.
> I use parser and sensors as the same thing, but from what you say I think I
> mean parser.
>
>
> On September 20, 2017 at 10:08:25, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> So,
> The distinction between ‘sensor’ and ‘parser’ is not clear to me either, if
> it is defined somewhere and I have missed it, please point me in the right
> direction.
> While I don’t want to fork the discussion, the question is where you find
> it so to speak, so about bundles and configurations.
>
> With the extension system you have two options for ‘config’ only
> parser/sensor installs.
>
> 1.  The previous mechanism of creating the configurations and manually
> pushing them zookeeper and possibly patterns still works, although they
> will not be managed as extensions.  This I believe is a feature gap that
> already exists and I did not address it as part of this effort.
> 2. Create a new parser from the archetype, but remove all the src/main code
> and make it configuration only.  This I think should be the recommended way
> to create configuration only parsers, since such parsers will still have
> the unit and integration tests.  Flows where you start with the archetype
> and refine through tests or manually deploy, refine and then build from the
> archetype can be imagined.
>
>
> The main question for this thread however, is should metron parsers and
>  3rd party parsers be treated the same -> they are all extensions,
> manageable through the extensions ui.
>
> I can demo for you whenever you are free if you don’t want to wait for the
> community demo.
>
>
> On September 20, 2017 at 09:52:56, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Otto,
>
> Can you just clarify what you mean by parsers in this instance. To my mind
> parsers in metron are be classes, and do not have any configuration
> settings. Instances of parsers are referred to in the ui as sensors, and
> are essentially concrete instances of parsers and as such do have config.
>
> The parser vs sensor distinction feels like a valuable one to make here.
> I'd really like a clearer understanding of the role of bundles in config,
> and how we maintain the ability to control config without the need for
> files in the bundle.
>
> Maybe some samples / a demo would really clear this up, but to be honest
> right now I'm a little confused about how this would work for an average
> SOC team who do not have maven available.
>
> Thanks,
> Simon
>
> Sent from my iPhone
>
> > On 20 Sep 2017, at 23:43, Otto Fowler  wrote:
> >
> > Note: Grok, CSV and JSONMap would be ‘always there’, as they are still
> > part of the system and not installed individually.
> >
> >
> >
> > On September 20, 2017 at 09:39:38, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > The question has come up about the metron parsers installation vs. parser
> > extension installation differences, and I’d like to get some comments.
> >
> > Right now ( let’s pretend the UI PR get’s merged to the feature branch
> for
> > a minute ) in the original take on this the metron parsers ( bro, yaf,
> > snort etc ) get installed
> > into the system basically as before with regards to zookeeper ( although
> > they ALL get installed since they all have demo compatible default
> configs
> > now). The parser extensions, installed after the fact through ui however
> > have a new parser extension configuration
> > that is registered into a new zookeeper area, and are listed and managed
> in
> > the new UI. They can be installed or uninstalled basically.
> >
> > The question is if the parsers installed by metron should *also* be
> > registered in the same way, such that the parser extension ui lists all
> of
> > the parsers.
> >
> > This would allow removal and installation of metron parsers installed by
> > the system. Also, following on we can customize the install to not
> install
> > everything. It may also be more simple concept wise.
> >
> > That is the basic thing. So the question is if we want to go in this
> > direction.
> >
> > The not so basic thing is to still deploy the extensions into the system
> as
> > packages, but not install them, such that you can add an extension from a
> > file *and also* from a ‘repository’ of
> > extensions. Down the line we can support local and remote repositories
> etc.
> >
> > So the options are:
> >
> > 1. As it is now on the feature branch ( still pretending ;)), the system
> > installed parsers are not the same as the extension installed parsers.
> > While the project can uninstall and replace these parsers, the user/ui
> > cannot.
> > 2. 

Re: [GitHub] metron issue #760: METRON-1188: Ambari global configuration management broke...

[zeo...@gmail.com]

Spun up fine now, thanks.

On Tue, Sep 19, 2017, 14:09 mmiklavc  wrote:

> Github user mmiklavc commented on the issue:
>
> https://github.com/apache/metron/pull/760
>
> A @JonZeolla fixing it now. Sorry about that - I missed one of the
> "patch_path -> patch_file" arg changes in the mpack.
>
>
> ---
>
-- 

Jon

Re: [ANNOUNCE] Apache Metron Release 0.4.1

[zeo...@gmail.com]

Great job everybody, this is a really top notch release.  Well done

Jon

On Tue, Sep 19, 2017, 15:53 Otto Fowler  wrote:

> Congratulations everyone, great job.  Thank you Matt!
>
>
> On September 19, 2017 at 15:22:21, Matt Foley (ma...@apache.org) wrote:
>
> I’m very happy to announce the public release of Apache Metron version
> 0.4.1.
>
> The release, with PGP signature and checksums, may be obtained from our
> public website at http://metron.apache.org/documentation/#releases
> (Note that you may need to refresh this web page in your browser, to
> assure the latest version.)
>
> or the source code may be cloned from our github repository at
> https://github.com/apache/metron.git (tag: apache-metron-0.4.1-release)
>
> The newest documentation book is at
> http://www.apache.org/dyn/closer.cgi/metron/0.4.1/site-book/index.html
>
> This release represents a great deal of work by the developer community,
> including over 100 enhancements, bug fixes, and innovations, since 0.4.0.
> The complete list of changes is at
> http://www.apache.org/dyn/closer.cgi/metron/0.4.1/CHANGES
>
> The release notes are at
> http://www.apache.org/dyn/closer.cgi/metron/0.4.1/RELEASE_NOTES
>
> Many thanks to all who contributed, and enjoy your new release!
>
> Warm regards,
> --Matt Foley
> release manager
>
>
>
> --

Jon

Re: Committing to the metron-bro-plugin-kafka repo

[zeo...@gmail.com]

Good point, I can take that task re migrating the revision history of the
folder.

Jon

On Fri, Sep 15, 2017, 12:14 Nick Allen <n...@nickallen.org> wrote:

> Hi Jon -
>
> I agree with you on the approach.  We should first copy everything as it is
> to the new repo.  We should maintain the revision history too.  I'm sure
> there is a way to do it, but would have to research a bit.  Then we apply
> your changes on top of that.
>
> Thanks
>
> On Thu, Sep 14, 2017 at 1:36 AM, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
> > So, I've been working on METRON-813
> > <https://issues.apache.org/jira/browse/METRON-813> lately and I have an
> > initial run at it ready to go here
> > <https://github.com/JonZeolla/metron-bro-plugin-kafka> (squashed
> history,
> > see a better history there
> > <https://github.com/JonZeolla/metron-bro-plugin-kafka/commits/bro-pkg>).
> > Since the metron-bro-plugin-kafka repo is empty, I can't open a PR
> against
> > it on GitHub for review.  Does anybody have a suggestion regarding how to
> > move forward?  I see two options:
> > 1. I make the initial commit a direct copy of the bro-plugin-kafka folder
> > <https://github.com/apache/metron/tree/master/metron-
> > sensors/bro-plugin-kafka>
> > (I believe this would require a new JIRA for a direct copy), and then
> open
> > a PR for the METRON-813 changes to get reviewed via the normal process.
> > 2. I make the initial commit the result of METRON-813, but review occurs
> > via the mailing list and using my fork.
> >
> > I prefer 1, but wanted to put it up for discussion.  Once we decide on
> the
> > correct approach then I would be happy to put together a testing plan for
> > the PR as well.
> >
> > Just to clarify, the general roadmap for getting this used in
> apache/metron
> > is:
> > 1.  Create a bro package in apache/metron-bro-plugin-kafka
> > 2.  Update the ansible bro setup
> > <https://github.com/apache/metron/tree/master/metron-
> > deployment/roles/bro/tasks>
> > to install/configure bro-pkg (`pip install bro-pkg && bro-pkg
> autoconfig`)
> > and use it to install the apache/metron-bro-plugin-kafka package.
> >
> > I will also be adding this to the official bro package manager
> > <https://github.com/bro/packages>, but out of an abundance of caution I
> > plan to setup ansible to pull the package directly from the
> > apache/metron-bro-plugin-kafka using bro-pkg instead of going through the
> > bro/packages package source (which removes the bro/packages dependency).
> >
> > Feedback on all of the above is welcome.
> >
> > Jon
> > --
> >
> > Jon
> >
>
-- 

Jon

Committing to the metron-bro-plugin-kafka repo

[zeo...@gmail.com]

So, I've been working on METRON-813
 lately and I have an
initial run at it ready to go here
 (squashed history,
see a better history there
).
Since the metron-bro-plugin-kafka repo is empty, I can't open a PR against
it on GitHub for review.  Does anybody have a suggestion regarding how to
move forward?  I see two options:
1. I make the initial commit a direct copy of the bro-plugin-kafka folder

(I believe this would require a new JIRA for a direct copy), and then open
a PR for the METRON-813 changes to get reviewed via the normal process.
2. I make the initial commit the result of METRON-813, but review occurs
via the mailing list and using my fork.

I prefer 1, but wanted to put it up for discussion.  Once we decide on the
correct approach then I would be happy to put together a testing plan for
the PR as well.

Just to clarify, the general roadmap for getting this used in apache/metron
is:
1.  Create a bro package in apache/metron-bro-plugin-kafka
2.  Update the ansible bro setup

to install/configure bro-pkg (`pip install bro-pkg && bro-pkg autoconfig`)
and use it to install the apache/metron-bro-plugin-kafka package.

I will also be adding this to the official bro package manager
, but out of an abundance of caution I
plan to setup ansible to pull the package directly from the
apache/metron-bro-plugin-kafka using bro-pkg instead of going through the
bro/packages package source (which removes the bro/packages dependency).

Feedback on all of the above is welcome.

Jon
-- 

Jon

Re: [VOTE] Metron Release Candidate 0.4.1-RC4

[zeo...@gmail.com]

+1 (binding)

- Verified the signature
- Verified all hashes
- mvn -q -T 2C surefire:test@unit-tests && mvn -q
surefire:test@integration-tests && mvn -q test --projects
metron-interface/metron-config && build_utils/verify_licenses.sh
- Spun up full-dev
- Manually reviewed the site-book.  Found some minor issues
 that I don't think are worth
failing the release.
- Reviewed files in HDFS and documents in ES
- Verified various logs and UIs for the Metron components
- Used the management UI

FYI, I also updated the Verifying Builds page to remove monit testing, add
testing of the management UI, and update the hdfs filepaths.

Jon

On Sun, Sep 10, 2017 at 1:55 PM Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> +1 (non-binding)
>
> * mvn clean package at root level
> * mvn clean package -Pbuild-rpms at metron-deployment level and generate
> RPMs
> * Brought up Metron stack on 12-node CentOS 7 openstack cluster using the
> generated RPMs
> * Verify all services come up fine [PASS]
> * Bro, YAF and snort - ingest into kafka topics and write indices [PASS]
> * Add squid telemetry, ingest into kafka topic and write indices [PASS]
> * Management UI and REST Swagger UI sanity check [PASS]
>
> Regards,
> Anand
>
>
>
>
> On 9/9/17, 10:36 AM, "Matt Foley"  wrote:
>
> >Colleagues,
> >This is a call to vote on releasing Apache Metron 0.4.1.
> >The release candidate is available at
> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/
> >
> >Full list of changes in this release:
> >https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/CHANGES
> >
> >The github tag to be voted upon is:
> >apache-metron-0.4.1-rc4
> >
> >The source archive being voted upon can be found here:
> >
> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/apache-metron-0.4.1-rc4.tar.gz
> >
> >The site-book is at:
> >
> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/site-book/index.html
> >
> >Other release files, signatures and digests can be found here:
> >https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/
> >
> >The release artifacts are signed with the following key:
> >4169 AA27 ECB3 1663 in
> https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/KEYS
> >
> >Please vote on releasing this package as Apache Metron 0.4.1
> >
> >When voting, please list the actions taken to verify the release.
> >
> >Recommended build validation and verification instructions are posted
> here:
> >https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >
> >This vote will be open until 5pm PDT Wednesday 13 Sep, due to the weekend.
> >Thanks,
> >--Matt
> >(release manager)
> >
> >
> >
>
-- 

Jon

Unclear recent commit

[zeo...@gmail.com]

I was looking through some of the recent commits and I noticed this[1],
anybody know what the back story is there?

1:
https://github.com/apache/metron/commit/c8e84fa3be89901013168d15df38b8a58265148a

Jon
-- 

Jon

Re: Ambari Metrics Collector failing...

[zeo...@gmail.com]

I wouldn't consider it a show stopper myself, happy to be persuaded
otherwise.  I'm not even 100% sure it's related to Metron.  I just put it
in Ambari's maintenance mode for now.

Jon

On Thu, Sep 7, 2017, 11:11 Laurens Vets <laur...@daemon.be> wrote:

> Is this something we need to fix for 0.4.1? Also, should I create  JIRA
> ticket?
>
> On 2017-09-06 16:45, zeo...@gmail.com wrote:
> > I'm seeing the same issue right now as well on my fresh bare metal
> > install
> > of HDP (no Metron yet), haven't dug into it further to troubleshoot.
> >
> > Jon
> >
> > On Wed, Sep 6, 2017, 18:22 Laurens Vets <laur...@daemon.be> wrote:
> >
> >> In preparation of 0.4.1-rc, I'm trying to install the current github
> >> master and I'm running into an issue with Ambari-metrics-collector.
> >> "Metrics Collector" seems to start, but immediately turns red again
> >> Ambari and stops.
> >>
> >> Any idea what might be going on or where I can start troubleshooting
> >> this?
> >>
> >> In /var/log/hbase-ams-master-metron1.log I see lots of:
> >>
> >> 2017-09-06 21:49:16,469 INFO  [HBase-Metrics2-1] impl.MetricsConfig:
> >> loaded properties from hadoop-metrics2-hbase.properties
> >> 2017-09-06 21:49:16,477 INFO  [HBase-Metrics2-1]
> >> timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics
> >> sink.
> >> 2017-09-06 21:49:16,478 INFO  [HBase-Metrics2-1]
> >> timeline.HadoopTimelineMetricsSink: Identified hostname = metron1,
> >> serviceName = ams-hbase
> >> 2017-09-06 21:49:16,478 INFO  [HBase-Metrics2-1]
> >> timeline.HadoopTimelineMetricsSink: Collector Uri:
> >> http://metron1:6188/ws/v1/timeline/metrics
> >> 2017-09-06 21:49:16,491 INFO  [HBase-Metrics2-1]
> >> impl.MetricsSinkAdapter: Sink timeline started
> >> 2017-09-06 21:49:16,500 INFO  [HBase-Metrics2-1]
> >> impl.MetricsSystemImpl:
> >> Scheduled snapshot period at 10 second(s).
> >> 2017-09-06 21:49:16,500 INFO  [HBase-Metrics2-1]
> >> impl.MetricsSystemImpl:
> >> HBase metrics system started
> >> 2017-09-06 21:49:16,518 WARN  [HBase-Metrics2-1] lib.Interns: Metrics
> >> intern cache overflow at 2011 for
> >> MetricsSystem={MetricsSystem=MetricsInfo
> >> Impl{name=MetricsSystem, description=MetricsSystem}, MetricsSystem
> >> record=MetricsInfoImpl{name=MetricsSystem, description=MetricsSystem
> >> record}}
> >> 2017-09-06 21:49:17,564 WARN
> >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn:
> >> caught end of stream exception
> >> EndOfStreamException: Unable to read additional data from client
> >> sessionid 0x15e5929270d0001, likely client has closed socket
> >>  at
> >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> >>  at
> >>
> >>
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> >>  at java.lang.Thread.run(Thread.java:745)
> >> 2017-09-06 21:49:17,565 INFO
> >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn:
> >> Closed socket connection for client /10.0.0.11:
> >> 45058 which had sessionid 0x15e5929270d0001q
> >>
> >> Additionally, I also see:
> >>
> >> 2017-09-06 22:19:16,468 INFO  [HBase-Metrics2-1]
> >> timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics
> >> sink.
> >> 2017-09-06 22:19:16,468 INFO  [HBase-Metrics2-1]
> >> timeline.HadoopTimelineMetricsSink: Identified hostname = metron1,
> >> serviceName = ams-hbase
> >> 2017-09-06 22:19:16,468 INFO  [HBase-Metrics2-1]
> >> timeline.HadoopTimelineMetricsSink: Collector Uri:
> >> http://metron1:6188/ws/v1/timeline/metrics
> >> 2017-09-06 22:19:16,470 INFO  [HBase-Metrics2-1]
> >> impl.MetricsSinkAdapter: Sink timeline started
> >> 2017-09-06 22:19:16,471 INFO  [HBase-Metrics2-1]
> >> impl.MetricsSystemImpl:
> >> Scheduled snapshot period at 10 second(s).
> >> 2017-09-06 22:19:16,471 INFO  [HBase-Metrics2-1]
> >> impl.MetricsSystemImpl:
> >> HBase metrics system started
> >> 2017-09-06 22:20:26,491 INFO  [timeline]
> >> timeline.HadoopTimelineMetricsSink: Unable to connect to collector,
> >> http://metron1:6188/ws/v1/timeline/metrics
> >> This exceptions will be ignored for next 100 times
> >>
> >> 2017-09-06 22:20:26,491 WARN  [timeline]
> >> timeline.HadoopTimelineMetricsSink: Unable to send metrics to
> >> collector
> >> by address:http://metron1:6188/ws/v1/timeline/metrics
> >>
> >> Which ok because metrics collector isn't working...
> >>
> >> Any ideas?
> >>
>
-- 

Jon

Re: Ambari Metrics Collector failing...

[zeo...@gmail.com]

I'm seeing the same issue right now as well on my fresh bare metal install
of HDP (no Metron yet), haven't dug into it further to troubleshoot.

Jon

On Wed, Sep 6, 2017, 18:22 Laurens Vets  wrote:

> In preparation of 0.4.1-rc, I'm trying to install the current github
> master and I'm running into an issue with Ambari-metrics-collector.
> "Metrics Collector" seems to start, but immediately turns red again
> Ambari and stops.
>
> Any idea what might be going on or where I can start troubleshooting
> this?
>
> In /var/log/hbase-ams-master-metron1.log I see lots of:
>
> 2017-09-06 21:49:16,469 INFO  [HBase-Metrics2-1] impl.MetricsConfig:
> loaded properties from hadoop-metrics2-hbase.properties
> 2017-09-06 21:49:16,477 INFO  [HBase-Metrics2-1]
> timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink.
> 2017-09-06 21:49:16,478 INFO  [HBase-Metrics2-1]
> timeline.HadoopTimelineMetricsSink: Identified hostname = metron1,
> serviceName = ams-hbase
> 2017-09-06 21:49:16,478 INFO  [HBase-Metrics2-1]
> timeline.HadoopTimelineMetricsSink: Collector Uri:
> http://metron1:6188/ws/v1/timeline/metrics
> 2017-09-06 21:49:16,491 INFO  [HBase-Metrics2-1]
> impl.MetricsSinkAdapter: Sink timeline started
> 2017-09-06 21:49:16,500 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl:
> Scheduled snapshot period at 10 second(s).
> 2017-09-06 21:49:16,500 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl:
> HBase metrics system started
> 2017-09-06 21:49:16,518 WARN  [HBase-Metrics2-1] lib.Interns: Metrics
> intern cache overflow at 2011 for
> MetricsSystem={MetricsSystem=MetricsInfo
> Impl{name=MetricsSystem, description=MetricsSystem}, MetricsSystem
> record=MetricsInfoImpl{name=MetricsSystem, description=MetricsSystem
> record}}
> 2017-09-06 21:49:17,564 WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn:
> caught end of stream exception
> EndOfStreamException: Unable to read additional data from client
> sessionid 0x15e5929270d0001, likely client has closed socket
>  at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>  at
>
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>  at java.lang.Thread.run(Thread.java:745)
> 2017-09-06 21:49:17,565 INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn:
> Closed socket connection for client /10.0.0.11:
> 45058 which had sessionid 0x15e5929270d0001q
>
> Additionally, I also see:
>
> 2017-09-06 22:19:16,468 INFO  [HBase-Metrics2-1]
> timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink.
> 2017-09-06 22:19:16,468 INFO  [HBase-Metrics2-1]
> timeline.HadoopTimelineMetricsSink: Identified hostname = metron1,
> serviceName = ams-hbase
> 2017-09-06 22:19:16,468 INFO  [HBase-Metrics2-1]
> timeline.HadoopTimelineMetricsSink: Collector Uri:
> http://metron1:6188/ws/v1/timeline/metrics
> 2017-09-06 22:19:16,470 INFO  [HBase-Metrics2-1]
> impl.MetricsSinkAdapter: Sink timeline started
> 2017-09-06 22:19:16,471 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl:
> Scheduled snapshot period at 10 second(s).
> 2017-09-06 22:19:16,471 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl:
> HBase metrics system started
> 2017-09-06 22:20:26,491 INFO  [timeline]
> timeline.HadoopTimelineMetricsSink: Unable to connect to collector,
> http://metron1:6188/ws/v1/timeline/metrics
> This exceptions will be ignored for next 100 times
>
> 2017-09-06 22:20:26,491 WARN  [timeline]
> timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector
> by address:http://metron1:6188/ws/v1/timeline/metrics
>
> Which ok because metrics collector isn't working...
>
> Any ideas?
>
-- 

Jon

Re: [DISCUSS] Metron release 0.4.1

[zeo...@gmail.com]

All set from my perspective.

Jon

On Sat, Sep 2, 2017 at 4:38 AM Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> Hello Matt,
>
> Simon's pull request supersedes mine (METRON-1139 /
> https://github.com/apache/metron/pull/722) and is already merged into
> master.
>
> Thanks,
> Anand
>
>
>
> On 9/1/17, 12:41 AM, "Matt Foley" <mfo...@hortonworks.com> wrote:
>
> >Please mark them 0.4.1, as that’s what the community says we want to call
> the upcoming release, and everything that’s there when I throw the switch
> will be included.
> >
> >Jon and Anand, will they be in by end/day Friday?
> >Thanks,
> >--Matt
> >
> >On 8/31/17, 7:45 AM, "Nick Allen" <n...@nickallen.org> wrote:
> >
> >Matt, et al - For JIRAs that are going into master, should we be
> marking
> >these as "Next + 1" or "0.4.1" ?
> >
> >On Thu, Aug 31, 2017 at 8:17 AM zeo...@gmail.com <zeo...@gmail.com>
> wrote:
> >
> >> Can I advocate to get METRON-1129 in the RC, and throw in a second
> vote for
> >> METRON-1134?  Both in an attempt to better support of prod/offline
> use.
> >> Happy to provide testing cycles for the former.
> >>
> >> Jon
> >>
> >> On Wed, Aug 30, 2017 at 11:41 AM Anand Subramanian <
> >> asubraman...@hortonworks.com> wrote:
> >>
> >> > Hi Matt,
> >> >
> >> > This defect needs to be included as a follow-on to METRON-1122:
> >> > * METRON-1141 (https://github.com/apache/metron/pull/723)
> >> >
> >> >
> >> > Thanks,
> >> > Anand
> >> >
> >> >
> >> >
> >> > On 8/30/17, 8:57 PM, "Michael Miklavcic" <
> michael.miklav...@gmail.com>
> >> > wrote:
> >> >
> >> > >I have some work around fixing how we handle config with Ambari
> that I'd
> >> > >like to see go in. No PR yet, but coming soon. I expect to have
> this by
> >> > the
> >> > >RC deadline.
> >> > >
> >> > >Mike
> >> > >
> >> > >On Wed, Aug 30, 2017 at 8:57 AM, Nick Allen <n...@nickallen.org>
> wrote:
> >> > >
> >> > >> The following PRs are usability enhancements for the
> Profiler.  They
> >> are
> >> > >> fairly simple and I think are very helpful for
> troubleshooting.  I
> >> don't
> >> > >> want to hold up the release, but it would be a "nice to have"
> to get
> >> > these
> >> > >> in.
> >> > >>
> >> > >> If anyone has cycles, I would appreciate some reviews of these
> PRs.
> >> > >> https://github.com/apache/metron/pull/721
> >> > >> https://github.com/apache/metron/pull/716
> >> > >>
> >> > >>
> >> > >>
> >> > >> On Tue, Aug 29, 2017 at 8:59 PM Matt Foley <ma...@apache.org>
> wrote:
> >> > >>
> >> > >> > Okay, just to be clear, you’re requesting we wait until
> Friday, if
> >> > >> > necessary, to cut an RC with 717 in it?
> >> > >> > Thanks,
> >> > >> > --Matt
> >> > >> >
> >> > >> > On 8/29/17, 11:45 AM, "Casey Stella" <ceste...@gmail.com>
> wrote:
> >> > >> >
> >> > >> > 709 is in and 717 is under concerted review by Otto.
> I'd like
> >> to
> >> > see
> >> > >> > it in
> >> > >> > by Friday.
> >> > >> >
> >> > >> > On Tue, Aug 29, 2017 at 1:35 PM, Matt Foley <
> ma...@apache.org>
> >> > >> wrote:
> >> > >> >
> >> > >> > > Hi all,
> >> > >> > > Thanks for your inputs.  The three PRs Nick mentioned
> have
> >> been
> >> > >> > > successfully closed.
> >> > >> > > Casey, do you have an ETA for these two PRs? (PR#709
> and 717)
> >> > >> > > Thanks,
> >> >

Re: [ANNOUNCE] Metron community meeting

[zeo...@gmail.com]

Apologies if this was in a separate thread, but I'm struggling to find it.
Were we planning to establish a standard cadence of these meetings?  I
would give that a big +1.

Jon

On Thu, Aug 24, 2017 at 11:26 AM Casey Stella <ceste...@gmail.com> wrote:

> I should be more clear, for discussions like this, I treat them open
> forever.  For things that are asking for consensus, it generally goes like
> a PR.  People discuss until the voices sort of die down and some loose
> consensus is built.  I normally go by the following conditions:
>
>- 1 full weekend plus a business day has been elapsed
>- Some general consensus is reached
>- A full day has gone by without any dissent
>
> If nobody responds, then treat that as consent by silence.  That's what I
> normally use. :)
>
> On Thu, Aug 24, 2017 at 11:09 AM, Casey Stella <ceste...@gmail.com> wrote:
>
> > forever as far as I'm concerned.  They don't really close so much as they
> > are forgotten. :)
> >
> > On Thu, Aug 24, 2017 at 11:08 AM, Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> >> Casey, what is the time frame for the discuss thread to ‘close’?
> >>
> >>
> >> On August 24, 2017 at 10:06:12, Casey Stella (ceste...@gmail.com)
> wrote:
> >>
> >> Yeah, sorry about that; technology (and webex in particular) is such a
> >> fragile thing sometimes. I think the synopsis is pretty complete. That
> >> being said, if there's anything that you want clarity on, we can hammer
> >> it
> >> out on the discuss thread too.
> >>
> >> On Wed, Aug 23, 2017 at 4:28 PM, James Sirota <jsir...@apache.org>
> >> wrote:
> >>
> >> > Hi Guys, apologies about this. I couldn't record yesterday, but Casey
> >> > posted a synopsis of the meeting.
> >> >
> >> > 22.08.2017, 10:27, "James Sirota" <jsir...@apache.org>:
> >> > > Yes, I will post a recording
> >> > >
> >> > > 21.08.2017, 14:57, "Kyle Richardson" <kylerichards...@gmail.com>:
> >> > >> Unfortunately I have a conflict. Will there be a recording posted
> >> > afterward
> >> > >> like previous meetings?
> >> > >>
> >> > >> -Kyle
> >> > >>
> >> > >> On Mon, Aug 21, 2017 at 10:35 AM Otto Fowler <
> >> ottobackwa...@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >>> Sounds good
> >> > >>>
> >> > >>> On August 21, 2017 at 09:43:25, James Sirota (jsir...@apache.org)
> >> > wrote:
> >> > >>>
> >> > >>> Hi Jon,
> >> > >>>
> >> > >>> Sure. Lets move it by a day. The reason it's at this time is to
> >> give
> >> > people
> >> > >>> in India and Europe a chance to attend live.
> >> > >>>
> >> > >>> So lets move it to the same time tomorrow
> >> > >>>
> >> > >>> I would like to schedule it for Tuesday, Aug. 22. at 10.30AM PST.
> >> > >>>
> >> > >>> I would like to have it over webex:
> >> > >>>
> >> > >>> https://hortonworks.webex.com/hortonworks/j.php?MTID=
> >> > m5621bcdbf6163c7df1568b41fc1a1d93
> >> > >>> Meeting number: 622 537 955
> >> > >>> Meeting password: biFTEuh2
> >> > >>>
> >> > >>> For global callers:
> >> > >>>
> >> > >>> https://hortonworks.webex.com/hortonworks/globalcallin.
> >> > php?serviceType=MC=590161912=1
> >> > >>>
> >> > >>> Thanks,
> >> > >>> James
> >> > >>>
> >> > >>> 18.08.2017, 11:02, "zeo...@gmail.com" <zeo...@gmail.com>:
> >> > >>> > Is it possible to reschedule this to later in the day or another
> >> > day?
> >> > >>> That
> >> > >>> > overlaps with the eclipse on the east cost of the US that some
> >> > people
> >> > >>> would
> >> > >>> > like to enjoy.
> >> > >>> >
> >> > >>> > Jon
> >> > >>> >
> >> > >>> > On Fri, Aug 18, 2017, 13:48 James Sirota <jsir...@apache.org>
> >> > wrote:
> 

Re: [DISCUSS] Metron release 0.4.1

[zeo...@gmail.com]

Looks like METRON-1134 made it in already and I just gave METRON-1129
a +1.  Just waiting on Ryan's feedback to merge it in.

Jon

On Thu, Aug 31, 2017 at 3:11 PM Matt Foley <mfo...@hortonworks.com> wrote:

> Please mark them 0.4.1, as that’s what the community says we want to call
> the upcoming release, and everything that’s there when I throw the switch
> will be included.
>
> Jon and Anand, will they be in by end/day Friday?
> Thanks,
> --Matt
>
> On 8/31/17, 7:45 AM, "Nick Allen" <n...@nickallen.org> wrote:
>
> Matt, et al - For JIRAs that are going into master, should we be
> marking
> these as "Next + 1" or "0.4.1" ?
>
> On Thu, Aug 31, 2017 at 8:17 AM zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
> > Can I advocate to get METRON-1129 in the RC, and throw in a second
> vote for
> > METRON-1134?  Both in an attempt to better support of prod/offline
> use.
> > Happy to provide testing cycles for the former.
> >
> > Jon
> >
> > On Wed, Aug 30, 2017 at 11:41 AM Anand Subramanian <
> > asubraman...@hortonworks.com> wrote:
> >
> > > Hi Matt,
> > >
> > > This defect needs to be included as a follow-on to METRON-1122:
> > > * METRON-1141 (https://github.com/apache/metron/pull/723)
> > >
> > >
> > > Thanks,
> > > Anand
> > >
> > >
> > >
> > > On 8/30/17, 8:57 PM, "Michael Miklavcic" <
> michael.miklav...@gmail.com>
> > > wrote:
> > >
> > > >I have some work around fixing how we handle config with Ambari
> that I'd
> > > >like to see go in. No PR yet, but coming soon. I expect to have
> this by
> > > the
> > > >RC deadline.
> > > >
> > > >Mike
> > > >
> > > >On Wed, Aug 30, 2017 at 8:57 AM, Nick Allen <n...@nickallen.org>
> wrote:
> > > >
> > > >> The following PRs are usability enhancements for the Profiler.
> They
> > are
> > > >> fairly simple and I think are very helpful for
> troubleshooting.  I
> > don't
> > > >> want to hold up the release, but it would be a "nice to have"
> to get
> > > these
> > > >> in.
> > > >>
> > > >> If anyone has cycles, I would appreciate some reviews of these
> PRs.
> > > >> https://github.com/apache/metron/pull/721
> > > >> https://github.com/apache/metron/pull/716
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Aug 29, 2017 at 8:59 PM Matt Foley <ma...@apache.org>
> wrote:
> > > >>
> > > >> > Okay, just to be clear, you’re requesting we wait until
> Friday, if
> > > >> > necessary, to cut an RC with 717 in it?
> > > >> > Thanks,
> > > >> > --Matt
> > > >> >
> > > >> > On 8/29/17, 11:45 AM, "Casey Stella" <ceste...@gmail.com>
> wrote:
> > > >> >
> > > >> > 709 is in and 717 is under concerted review by Otto.  I'd
> like
> > to
> > > see
> > > >> > it in
> > > >> > by Friday.
> > > >> >
> > > >> > On Tue, Aug 29, 2017 at 1:35 PM, Matt Foley <
> ma...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > Hi all,
> > > >> > > Thanks for your inputs.  The three PRs Nick mentioned
> have
> > been
> > > >> > > successfully closed.
> > > >> > > Casey, do you have an ETA for these two PRs? (PR#709
> and 717)
> > > >> > > Thanks,
> > > >> > > --Matt
> > > >> > >
> > > >> > > On 8/29/17, 9:34 AM, "Casey Stella" <ceste...@gmail.com
> >
> > wrote:
> > > >> > >
> > > >> > > For my PRs, I'd vote for METRON-1122 being in
> (commit very
> > > >> > imminent).
> > > >> > > I'd very much like METRON-1134 to be in as well.
> > > >> > &

Re: [DISCUSS] Metron release 0.4.1

[zeo...@gmail.com]

Can I advocate to get METRON-1129 in the RC, and throw in a second vote for
METRON-1134?  Both in an attempt to better support of prod/offline use.
Happy to provide testing cycles for the former.

Jon

On Wed, Aug 30, 2017 at 11:41 AM Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> Hi Matt,
>
> This defect needs to be included as a follow-on to METRON-1122:
> * METRON-1141 (https://github.com/apache/metron/pull/723)
>
>
> Thanks,
> Anand
>
>
>
> On 8/30/17, 8:57 PM, "Michael Miklavcic" 
> wrote:
>
> >I have some work around fixing how we handle config with Ambari that I'd
> >like to see go in. No PR yet, but coming soon. I expect to have this by
> the
> >RC deadline.
> >
> >Mike
> >
> >On Wed, Aug 30, 2017 at 8:57 AM, Nick Allen  wrote:
> >
> >> The following PRs are usability enhancements for the Profiler.  They are
> >> fairly simple and I think are very helpful for troubleshooting.  I don't
> >> want to hold up the release, but it would be a "nice to have" to get
> these
> >> in.
> >>
> >> If anyone has cycles, I would appreciate some reviews of these PRs.
> >> https://github.com/apache/metron/pull/721
> >> https://github.com/apache/metron/pull/716
> >>
> >>
> >>
> >> On Tue, Aug 29, 2017 at 8:59 PM Matt Foley  wrote:
> >>
> >> > Okay, just to be clear, you’re requesting we wait until Friday, if
> >> > necessary, to cut an RC with 717 in it?
> >> > Thanks,
> >> > --Matt
> >> >
> >> > On 8/29/17, 11:45 AM, "Casey Stella"  wrote:
> >> >
> >> > 709 is in and 717 is under concerted review by Otto.  I'd like to
> see
> >> > it in
> >> > by Friday.
> >> >
> >> > On Tue, Aug 29, 2017 at 1:35 PM, Matt Foley 
> >> wrote:
> >> >
> >> > > Hi all,
> >> > > Thanks for your inputs.  The three PRs Nick mentioned have been
> >> > > successfully closed.
> >> > > Casey, do you have an ETA for these two PRs? (PR#709 and 717)
> >> > > Thanks,
> >> > > --Matt
> >> > >
> >> > > On 8/29/17, 9:34 AM, "Casey Stella"  wrote:
> >> > >
> >> > > For my PRs, I'd vote for METRON-1122 being in (commit very
> >> > imminent).
> >> > > I'd very much like METRON-1134 to be in as well.
> >> > >
> >> > > Beyond that, I'm ok
> >> > >
> >> > > On Tue, Aug 22, 2017 at 4:37 PM, Nick Allen <
> >> n...@nickallen.org>
> >> > > wrote:
> >> > >
> >> > > > Thanks for starting the process, Matt.
> >> > > >
> >> > > > These are my own open PRs that I would most like to see
> get
> >> in.
> >> > > They all
> >> > > > relate to the Profiler. My other outstanding PRs are less
> >> > important.
> >> > > >
> >> > > >- https://github.com/apache/metron/pull/705
> >> > > >- https://github.com/apache/metron/pull/707
> >> > > >- https://github.com/apache/metron/pull/708
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Tue, Aug 22, 2017 at 4:17 PM Matt Foley <
> ma...@apache.org
> >> >
> >> > wrote:
> >> > > >
> >> > > > > Hello all,
> >> > > > > At the meeting this morning, the community expressed the
> >> > wish to
> >> > > create a
> >> > > > > new release, to be a point release and not including
> >> > METRON-777.
> >> > > > >
> >> > > > > Therefore, I propose to create release 0.4.1 from Metron
> >> > master
> >> > > branch,
> >> > > > > with whatever additional commits the community considers
> >> > vital and
> >> > > can
> >> > > > get
> >> > > > > in in the next few days.
> >> > > > >
> >> > > > > There have been 74 commits since the 0.4.0 release (list
> >> > appended
> >> > > to end
> >> > > > > of this message).
> >> > > > > I have not yet done a compare to make sure all 74
> completed
> >> > > commits have
> >> > > > > closed their Jiras, but I will follow up with that info.
> >> > > > >
> >> > > > > There are 35 open PRs, see
> >> > https://github.com/apache/metron/pulls
> >> > > ,
> >> > > > which
> >> > > > > is getting a little clunky.  Surely not all of them can
> get
> >> > in in
> >> > > the
> >> > > > next
> >> > > > > few days, but do we want to try to close and include
> some
> >> of
> >> > them?
> >> > > > >
> >> > > > > Please respond to the list with your requests to include
> >> > specific
> >> > > > > additional PRs before we cut the 0.4.1 release
> candidate.
> >> > Let’s
> >> > > keep it
> >> > > > > practical so we don’t delay more than a week or so.
> >> > > > >
> >> > > > > TIMEFRAME:  Tentatively I would like to start the
> process
> >> of
> >> > > cutting an
> >> > > > RC
> >> > > > > this coming 

Re: [DISCUSS] METRON-777 and the road to perditi... er enlightenment

[zeo...@gmail.com]

This is all great stuff.  As far as feature branch naming, I would suggest
something like feature/$brief_explanation accompanied with a feature branch
JIRA that explains the original intent of the branch and its
goals/"complete" indicators.

Along the lines of the FEATURE.md, I feel like at the very least we should
update the dev guidelines
,
but doing more than that could potentially be warranted.

+1

Jon

On Wed, Aug 23, 2017 at 12:38 PM Nick Allen  wrote:

> +1 I like it all, Otto.  You deserve a freakin' medal.
>
>
>
>
>
> On Wed, Aug 23, 2017 at 10:04 AM Otto Fowler 
> wrote:
>
> > WRT : regression fixes, I would also like us to consider putting these
> the
> > initial 777 to feature branch PR as an option.
> >
> >
> > On August 23, 2017 at 09:56:33, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> >
> >
> > A feature branch
> >
> > As discussed in the community meeting we are going to create a ‘feature’
> > branch for METRON–777 and it’s associated PR’s. The purpose behind the
> > feature branch is to more formally stage a large PR or set of PR’s to
> allow
> > for for better review participation ( of the pr and changes due to
> feedback
> > ) and more formal collaboration and testing.
> >
> > In order to accomplish this, there are several things that will have to
> be
> > done. I believe they are:
> >
> >1. A new branch shall be made in the apache-git repo.
> >2. I will rebase my METRON–777 branch to that new branch and submit a
> PR
> >to it in github, which we will land.
> >3. I will also follow the same procedure for METRON–942 with the
> >expectation of landing it.
> >
> > This will give us the full set of functionality as currently set of out
> for
> > extensions.
> >
> > Currently there are two other branches that I have to fix regressions or
> > missing required functionality.
> >
> >1. I will merge these into one ( they are related ).
> >2. I will then submit them are PR’s
> >
> > Hopefully we can get these into the branch, so that we can then test the
> > whole end to end functionality, and have the ability to review and
> discuss
> > the design decisions made against a working system.
> >
> > To that end, after these PR sets land, I will make a video around using
> the
> > features.
> >
> > The end result of the feature branch and the effort will be a branch that
> > has met all the criteria and and acceptance standards set out for it.
> Will
> > will have some community activity at that time as a means of introduction
> > and demonstration.
> > Questions:
> >
> >- What should the naming convention for the branch be? feature-?
> >- Should there be a new FEATURE.md that has the equivalent of the PR
> >description? or should the README.md be modified ( to be changed back
> >before merge ) so that it is prominent?
> >- We need to set some criteria for ‘done’, and record that somewhere.
> >Should there be a feature branch jira? I think we need one for the
> > eventual
> >merge anyways.
> >- other?
> >
> > Please provide any feedback or other questions so that we can proceed
> > without much more delay.
> > Refresher : What is METRON–777?
> >
> > METRON–777 one of two PR’s to implement the side loading of Metron
> parsers
> > ( METRON–258 ).
> >
> > In order to provide a working, testable build that meets our pr
> > requirements, the functionality was only broken down into two PR’s:
> >
> >-
> >
> >[#530 METRON-777 Metron Extension System and Parser Extensions](
> >https://github.com/apache/metron/pull/530)
> >-
> >
> >[#590 METRON-942 Rest api and configuration for Metron parser
> > extensions]
> >(https://github.com/apache/metron/pull/580)
> >
> > METRON–777 provides the architecture support and changes and moves the
> > current parsers over to the new architecture.
> >
> > METRON–942 provides the rest api to install a 3rd party parser extension,
> > and a stellar command for reading extension configurations
> >
> > it should be noted that this set of features is still incomplete, as the
> > metron-config ui should be changed to provide a ui front end for
> extension
> > management
> >
>
-- 

Jon

Re: [DISCUSS] Synopsis of Community Meeting on 8/22/2017

[zeo...@gmail.com]

Was there any discussion about future features of Metron aside from
777/942? In the initial announce thread the agenda mentioned where want to
take the project long-term and feature requests and comments on existing
features.

My thoughts on the topic are that I would like to see a move quickly after
0.4.1 to upgrade the stack, from centos 7, to elasticsearch 5, bro 2.5.1,
HDP 2.6.1, etc.  There's a growing list of issues due to the older
platforms we use.  I have already started work on some of this, and I know
we have a PR open for an ES upgrade as well.

More forward looking, I would also like to see a push for exactly once
processing through the stack, which I expect will be available with HDP
3.x?

I wholeheartedly agree that we should get back on top of the community
demos.  They have been helpful for me to watch and are very good materials
for pointing people new to the project to, in addition to our site-book.

Jon

On Tue, Aug 22, 2017, 15:00 Casey Stella  wrote:

> Introduction by James Sirota and hand off to me.
>
> Topic: Apache Administrivia
>
>- We have passed our 3 month window of submitting board reports without
>any serious incident
>- We need and want more committers.  Anyone who is so inclined and
>interested should pitch in, even without code contributions
>
> Topic: Release
>
>- General consensus that a release is in order
>- The release should be a point release and pre-777
>
> Topic: The Next Steps for METRON-777
>
>- Proposed: We should move METRON-777 and METRON-942 into a feature
>branch
>- Proposed: Once working sufficiently, the first milestone is for Otto
>to create a youtube video showing off his work and have a discuss thread
>for a second pass of architecture/feature review
>- Proposed: This feature branch should have no less stringent
>requirements than we have for master
>- Proposed: Documentation, testing, and the normal accompanying
>artifacts for any other feature should be in place before we merge.
> Other
>more feature-specific criteria should be discussed in separate discuss
>threads.
>- No dissenting opinions on the call.  Otto to create a discuss thread
>about the feature branch and the mechanics of actually creating it.
>
> Topic: Community Demos
>
>- Proposed: We should have community demos again on a monthly basis.
>- General assent to this as well.
>- James to set up a webex for one month hence.
>
> Anything that I got wrong or missed, please respond to this discuss thread.
>
> Best,
>
> Casey
>
-- 

Jon

Re: [ANNOUNCE] Metron community meeting

[zeo...@gmail.com]

Thanks, I'll be there.

Jon

On Mon, Aug 21, 2017 at 9:43 AM James Sirota <jsir...@apache.org> wrote:

> Hi Jon,
>
> Sure.  Lets move it by a day.  The reason it's at this time is to give
> people in India and Europe a chance to attend live.
>
> So lets move it to the same time tomorrow
>
>  I would like to schedule it for Tuesday, Aug. 22. at 10.30AM PST.
>
>  I would like to have it over webex:
>
>
>
> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93
>  Meeting number: 622 537 955
>  Meeting password: biFTEuh2
>
>  For global callers:
>
>
> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1
>
> Thanks,
> James
>
> 18.08.2017, 11:02, "zeo...@gmail.com" <zeo...@gmail.com>:
> > Is it possible to reschedule this to later in the day or another day?
> That
> > overlaps with the eclipse on the east cost of the US that some people
> would
> > like to enjoy.
> >
> > Jon
> >
> > On Fri, Aug 18, 2017, 13:48 James Sirota <jsir...@apache.org> wrote:
> >
> >>  I would like to propose a meeting with the following set of topics:
> >>
> >>  - What do we want in the next build
> >>  - Where do we want to take the project long-term
> >>  - Any feature requests/comments on existing feature set
> >>  - Feedback on some of the early UI work
> >>
> >>  Please suggest if there is anything else you want to talk about.
> >>
> >>  I would like to schedule it for Monday, Aug. 21. at 10.30AM PST.
> >>
> >>  I would like to have it over webex:
> >>
> >>
> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93
> >>  Meeting number: 622 537 955
> >>  Meeting password: biFTEuh2
> >>
> >>  For global callers:
> >>
> >>
> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1
> >>
> >>  Anyone is welcome to join.
> >>  ---
> >>  Thank you,
> >>
> >>  James Sirota
> >>  PPMC- Apache Metron (Incubated and Hatched)
> >>  jsirota AT apache DOT org
> > --
> >
> > Jon
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
-- 

Jon

Re: [ANNOUNCE] Metron community meeting

[zeo...@gmail.com]

Is it possible to reschedule this to later in the day or another day?  That
overlaps with the eclipse on the east cost of the US that some people would
like to enjoy.

Jon

On Fri, Aug 18, 2017, 13:48 James Sirota  wrote:

> I would like to propose a meeting with the following set of topics:
>
> - What do we want in the next build
> - Where do we want to take the project long-term
> - Any feature requests/comments on existing feature set
> - Feedback on some of the early UI work
>
> Please suggest if there is anything else you want to talk about.
>
> I would like to schedule it for Monday, Aug. 21. at 10.30AM PST.
>
> I would like to have it over webex:
>
>
> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93
> Meeting number: 622 537 955
> Meeting password: biFTEuh2
>
> For global callers:
>
> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1
>
>
> Anyone is welcome to join.
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubated and Hatched)
> jsirota AT apache DOT org
>
-- 

Jon

Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?

[zeo...@gmail.com]

I'm grasping at straws but maybe play with your batch sizes?  I hear now
there's a time based option too that I haven't played with.

It would be helpful to have more metrics about resource utilization over
time in your cluster, are you able to gather anything and maybe put it into
a visualization tool?

Jon

On Thu, Aug 17, 2017, 16:35 Laurens Vets <laur...@daemon.be> wrote:

> That seems close to the issue that I'm having... But there's a part I
> don't quite understand in my case.
>
> Everything's working fine for days and then suddenly, Java throws an
> error (see other mail to the mailing list) and I can't get Metron stable
> again. It's always either the indexingBolt or hdfsIndexingBolt. The
> error I see the most after this is "java.lang.OutOfMemoryError: GC
> overhead limit exceeded"...
>
> It seems that somewhere there's a big pileup of messages which Metron
> suddenly can't process.
>
> Any ideas on how to further troubleshoot this?
>
> On 2017-08-17 11:10, zeo...@gmail.com wrote:
> > I used to run into similar issues when my environment was resource
> > constrained but never ran it to root cause.  It has been a long time
> > since
> > I was in this scenario to re-test.
> >
> > https://issues.apache.org/jira/projects/METRON/issues/METRON-485
> >
> > Jon
> >
> > On Thu, Aug 17, 2017 at 12:49 PM Laurens Vets <laur...@daemon.be>
> > wrote:
> >
> >> Hi,
> >>
> >> Simple question, when I stop Metron, Kafka & Storm via Ambari, I still
> >> see the storm worker processes running, is this expected?
> >>
>
-- 

Jon

Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?

[zeo...@gmail.com]

I used to run into similar issues when my environment was resource
constrained but never ran it to root cause.  It has been a long time since
I was in this scenario to re-test.

https://issues.apache.org/jira/projects/METRON/issues/METRON-485

Jon

On Thu, Aug 17, 2017 at 12:49 PM Laurens Vets  wrote:

> Hi,
>
> Simple question, when I stop Metron, Kafka & Storm via Ambari, I still
> see the storm worker processes running, is this expected?
>
-- 

Jon

Re: Upgrade vagrant base to centos 7

[zeo...@gmail.com]

Ahh, perfect, I didn't know we stored that in the repo too.  I will take a
look thanks.

Jon

On Mon, Aug 7, 2017 at 8:18 AM David Lyle <dlyle65...@gmail.com> wrote:

> Hi Jon,
>
> I'm not sure how to answer your question, but the base image (used for
> full-dev) we use is defined here:
>
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/packer-build/base-centos-6.7.json
> .
> It's forked from  bento/centos6.7, but not derived from it. Same deal with
> quick-dev.
>
> Readme for building the boxes is here:
>
> https://github.com/apache/metron/blob/master/metron-deployment/packaging/packer-build/README.md
> .
>
> -D...
>
>
> On Sun, Aug 6, 2017 at 10:34 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > https://issues.apache.org/jira/browse/METRON-667
> >
> >
> >
> > On August 6, 2017 at 08:54:50, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
> >
> > I'm working on a few updates/changes to the bro portions of Metron, in
> > preparation for BroCon <https://www.bro.org/community/brocon2017.html>
> in
> > September. I'm running into a couple of dependency issues which would be
> > most cleanly resolved by a move to a centos 7 base, so I was going to
> take
> > on METRON-559 <https://issues.apache.org/jira/browse/METRON-559> first,
> > before moving on METRON-1088
> > <https://issues.apache.org/jira/browse/METRON-1088> and METRON-813
> > <https://issues.apache.org/jira/browse/METRON-813>.
> >
> > It looks like as of 3/10
> > <
> > https://github.com/apache/metron/commit/68a334a8717d2a6b82f7f9651c57bc
> > 75e256ba97>
> >
> > we cut over from using bento on full-dev to using our own base vagrant
> > box. Do we have any information regarding what was run on that image, so
> I
> > could attempt to migrate those changes to a new base image? I saw in the
> > commentary of the PR that maybe this is just a version pinned bento
> > centos6.7 box with some minor swap space changes?
> >
> > Thanks,
> >
> > Jon
> > --
> >
> > Jon
> >
>
-- 

Jon

Re: [DISCUSS] Relocate Docker

[zeo...@gmail.com]

I agree to moving it to a contrib or contrib-like area.

Jon

On Thu, Jul 13, 2017 at 12:38 PM Kyle Richardson 
wrote:

> I completely support the idea of moving metron-docker down in the tree. I
> do like the idea of a contrib/ area for things like this that are not as
> frequently updated or maintained. Are there any other pieces of the code
> base that would fit into this type of area?
>
> -Kyle
>
> On Thu, Jul 13, 2017 at 12:30 PM Laurens Vets  wrote:
>
> > On 2017-07-13 09:04, Nick Allen wrote:
> > > Having metron-docker at the top-level of the project seems to catch the
> > > attention of new users.  Some then start using metron-docker to
> > > explore/try-out/demo Metron.
> > >
> > > The metron-docker code that we have is not well-suited for this
> > > purpose.
> > > It is only really useful for development.  It is not regularly tested
> > > and
> > > maintained like our Vagrant environment.
> > >
> > > I am proposing that we move the top-level "metron-docker" directory to
> > > live
> > > under "metron-deployment" to avoid confusing new users.  This also
> > > seems
> > > more logical to me. I would also pair this move with some better
> > > "Getting
> > > Started" steps for new users in the top-level README.
> > >
> > > cd $METRON_HOME
> > > mv metron-docker metron-deployment/docker
> > >
> > >
> > > Do you agree/disagree?  Is there a better solution?
> >
> > Maybe under "Other Examples"?
> >
>
-- 

Jon

Re: [Request for Consensus Approval] dev branch for Stellar additional work

[zeo...@gmail.com]

That all sounds pretty reasonable to me.  My biggest concern would be
attribution during step 5 - we would need to make sure it isn't squash
merged like we typically do (assuming we do properly squash merge into
the speculative
branch).  Not a big issue though, I guess, just need to make sure it isn't
overlooked.

Jon

On Wed, Jul 5, 2017 at 4:40 AM Matt Foley  wrote:

> Now that METRON-877 is in, I would like to proceed with Steps 3-6 of the
> remaining work to separate out Stellar functionality as an independent
> module.  A couple people have suggested that this further development
> should be done in a Metron “dev branch”, where:
> a) changes are more visible than in a single person’s private development
> branch, and
> b) work can proceed for several days or a couple weeks on a branch that
> the collaborators may choose to keep stable for the duration (ie, without
> constantly updating to master).
>
> This concept was discussed as a “speculative branch” in this email thread:
> https://lists.apache.org/thread.html/391e15347ad625c4aa61e81f5dd238c0acb4048b8d77f93313298263@%3Cdev.metron.apache.org%3E
> but I don’t see that we ever actually changed our bylaws to mention it.
>
> Nevertheless, it falls within the purview of the PMC to create new
> branches in our code tree, and I request PMC members to give me a lazy
> consensus vote to do so.  Please +1 this email if you agree.
>
> The proposed rules of engagement are (drawn from issues raised in that
> email thread):
> 1. Commits to this branch to have the same rules as to master:  Jira, PR,
> and at least one +1 from a knowledgeable reviewer, and no -1’s.
> 2. +1 reviews may come from any participating contributor, not only
> current committers.  But commits still have to be made by a committer, so
> we don’t have to create new auth infra for this branch.
> 3. The branch should be updated from master at least every second week, or
> more frequently.  This may be adjusted to avoid disruption of work in
> progress.
> 4. PR’s to master will be posted for review as soon as self-consistent
> chunks of useful functionality are done.  The collaborators will define
> those chunks, but a rough goal is every two weeks.  The goal is to avoid
> mega-patches to review.
> 5. PR’s to master will be posted by a single developer from their home
> github repo, not directly from the speculative branch, so that
> collaborative work can proceed on the speculative branch.
> 6. The PR’s will be credited equally to all collaborators active during
> that “chunk” of work.
> 7. PR’s to master have to be reviewed and agreed to as though they were
> new patches. (The fact they were previously accepted into the speculative
> branch is at most a recommendation, not an a priori decision to let them
> into master.) The usual rules apply.  While collaborators will likely want
> to +1 such PRs, sufficient time must be provided for other community
> members to review and raise issues.
>
> Thanks,
> --Matt
>
>
>
> --

Jon

Re: [DISCUSS] Mutation of Indexed Data

[zeo...@gmail.com]

The key should be a solved problem as of METRON-765
,
right?  It provides a single key for a given message that globally stored
with the message, regardless of where/how.

Jon

On Thu, Jun 22, 2017 at 9:01 AM Justin Leet  wrote:

> First off, I agree with the characteristics.
>
> For the data stores, we'll need to be able to make sure we can actually
> handle the collapsing of the updates into a single view.  Casey mentioned
> making the long term stores transparent, but there's potentially work for
> the near real time stores: we need to make sure we actually do updates,
> rather than create new docs that aren't linked to the old ones. This should
> be entirely transparent and handled by a service layer, rather than
> anything hardcoded to a datastore.
>
> For ES at least, the only way to do this is to retrieve, mutate it, and
> then reindex (even the updates API does that dance under the hood for you,
> and since we're potentially doing non trivial changes we might need to
> manage it ourselves).  This implies the existence of a key, even if one
> isn't enforced by ES (Which I don't believe it will be).  We need to be
> able to grab the doc(s?) to be updated, not end up with similar ones that
> shouldn't be mutated.  I assume this is also true (at least the
> generalities) of Solr as well.
>
> In concert with your other thread, couldn't part of this key end up being
> metadata (either user defined or environment defined)?  For example, in a
> situation where customer id is applied as metadata, it's possible two
> customers feed off the same datasource, but may need to mutate
> independently.  At this point, we have metadata that is effectively keyed.
> We don't want to update both docs, but there's not a real way to
> distinguish them.  And maybe that's something we push off for the short
> term, but it seems potentially nontrivial.
>
> In terms of consistency, I'd definitely agree that the long-term storage
> can be eventually consistent.  Any type of bulk spelunking, Spark jobs,
> dashboarding, etc. shouldn't need up to the millisecond data.
>
> Basically, I'm thinking the real time store is the snapshot of current
> state, and the long term store is the full record complete with the lineage
> history.
>
> I'm also interested in people's opinions on how we want to manage HDFS.
> Assuming we do use HBase to store our updates, that means that every HDFS
> op has to join onto that HBase table to get any updates that HDFS is
> missing (unless we implement some writeback and merge for HDFS data).  I'm
> worried that our two datastores are really: ES, HDFS+HBase.  And that
> keeping that data actually synced to end users is going to be painful.
>
> Justin
>
>
> On Wed, Jun 21, 2017 at 10:18 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > I'd say that was an excellent set of requirements (very similar to the
> one
> > we arrived on with the last discuss thread on this)
> >
> > My vote remains a transaction log in hbase given the relatively low
> volume
> > (human scale) i would not expect this to need anything fancy like
> > compaction into hdfs state, but that does make a good argument for a long
> > term dataframe solution for spark, with a short term stop gap using a
> > joined data frame and shc.
> >
> > Simon
> >
> > Sent from my iPhone
> >
> > > On 22 Jun 2017, at 05:11, Otto Fowler  wrote:
> > >
> > > Can you clarify what data stores are at play here?
> > >
> > >
> > > On June 21, 2017 at 17:07:42, Casey Stella (ceste...@gmail.com) wrote:
> > >
> > > Hi All,
> > >
> > > I know we've had a couple of these already, but we're due for another
> > > discussion of a sensible approach to mutating indexed data. The
> > motivation
> > > for this is users will want to update fields to correct and augment
> data.
> > > These corrections are invaluable for things like feedback for ML models
> > or
> > > just plain providing better context when evaluating alerts, etc.
> > >
> > > Rather than posing a solution, I'd like to pose the characteristics of
> a
> > > solution and we can fight about those first. ;)
> > >
> > > In my mind, the following are the characteristics that I'd look for:
> > >
> > > - Changes should be considered additional or replacement fields for
> > > existing fields
> > > - Changes need to be available in the web view in near real time (on
> the
> > > order of milliseconds)
> > > - Changes should be available in the batch view
> > > - I'd be ok with eventually consistent with the web view, thoughts?
> > > - Changes should have lineage preserved
> > > - Current value is the optimized path
> > > - Lineage search is the less optimized path
> > > - If HBase is part of a solution
> > > - maintain a scan-free solution
> > > - maintain a coprocessor-free solution
> > >
> > > Most of what I've thought of is something along the lines:
> > >
> > > - Diffs 

Re: [Discussion] About the wiki….

[zeo...@gmail.com]

I suggested in the past and got some buy in, but never had time to move
everything into GitHub.  I vote to mostly or entirely archive the wiki.

Jon

On Tue, Jun 13, 2017, 5:19 PM Laurens Vets  wrote:

> On 2017-06-13 14:09, Otto Fowler wrote:
> > I think there are things in the wiki that are very very out of date, to
> > the
> > extent that they are confusing people looking at Metron.
> > Basically anyone going to DOCS HOME from the site is being thrown into
> > documentation that is sure to confuse them.
> >
> > Does anyone have any ideas how we can address this?   Can we prune the
> > wiki
> > and remove the out of date information ( better less than wrong )?
> > Should
> > we replace DOCS HOME with the site book link?
>
> Throw out the old/wrong stuff asap.
>
> I also wouldn't mind trying to get some structure in it but is the wiki
> still necessary if everything will be in markdown in the source
> eventually?
>
-- 

Jon

Re: Installation problem with Docker and processor that does not support virtualization

[zeo...@gmail.com]

Moving to user list

Jon

On Thu, Jun 8, 2017, 5:51 AM <sml...@libero.it> wrote:

> Hei Jon,
>
> thank you.
>
> I don't know which are the differences between 0.3.1 and 0.4.0.
>
> Unfortunately for me, my CPU does not support virtualization.  That means
> that I cannot use Docker.
>
> The only workaround that I found is to use AWS directly but for me that I
> have never used Mentor it could be a so big step...
>
> So the question is, do I lose many things if I start with Mentor 0.3.1
> into a single VM without Docker?
>
> Best regards,
>
> Simone
>
>
> Il 8 giugno 2017 alle 11.37 "zeo...@gmail.com" <zeo...@gmail.com> ha
> scritto:
>
> If I recall properly, 0.3.1 does not require docker yet.  That will come
> with 0.4.0/master.  It still does require virtualization, however, to spin
> up the dev environments (excluding if you were to run it on AWS).
>
> Jon
>
> On Thu, Jun 8, 2017, 2:03 AM <sml...@libero.it> wrote:
>
> Hello,
>
> I didn't check the BIOS if I can enable it.
>
> Then, I see that yesterday has been updated the installation guideline for
> Metron 0.3.1 as follows:
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
>
> Reading that, there is no mention to Docker.
> Maybe for my experiments (I do have to test some ML algo's) and improve my
> knowledge on this tool that installation should be enough.
>
> What do you think about?
>
> Thanks.
>
> Simone
>
>
> Il 7 giugno 2017 alle 23.32 "zeo...@gmail.com" <zeo...@gmail.com> ha
> scritto:
>
> If your processor doesn't support virtualization right now I would suggest
> looking into if it is simply disabled in your BIOS/UEFI (most processers
> have supported this for 10+ years, excluding some processors of course).
> Docker is integrated into the build process right now and is considered
> mandatory (although you technically could work around it with some effort).
>
> Assuming you are spinning up full-dev, vagrant should create a centos 6 VM
> to run Metron in.
>
> Metron is the repository that you cloned from GitHub, downloaded from
> Apache, etc. If you didn't do this, you will need to. Here is our last
> release - http://metron.apache.org/documentation/#releases
>
> Hope that helps
>
> Jon
>
> On Wed, Jun 7, 2017, 4:00 PM <sml...@libero.it> wrote:
>
> Dear All,
>
> I'm installing Metron, following the instructions found here:
>
>
> https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform
>
> Unfortunately, my processor does not support virtualization and I'm not
> able to launch Docker.
>
> Is there any workaround?
>
> I installed Vagrant on my OSX and I assumed to use Vagrant to create a VM
> with Ubuntu in which I would run Metron. Is it right?
>
> Another question about the instructions, I do not really understand where
> I get Metron.
>
> In this point:
>
>
>1. Deploy Metron
>
> cd metron-deployment/vagrant/full-dev-platform
> vagrant up
>
> I understood that I should have already downloaded Metron, but I don't.
> Where is Metron?
>
> Thank you.
> Simone
>
> --
>
> Jon
>
> --
>
> Jon
>
> --

Jon

Re: Installation problem with Docker and processor that does not support virtualization

[zeo...@gmail.com]

If your processor doesn't support virtualization right now I would suggest
looking into if it is simply disabled in your BIOS/UEFI (most processers
have supported this for 10+ years, excluding some processors of course).
Docker is integrated into the build process right now and is considered
mandatory (although you technically could work around it with some effort).

Assuming you are spinning up full-dev, vagrant should create a centos 6 VM
to run Metron in.

Metron is the repository that you cloned from GitHub, downloaded from
Apache, etc.  If you didn't do this, you will need to.  Here is our last
release - http://metron.apache.org/documentation/#releases

Hope that helps

Jon

On Wed, Jun 7, 2017, 4:00 PM  wrote:

> Dear All,
>
> I'm installing Metron, following the instructions found here:
>
>
> https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform
>
> Unfortunately, my processor does not support virtualization and I'm not
> able to launch Docker.
>
> Is there any workaround?
>
> I installed Vagrant on my OSX and I assumed to use Vagrant to create a VM
> with Ubuntu in which I would run Metron. Is  it right?
>
> Another question about the instructions, I do not really understand where
> I get Metron.
>
> In this point:
>
> 2. Deploy Metron
>
> cd metron-deployment/vagrant/full-dev-platform
> vagrant up
>
> I understood that I should have already downloaded Metron, but I don't.
> Where is Metron?
>
> Thank you.
> Simone
>
-- 

Jon

Re: [INCOMING] Metron 0.4.0 release (RC3)

[zeo...@gmail.com]

What about 976, which follows the Kerberized trend for this release?

Jon

On Thu, Jun 1, 2017, 6:03 PM Nick Allen  wrote:

> Sounds good, Matt.  Looking forward to cutting this release.
>
> On Thu, Jun 1, 2017 at 5:17 PM, Matt Foley  wrote:
>
> > Hi all,
> >
> > Now that METRON-844 is in, I plan to proceed with the 0.4.0 release
> > candidate.  I think 844 was the last item we considered a must-have for
> the
> > 0.4.0 release, and we want to post this release before incorporating
> > several broad changes that should go in the NEXT release not this one
> > (METRON-777, 942, 975, 876, etc.).
> >
> >
> >
> > Let me know if anything else is considered must-have for 0.4.0, but I’m
> > going to proceed unless I hear otherwise.  Current HEAD is SHA1=
> 85872bd686
> > (METRON-858), I’ll go from there.
> >
> >
> >
> > Thanks,
> >
> > --Matt
> >
> >
>
-- 

Jon

METRON-777

[zeo...@gmail.com]

I was wondering, is anybody planning to or currently taking a look at
Metron 777?  I think this is a great contribution and very important to
improving the usability of the platform (along with some of it's follow on
PRs).  I would be happy to help with functional testing and security static
code analysis but unfortunately I'm not familiar enough with the existing
code base to give it a +1 based off of the architectural changes.

Jon
-- 

Jon

Re: [DISCUSS] Metron IRC channel

[zeo...@gmail.com]

AFAIK it gives full edit capabilities to the bot, and allows delegation of
additional access to others.

https://wilderness.apache.org/manual.html#karma

Jon

On Wed, May 24, 2017 at 4:05 PM Otto Fowler <ottobackwa...@gmail.com> wrote:

> What is the karma for?
>
>
> On May 24, 2017 at 14:49:32, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> Perhaps we could think about adding what I wrote below this to the ticket
>
> <https://issues.apache.org/jira/browse/INFRA-12931>? Please feel free to
>
> double check it. Also, if someone is a moderator of the dev@metron list
> we
> can send email to
> dev-allow-subscribe-asfbot=urd.zones.apache@metron.apache.org to set
> up
> the JIRA reporting, and remove the "In order..." part of the below
> message.
>
>
>
> We discussed this on the mailing list and we'd like to add the ASFBot to
> #apache-metron with Git and JIRA reporting
>
> <https://wilderness.apache.org/manual.html#commits> enabled. I believe
>
> this would look something like this, for clarity of intent:
>
> ```
> [Channel:#apache-metron]
> tags: metron JIRA:METRON
> jiraName: METRON
> ```
>
> In order for this to function I think we'll also need `
> asf...@urd.zones.apache.org` to be a recipient of emails pertaining to
> changes to the METRON JIRA tickets.
>
> Finally, can we establish `cstella` as having Level 10 Karma in the
> config? Thanks.
>
>
>
> Jon
>
> On Tue, May 2, 2017 at 8:54 AM zeo...@gmail.com <zeo...@gmail.com> wrote:
>
> > Per the INFRA ticket, perhaps we should reopen and ask for what we
> > mentioned above?
> >
> > Jon
> >
> > On Sun, Dec 18, 2016 at 1:58 PM Kyle Richardson <
> kylerichards...@gmail.com>
> > wrote:
> >
> >> I'll second the JIRA and Git integrations. I also like the meeting
> minutes
> >> feature. Maybe it's something we could think about for the future.
> >>
> >> -Kyle
> >>
> >> On Fri, Dec 16, 2016 at 3:57 PM, Casey Stella <ceste...@gmail.com>
> wrote:
> >>
> >> > I'll leave this open til monday and update the INFRA jira with the
> >> results.
> >> >
> >> > On Fri, Dec 16, 2016 at 3:46 PM, zeo...@gmail.com <zeo...@gmail.com>
> >> > wrote:
> >> >
> >> > > What they said ^.
> >> > >
> >> > > On Fri, Dec 16, 2016 at 3:39 PM Michael Miklavcic <
> >> > > michael.miklav...@gmail.com> wrote:
> >> > >
> >> > >> Jira search
> >> > >>
> >> > >> On Fri, Dec 16, 2016 at 11:50 AM, Otto Fowler <
> >> ottobackwa...@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >> > Start with jira and git?
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On December 16, 2016 at 13:17:06, Casey Stella (
> ceste...@gmail.com
> >> )
> >> > >> wrote:
> >> > >> >
> >> > >> > Hi all,
> >> > >> >
> >> > >> > Any ideas of what features we would like to add? The options are
> >> at:
> >> > >> > http://wilderness.apache.org/manual.html
> >> > >> >
> >> > >> > On Wed, Nov 16, 2016 at 3:14 PM, Casey Stella <
> ceste...@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > Done
> >> > >> > >
> >> > >> > > https://issues.apache.org/jira/browse/INFRA-12931
> >> > >> > >
> >> > >> > >
> >> > >> > > On Wed, Nov 16, 2016 at 12:34 PM, Yohann Lepage <
> >> yoh...@lepage.info
> >> > >
> >> > >> > > wrote:
> >> > >> > >
> >> > >> > >> Could an official member of the project fill an issue on ASF
> >> INFRA
> >> > as
> >> > >> > >> described on https://reference.apache.org/pmc/github to get
> the
> >> > bot
> >> > >> on
> >> > >> > >> #apache-metron?
> >> > >> > >>
> >> > >> > >>
> >> > >> > >> 2016-11-04 18:18 GMT+01:00 James Sirota <jsir...@apache.org>:
>
> >> > >> > >>
> >> > >> > >> > We tried using slack during 

Re: [DISCUSS] Metron IRC channel

[zeo...@gmail.com]

Perhaps we could think about adding what I wrote below this to the ticket
<https://issues.apache.org/jira/browse/INFRA-12931>?  Please feel free to
double check it.  Also, if someone is a moderator of the dev@metron list we
can send email to
dev-allow-subscribe-asfbot=urd.zones.apache@metron.apache.org to set up
the JIRA reporting, and remove the "In order..." part of the below message.



We discussed this on the mailing list and we'd like to add the ASFBot to
#apache-metron with Git and JIRA reporting
<https://wilderness.apache.org/manual.html#commits> enabled.  I believe
this would look something like this, for clarity of intent:

```
[Channel:#apache-metron]
 tags: metron JIRA:METRON
 jiraName: METRON
```

In order for this to function I think we'll also need `
asf...@urd.zones.apache.org` to be a recipient of emails pertaining to
changes to the METRON JIRA tickets.

Finally, can we establish `cstella` as having Level 10 Karma in the
config?  Thanks.



Jon

On Tue, May 2, 2017 at 8:54 AM zeo...@gmail.com <zeo...@gmail.com> wrote:

> Per the INFRA ticket, perhaps we should reopen and ask for what we
> mentioned above?
>
> Jon
>
> On Sun, Dec 18, 2016 at 1:58 PM Kyle Richardson <kylerichards...@gmail.com>
> wrote:
>
>> I'll second the JIRA and Git integrations. I also like the meeting minutes
>> feature. Maybe it's something we could think about for the future.
>>
>> -Kyle
>>
>> On Fri, Dec 16, 2016 at 3:57 PM, Casey Stella <ceste...@gmail.com> wrote:
>>
>> > I'll leave this open til monday and update the INFRA jira with the
>> results.
>> >
>> > On Fri, Dec 16, 2016 at 3:46 PM, zeo...@gmail.com <zeo...@gmail.com>
>> > wrote:
>> >
>> > > What they said ^.
>> > >
>> > > On Fri, Dec 16, 2016 at 3:39 PM Michael Miklavcic <
>> > > michael.miklav...@gmail.com> wrote:
>> > >
>> > >> Jira search
>> > >>
>> > >> On Fri, Dec 16, 2016 at 11:50 AM, Otto Fowler <
>> ottobackwa...@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > Start with jira and git?
>> > >> >
>> > >> >
>> > >> >
>> > >> > On December 16, 2016 at 13:17:06, Casey Stella (ceste...@gmail.com
>> )
>> > >> wrote:
>> > >> >
>> > >> > Hi all,
>> > >> >
>> > >> > Any ideas of what features we would like to add? The options are
>> at:
>> > >> > http://wilderness.apache.org/manual.html
>> > >> >
>> > >> > On Wed, Nov 16, 2016 at 3:14 PM, Casey Stella <ceste...@gmail.com>
>> > >> wrote:
>> > >> >
>> > >> > > Done
>> > >> > >
>> > >> > > https://issues.apache.org/jira/browse/INFRA-12931
>> > >> > >
>> > >> > >
>> > >> > > On Wed, Nov 16, 2016 at 12:34 PM, Yohann Lepage <
>> yoh...@lepage.info
>> > >
>> > >> > > wrote:
>> > >> > >
>> > >> > >> Could an official member of the project fill an issue on ASF
>> INFRA
>> > as
>> > >> > >> described on https://reference.apache.org/pmc/github to get the
>> > bot
>> > >> on
>> > >> > >> #apache-metron?
>> > >> > >>
>> > >> > >>
>> > >> > >> 2016-11-04 18:18 GMT+01:00 James Sirota <jsir...@apache.org>:
>> > >> > >>
>> > >> > >> > We tried using slack during the early days of the project and
>> it
>> > >> was
>> > >> > >> > frowned upon by Apache. So we abandoned it in favor of IRC and
>> > >> message
>> > >> > >> > lists.
>> > >> > >> >
>> > >> > >> > 04.11.2016, 04:54, "zeo...@gmail.com" <zeo...@gmail.com>:
>> > >> > >> > > Is anybody interested in migrating this to slack? I'm
>> > personally
>> > >> a
>> > >> > >> fan of
>> > >> > >> > > the benefits this provides - just wanted to bring it up and
>> see
>> > >> if
>> > >> > >> anyone
>> > >> > >> > > else was thinking the same thing. If not, no biggie.
>> > >> > >> > >
>

Re: [Discuss] Cyber Security Asset Management for Metron

[zeo...@gmail.com]

I would be very interested in a graph db that could leverage the
ip_src_addr and ip_dst_addr fields in a broad sense (who is talking to who,
visualize top talkers, etc.).  In order to be very useful it would need to
have the ability to apply filters (IPs, ports, connection durations, bytes
transferred, etc.) and to narrow down certain time-based windows.  I
probably have an environment where I could test this at semi-scale (a
couple billion messages per day) and flesh out some of the performance
concerns if this turns into something.  Even if it was very early in
development, as I frequently rebuild that environment from scratch for
testing things.

Jon

On Wed, May 24, 2017 at 12:46 PM Nick Allen  wrote:

> I think the addition of a graph capability would be very powerful.  I know
> many who would love the idea, but I know of no implementations that have
> occurred.
>
> It might be good to discuss in the community specific use cases that would
> be enabled by a graph database.  That might help to flesh out the technical
> aspects of it.
>
>
>
>
>
> On Wed, May 24, 2017 at 10:08 AM, Ali Nazemian 
> wrote:
>
> > Hi all,
> >
> > We are going to design and develop an asset database for Metron. For this
> > purpose, I have been thinking of a graph schema model to map assets as
> > Nodes and provide relations as Edges. This can be extended to event level
> > to have a particular relation to assets as well as an event to event
> > relation. Regarding technology, I was thinking of using Titan Graph
> > Database (probably JanusGraph) and using HBase and Elasticsearch/Solr as
> > backends. However, there might be a performance issue regarding this
> > decision if we want to use lots of Composite Indices. The problem we will
> > be facing would be the fact that Titan creates separate column family for
> > each Composite Index which HBase is not very good for it. Basically, it
> > would be better to use Cassandra for this purpose.
> >
> > I would like to understand what work have been done already regarding
> this
> > problem and what the roadmap will be, so I can make sure we will follow
> the
> > same strategy.
> >
> > Regards,
> > Ali
> >
>
-- 

Jon

Re: [DISCUSS] Enrichment Split/Join issues

[zeo...@gmail.com]

The field stub also gives something that can potentially be used in the
error dashboard (or similar) to graph, allowing failed enrichments to
"shout" louder to the end user.

Jon

On Tue, May 16, 2017 at 12:34 PM Nick Allen  wrote:

> > but also adds a field stub to indicate failed enrichment. This is then an
> indicator to an operator or investigator as well that something is missing,
> and could drive things like replay of the message to retrospectively enrich
> when things have calmed down.
>
> Yes, I like the idea of a "field stub".  You need some way to distinguish
> "did I configure this wrong" versus "something bad happened outside of my
> control".
>
>
>
> On Tue, May 16, 2017 at 12:27 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > Nick, I’d tend to agree with you there.
> >
> > How about:
> > If an enrichment fails / effectively times out, the join bolt emits the
> > message before cache eviction (as Nick’s point 2), but also adds a field
> > stub to indicate failed enrichment. This is then an indicator to an
> > operator or investigator as well that something is missing, and could
> drive
> > things like replay of the message to retrospectively enrich when things
> > have calmed down.
> >
> > Simon
> >
> > > On 16 May 2017, at 17:25, Nick Allen  wrote:
> > >
> > > Ah, yes.  Makes sense and I can see the value in the parallelism that
> the
> > > split/join provides.  Personally, I would like to see the code do the
> > > following.
> > >
> > > (1) Scream and shout when something in the cache expires.  We have to
> > make
> > > sure that it is blatantly obvious to a user what happened.  We also
> need
> > to
> > > make it blatantly obvious to the user what knobs they can turn to
> correct
> > > the problem.
> > >
> > > (2) Enrichments should be treated as best-effort.  When the cache
> > expires,
> > > it should pass on the message without the enrichments that have not
> > > completed.  If I am relying on an external system for an enrichment, I
> > > don't want an external system outage to fail all of my telemetry.
> > >
> > >
> > >
> > >
> > >
> > > On Tue, May 16, 2017 at 12:05 PM, Casey Stella 
> > wrote:
> > >
> > >> We still do use split/join even within stellar enrichments.  Take for
> > >> instance the following enrichment:
> > >> {
> > >>  "enrichment" : {
> > >>"fieldMap" : {
> > >>  "stellar" : {
> > >> "config" : {
> > >> "parallel-task-1" : {
> > >> "my_field" : "PROFILE_GET()"
> > >> },
> > >> "parallel-task-2" : {
> > >> "my_field2" : "PROFILE_GET()"
> > >> }
> > >> }
> > >>  }
> > >>}
> > >>  }
> > >>
> > >> Messages will get split between two tasks of the Stellar enrichment
> bolt
> > >> and the stellar statements in "parallel-task-1" will be executed in
> > >> parallel to those in "parallel-task-2".  This is to enable people to
> > >> separate computationally intensive or otherwise high latency tasks
> that
> > are
> > >> independent across nodes in the cluster.
> > >>
> > >> I will agree wholeheartedly, though, that my personal desire would be
> to
> > >> have just stellar enrichments, though.  You can do every one of the
> > other
> > >> enrichments in Stellar and it would greatly simplify that config
> above.
> > >>
> > >>
> > >>
> > >> On Tue, May 16, 2017 at 11:59 AM, Nick Allen 
> > wrote:
> > >>
> > >>> I would like to see us just migrate wholly to Stellar enrichments and
> > >>> remove the separate HBase and Geo enrichment bolts from the
> Enrichment
> > >>> topology.  Stellar provides a user with much greater flexibility than
> > the
> > >>> existing HBase and Geo enrichment bolts.
> > >>>
> > >>> A side effect of this would be to greatly simplify the Enrichment
> > >>> topology.  I don't think we would not need the split/join pattern if
> we
> > >> did
> > >>> this. No?
> > >>>
> > >>> On Tue, May 16, 2017 at 11:54 AM, Casey Stella 
> > >> wrote:
> > >>>
> >  The problem is that an enrichment type won't necessarily have a
> fixed
> >  performance characteristic.  Take stellar enrichments, for instance.
> > >>> Doing
> >  a HBase call for one sensor vs doing simple string munging will have
> > >>> vastly
> >  differing performance.  Both of them are functioning within the
> > stellar
> >  enrichment bolt.  Also, some enrichments may call for multiple calls
> > to
> >  HBase.  Parallelizing those, would make some sense, I think.
> > 
> >  I do take your point, though, that it's not as though it's strictly
> > >>> serial,
> >  it's just that the unit of parallelism is the message, rather than
> the
> >  enrichment per message.
> > 
> >  On Tue, May 16, 2017 at 11:47 AM, Christian Tramnitz <
> > >> tramn...@trasec.de
> > 
> >  wrote:
> > 
> > > I’m glad you bring this up. 

Re: we currently have 31 PR’s that are not landed

[zeo...@gmail.com]

Assuming the unincubating process is almost completed (I don't know if
that's true or not), I think there are some simple, obvious priorities
based on our pending 0.4.0 release.  Things like METRON-833, METRON-819,
and METRON-953 should probably get finalized and merged in asap.

Also, we have some big wins, like METRON-508, METRON-950, METRON-844,
METRON-891, etc. (not an authoritative review) which I would love to see in
the release.

That said, doing a general call to review PRs without huge changes
(METRON-777, I'm lookin' at you) and get them merged is probably a good
idea.

Jon

On Tue, May 16, 2017 at 9:53 AM Otto Fowler  wrote:

> https://github.com/apache/metron/pulls
>
> This seems a little large given that I *think* we have been at around 19 or
> so consistently.
>
-- 

Jon

Re: integration testing framework

[zeo...@gmail.com]

The standard has been centos6 for installing Metron up to this point.
There are some Ubuntu guides floating around as well.

Jon

On Mon, May 15, 2017, 8:07 AM moshe jarusalem  wrote:

> I would like to ask another question related to this topic.
> If I am going to install metron on a single machine (the same machine as
> development) which operating distribution would be best centos7 or ubuntu
> 16.4 or another?
> And is there any document to follow for such an installation ?
>
> Regards,
>
> On Sat, May 13, 2017 at 1:50 PM, moshe jarusalem  wrote:
>
> > Hi All,
> > I have been looking for an easy way to test metron in the development
> > environment. Using "full-dev-environment" is a bit painful because you
> each
> > time need to copy artifacts etc. I tried to understand integration
> testing
> > framework but it a little complex for a newbie.
> >
> > Would you guys describe how to utilize it? broadly how to quickly write
> > and test? How do you manage?
> >
> > Regards,
> >
>
-- 

Jon

Infosec training (including Metron)

[zeo...@gmail.com]

If anybody is interested, I'll be touching on Metron as a part of some
security training I'll be doing as at BSides Pittsburgh 2017 on June 8th
(main conference is June 9).  It's a whole day of infosec training for only
$100, feel free to come check it out!

https://www.bsidespgh.com/training/

Jon
-- 

Jon

Re: Parser Docs

[zeo...@gmail.com]

Definitely worthwhile.  I discussed something similar (but more general) a
little while back here
.
Totally worth the effort IMO.

Jon

On Mon, May 8, 2017 at 7:36 PM Casey Stella  wrote:

> +1 for parser docs
>
> On Mon, May 8, 2017 at 7:35 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > Quick thought, and please shoot me down if this has already been thought
> > of, but….
> >
> > Would it be worthwhile if we put some per parser docs into the repo,
> > essentially a README.md per major parser class, which would in theory be
> > picked up by the docbook? We don’t really have much documentation right
> now
> > on the effect of parserConfig settings for example.
> >
> > Similarly, is this something worth considering in the package format, or
> > at least in the archetype proposed by METRON-777 (paging Ottto!)
> >
> > Worth the effort? I’m happy to do at least a few of the more recent ones
> > I’ve spent meaningful time with.
> >
> > Simon
>
-- 

Jon

Re: [DISCUSS] Update Metron Release Documentation

[zeo...@gmail.com]

So I'm not hearing any opposition to the idea.  What do we need to handle
for the eventual 0.4.0 release?  I really only see a need to update the
release documentation to account for this.  Thanks,

Jon

On Mon, May 1, 2017 at 2:39 PM Matt Foley <ma...@apache.org> wrote:

> In previous discussion threads, I proposed that the site should have a
> pull-down menu for all existing docs versions, so user could choose which
> one they want to access.  Each choice would point into the release repo.
> This implies the Release Manager has to make a one-line edit to add each
> new release’s book to the pull-down.
>
> Better, we can do what many projects do and have a docs area under the
> release repo, separate from the code area.  The menu in question can then
> be automatically constructed from the available set of choices.  Not a bad
> thing to make older versions of the code accessible similarly.
>
> --Matt
>
> On 5/1/17, 9:20 AM, "Nick Allen" <n...@nickallen.org> wrote:
>
> One major benefit of the site-book is that we can maintain docs for
> previous releases of Metron. Unless there is a major technical hurdle,
> I
> think we should do so.
>
> On Mon, May 1, 2017 at 10:06 AM, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
> > Just bringing up this thread again, as we're going to have two books
> as of
> > the 0.4.0 release.  I don't have any strong opinions on the topic
> >
> > Jon
> >
> > On Wed, Feb 15, 2017 at 10:13 AM Casey Stella <ceste...@gmail.com>
> wrote:
> >
> > > On the subject, we should also document updating the releases page
> after
> > a
> > > release and figure out how old books are stored/served up.  Anyone
> have
> > > thoughts on that?
> > >
> > > On Wed, Feb 15, 2017 at 9:53 AM, Casey Stella <ceste...@gmail.com>
> > wrote:
> > >
> > > > Yeah I agree we need to document it there.
> > > >
> > > > On Tue, Feb 14, 2017 at 09:57 zeo...@gmail.com <zeo...@gmail.com
> >
> > wrote:
> > > >
> > > >> As a follow-up to METRON-716, I would like to suggest that we
> update
> > our
> > > >> Metron
> > > >> Release documentation
> > > >> <
> https://cwiki.apache.org/confluence/display/METRON/Release+Process>
> > to
> > > >> account for the site-book.  Specifically, I think that Step 4
> and
> > Step 9
> > > >> need a bit of a refresher.
> > > >>
> > > >> In the most recent build, Casey appears to have handled this by
> > building
> > > >> the site-book and then releasing it to
> > > >> https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.
> > > >> 1-RC4-incubating/book-site,
> > > >> documenting that in the VOTE thread.
> > > >>
> > > >> My initial question is, is there a reason to use the "book-site"
> > folder
> > > >> name, as opposed to "site-book"?  I would prefer to pick a
> standard
> > and
> > > >> stick with it, if possible.
> > > >>
> > > >> Regardless, I am suggesting that under Step 4 we add the
> following
> > > bullet
> > > >> under the "The artifacts for a release" section:
> > > >>
> > > >> "- The site-book documentation, as generated using the most
> recent
> > > >> documentation under the site-book/README.md."
> > > >>
> > > >> And under Step 9 we add the following:
> > > >>
> > > >> "- Update the Metron site documentation links to point to the
> > > >> documentation
> > > >> for the most recent release."
> > > >>
> > > >> Right now the website points to the wiki
> > > >> <
> https://cwiki.apache.org/confluence/display/METRON/Documentation>.
> > > >> Thoughts?
> > > >>
> > > >> Jon
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >> Sent from my mobile device
> > > >>
> > > >
> > >
> > --
> >
> > Jon
> >
>
>
>
> --

Jon

Re: Request double-check on Ambari config logic (ES network_host)

[zeo...@gmail.com]

t; clients and other servers to connect to. It will publish only one address.
> If you give it a set of addresses, it picks the most “desirable” of the set
> – it assures it actually is accessible, and it prefers ipv4 (or 6,
> depending on another config), then  global, then site-local, then
> link-local, then loopback. Within each category it orders by numeric
> magnitude of the IP address, which is hardly meaningful.  This means the
> published address can be wrong on a multi-homed server or VM, if you don’t
> appropriately constrain it.
> -The parameter values can be network addresses, network interface
> names, host names (to be dereferenced via DNS), “special” names denoting
> predefined sets of addresses, and combinations of the above.
> -Wildcard and loopback addresses are allowed.
> -If the wildcard is provided it must be the ONLY value provided (list
> of length == 1), or ES will throw an error.
>
> Discussion item 1:  If you use network.host, the same list of addresses
> get sent to both network.bind_host and network.publish_host.  The algorithm
> for picking the single publish_host address is not good enough, at least in
> ES 2.3, to give certainty that the right address will be published, on
> multi-homed servers or VMs (although on non-multi-homed, it should
> generally work fine).
>
> It seems to me that specifying exactly one of _local_, _site_, or _global_
> will usually give the right result, but that too can fail if the server has
> multiple addresses within the same category.
>
> I think network.bind_host and network.publish_host should be separately
> configured, as they are with Hadoop.
> There’s an article here:
> https://community.hortonworks.com/content/kbentry/24277/parameters-for-multi-homing.html
> that discusses these issues at some length, and clarifies why they must be
> separately configured.
>
> What do you-all think?
>
> Discussion item 2:  While it’s fine to use 0.0.0.0 for the bind address,
> it gives no guidance at all to the needed publish_host value. Using _local_
> for QuickDev and single-node deployments, and _site_  for FullDev
> deployments and all cluster deployments, is probably a reasonable choice
> for publish_host.
>
> What do you-all think?
>
> Discussion item 3: Should we attempt to further the “hadoop style” of
> config parameter, and silently add the square brackets and perhaps
> substring quotes in python processing?  Or should we say users need to
> understand ES configuration, and tell them to put the list in square
> brackets themselves, if they need a list entry in this parameter, per
> https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-network.html
> ?
>
> Please share your thoughts,
> Thanks,
> --Matt
>
>
> On 5/2/17, 9:57 PM, "Matt Foley" <mfo...@hortonworks.com> wrote:
>
> Hi Otto,
> This event derives from this line of code:
> https://github.com/elastic/elasticsearch/blob/2.3/core/src/main/java/org/elasticsearch/action/support/master/TransportMasterNodeAction.java#L148
> which suggests that a cluster action has been requested on a local
> (loopback) address.  This is not
> surprising given what I’ve learned about the semantics of network.host
> with wildcard address.
> See next message, item C.  Basically, while the wildcard causes ES to
> “listen” on all IP addresses, it
> only *publishes* one, and on a multi-homed server it can be the wrong
> one.  I can’t be certain
> this causes what you’re seeing, but it seems feasible.
>
> From: Otto Fowler <ottobackwa...@gmail.com>
> Date: Tuesday, May 2, 2017 at 8:30 PM
> To: "d...@metron.incubator.apache.org" <d...@metron.incubator.apache.org>,
> Matt Foley <mfo...@hortonworks.com>, "dev@metron.apache.org" <
> dev@metron.apache.org>, "zeo...@gmail.com" <zeo...@gmail.com>
> Subject: Re: Request double-check on Ambari config logic (ES
> network_host)
>
> OK.
> I tried it using this method, and master ( adding [] ).  In both
> cases, I can hit 9200 from other machines, but in both cases I’m getting ES
> master errors:
>
> ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not
> recovered / initialized];]
> at
> org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:174)
> at
> org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:66)
> at
> org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.checkBlock(TransportCreateIndexAction.java:41)
> at
> org.elasticsearch.action.support.master.TransportMasterNodeAction

Re: Request double-check on Ambari config logic (ES network_host)

[zeo...@gmail.com]

As somebody who has systems that have globally-scoped addresses directly
addressed onto servers, I would prefer using using "_local_", "_site_".

Jon

On Tue, May 2, 2017 at 1:36 PM Matt Foley <mfo...@hortonworks.com> wrote:

> Hi Otto,
> The basic change to use “0.0.0.0” as the default binding, and put the
> square brackets in the template text instead of the parameter value, is now
> available in
> https://github.com/mattf-horton/incubator-metron  branch METRON-905
> commit e879719a0c3fb
>
> I’m having some trouble with my test env, so if you wanted to give it a
> try, that would be great.
> If the “0.0.0.0” doesn’t work, then we should use
> "_local_", "_site_"
> that being the ES special values that mean aprx the same.
>
> I’m going to have to do trial-and-error to determine the exact behavior of
> multi-item lists, and then write the python code to strip redundant square
> brackets if included in the parameter value.
> Thanks,
> --Matt
>
>
> On 5/2/17, 6:44 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote:
>
> I am working on a centos 7 cluster deploy for testing the steps.
> I have this issue ( along with the wrong interface name ) and can test
> when
> you have it.
>
> An eta would help?
>
>
> On May 2, 2017 at 09:14:10, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> Are you working on this one? The JIRA doesn't look like it's currently
> assigned. Thanks,
>
> Jon
>
> On Mon, May 1, 2017 at 6:40 PM Matt Foley <mfo...@hortonworks.com>
> wrote:
>
> > Ah, I see I mis-read METRON-897, and Nick specifically says
> > "lo:ipv4","eth0:ipv4" did not work for him, but
> ["_lo:ipv4_","_eth0:ipv4_"]
> > did work.
> >
> > So I went back and dug a little deeper, and realized that in the
> > environment where "lo:ipv4","eth0:ipv4" worked for me, I had
> modified the
> > yaml.j2 template to include the square brackets.
> >
> > So the below theory is wrong. Back to the drawing board.
> > Thanks,
> > --Matt
> >
> > On 5/1/17, 3:08 PM, "Matt Foley" <ma...@apache.org> wrote:
> >
> > Hi, there have been widely varying statements about what needs to be
> > in the Elasticsearch config parameter “network_host”. I think I may
> have
> a
> > rationale for what works and what doesn’t, but I’d like your input or
> > correction.
> >
> > I am focusing on what worked in terms of punctuation (quotes and
> > square brackets) with the old _lo:ip4_,_eth0:ip4_. I would like to
> ignore
> > for the moment, please, whether eth0 was the correct name for a given
> env,
> > and whether we can use 0.0.0.0. Instead, for systems where eth0 WAS
> the
> > correct name, I’d like to understand what worked and why.
> >
> > It’s complicated because the value starts out in xml, is read into
> > python, printed by jinja, then consumed by yaml.
> >
> > I think there were two constructs that actually worked for this
> > param. Please say whether this is consistent or inconsistent with
> your
> > experience:
> >
> > "_lo:ip4_","_eth0:ip4_"
> > This worked for me. I think this was read from XML into python as a
> > list of strings, then output in jinja ‘print statement‘
> > {{ network_host }} as a python literal list with form:
> > [ "_lo:ip4_", "_eth0:ip4_" ]
> > In other words, the print statement for a python list object injected
> > the needed square brackets.
> >
> > and
> > "[ _lo:ip4_, _eth0:ip4_ ]"
> > Nick and Anand, please confirm if this is the form that worked for
> > you. I think this was read from XML into python as a single string,
> and
> > output in the same jinja print statement as:
> > [ _lo:ip4_, _eth0:ip4_ ]
> > because the print statement for a python string object does not
> > produce quote marks.
> >
> > In either case, yaml (the consumer of the jinja output) saw what it
> > interprets as a list of strings (since quotes are optional for yaml
> > strings).
> >
> > What didn’t work was:
> >
> > * "_lo:ip4_, _eth0:ip4_"
> > This would be read in and output as a single string, and no square
> > brackets would ever be introduced.
> >
> > * _lo:ip4_, _eth0:ip4_ or [ _lo:ip4_, _eth0:ip4_ ]
> > (without quotes) I think the unquoted colons messed up the python
> > parsing
> >
> > Finally, I don’t know whether
> > * [ "_lo:ip4_", "_eth0:ip4_" ]
> > worked or not, I’m not sure anyone ever tried it. By the above logic
> > it probably should work.
> >
> > Please give me your input if you have touched on these issues.
> > Thanks,
> > --Matt
> >
> >
> >
> >
> >
> >
> > --
>
> Jon
>
>
> --

Jon

Re: Request double-check on Ambari config logic (ES network_host)

[zeo...@gmail.com]

Are you working on this one?  The JIRA doesn't look like it's currently
assigned.  Thanks,

Jon

On Mon, May 1, 2017 at 6:40 PM Matt Foley  wrote:

> Ah, I see I mis-read METRON-897, and Nick specifically says
> "lo:ipv4","eth0:ipv4" did not work for him, but ["_lo:ipv4_","_eth0:ipv4_"]
> did work.
>
> So I went back and dug a little deeper, and realized that in the
> environment where "lo:ipv4","eth0:ipv4" worked for me, I had modified the
> yaml.j2 template to include the square brackets.
>
> So the below theory is wrong.  Back to the drawing board.
> Thanks,
> --Matt
>
> On 5/1/17, 3:08 PM, "Matt Foley"  wrote:
>
> Hi, there have been widely varying statements about what needs to be
> in the Elasticsearch config parameter “network_host”.  I think I may have a
> rationale for what works and what doesn’t, but I’d like your input or
> correction.
>
> I am focusing on what worked in terms of punctuation (quotes and
> square brackets) with the old _lo:ip4_,_eth0:ip4_.  I would like to ignore
> for the moment, please, whether eth0 was the correct name for a given env,
> and whether we can use 0.0.0.0.  Instead, for systems where eth0 WAS the
> correct name, I’d like to understand what worked and why.
>
> It’s complicated because the value starts out in xml, is read into
> python, printed by jinja, then consumed by yaml.
>
> I think there were two constructs that actually worked for this
> param.  Please say whether this is consistent or inconsistent with your
> experience:
>
> "_lo:ip4_","_eth0:ip4_"
> This worked for me.  I think this was read from XML into python as a
> list of strings, then output in jinja ‘print statement‘
> {{ network_host }} as a python literal list with form:
> [ "_lo:ip4_", "_eth0:ip4_" ]
> In other words, the print statement for a python list object injected
> the needed square brackets.
>
> and
> "[ _lo:ip4_, _eth0:ip4_ ]"
> Nick and Anand, please confirm if this is the form that worked for
> you.  I think this was read from XML into python as a single string, and
> output in the same jinja print statement as:
> [ _lo:ip4_, _eth0:ip4_ ]
> because the print statement for a python string object does not
> produce quote marks.
>
> In either case, yaml (the consumer of the jinja output) saw what it
> interprets as a list of strings (since quotes are optional for yaml
> strings).
>
> What didn’t work was:
>
> *"_lo:ip4_, _eth0:ip4_"
> This would be read in and output as a single string, and no square
> brackets would ever be introduced.
>
> *_lo:ip4_, _eth0:ip4_or[ _lo:ip4_, _eth0:ip4_ ]
> (without quotes)  I think the unquoted colons messed up the python
> parsing
>
> Finally, I don’t know whether
> *[ "_lo:ip4_", "_eth0:ip4_" ]
> worked or not, I’m not sure anyone ever tried it.  By the above logic
> it probably should work.
>
> Please give me your input if you have touched on these issues.
> Thanks,
> --Matt
>
>
>
>
>
>
> --

Jon

Re: introduction

[zeo...@gmail.com]

Welcome, Christian!  Best of luck with everything, feel free to shoot an
email or hop on our IRC channel #apache-metron on freenode if you'd like to
chat further.

Jon

On Tue, May 2, 2017 at 7:55 AM Christian Tramnitz 
wrote:

> Hello Metron developers,
>
> I thought I’d first introduce myself before I ask any questions:
> I’m new to Metron, but we are running a fairly complex OpenSOC-based
> installation for a customer. Since Metron evolved from OpenSOC our
> code-base drifted apart somewhat and I don’t know if we can contribute back
> some of the stuff we added with that customer, but in parallel I would like
> to make myself familiar with Metron for internal use, other projects and
> reusing a few components (i.e. the indexing topology) in the existing
> environment.
>
>
>
> Best regards,
>Christian Tramnitz
>
> --

Jon

Recent commit without JIRA in commit message

[zeo...@gmail.com]

It looks like METRON-799 
(#518 ) got commit

without having the JIRA in the title.  Is this enough of a concern
to reword the commit message?  I know there have been some concerns with
changing ever changing history, no matter how small, but I figured I would
bring it up to discuss.  Thanks,

Jon
-- 

Jon

Re: Started the infrastructure requests to move to TLP

[zeo...@gmail.com]

Does this officially mean 0.4.0 is on hold until the migration is
complete?  I assume the vote based on RC2 failed (for more reason than
one), but also can we assume that work in progress now is more likely to
get into 0.4.0?  I would love some more time to QA master as it currently
is - which I'm working on.  I have fumbled upon a fair amount of issues
lately and I'm concerned that the more I look the more I'll find.  That
said, I definitely would not vote to hold things up to wait on my efforts
because it could be weeks before it's entirely done and tested
appropriately, I'm just wondering if that will be a natural byproduct of
our graduation.

Jon

On Sat, Apr 29, 2017, 3:24 AM Matt Foley  wrote:

> A brief guide to changing our github remotes from
> “apache/metron-incubating” to “apache/metron”
> [also applies to “incubator-metron-bro-plugin-kafka” →
> “metron-bro-plugin-kafka” ]
>
> Hey all, as a result of un-incubating, the github apache repo names will
> change.  Some people have expressed concerns whether this will impact their
> development environments.  This potentially affects four areas:
> A. Changing local repo names to match
> B. Changing local repos’ upstream pointers to point at the new apache repo
> names
> C. Changing the names of personal fork repos in Github to match
> D. Changing the upstream “source” apache repo that your personal fork
> repos in Github point at for purposes of Pull Requests
>
> ‘B’ is the only important one, but we’ll talk about them in order.
>
> ‘A’ is the least important.  You can call a local repo anything you want,
> because the name of the local clone is just a directory name, and not
> strictly part of the repository at all.  If you want to change its name,
> just `mv` the base directory name.  It doesn’t matter.  You probably
> already call them something other than “incubator-metron”  anyway :-)
>
> ‘B’ may or may not be important, depending on how Apache INFRA goes about
> changing the name of the Github apache/metron mirror repo.  If they use the
> standard Github owner’s command for renaming a repo, then Github maintains
> semantics for users of the prior name.  Then, you wouldn’t have to do
> anything about the name change unless you wanted to.
>
> However, if they use a technique that doesn’t respect Github’s protocol
> (or if that protocol doesn’t work with mirror repos), or if you just want
> to keep things neat and avoid confusion, here is how you change local
> repos’ upstream pointers to point at the new apache repo names:
>
> 1. cd to the base directory of the local git repo
> 2. Give the command `git remote -v`
> 3. You’ll probably see something like:
>
> apache  https://github.com/apache/incubator-metron.git (fetch)
> apache  https://github.com/apache/incubator-metron.git (push)
> originhttps://github.com/mattf-horton/incubator-metron.git (fetch)
> originhttps://github.com/mattf-horton/incubator-metron.git (push)
>
> 4. In this case, you could give the command:
> `git remote set-url apache https://github.com/apache/metron.git`
> 
> 5. Now the command `git remote -v` will show:
>
> apache  https://github.com/apache/metron.git (fetch)
> apache  https://github.com/apache/metron.git (push)
> originhttps://github.com/mattf-horton/incubator-metron.git (fetch)
> originhttps://github.com/mattf-horton/incubator-metron.git (push)
>
> and thereafter, every reference to the remote named “apache” would refer
> to apache/metron instead of apache/incubator-metron.
>
> ‘C’ is if you want to rename your personal fork repositories in Github.
> Again it doesn’t matter that much, but you may wish to do so for
> appearance’ sake, or to avoid confusion.  Instructions for renaming a
> personal repo are here:
> https://help.github.com/articles/renaming-a-repository/ .  (Note you must
> have “owner” privs for the repo.)  The instructions say that if you follow
> these instructions, and don’t create a new repo with the old name, then
> Github will maintain semantics for users of the prior name, so you don’t
> have to do any other adjustments.  Nevertheless, you would then probably
> want to do something like this command within the above example repo:
> `git remote set-url origin https://github.com/mattf-horton/metron.git`
> 
> Thereafter references to “origin” in your local repo would refer directly
> to your renamed fork repo in Github.
>
> ‘D’ finally is perhaps the most difficult question.  Every fork repo in
> Github “knows” what repo it was forked from, and uses that as the default
> for Pull Requests generated from the fork.  Can this be changed?
>
> If Apache INFRA respects the Github protocol for repo renaming, then we
> don’t need to change anything.  We can just rely on the semantic
> preservation in the protocol and new PRs generated will automatically point
> at the renamed Apache repository, directly or indirectly.  That would be
> best, 

Re: Ambari Wizard: Repo Tab

[zeo...@gmail.com]

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler  wrote:

> I am following
>
> https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup 1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node, index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host, we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > ; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> > ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: Refer
> > to
> > the log for complete error details.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service: main process exited, code=exited, status=1/FAILURE
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]: Unit
> > elasticsearch.service entered failed state.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service failed.
> >
> > INFO 2017-04-26 08:55:19,962
> >  Contr
> >
> >
> > On April 26, 2017 at 08:27:27, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Can you describe what ES issues you are working on?
> > Fixing the repos got everything installed, but my ES components don’t
> stay
> > running.
> > I need to harvest the errors.
> >
> >
> >
> > On April 25, 2017 at 16:46:00, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Nm. sorry. I fixed it.
> >
> >
> >
> 

Re: auto-install on bare metal

[zeo...@gmail.com]

I can verify that I've used
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361 to
install Metron on a bare metal cluster before the docker requirement was
imposed.

Jon

On Wed, Apr 26, 2017 at 9:59 AM Otto Fowler  wrote:

> Right, I think this :
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361
> is the flow,
> but I need to verify it post recent changes to allow building in docker
> again.
>
>
> On April 26, 2017 at 09:54:26, Nick Allen (n...@nickallen.org) wrote:
>
> Here is an example of how you might do that. I created this quite a while
> ago, but it shows you the structure and how you could manage multiple
> environments with this method.
>
> https://github.com/nickwallen/metron-environments
>
> On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler 
> wrote:
>
> > I failed at this today, but maybe it was the way I tried.
> > An example would be great.
> >
> >
> >
> > On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com) wrote:
> >
> > Hi Dima,
> >
> > The same Ansible playbooks that work for EC2 and Vagrant will work for
> bare
> > metal installations. The only difference is that you would need to
> > pre-provision your machines and hand-build your inventory file. The AWS
> > playbooks only provision the machines. All deployment of Metron is
> handled
> > (for all installation types) by the metron_full_install playbook [1].
> >
> > -D...
> >
> > [1]
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/playbooks/metron_full_install.yml
> >
> > On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov 
> > wrote:
> >
> > > Hello Metron Team,
> > >
> > > We have developed a script that performs auto-install of the Metron on
> > > bare metal machines, but still working on few issues here and there.
> > >
> > > I am curios as to what automate solutions we do have for Metron
> > > installation right now?
> > > The ones I am aware of are in
> > > https://github.com/apache/incubator-metron/tree/master/
> > metron-deployment/:
> >
> > > a) AWS Ansible install (1 or 10 nodes)
> > > b) Vagrant local VM setup
> > >
> > > Is there any other solution available? Has anyone managed to use AWS
> > > Ansible playbooks for bare metal installation?
> > >
> > > - Dima
> > >
> > >
> > >
> >
>
-- 

Jon

Re: [DISCUSS] Regression introduced in Full Dev

[zeo...@gmail.com]

Interesting.  I found it via pony mail -
https://lists.apache.org/thread.html/82e194ad8f8b8378676a28c09b074f45dee82820ead6ff8ee8fbebcc@


But nothing in my inbox.  I suspected it was @metron.incubator.apache.org
vs @metron.apache.org but when I attempted to subscribe to the top level
mailing list I was told I'm already subscribed.  Same with User.

Jon

On Wed, Apr 26, 2017, 7:39 AM Justin Leet <justinjl...@gmail.com> wrote:

> I have it (and had it yesterday). Subject is:  "[DISCUSS] The various
> methods and incantations to spin up Metron".
>
> On Wed, Apr 26, 2017 at 7:33 AM, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
> > Yeah, I don't see the other thread either.  Stuck in the outbox Casey?
> >
> > Jon
> >
> > On Wed, Apr 26, 2017, 6:53 AM Otto Fowler <ottobackwa...@gmail.com>
> wrote:
> >
> > > What other thread?
> > >
> > >
> > > On April 25, 2017 at 19:56:56, Casey Stella (ceste...@gmail.com)
> wrote:
> > >
> > > Ok, I spun up that discussion in another thread. Hopefully we can get
> > some
> > > better sense about the various ways to spin up metron and a centralized
> > > place to direct people to along with with guidance on when some
> approach
> > > would be better than another.
> > >
> > > I'll be honest, I've totally lost track and never really consider
> > anything
> > > outside of full-dev anymore since it's the one that is generally stable
> > > (quick-dev gets out of date quickly because mpack changes cause it to
> get
> > > stale) and is sufficient for validating PRs. Most of the other ones
> tend
> > > to either not have all of the system spun up (i.e. the hadoop
> components)
> > > and therefore end up with me having to test in full-dev anyway or just
> > > weren't apparent to me and have unknown pros and cons. ;)
> > >
> > > On Tue, Apr 25, 2017 at 7:21 PM, Casey Stella <ceste...@gmail.com>
> > wrote:
> > >
> > > > Yeah, I tend to agree that a rundown of the various methods and when
> > you
> > > > would use them is in order. I will say that full-dev is especially
> > > > important to have working since it is required for validating PRs.
> > > >
> > > > On Tue, Apr 25, 2017 at 18:56 zeo...@gmail.com <zeo...@gmail.com>
> > wrote:
> > > >
> > > >> Can somebody map out all of the current methods and procedures to
> spin
> > > up
> > > >> Metron components? I swear I find out about new ones every month.
> > > >> Metron-docker, the 4 vagrants, rpm-docker, ansible-docker, any
> others?
> > > >> Perhaps an agreed upon write up of when to use what would be
> helpful.
> > > >>
> > > >> Jon
> > > >>
> > > >> On Tue, Apr 25, 2017, 6:17 PM Ryan Merriman <merrim...@gmail.com>
> > > wrote:
> > > >>
> > > >> > A regression was introduced recently that breaks full dev. I've
> > > >> narrowed
> > > >> > down the commit that introduced it and have submitted a PR to
> revert
> > > >> that
> > > >> > commit: https://github.com/apache/incubator-metron/pull/549.
> > > >> >
> > > >> > Given there has been confusion recently over our deployment build
> > > >> process,
> > > >> > I think it's appropriate that we discuss and come to a consensus
> and
> > > on
> > > >> > how this should work.
> > > >> >
> > > >> > Ryan
> > > >> >
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > >
> > --
> >
> > Jon
> >
>
-- 

Jon

Re: Status of METRON-153

[zeo...@gmail.com]

Just tagging on here to indicate my interest in this - in order to have
someone other than me manage the OSs in my Metron cluster, I must run on
RHEL 7.  I assume that will be common across many enterprises.
Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a rough
go and I dropped it down on my priority list.

Jon

On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler 
wrote:

> https://issues.apache.org/jira/browse/METRON-153
>
> What is the other approach that is mentioned here?  Was it implemented?
> The ansible build still does not support this and I was thinking of looking
> at it.
>
-- 

Jon

Re: So we graduated...

[zeo...@gmail.com]

Well done everybody!  Congrats

Jon

On Thu, Apr 20, 2017 at 8:55 PM Matt Foley  wrote:

> Really exciting!  Congrats to the founding team!
> --Matt
>
>
> On 4/20/17, 4:02 PM, "Houshang Livian"  wrote:
>
> Congratulations Team. Great work!
>
>
>
>
> On 4/20/17, 2:55 PM, "larry mccay"  wrote:
>
> >Wonderful news and well deserved!
> >This community has embraced and committed to the Apache way so
> quickly.
> >
> >
> >On Thu, Apr 20, 2017 at 5:39 PM, Kyle Richardson <
> kylerichards...@gmail.com>
> >wrote:
> >
> >> That's awesome! Congratulations everyone. Looking forward to the
> official
> >> announcement on Monday.
> >>
> >> -Kyle
> >>
> >> > On Apr 20, 2017, at 5:15 PM, David Lyle 
> wrote:
> >> >
> >> > Outstanding! Great work everyone. Building a TLP worthy community
> is
> >> > difficult and worthy work, congratulations all!
> >> >
> >> > -D...
> >> >
> >> >> On Thu, Apr 20, 2017 at 5:12 PM, Casey Stella <
> ceste...@gmail.com>
> >> wrote:
> >> >>
> >> >> For anyone paying attention to incubator-general, it will come
> as no
> >> >> surprise that we graduated as of last night's board meeting.  We
> have a
> >> >> press released queued up and planned for monday along with a PR
> >> (METRON-687
> >> >> at https://github.com/apache/incubator-metron/pull/539).
> >> >>
> >> >> It escaped my notice that the graduation was talked about on
> >> >> incubator-general; otherwise I'd have sent this email earlier
> and been
> >> less
> >> >> cagey in 687's description.  Even so, I'd like to ask that
> everyone
> >> keep it
> >> >> to themselves until monday morning after the press release gets
> out the
> >> >> door.  I know the cat is out of the bag, but it'd be nice to
> have a bit
> >> of
> >> >> an embargo.
> >> >>
> >> >> Thanks!
> >> >>
> >> >> Casey
> >> >>
> >>
>
>
>
> --

Jon

Re: Failing build

[zeo...@gmail.com]

I was just looking at the commit history and noticed that out of the last
commits that had travis runs, at least 7 of them failed (mine is still
running).  Given that we require a clean build on the branch prior to
merging, that's a pretty unfortunate track record.

Jon

On Thu, Apr 6, 2017 at 1:31 PM Casey Stella <ceste...@gmail.com> wrote:

> Yeah, this is an intermittent test failure and has to do with the migration
> to storm 1.0.3 and how it handles shutting down in local mode.  The slot
> worker refuses to shut down and freezes and therefore we wait forever.  I
> thought I had it fixed by manually clobbering the slots, but alas it
> appears that I only made the problem less frequent.  When I found it as
> part of METRON-793, I ran the fix 20 times locally as well as at least 10
> times on my personal travis without repetition.
>
> Bottom line: we should correct it.  I'll have to think a bit more about how
> to fix it and if anyone else wants to take a crack at it, feel free. :)
>
> On Thu, Apr 6, 2017 at 1:26 PM, zeo...@gmail.com <zeo...@gmail.com> wrote:
>
> > We appear to have a failed build again:
> >
> > No output has been received in the last 10m0s, this potentially
> indicates a
> > stalled build or something wrong with the build itself.
> >
> > https://travis-ci.org/apache/incubator-metron/builds/219261745
> >
> > Jon
> > --
> >
> > Jon
> >
>
-- 

Jon