Re: [DISCUSS] Upgrading HBase and Kafka support

[Kyle Richardson]

+1 to Docker and looking at testcontainers.org

On Fri, Mar 8, 2019 at 3:03 PM Otto Fowler  wrote:

> https://github.com/apache/metron-bro-plugin-kafka/tree/master/docker
>
>
> On March 8, 2019 at 14:28:20, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> So most importantly I want to make sure to give Otto credit for being the
> one who cleaned up the rudimentary testing steps we had for testing the
> plugin and turned it into the docker end to end. Right now we manually run
> the tests, as there were a few follow-ons we needed to work through before
> it's ready for Travis. In my opinion, once METRON-2003 (PR 26) gets in
> it'll be ready to have Travis. There isn't any current Maven use
>
> Jon
>
> On Fri, Mar 8, 2019 at 12:26 PM Otto Fowler 
> wrote:
>
> > I believe that the TestContainers allows the ide case
> >
> >
> > On March 8, 2019 at 11:38:24, Michael Miklavcic (
> > michael.miklav...@gmail.com)
> > wrote:
> >
> > I'm -1 on #1 unless there's some desperately compelling reason to go that
> > route. It would be a regression in our test coverage, and at that point
> > it's really just duplicating our unit tests as opposed to checking our
> > integration.
> >
> > I'm good with 3. Gating factors for a successful implementation would be
> > that as a developer I can:
> >
> > 1. Run it in my IDE without having to do anything extra (the beauty of
> > the in-mem component is that @BeforeClass spins it up automatically - we
> > should keep doing something along those lines)
> > 2. Run it via Maven cli
> > 3. Run it in Travis as part of our normal build
> >
> > It's probably worth looking at Kafka's testing infrastructure straight
> from
> > the source - https://github.com/apache/kafka/blob/trunk/tests/README.md.
> > They leverage Docker containers now for system tests.
> >
> > Best,
> > Mike
> >
> >
> > On Fri, Mar 8, 2019 at 7:47 AM Ryan Merriman 
> wrote:
> >
> > > I have been researching the effort involved to upgrade to HDP 3. Along
> > the
> > > way I've found a couple challenging issues that we will need to solve,
> > both
> > > involving our integration testing strategy.
> > >
> > > The first issue is Kafka. We are moving from 0.10.0 to 2.0.0 and there
> > > have been significant changes to the API. This creates an issue in the
> > > KafkaComponent class, which we use as an in-memory Kafka server in
> > > integration tests. Most of the classes that were previously used have
> > gone
> > > away, and to the best of my knowledge, were not supported as public
> APIs.
> > > I also don't see any publicly documented APIs to replace them.
> > >
> > > The second issue is HBase. We are moving from 1.1.2 to 2.0.2 so another
> > > significant change. This creates an issue in the MockHTable class
> > > becausethe HTableInterface class has changed to Table, essentially
> > > requiring that MockHTable be rewritten to conform to the new interface.
> > > It's my opinion that this class is complicated and difficult to
> maintain
> > as
> > > it is anyways.
> > >
> > > These 2 issues have the potential to add a significant amount of work
> to
> > > upgrading Metron to HDP 3. I want to take a step back and review our
> > > options before we move forward. Here are some initial thoughts I had on
> > > how to approach this. For HBase:
> > >
> > > 1. Update MockHTable to work with the new HBase API. We would continue
> > > using a mock server approach for HBase.
> > > 2. Research replacing MockHTable with an in-memory HBase server.
> > > 3. Replace MockHTable with a Docker container running HBase.
> > >
> > > For Kafka:
> > >
> > > 1. Replace KafkaComponent with a mock server implementation.
> > > 2. Update KafkaComponent to work with the new API. We would probably
> > > need to leverage some internal Kafka classes. I do not see a testing
> > > API
> > > documented publicly.
> > > 3. Replace KafkaComponent with a Docker container running Kafka.
> > >
> > > What other options are there? Whatever we choose I think we should
> follow
> > > a similar approach for both (mock servers, in memory servers, Docker,
> > other
> > > options I'm not thinking of).
> > >
> > > This will not shock anyone but I would be in favor of Docker
> containers.
> > > They have the advantage of classpath isolation, easy upgrades, and
> > accurate
> > > integration testing. The downside is we will have to adjusts our tests
> > and
> > > travis script to incorporate these Docker containers into our build
> > > process. We have discussed this at length in the past and it has
> > generally
> > > stalled for various reasons. Maybe if we move a few services at a time
> it
> > > might be more palatable? As for the other 2 approaches, I think if
> either
> > > worked well we wouldn't be having this discussion. Mock servers are
> hard
> > > to maintain and I don't see in memory testing classes documented in
> > > javadocs for either service.
> > >
> > > Thoughts?
> > >
> >
> --
>
> Jon Zeolla
>
-- 
-Kyle

Re: [DISCUSS] Getting to a 1.0 release

[Kyle Richardson]

+1 to separating developer docs and user docs. How should we approach that.
Have a separate doc book? I haven’t had a ton of time to contribute to code
lately but I’d be happy to help write some of these.

On Sat, Aug 18, 2018 at 9:48 AM Nick Allen  wrote:

> Personally, I think the state of our docs and web presence is an inhibitor
> to growing the Metron community.  Unless we can offer concise, compelling
> answers to the basic questions (What can I do with Metron?  Who does it
> help? How do I do that?), potential users and contributors are unable to
> see the value of Metron.
>
>
>
> On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen  wrote:
>
> > I'd like to see us focus on improving our docs before a version 1.0.
> > Right now we just stitch together a bunch of READMEs, which is a great
> > stride from where we started, but is not ideal.
> >
> > Our docs should focused on the user and use cases; What can I do with
> > Metron?  Who does it help? How do I do that?
> >
> > The docs should be separate from the code base to allow for an
> > organization that is focused on the user rather than the implementation.
> > This allows the READMEs to focus on the developer and the implementation,
> > which should make them more digestible too.  The docs should be version
> > controlled and maintained through PRs, just like the code.  We should
> take
> > just as much pride in our docs as we do in our code.
> >
> >
> >
> > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> Agreed, should we add TDE by default, and get the ranger policies on by
> >> default? That leaves secured in Kafka, which would have to be built into
> >> the consumers and producers to encrypt into the on disk Kafka topics.
> Does
> >> that seem necessary to people? It would have performance implications
> for
> >> sure.
> >>
> >> Simon
> >>
> >> > On 15 Aug 2018, at 21:26, Otto Fowler 
> wrote:
> >> >
> >> > Well, I look at it like this.
> >> >
> >> > The Secure Vault was part of the original metron pitch, and many may
> >> have used that as part of their evaluations.
> >> > “Look, it is going to have a security vault type thing, it is on the
> >> roadmap”.
> >> >
> >> > Regardless of the implementation, conceptually, security of data at
> >> rest is important, and is a major outstanding item or the core metron
> >> proposition.
> >> >
> >> >
> >> >
> >> >
> >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball (
> >> si...@simonellistonball.com) wrote:
> >> >>
> >> >> That’s going back a way. I always saw that concept as begin about the
> >> formats, e.g. Orc, and meta data around it plus the data service api to
> get
> >> at it. I’m all for that too, but think it needs more thought than the
> >> ticket captures.
> >> >>
> >> >> Simon
> >> >>
> >> >> On 15 Aug 2018, at 20:53, Otto Fowler 
> wrote:
> >> >>
> >> >>> https://issues.apache.org/jira/browse/METRON-343
> >> >>>
> >>  On August 15, 2018 at 15:47:24, Simon Elliston Ball (
> >> si...@simonellistonball.com) wrote:
> >> 
> >>  What would you see as secure? I’ve seen people use TDE for the HDFS
> >> store, but it’s harder to encrypt storage with solr / es. Something I
> was
> >> thinking of doing to follow up on the Knox Feature was to add Ranger
> >> integration for securing and auditing configs, and potentially
> extending to
> >> the index destinations. Do you think that would cover the secure storage
> >> concept?
> >> 
> >>  Simon
> >> 
> >>  > On 15 Aug 2018, at 20:39, Otto Fowler 
> >> wrote:
> >>  >
> >>  > Secure storage off the top of my head
> >>  >
> >>  > On August 15, 2018 at 14:49:26, zeo...@gmail.com (
> zeo...@gmail.com)
> >> wrote:
> >>  >
> >>  > So, as has been discussed in a few
> >>  > <
> >>  > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a
> >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
> >>  >
> >>  > other
> >>  > <
> >>  > https://lists.apache.org/thread.html/427a20c22207e84331b94e8
> >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
> >>  >
> >>  > recent dev list threads, I would like to discuss what a Metron
> 1.0
> >> release
> >>  > looks like.
> >>  >
> >>  > In order to kick off the conversation, I would like to make a few
> >>  > suggestions regarding "what 1.0 means to me," but I'm very
> >> interested to
> >>  > hear everybody else's opinions.
> >>  >
> >>  > In order to go 1.0 I believe we should have:
> >>  > 1. A clear, supported method of upgrading from one version of
> >> Metron to the
> >>  > next. We have attempted
> >>  >  to
> >> make this
> >>  > easier in the past, but it is currently not
> >>  > <
> >>  > https://github.com/apache/metron/tree/master/metron-deployme
> >> nt/packaging/ambari/metron-mpack#limitations>
> >>  >
> >>  > supported
> 

Re: Metron User Community Meeting Call

[Kyle Richardson]

Thanks! I'll be there. Excited to hear Ahmed's successes and challenges.

-Kyle

On Thu, Jan 25, 2018 at 7:44 PM zeo...@gmail.com  wrote:

> Thanks Otto, I'm in to attend at that time/place.
>
> Jon
>
> On Thu, Jan 25, 2018, 14:45 Otto Fowler  wrote:
>
>> I would like to propose a Metron user community meeting. I propose that
>> we set the meeting next week, and will throw out Wednesday, January 31st at
>> 09:30AM PST, 12:30 on the East Coast and 5:30 in London Towne. This meeting
>> will be held over a web-ex, the details of which will be included in the
>> actual meeting notice.
>> Topics
>>
>> We have a volunteer for a community member presentation:
>>
>> Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR -
>> Cybersecurity Operations Center Carleton University - cugcr.com
>>
>> Ahmed would like to talk to the community about
>>
>>-
>>
>>Who the GCR group is
>>-
>>
>>How they use Metron 0.4.1
>>-
>>
>>Walk through their dashboards, UI management screen, nifi
>>-
>>
>>Challenges we faced up until now
>>
>> I would like to thank Ahmed for stepping forward for this meeting.
>>
>> If you have something you would like to present or talk about please
>> reply here! Maybe we can have people ask for “A better explanation of
>> feature X” type things?
>> Metron User Community Meetings
>>
>> User Community Meetings are a means for realtime discussion of
>> experiences with Apache Metron, or demonstration of how the community is
>> using or will be using Apache Metron.
>>
>> These meetings are geared towards:
>>
>>-
>>
>>Demonstrations and knowledge sharing as opposed to technical
>>discussion or implementation details from members of the Apache Metron
>>Community
>>-
>>
>>Existing Feature demonstrations
>>-
>>
>>Proposed Feature demonstrations
>>-
>>
>>Community feedback
>>
>> These meetings are *not* for :
>>
>>-
>>
>>Support discussions. Those are best left to the mailing lists.
>>-
>>
>>Development discussions. There is another type of meeting for that.
>>
>>
>>
>>
>
> --
>
> Jon
>

Re: [DISCUSS] Community Meetings

[Kyle Richardson]

+1 Thanks for organizing, Otto!

-Kyle

On Thu, Dec 14, 2017 at 1:56 PM, zeo...@gmail.com  wrote:

> This sounds great Otto, thanks.
>
> Jon
>
> On Thu, Dec 14, 2017 at 11:24 AM Laurens Vets  wrote:
>
> >
> > Sounds good to me :)
> >
> > On 2017-12-14 05:59, Otto Fowler wrote:
> > > Ok,
> > >
> > > So we will be concerned with two types of meetings.  I’ll take
> > > responsibility for calling the meetings and ‘moderation’.
> > >
> > > Dev meetings
> > >  - feedback on how things are going overall
> > >  - discussions on specific technical problems
> > >  - discussion of possible improvements
> > >
> > > User meetings
> > >  - demos
> > >  - user content ( how I’m using metron )
> > >  - some unavoidable discussion on problems
> > >  - some requirements gathering triage
> > >
> > > ALL
> > >  - I will call
> > >  - I will gather input for agenda
> > >  - I will distribute the agenda
> > >  - I will distribute the notes to the list and on confluence
> > >  - No decisions will be made, only discussed and then put to list
> > >  - besides general nodes, breakout messages for topical discussion or
> > > decisions
> > >
> > >
> > >
> > > How does that sound?
> > >
> > >
> > > On December 13, 2017 at 16:41:29, Otto Fowler (ottobackwa...@gmail.com
> )
> > > wrote:
> > >
> > > +1
> > >
> > >
> > > On December 13, 2017 at 16:39:52, James Sirota (jsir...@apache.org)
> > > wrote:
> > >
> > > I can set up a dedicated Zoom room with a recurrent meeting and give
> > > PMC
> > > members rights to the room. I think hosting these meetings should not
> > > be a
> > > problem. I would vote not to record them, but rather provide the notes
> > > after the meeting. It's a lot easier to skim through the notes than
> > > jump
> > > around in a recording. As Simon mentioned, I would also make it
> > > explicitly
> > > clear that the meetings are dev meetings. These are not user Q&A and
> > > are
> > > not meant to be overviews of how different features of Metron work. If
> > > we
> > > want to do feature demos or provide user content I would want that to
> > > be in
> > > its own separate meeting.
> > >
> > > Thanks,
> > > James
> > >
> > > 13.12.2017, 05:00, "Otto Fowler" :
> > >> I am ok with just notes and no recording.
> > >>
> > >> On December 13, 2017 at 04:37:20, Simon Elliston Ball (
> > >> si...@simonellistonball.com) wrote:
> > >>
> > >> Good points Larry, we would need to get consent from everyone on the
> > >> call
> > >> to record to properly comply with regulations in some countries. We
> > >> would
> > >> definitely need someone to step up as note taker.
> > >>
> > >> Something else to think about is intended audience. Previously we’ve
> > >> had
> > >> meeting like this which have been very detailed Dev@ focussed (which
> > >> is a
> > >> great thing) but have rather alienated participants in User@ land. We
> > >> need
> > >> to make it clear what level we’re talking about to be inclusive.
> > >>
> > >> Simon
> > >>
> > >>>  On 13 Dec 2017, at 00:44, larry mccay  wrote:
> > >>>
> > >>>  Not sure about posting the recordings - you will need to check and
> > >>> make
> > >>>  sure that doesn't violate anything.
> > >>>
> > >>>  Just a friendly reminder...
> > >>>  It is important that meetings have notes and a summary that is sent
> > >>> out
> > >>>  describing topics to be decided on the mailing list.
> > >>>  No decisions can be made in the community meeting itself - this
> > >>> gives
> > >>>  others in other timezones and commitments review and voice in the
> > >>
> > >> decisions.
> > >>>  If it didn't happen on the mailing lists then it didn't happen. :)
> > >>>
> > >>>  On Tue, Dec 12, 2017 at 1:39 PM, Simon Elliston Ball <
> > >>>  si...@simonellistonball.com> wrote:
> > >>>
> >   Yes, I do.
> > 
> >   I suspect the best bet will be to post recordings somewhere on the
> >   apache.org  metron site.
> > 
> >   Simon
> > 
> > >  On 12 Dec 2017, at 18:36, Otto Fowler 
> > > wrote:
> > >
> > >  Excellent, do you have the > 40 min + record option?
> > >
> > >  On December 12, 2017 at 13:19:55, Simon Elliston Ball (
> > >  si...@simonellistonball.com) wrote:
> > >
> > >  Happy to volunteer a zoom room. That seems to have worked for most
> > > in
> > >>
> > >> the
> > >  past.
> > >
> > >  Simon
> > >
> > >>  On 12 Dec 2017, at 18:09, Otto Fowler 
> > > wrote:
> > >>
> > >>  Thanks! I think I’d like something hosted though.
> > >>
> > >>  On December 12, 2017 at 11:18:52, Ahmed Shah (
> >   ahmeds...@cmail.carleton.ca)
> > >>  wrote:
> > >>
> > >>  Hello,
> > >>
> > >>  wrt "- How are we going to host it"...
> > >>
> > >>  I've used BigBlueButton as an end user at our University.
> > >>
> > >>  It is LGPL open source.
> > >>
> > >>  https://bigbluebutton.org/
> > >>  https://bigbluebutton.org/developers/
> > >>
> > >>  -Ahmed
> > >

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Kyle Richardson]

I agree with Justin. I suggest we move forward with option (a). I see it as
simpler and more flexible moving forward.

-Kyle

On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet  wrote:

> Do we have any further discussion on this?  Pardon me if I misstate
> anyone's position, but it seems like we have a couple people (Otto and Jon
> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
> section of people like myself without a particular horse in the race.
>
> It seems like we need to come to some sort of consensus so that we can get
> the release bus moving again, and right now it seems like (a) is gathering
> more explicit support.  Do we have a compelling reason to not do (a)? To be
> honest, my main worry is more "If we do (a) are we going to be miserable if
> we need to iterate or adjust?" I'm not seeing anything that suggests
> anything too terrible, so unless we see some more discussion, I suggest we
> move forward with (a).
>
> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com  wrote:
>
> > I would prefer a, but I was initially thinking of doing the plugin first
> > and then get in the two PRs out to use this new tag, which are already
> +1'd
> > and just waiting on this conversation.  For reference,
> > https://github.com/apache/metron/pull/847 and
> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
> >
> > Jon
> >
> > On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:
> >
> > > It seems to me, as I believe I have stated before that a) feels like
> the
> > > proper way to handle this.  It is how I have seen other projects like
> > NiFi
> > > handle things as well.
> > >
> > >
> > >
> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
> > >
> > > Okay, looking at this from the perspective of making a release:
> > >
> > >
> > >
> > > We have two choices:
> > >
> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > > metron-bro-plugin-kafka, at the same time and using the same process
> > > (modulo the necessary) as Metron.  This is dirt simple.
> > >
> > > b) I or someone needs to:
> > >
> > > - open a jira,
> > >
> > > - add the submodule to the Metron code tree,
> > >
> > > - possibly (optionally) add build mechanism to the maven poms, and
> > >
> > > - document as much as we think appropriate regarding what it is,
> how
> > to
> > > build it, and how to update it,
> > >
> > > and commit that before the 0.4.2 release.
> > >
> > >
> > >
> > > What is the will of the community?
> > >
> > > Thanks,
> > >
> > > --Matt
> > >
> > >
> > >
> > > From: Nick Allen 
> > > Reply-To: "dev@metron.apache.org" 
> > > Date: Tuesday, November 28, 2017 at 9:09 AM
> > > To: "dev@metron.apache.org" 
> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> > Bro'
> > >
> > >
> > >
> > > I'll add a bit to Jon's technical comments.
> > >
> > >
> > >
> > > * We only created a separate repo because it was a technical
> requirement
> > to
> > > leverage the bro-pkg mechanism.
> > >
> > > * Leveraging the new bro-pkg mechanism has many advantages as outlined
> by
> > > Jon.
> > >
> > > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> > the
> > > plugin exactly how we use to.
> > >
> > >
> > >
> > > While I agree with Jon's technical comments, I disagree with the
> > > non-technical ones.
> > >
> > >
> > >
> > > (1) I do not want to change our release management process. While we
> > needed
> > > to make a new repo (a technical change), I did not want that to change
> > how
> > > we operate as a community (our procedures, policies, versioning and
> > release
> > > cycles).
> > >
> > >
> > >
> > > (2) I see no value in adopting a separate release management process
> for
> > > the Bro plugin alone. Having a separate release process does not make
> the
> > > plugin *more* available to the Bro community.
> > >
> > >
> > >
> > > (3) I also see no value in positioning the plugin to be spun-out of the
> > > Metron project. It is a part of Metron and I want to see it benefit and
> > > evolve "the Apache-way".
> > >
> > >
> > >
> > > In my mind, the best way to accommodate the additional repo, while
> > > minimizing changes to our release management process, is to treat the
> new
> > > repo as a submodule. I fail to see significant downsides to this
> > approach.
> > > A few extract command-line options do not seem overly onerous to me.
> > >
> > >
> > >
> > > Many thanks go to Jon for all the hard work he has put in enhancing the
> > > plugin.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com 
> > > wrote:
> > >
> > > In an attempt to keep this from becoming unbearably long, I will try to
> > > keep my responses short, but I would be happy to elaborate. That's a
> > fairly
> > > good timeline and summary, but here are some clarifications in
> > > corresponding order:
> > >
> > >
> > >
> > > - The plugin history is quite short and you can probably get a good bit
> > of
> > > context 

Re: New PMC members

[Kyle Richardson]

Congratulations guys! Well deserved.

-Kyle

On Thu, Dec 7, 2017 at 10:18 AM, Nick Allen  wrote:

> Congrats to all 3 for joining the PMC!
>
> On Thu, Dec 7, 2017 at 10:12 AM, Casey Stella  wrote:
>
> > Well, obviously, I meant Metron instead of Impala.  To this point, we
> > should have a wiki page around templates for this, similar to the impala
> > project. :)
> >
> > On Thu, Dec 7, 2017 at 10:06 AM, Casey Stella 
> wrote:
> >
> > > The Project Management Committee (PMC) for Apache Impala has invited
> Otto
> > > Fowler, Michael Miklavcic and Justin Leet  to become a PMC member and
> we
> > > are pleased to announce that they have accepted.
> > >
> > > Congratulations and welcome!
> > >
> > >
> >
>

Re: Wiki Docs links seem wrong

[Kyle Richardson]

Fixed.

-Kyle

On Thu, Dec 7, 2017 at 7:20 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> https://cwiki.apache.org/confluence/display/METRON/
> Metron+User+Guide+-+per+release  confluence/display/METRON/Metron+User+Guide+-+per+release>
>
> The links don’t seem to correspond to the versions on this page. Would be
> happy to fix, but I don’t have wiki perms.
>
> Simon

Re: [DISCUSS] - Remove Kibana

[Kyle Richardson]

I take your point. If another mpack would be that resource intensive, it
doesn't make much sense to go down that road. There are plenty of other
features the community would get greater value from for the time investment.

-Kyle

On Wed, Nov 1, 2017 at 10:14 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I think the suggestion is creative, however I am strongly opposed to 2
> mpacks. It has been a fair amount of work through upgrades to maintain the
> one we have and I think splitting them might make it even worse. I'd much
> prefer to keep the dep on Kibana as is rather than make a change that
> doesn't simplify things in the end.
>
> On Nov 1, 2017 6:25 PM, "Kyle Richardson" 
> wrote:
>
> > I'll second Otto's suggestion. I like the idea of "splitting" the ES and
> > Kibana components from the pure Metron components. I suppose that would
> > mean having two mpacks to build for a while though.
> >
> > I agree with others that, at least for now, Kibana is an integral part of
> > the Metron user experience.
> >
> > -Kyle
> >
> > On Wed, Nov 1, 2017 at 7:37 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > It would not be installed with/by Metron. You'd install and manage
> Kibana
> > > on your own. Some things can be done with the head plugin, but it
> > wouldn't
> > > be as pretty.
> > >
> > > From the sounds of it the community still uses and wants Kibana, so
> we'll
> > > hold off until the UI can manage more of this functionality. Thanks
> > > everyone for the input.
> > >
> > > On Nov 1, 2017 4:18 PM, "Laurens Vets"  wrote:
> > >
> > > > How would I do that without Kibana? Having a SIEM without the ability
> > to
> > > > see raw processed events (whether they are alerts or not), would be a
> > big
> > > > issue I think.
> > > >
> > > > Or would Kibana always be required, just not installed by Metron?
> > > >
> > > > On 2017-11-01 11:34, Michael Miklavcic wrote:
> > > >
> > > >> You could absolutely still do it, I'm simply saying it would not be
> > > >> managed
> > > >> by us.
> > > >>
> > > >> On Nov 1, 2017 12:20 PM, "Laurens Vets"  wrote:
> > > >>
> > > >> If there's a viable way of looking at raw processed events (not
> > > >>> necessarily alerts), then I'm all for removeing Kibana. I use
> > Discover
> > > a
> > > >>> lot to filter and look at events and create new policies from that.
> > > >>>
> > > >>> Is there currently a simple way to do this without Kibana?
> > > >>>
> > > >>> On 2017-11-01 09:13, Michael Miklavcic wrote:
> > > >>>
> > > >>> As part of the ES upgrade, I got to thinking that it makes sense to
> > > >>>> remove
> > > >>>> Kibana and the dashboards we're currently bundling in the MPack.
> To
> > be
> > > >>>> clear, this would not remove the ability to independently install
> > and
> > > >>>> use
> > > >>>> Kibana if the user so chooses, it would only remove the
> dashboards,
> > > and
> > > >>>> potentially, the Ambari/MPack management support that we ship.
> > > >>>>
> > > >>>> *pros*
> > > >>>> Removes need to support tooling outside our wheelhouse
> > > >>>> Smaller testing effort for ongoing support and future upgrades
> > > >>>> Simplifies our base Metron install via Ambari. Also simplifies
> full
> > > dev
> > > >>>> setup.
> > > >>>>
> > > >>>> *cons*
> > > >>>> User would need to install and setup their own Kibana instance and
> > > >>>> dashboards.
> > > >>>> If any existing users are using this, they'd need to backup and
> > manage
> > > >>>> their own Kibana dashboards going forward. They would also need to
> > > >>>> handle
> > > >>>> any upgrade issues with Kibana post-Metron ES 5.x upgrade.
> > > >>>>
> > > >>>> Any concerns?
> > > >>>>
> > > >>>> Mike
> > > >>>>
> > > >>>
> > >
> >
>

Re: [DISCUSS] - Remove Kibana

[Kyle Richardson]

I'll second Otto's suggestion. I like the idea of "splitting" the ES and
Kibana components from the pure Metron components. I suppose that would
mean having two mpacks to build for a while though.

I agree with others that, at least for now, Kibana is an integral part of
the Metron user experience.

-Kyle

On Wed, Nov 1, 2017 at 7:37 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> It would not be installed with/by Metron. You'd install and manage Kibana
> on your own. Some things can be done with the head plugin, but it wouldn't
> be as pretty.
>
> From the sounds of it the community still uses and wants Kibana, so we'll
> hold off until the UI can manage more of this functionality. Thanks
> everyone for the input.
>
> On Nov 1, 2017 4:18 PM, "Laurens Vets"  wrote:
>
> > How would I do that without Kibana? Having a SIEM without the ability to
> > see raw processed events (whether they are alerts or not), would be a big
> > issue I think.
> >
> > Or would Kibana always be required, just not installed by Metron?
> >
> > On 2017-11-01 11:34, Michael Miklavcic wrote:
> >
> >> You could absolutely still do it, I'm simply saying it would not be
> >> managed
> >> by us.
> >>
> >> On Nov 1, 2017 12:20 PM, "Laurens Vets"  wrote:
> >>
> >> If there's a viable way of looking at raw processed events (not
> >>> necessarily alerts), then I'm all for removeing Kibana. I use Discover
> a
> >>> lot to filter and look at events and create new policies from that.
> >>>
> >>> Is there currently a simple way to do this without Kibana?
> >>>
> >>> On 2017-11-01 09:13, Michael Miklavcic wrote:
> >>>
> >>> As part of the ES upgrade, I got to thinking that it makes sense to
>  remove
>  Kibana and the dashboards we're currently bundling in the MPack. To be
>  clear, this would not remove the ability to independently install and
>  use
>  Kibana if the user so chooses, it would only remove the dashboards,
> and
>  potentially, the Ambari/MPack management support that we ship.
> 
>  *pros*
>  Removes need to support tooling outside our wheelhouse
>  Smaller testing effort for ongoing support and future upgrades
>  Simplifies our base Metron install via Ambari. Also simplifies full
> dev
>  setup.
> 
>  *cons*
>  User would need to install and setup their own Kibana instance and
>  dashboards.
>  If any existing users are using this, they'd need to backup and manage
>  their own Kibana dashboards going forward. They would also need to
>  handle
>  any upgrade issues with Kibana post-Metron ES 5.x upgrade.
> 
>  Any concerns?
> 
>  Mike
> 
> >>>
>

Re: Stellar support for switch/case style conditionals

[Kyle Richardson]

Gotcha. Yes, that makes perfect sense.

-Kyle

On Tue, Oct 24, 2017 at 11:39 AM, Casey Stella  wrote:

> Right.  My point is that you never want the values of the match statement
> to be evaluated prior to evaluating the match (otherwise you cannot short
> circuit them).  Consider the following:
>
> match {foo == 0 : CHEAP_FUNC(), foo != 0 : EXPENSIVE_FUNC() }
>
> you don't want CHEAP_FUNC() AND EXPENSIVE_FUNC() to be run every time it's
> evaluated and have the values returned depending on foo's value.  Instead,
> you'd want to evaluate foo and choose which of EXPENSIVE_FUNC() or
> CHEAP_FUNC() to evaluate (essentially treating the values as implicit
> lambdas).  This is how if/then/else's work now.  For chained conditionals,
> you aren't going to evaluate the else if the if condition is true.  I just
> want to make sure this fits that paradigm.
>
> Casey
>
>
> On Tue, Oct 24, 2017 at 11:34 AM, Kyle Richardson <
> kylerichards...@gmail.com
> > wrote:
>
> > I guess I read it as supporting both. As a user, I certainly would prefer
> > to use the less explicit syntax ( e.g. match { foo == 0: true, bar == 1:
> > false, default: false } ).
> >
> > -Kyle
> >
> > On Tue, Oct 24, 2017 at 11:30 AM, Casey Stella 
> wrote:
> >
> > > So, I do like it.  My only issue is the explicit lambda syntax in the
> > > values there (e.g. foo == 0 : () -> true)  I'm afraid that when we
> > migrate
> > > to a less explicit lambda syntax ( foo == 0 : true ), we will cause
> > people
> > > to have to transition twice in a row.
> > >
> > > Also, it goes without saying, but the syntax must be short-circuiting
> > IMO.
> > >
> > > Casey
> > >
> > > On Tue, Oct 24, 2017 at 11:26 AM, Kyle Richardson <
> > > kylerichards...@gmail.com
> > > > wrote:
> > >
> > > > I like the way you have this laid out. Very useful to see it in test
> > > cases.
> > > > I'm +1 for this syntax addition.
> > > >
> > > > -Kyle
> > > >
> > > > On Mon, Oct 23, 2017 at 4:16 PM, Otto Fowler <
> ottobackwa...@gmail.com>
> > > > wrote:
> > > >
> > > > > What I would like to do for the first PR is introduce match with
> the
> > > > > following syntax
> > > > >
> > > > >
> > > > > match{ logical expression   :  transformation expression, ….. ,
> > > default :
> > > > > transformation expression}
> > > > >
> > > > > Such that the following work  for example:
> > > > >
> > > > > @Test
> > > > > public void testMatch() {
> > > > >  Assert.assertTrue(runPredicate("match { 1 >= 0 : ()-> true }",
> new
> > > > > HashMap(){{
> > > > >put("foo", 0);
> > > > >   }}));
> > > > >   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> true,
> > > > > default : ()-> false }", new HashMap(){{
> > > > > put("foo", 0);
> > > > >   }}));
> > > > >   Assert.assertFalse(runPredicate("match { foo == 0 : ()-> true,
> > > > > default : ()-> false }", new HashMap(){{
> > > > > put("foo", 1);
> > > > >   }}));
> > > > >
> > > > >   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> false,
> foo
> > > > > == 1 : ()-> true, default : ()-> false }", new HashMap(){{
> > > > > put("foo", 1);
> > > > >   }}));
> > > > >
> > > > >   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> bFalse,
> > foo
> > > > > == 1 : ()-> bTrue, default : ()-> bFalse }", new HashMap(){{
> > > > > put("foo", 1);
> > > > > put("bFalse", new Boolean(false));
> > > > > put("bTrue", new Boolean(true));
> > > > >   }}));
> > > > >
> > > > >   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> bFalse,
> > foo
> > > > > == 1 : ()-> bTrue, default : ()-> bFalse }", new HashMap(){{
> > > > > put("foo", 1);
> > > > > put("bFalse", new Boolean(false));
> > > > > put("bTrue", new Boolean(true));
> > > > >   }}));

Re: Stellar support for switch/case style conditionals

[Kyle Richardson]

I guess I read it as supporting both. As a user, I certainly would prefer
to use the less explicit syntax ( e.g. match { foo == 0: true, bar == 1:
false, default: false } ).

-Kyle

On Tue, Oct 24, 2017 at 11:30 AM, Casey Stella  wrote:

> So, I do like it.  My only issue is the explicit lambda syntax in the
> values there (e.g. foo == 0 : () -> true)  I'm afraid that when we migrate
> to a less explicit lambda syntax ( foo == 0 : true ), we will cause people
> to have to transition twice in a row.
>
> Also, it goes without saying, but the syntax must be short-circuiting IMO.
>
> Casey
>
> On Tue, Oct 24, 2017 at 11:26 AM, Kyle Richardson <
> kylerichards...@gmail.com
> > wrote:
>
> > I like the way you have this laid out. Very useful to see it in test
> cases.
> > I'm +1 for this syntax addition.
> >
> > -Kyle
> >
> > On Mon, Oct 23, 2017 at 4:16 PM, Otto Fowler 
> > wrote:
> >
> > > What I would like to do for the first PR is introduce match with the
> > > following syntax
> > >
> > >
> > > match{ logical expression   :  transformation expression, ….. ,
> default :
> > > transformation expression}
> > >
> > > Such that the following work  for example:
> > >
> > > @Test
> > > public void testMatch() {
> > >  Assert.assertTrue(runPredicate("match { 1 >= 0 : ()-> true }", new
> > > HashMap(){{
> > >put("foo", 0);
> > >   }}));
> > >   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> true,
> > > default : ()-> false }", new HashMap(){{
> > > put("foo", 0);
> > >   }}));
> > >   Assert.assertFalse(runPredicate("match { foo == 0 : ()-> true,
> > > default : ()-> false }", new HashMap(){{
> > > put("foo", 1);
> > >   }}));
> > >
> > >   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> false, foo
> > > == 1 : ()-> true, default : ()-> false }", new HashMap(){{
> > > put("foo", 1);
> > >   }}));
> > >
> > >   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> bFalse, foo
> > > == 1 : ()-> bTrue, default : ()-> bFalse }", new HashMap(){{
> > > put("foo", 1);
> > > put("bFalse", new Boolean(false));
> > > put("bTrue", new Boolean(true));
> > >   }}));
> > >
> > >   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> bFalse, foo
> > > == 1 : ()-> bTrue, default : ()-> bFalse }", new HashMap(){{
> > > put("foo", 1);
> > > put("bFalse", new Boolean(false));
> > > put("bTrue", new Boolean(true));
> > >   }}));
> > >
> > >   Assert.assertTrue(runPredicate("match { foo == 0 : bFalse, foo == 1
> > > : bTrue, default : false }", new HashMap(){{
> > > put("foo", 1);
> > > put("bFalse", new Boolean(false));
> > > put("bTrue", new Boolean(true));
> > >   }}));
> > >
> > >   Assert.assertTrue(runPredicate("match { foo == 0  OR bar == 'yes' :
> > > ()-> true, default : ()-> false }", new HashMap(){{
> > > put("foo", 1);
> > > put("bar", "yes");
> > >   }}));
> > >
> > >   Assert.assertEquals("warning", run("match{ threat.triage.level < 10
> > > : 'info', threat.triage.level < 20 : 'warning', default : 'critical'
> > > }", new HashMap(){{
> > > put("threat.triage.level", 15);
> > >   }}));
> > > }
> > >
> > >
> > > So, the transformation expression will include support for zero arg
> > lambda
> > > syntax.  The work to support the AS aliasing statement and lambda
> support
> > > for parameters may page in assignment and some other things,
> > > and I would like to make that a follow on with some discussion after
> > > review.
> > >
> > > Thoughts?
> > >
> > >
> > >
> > > On October 17, 2017 at 14:31:07, Otto Fowler (ottobackwa...@gmail.com)
> > > wrote:
> > >
> > > OK
> > >
> > >
> > > On October 17, 2017 at 14:28:02, Casey Stella (ceste...@gmail.com)
> > wrote:
> > >
> > > Yeah, default would be a keyword.  We co

Re: Stellar support for switch/case style conditionals

[Kyle Richardson]

I like the way you have this laid out. Very useful to see it in test cases.
I'm +1 for this syntax addition.

-Kyle

On Mon, Oct 23, 2017 at 4:16 PM, Otto Fowler 
wrote:

> What I would like to do for the first PR is introduce match with the
> following syntax
>
>
> match{ logical expression   :  transformation expression, ….. , default :
> transformation expression}
>
> Such that the following work  for example:
>
> @Test
> public void testMatch() {
>  Assert.assertTrue(runPredicate("match { 1 >= 0 : ()-> true }", new
> HashMap(){{
>put("foo", 0);
>   }}));
>   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> true,
> default : ()-> false }", new HashMap(){{
> put("foo", 0);
>   }}));
>   Assert.assertFalse(runPredicate("match { foo == 0 : ()-> true,
> default : ()-> false }", new HashMap(){{
> put("foo", 1);
>   }}));
>
>   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> false, foo
> == 1 : ()-> true, default : ()-> false }", new HashMap(){{
> put("foo", 1);
>   }}));
>
>   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> bFalse, foo
> == 1 : ()-> bTrue, default : ()-> bFalse }", new HashMap(){{
> put("foo", 1);
> put("bFalse", new Boolean(false));
> put("bTrue", new Boolean(true));
>   }}));
>
>   Assert.assertTrue(runPredicate("match { foo == 0 : ()-> bFalse, foo
> == 1 : ()-> bTrue, default : ()-> bFalse }", new HashMap(){{
> put("foo", 1);
> put("bFalse", new Boolean(false));
> put("bTrue", new Boolean(true));
>   }}));
>
>   Assert.assertTrue(runPredicate("match { foo == 0 : bFalse, foo == 1
> : bTrue, default : false }", new HashMap(){{
> put("foo", 1);
> put("bFalse", new Boolean(false));
> put("bTrue", new Boolean(true));
>   }}));
>
>   Assert.assertTrue(runPredicate("match { foo == 0  OR bar == 'yes' :
> ()-> true, default : ()-> false }", new HashMap(){{
> put("foo", 1);
> put("bar", "yes");
>   }}));
>
>   Assert.assertEquals("warning", run("match{ threat.triage.level < 10
> : 'info', threat.triage.level < 20 : 'warning', default : 'critical'
> }", new HashMap(){{
> put("threat.triage.level", 15);
>   }}));
> }
>
>
> So, the transformation expression will include support for zero arg lambda
> syntax.  The work to support the AS aliasing statement and lambda support
> for parameters may page in assignment and some other things,
> and I would like to make that a follow on with some discussion after
> review.
>
> Thoughts?
>
>
>
> On October 17, 2017 at 14:31:07, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> OK
>
>
> On October 17, 2017 at 14:28:02, Casey Stella (ceste...@gmail.com) wrote:
>
> Yeah, default would be a keyword.  We could also do match(variable1 as x,
> variable2 as y) if you want to alias your fields *or* you could do match {
> ... } if you dont' want to alias your variables.
>
> e.g. if you had a field threat.triage.level either of these would work:
>
> match(threat.triage.level -> x) { x < 10 : 'info', x < 20 : 'warning',
> default : 'critical' }
> OR
> match { threat.triage.level < 10 : 'info', threat.triage.level < 20 :
> 'warning, default : 'critical' }
>
>
>
> On Tue, Oct 17, 2017 at 2:24 PM, Otto Fowler 
> wrote:
>
> > No that is it.
> >
> > So default would be a keyword?
> >
> > and a lambda that uses x can be used on the right side of the :
> >
> >
> >
> > On October 17, 2017 at 14:21:01, Casey Stella (ceste...@gmail.com)
> wrote:
> >
> > So, just to map this onto the example, you mean:
> > match(longer_variable -> x) { x < 10 : 'info', x <= 20 : 'warn', default:
> > 'critical' } ?  I took the liberty of adding a default keyword there  the
> > evaluation of the conditionals are considered lambda functions also.
> >
> > Did I catch the spirit of the suggestion or did I miss anything?
> >
> > On Tue, Oct 17, 2017 at 2:18 PM, Otto Fowler 
> > wrote:
> >
> >> How about this:
> >>
> >> match(VAR_TO_VAL_ASSIGNMENT+) { BOOLEAN_STATEMENT(VALS) : LAMBDA(VALS),
> >> BOOLEAN_STATEMENT(VALS) : LAMBDA(VALS) , LAMBDA(VALS)}
> >>
> >> * match = new keyword
> >> * match takes variable number of assignments, where the val assigned to
> >> is available in the evaluation and the lambdas
> >> * match {} contains comma separated list of a statement that evaluates
> to
> >> a boolean and a lambda
> >> * LAMBDA is executed on match, and it’s value is returned
> >> * no matches returns null or return of optional final statement, which
> is
> >> a LAMBDA without a BOOLEAN_STATEMENT
> >>
> >>
> >> On October 17, 2017 at 12:06:05, Casey Stella (ceste...@gmail.com)
> wrote:
> >>
> >> Ugh, I forgot to preface this with DISCUSS: Sorry!
> >>
> >> On Tue, Oct 17, 2017 at 12:05 PM, Casey Stella 
> >> wrote:
> >>
> >> > Hi All,
> >> >
> >> > It's high time that Stellar supports some form of conditional that is
> >> > beyond if/then/else. Right now, the way to do fall-through
> conditionals
> >> is:
> >> >
> >> > if x < 10 then 'info' else if x >= 10 && x <= 20 then 'warn' else
> >> > 'critical'
> >> >
> >> > That be

Re: [DISCUSS] Release Process Update

[Kyle Richardson]

+1 I agree with Matt and Justin. The Travis build widget doesn't make sense
in the published release documentation. No sense in fixing retrospectively.

-Kyle

On Mon, Oct 23, 2017 at 3:13 PM Matt Foley  wrote:

> I agree with Justin.  This micro-feature is intended as a github widget,
> which causes the top-level README to give all viewers an immediate flag
> whether the build is healthy or not.  It does not belong in a rendered
> site-book.
>
> Removing the widget during site-book build, can be done with a one-line
> addition to HREF_REWRITE_LIST in:
>
> https://github.com/apache/metron/blob/master/site-book/bin/generate-md.sh#L75
>
> Recommend not worrying about historical site-books (they naturally
> obsolesce out of the “dist/release” area).
>
> Cheers,
> --Matt
>
> On 10/23/17, 6:38 AM, "Justin Leet"  wrote:
>
> I'd argue it shouldn't be in the site-book at all. Presumably we aren't
> releasing while Travis is broken, so it's not useful information for
> anyone
> looking at docs. It just carries over from the main README.  Seems
> like we
> should just scrub it when we do the other fixes to the READMEs to make
> them
> suitable for site-book.  At that point it's just gone entirely. from
> the
> next release.
>
> Doesn't solve the problem of prior releases (assuming we care enough
> to do
> anything).
>
> On Mon, Oct 23, 2017 at 9:32 AM, zeo...@gmail.com 
> wrote:
>
> > Today I was poking around the Metron site and documentation, and I
> noticed
> > that the site-book's travis build status image is pointing to master
> for
> > all of our releases.  We should probably update the release process
> > 
> to
> > pin
> > this to the specific release.
> >
> > I will happily handle the documentation update but I wanted to ask,
> what
> > should it be pointing to?  Repointing is incredibly straightforward
> > , but I'm not sure
> would be
> > more appropriate to use - the release branches (e.g. Metron_0.4.1),
> or
> > the release tags (e.g. apache-metron-0.4.1-release)?  I'm not clear
> on
> > their specific uses in our environment.  In reviewing our current
> process,
> > it appears that we _could_ use either.
> >
> > I also wanted to ask, does anybody think that this should get fixed
> > historically?  I think that this might be an excessive amount of
> hassle,
> > but I wanted to put it out there since I'm not intimately familiar
> with
> > what we'd need to do in order to clean this up.
> >
> > Jon
> > --
> >
> > Jon
> >
>
>
>

Re: Can we close old inactive PR’s

[Kyle Richardson]

I was wondering that too. There are quite a few that have had no response
in a long time.

-Kyle

On Tue, Oct 17, 2017 at 9:00 AM, Otto Fowler 
wrote:

> Can we close PR’s that are inactive for long periods of time, with no
> response to queries?
>
> Looking at : https://github.com/apache/metron/pull/278 for example.
>

Re: ENRICHMENT_GET from REPL throwing exception

[Kyle Richardson]

Thanks to Casey for some fast diagnosis on IRC. This is patched in master
via METRON-1185 (PR#755). Updates to $METRON_HOME/bin/stellar from that PR
fixed the issue.

Thanks again!

-Kyle

On Tue, Oct 10, 2017 at 12:27 PM, Kyle Richardson  wrote:

> Ok, so I've admittedly been absent for a while and may have missed a few
> things here and there, but I've run into an error that I'm looking for some
> input on. Might be a bug, might be user error.
>
> Short background info. I created and populated an HBase table called
> 'threatintel_ip' with IOCs that I'd like to use for enrichment. When I
> tried to test the lookup in the Stellar REPL using ENRICHMENT_GET, I get an
> exception thrown that seems like it might be related to the resolution
> order of the guava library.
>
> The full stack trace is here, https://gist.github.com/kylerichardson/
> f2e34e400073eb9b7521aa080574678c.
>
> Has anyone else seen this? Am I missing something simple?
>
> I'm running the 0.4.1 released code on a 3 node dev cluster with HDP
> 2.5.3.0.
>
> Thanks,
> Kyle
>
>

ENRICHMENT_GET from REPL throwing exception

[Kyle Richardson]

Ok, so I've admittedly been absent for a while and may have missed a few
things here and there, but I've run into an error that I'm looking for some
input on. Might be a bug, might be user error.

Short background info. I created and populated an HBase table called
'threatintel_ip' with IOCs that I'd like to use for enrichment. When I
tried to test the lookup in the Stellar REPL using ENRICHMENT_GET, I get an
exception thrown that seems like it might be related to the resolution
order of the guava library.

The full stack trace is here,
https://gist.github.com/kylerichardson/f2e34e400073eb9b7521aa080574678c.

Has anyone else seen this? Am I missing something simple?

I'm running the 0.4.1 released code on a 3 node dev cluster with HDP
2.5.3.0.

Thanks,
Kyle

Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

[Kyle Richardson]

I've been stretched a little thin lately but am trying to jump back in.
2:30pm EDT works well for me too.

-Kyle

On Mon, Sep 25, 2017 at 8:38 PM, Otto Fowler 
wrote:

> 11:30 your time.  Sorry I have to pick my kids up from school.  2:30 mine.
>
>
> On September 25, 2017 at 19:41:28, James Sirota (jsir...@apache.org)
> wrote:
>
> Oh sorry, didn't notice that. Otto, when is a good time for you?
>
> 25.09.2017, 16:35, "zeo...@gmail.com" :
> > When is the meeting, given Otto mentioned he can't make 10am? Or did that
> > change
> >
> > Jon
> >
> > On Mon, Sep 25, 2017, 19:19 James Sirota  wrote:
> >
> >>  Great. Thank you, Otto. I would encourage everyone to watch it so that
> we
> >>  have constructive feedback for tomorrow and are able to arrive to a
> decision
> >>
> >>  Thanks,
> >>  James
> >>
> >>  25.09.2017, 08:27, "Otto Fowler" :
> >>  > https://youtu.be/-ISycoP3TVA
> >>  >
> >>  > The video is short and simple. Hopefully it is what you are looking
> for.
> >>  >
> >>  > On September 21, 2017 at 16:54:13, zeo...@gmail.com (
> zeo...@gmail.com)
>
> >>  > wrote:
> >>  >
> >>  > I won't be able to make it and would really like to make sure there's
> a
> >>  > recording for this one, if possible. I'm unavailable until Thursday
> of
> >>  > next week, but not necessarily suggesting this gets moved.
> >>  >
> >>  > Jon
> >>  >
> >>  > On Thu, Sep 21, 2017, 15:04 Otto Fowler 
> wrote:
> >>  >
> >>  >> I can’t make that time, can we make it later in the day?
> >>  >>
> >>  >> On September 21, 2017 at 11:40:37, James Sirota (jsir...@apache.org
> )
> >>  >> wrote:
> >>  >>
> >>  >> https://hortonworks.webex.com/meet/jsirota
> >>  > --
> >>  >
> >>  > Jon
> >>
> >>  ---
> >>  Thank you,
> >>
> >>  James Sirota
> >>  PPMC- Apache Metron (Incubating)
> >>  jsirota AT apache DOT org
> > --
> >
> > Jon
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>

Re: [ANNOUNCE] Metron community meeting

[Kyle Richardson]

Unfortunately I have a conflict. Will there be a recording posted afterward
like previous meetings?

-Kyle

On Mon, Aug 21, 2017 at 10:35 AM Otto Fowler 
wrote:

> Sounds good
>
>
> On August 21, 2017 at 09:43:25, James Sirota (jsir...@apache.org) wrote:
>
> Hi Jon,
>
> Sure. Lets move it by a day. The reason it's at this time is to give people
> in India and Europe a chance to attend live.
>
> So lets move it to the same time tomorrow
>
> I would like to schedule it for Tuesday, Aug. 22. at 10.30AM PST.
>
> I would like to have it over webex:
>
>
>
> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93
> Meeting number: 622 537 955
> Meeting password: biFTEuh2
>
> For global callers:
>
>
> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC&ED=590161912&tollFree=1
>
> Thanks,
> James
>
> 18.08.2017, 11:02, "zeo...@gmail.com" :
> > Is it possible to reschedule this to later in the day or another day?
> That
> > overlaps with the eclipse on the east cost of the US that some people
> would
> > like to enjoy.
> >
> > Jon
> >
> > On Fri, Aug 18, 2017, 13:48 James Sirota  wrote:
> >
> >>  I would like to propose a meeting with the following set of topics:
> >>
> >>  - What do we want in the next build
> >>  - Where do we want to take the project long-term
> >>  - Any feature requests/comments on existing feature set
> >>  - Feedback on some of the early UI work
> >>
> >>  Please suggest if there is anything else you want to talk about.
> >>
> >>  I would like to schedule it for Monday, Aug. 21. at 10.30AM PST.
> >>
> >>  I would like to have it over webex:
> >>
> >>
>
> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93
> >>  Meeting number: 622 537 955
> >>  Meeting password: biFTEuh2
> >>
> >>  For global callers:
> >>
> >>
>
> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC&ED=590161912&tollFree=1
> >>
> >>  Anyone is welcome to join.
> >>  ---
> >>  Thank you,
> >>
> >>  James Sirota
> >>  PPMC- Apache Metron (Incubated and Hatched)
> >>  jsirota AT apache DOT org
> > --
> >
> > Jon
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>

Re: [DISCUSS] Relocate Docker

[Kyle Richardson]

I completely support the idea of moving metron-docker down in the tree. I
do like the idea of a contrib/ area for things like this that are not as
frequently updated or maintained. Are there any other pieces of the code
base that would fit into this type of area?

-Kyle

On Thu, Jul 13, 2017 at 12:30 PM Laurens Vets  wrote:

> On 2017-07-13 09:04, Nick Allen wrote:
> > Having metron-docker at the top-level of the project seems to catch the
> > attention of new users.  Some then start using metron-docker to
> > explore/try-out/demo Metron.
> >
> > The metron-docker code that we have is not well-suited for this
> > purpose.
> > It is only really useful for development.  It is not regularly tested
> > and
> > maintained like our Vagrant environment.
> >
> > I am proposing that we move the top-level "metron-docker" directory to
> > live
> > under "metron-deployment" to avoid confusing new users.  This also
> > seems
> > more logical to me. I would also pair this move with some better
> > "Getting
> > Started" steps for new users in the top-level README.
> >
> > cd $METRON_HOME
> > mv metron-docker metron-deployment/docker
> >
> >
> > Do you agree/disagree?  Is there a better solution?
>
> Maybe under "Other Examples"?
>

[REQUEST] Wiki Edit Rights

[Kyle Richardson]

Could a PMC member please grant my account edit rights to the Metron space
in Confluence? I'd like to start helping with the clean up efforts there.

Thanks,
Kyle

Re: Question about Metron-Docker

[Kyle Richardson]

I use it occasionally. I like it for quickly testing parser tweaks. 
Unfortunately I've had zero time lately so haven't run it off master for a 
while.

-Kyle

> On Jun 28, 2017, at 11:10 AM, Otto Fowler  wrote:
> 
> Has anyone run it off master?
> Is anyone using it?
> 
> 

Re: performance benchmarks on the asa parser

[Kyle Richardson]

I like the pre-compile idea. One concern is I see the number of grok objects 
growing over time. This parser does not account for nearly all of the possible 
ASA message types, currently only the most common ones. Is there a middle 
ground implementation where we can compile on first use of a grok and then hold 
in memory? Avoids the up front burden but should also boost performance.

-Kyle

> On Jun 8, 2017, at 8:56 PM, Simon Elliston Ball  
> wrote:
> 
> The changes are pretty simple (pre-compile the grok, duh). Most other grok 
> parser just use a single expression, which is already pre-compiled (/checks 
> assumption in code) so really it’s just the ASA one because of it’s strange 
> two stage grok. 
> 
> Shame, it would have been nice to find some more low hanging fruit.
> 
> Simon
> 
>> On 9 Jun 2017, at 01:52, Otto Fowler  wrote:
>> 
>> Are these changes that all grok parsers can benefit from?  Are your changes 
>> to the base classes that they use or asa only?
>> 
>> 
>> 
>>> On June 8, 2017 at 20:49:49, Simon Elliston Ball 
>>> (si...@simonellistonball.com ) wrote:
>>> 
>>> I got mildly interested in parser performance as a result of some recent 
>>> work on tuning, and did some very quick benchmarking with Predfix on the 
>>> ASA parser (which I hadn’t really cared about enough due to relatively low 
>>> volume previously). 
>>> 
>>> That said, it’s not exactly perf optimised. 3 runs of 1000 iterations on my 
>>> laptop as a micro-benchmark in Predfix (I know, scientific, right), with 
>>> some changes (basically pushing all the grok statements up to pre-compile 
>>> in init (the parser currently uses one grok to do the syslog bit and figure 
>>> out which grok it needs for the second half, so this makes for a large 
>>> number of Grok objects upfront, which I think we can live with.  
>>> 
>>> Do you think we should do this benchmarking properly, and extend? Anyone 
>>> have thoughts about how to build parser benchmarks in to our test suite 
>>> properly? 
>>> 
>>> Also, since these are showing approx 20 times improvement on the P95 
>>> interval, do we think it’s worth the memory (not measured, but 39 Grok 
>>> objects hanging around? If so I’ll get it JIRAed up and push my new 
>>> version. 
>>> 
>>> Run results:- 
>>> 
>>> Base line (current master as is) 
>>> |= Benchmark 
>>> ==|
>>>  
>>> | - | unit | sum | min | max | avg | stddev | conf95 | runs | 
>>> |= TimeMeter 
>>> ==| 
>>> |. AsaBenchmark 
>>> ...|
>>>  
>>> | parserBenchmark | ms | 5597.98 | 04.90 | 159.02 | 05.60 | 04.89 | 
>>> [05.01-06.20] | 1000.00 | 
>>> | parserBenchmark | ms | 5503.91 | 04.82 | 149.60 | 05.50 | 04.59 | 
>>> [05.00-05.90] | 1000.00 | 
>>> | parserBenchmark | ms | 5620.90 | 04.80 | 152.83 | 05.62 | 04.71 | 
>>> [04.98-06.73] | 1000.00 | 
>>> |==|
>>>  
>>> 
>>> Syslog element of Grok pulled out and pre-compiled 
>>> 
>>> |= Benchmark 
>>> ==|
>>>  
>>> | - | unit | sum | min | max | avg | stddev | conf95 | runs | 
>>> |= TimeMeter 
>>> ==| 
>>> |. AsaBenchmark 
>>> ...|
>>>  
>>> | parserBenchmark | ms | 4299.91 | 03.29 | 120.06 | 04.30 | 03.89 | 
>>> [03.36-07.10] | 1000.00 | 
>>> | parserBenchmark | ms | 4206.98 | 03.31 | 129.41 | 04.21 | 04.07 | 
>>> [03.46-05.44] | 1000.00 | 
>>> | parserBenchmark | ms | 3843.05 | 03.28 | 119.39 | 03.84 | 03.79 | 
>>> [03.33-04.55] | 1000.00 | 
>>> |==|
>>>  
>>> 
>>> With all precompiled in a hash map (more memory use, but not by a lot) 
>>> 
>>> |= Benchmark 
>>> =|
>>>  
>>> | - | unit | sum | min | max | avg | stddev | conf95 | runs | 
>>> |= TimeMeter 
>>> =| 
>>> |. AsaBenchmark 
>>> ..|
>>>  
>>> | parserBenchmark | ms | 514.68 | 00.22 | 112.35 | 00.51 | 03.55 | 
>>> [00.24-00.79] | 1000.00 | 
>>> | parserBenchmark | ms | 472.42 | 00.22 | 105.19 | 00.47 | 03.32 | 
>>> [00.23-00.70] | 1000.00 | 
>>> | parserBenchmark | ms | 484.40 | 00.21 | 103.71 | 00.48 | 03.27 | 
>>> [00.24-00.76] | 1000.00 | 
>>> |==|
>>>  
>>> 
>>> Simon
> 

Re: Infrastructure migrated

[Kyle Richardson]

Thanks, Casey!

-Kyle

On Mon, May 15, 2017 at 10:02 AM, Otto Fowler 
wrote:

> Thanks Casey.
>
>
> On May 15, 2017 at 09:51:21, Casey Stella (ceste...@gmail.com) wrote:
>
> It appears that committers who are not PMC members did not have their
> committer status migrated over to the new repo. I have filed an INFRA
> ticket to correct this: https://issues.apache.org/jira/browse/INFRA-14159
>
> I have called out certain people that I remembered. If you are not on the
> list in the JIRA, please let me know and I'll add you.
>
> On Mon, May 15, 2017 at 9:30 AM, Casey Stella  wrote:
>
> > For those who haven't noticed, the git infrastructure has migrated over.
> > If you are a committer, you will need to adjust the URLs for the apache
> > remote repo via:
> >
> > git remote set-url apache https://git-wip-us.apache.org/
> > repos/asf/metron.git
> >
> > Obviously substitute "apache" for "upstream" if you choose that
> > nomenclature, but the general change is the URL of the apache repo
> > changed. I just tried and commits worked. If they do not work for you
> > upon the next commit, let me know and I'll look into it.
> >
> > Best,
> >
> > Casey
> >
>

Re: [DISCUSS] Code Style

[Kyle Richardson]

+1 Thanks, Justin. I'm all for adopting the Google style guide.

-Kyle

On Mon, May 8, 2017 at 10:24 AM, Justin Leet  wrote:

> Sigh, the message was hidden below the fold.
>
> Part of what I've been playing around with is doing this.  It's pretty easy
> to set up both autoformatting and warnings in idea.  There's a checkstyle
> plugin, and you can just import the appropriate file on each.  As a fair
> warning, until we do the blanket reformatting there will be a lot of
> warning highlighting in a lot of the Java files.
>
> I'll be documenting that as part of the ticket (regardless of checkstyle
> configuration specifics), and I'll also update the wiki (or point to links
> for it, I believe Checkstyle's docs walk through how to do it) as part of
> the exercise.
>
> Sorry about spamming the list.
>
> On Mon, May 8, 2017 at 10:21 AM, Justin Leet 
> wrote:
>
> > Accidentally sent this to only Otto, instead of the whole group.
> >
> > Part of what I've been playing around with is doing this.  It's pretty
> > easy to set up both autoformatting and warnings in idea.  There's a
> > checkstyle plugin, and you can just import the appropriate file on each.
> > As a fair warning, until we do the blanket reformatting there will be a
> lot
> > of warning highlighting in a lot of the Java files.
> >
> > I'll be documenting that as part of the ticket (regardless of checkstyle
> > configuration specifics), and I'll also update the wiki (or point to
> links
> > for it, I believe Checkstyle's docs walk through how to do it) as part of
> > the exercise.
> >
> > On Mon, May 8, 2017 at 9:40 AM, Otto Fowler 
> > wrote:
> >
> >> +1.
> >> Does anyone have an idea setup for this?
> >>
> >>
> >> On May 8, 2017 at 09:29:15, Justin Leet (justinjl...@gmail.com) wrote:
> >>
> >> I've been taking a look at setting up checkstyle per
> >> https://issues.apache.org/jira/browse/METRON-746.
> >>
> >> Given that we don't actually enforce any style right now (saying we use
> >> Sun's is not the same as enforcing it!), and plan to do a blanket format
> >> after we get checkstyle setup, this seems like a good opportunity to
> >> discuss what we want to actually implement.
> >>
> >> In particular, I'm proposing moving to Google's code style guide and
> away
> >> from Sun's.
> >>
> >> - Google's style guide already address the (admittedly) minor issues we
> >> have with the Sun Style, in particular two spaces and line length (100
> in
> >> Google's). I'd prefer to just use something built-in as much as
> possible,
> >> and it seems like Google's style is closer to what we're looking for out
> >> of
> >> the box.
> >> - Not only is it built in, the explanation for each rule (and where
> >> checkstyle falls short) actually exists.
> >> - http://checkstyle.sourceforge.net/google_style.html
> >> - http://checkstyle.sourceforge.net/sun_style.html
> >> - Storm uses it, so it makes our code's style more compatible with
> >> one of our core dependencies (and one we've had to dig into repeatedly
> >> during upgrades and such).
> >> - https://github.com/apache/storm/blob/master/pom.xml#L1100
> >> - I personally find Google's guide easier to read and digest. I'm
> >> curious how other people feel.
> >>
> >> For ease of comparison:
> >> Sun's guide: http://www.oracle.com/technetwork/java/codeconvtoc-136057.
> >> html
> >> Google's guide: https://google.github.io/styleguide/javaguide.html
> >>
> >>
> >
>

Re: So we graduated...

[Kyle Richardson]

That's awesome! Congratulations everyone. Looking forward to the official 
announcement on Monday.

-Kyle

> On Apr 20, 2017, at 5:15 PM, David Lyle  wrote:
> 
> Outstanding! Great work everyone. Building a TLP worthy community is
> difficult and worthy work, congratulations all!
> 
> -D...
> 
>> On Thu, Apr 20, 2017 at 5:12 PM, Casey Stella  wrote:
>> 
>> For anyone paying attention to incubator-general, it will come as no
>> surprise that we graduated as of last night's board meeting.  We have a
>> press released queued up and planned for monday along with a PR (METRON-687
>> at https://github.com/apache/incubator-metron/pull/539).
>> 
>> It escaped my notice that the graduation was talked about on
>> incubator-general; otherwise I'd have sent this email earlier and been less
>> cagey in 687's description.  Even so, I'd like to ask that everyone keep it
>> to themselves until monday morning after the press release gets out the
>> door.  I know the cat is out of the bag, but it'd be nice to have a bit of
>> an embargo.
>> 
>> Thanks!
>> 
>> Casey
>>