Re: Change field separator in Metron to make it Hive and ORC friendly

2018-08-15 Thread Ali Nazemian
Hi Simon, I think it is a hard trade-off. Even right now without any ability to customise separator/Metron internal field names, Metron users need to put a mapping in place at the integration layer (At least this is what we are doing :) ). Every organisation/user may need to follow different

Re: [DISCUSS] Pcap query branch completion

2018-08-15 Thread Ryan Merriman
Otto, I believe the items you requested are in the feature branch now. Is there anything outstanding that we missed? The Jiras for the Pcap feature branch should be up to date: https://issues.apache.org/jira/browse/METRON-1554 On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman wrote: > - Date

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Simon Elliston Ball
Agreed, should we add TDE by default, and get the ranger policies on by default? That leaves secured in Kafka, which would have to be built into the consumers and producers to encrypt into the on disk Kafka topics. Does that seem necessary to people? It would have performance implications for

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Otto Fowler
Well, I look at it like this. The Secure Vault was part of the original metron pitch, and many may have used that as part of their evaluations. “Look, it is going to have a security vault type thing, it is on the roadmap”. Regardless of the implementation, conceptually, security of data at rest

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Simon Elliston Ball
+1 to that. That’s more the TDE bit, we would also need Kafka SSL, and the Knox stuff (METRON-1663 adds SSL to all the UI and rest api stuff) Simon > On 15 Aug 2018, at 21:03, Otto Fowler wrote: > > https://issues.apache.org/jira/browse/METRON-106 > At least making sure it is met and closing

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Simon Elliston Ball
That’s going back a way. I always saw that concept as begin about the formats, e.g. Orc, and meta data around it plus the data service api to get at it. I’m all for that too, but think it needs more thought than the ticket captures. Simon > On 15 Aug 2018, at 20:53, Otto Fowler wrote: > >

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Otto Fowler
https://issues.apache.org/jira/browse/METRON-106 At least making sure it is met and closing it On August 15, 2018 at 15:53:02, Otto Fowler (ottobackwa...@gmail.com) wrote: https://issues.apache.org/jira/browse/METRON-343 On August 15, 2018 at 15:47:24, Simon Elliston Ball (

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Otto Fowler
https://issues.apache.org/jira/browse/METRON-343 On August 15, 2018 at 15:47:24, Simon Elliston Ball ( si...@simonellistonball.com) wrote: What would you see as secure? I’ve seen people use TDE for the HDFS store, but it’s harder to encrypt storage with solr / es. Something I was thinking of

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Simon Elliston Ball
What would you see as secure? I’ve seen people use TDE for the HDFS store, but it’s harder to encrypt storage with solr / es. Something I was thinking of doing to follow up on the Knox Feature was to add Ranger integration for securing and auditing configs, and potentially extending to the

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Otto Fowler
Secure storage off the top of my head On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote: So, as has been discussed in a few < https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E> other <

Re: [DISCUSS] Release cadence

2018-08-15 Thread Michael Miklavcic
Works for me, that would be great. On Wed, Aug 15, 2018 at 12:22 PM Casey Stella wrote: > If you like, I can volunteer to kick off a discuss thread when I submit the > board report. > > On Wed, Aug 15, 2018 at 2:21 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > I'm also a

[DISCUSS] Getting to a 1.0 release

2018-08-15 Thread zeo...@gmail.com
So, as has been discussed in a few other recent

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Otto Fowler
Done On August 15, 2018 at 14:22:45, Vets, Laurens (laur...@daemon.be) wrote: Could I be invited? On 15-Aug-18 09:48, Michael Miklavcic wrote: > + Metron user list > > On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Turns out we are able to

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Vets, Laurens
Could I be invited? On 15-Aug-18 09:48, Michael Miklavcic wrote: > + Metron user list > > On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Turns out we are able to invite folks on an ad-hoc basis. See instructions >> here - >>

Re: [DISCUSS] Release cadence

2018-08-15 Thread Casey Stella
If you like, I can volunteer to kick off a discuss thread when I submit the board report. On Wed, Aug 15, 2018 at 2:21 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I'm also a fan of the 2-3 month time frame for releases. And I agree it > fits nicely with our board report. That

Re: [DISCUSS] Release cadence

2018-08-15 Thread Michael Miklavcic
I'm also a fan of the 2-3 month time frame for releases. And I agree it fits nicely with our board report. That said, I think we should minimally kick off a DISCUSS at least every 2 months per the recommendations above. If it's warranted, great. If not, then we bring it up at a stated later time

Re: [DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread Michael Miklavcic
+1 here as well to the proposed releases. On Wed, Aug 15, 2018 at 11:06 AM Casey Stella wrote: > +1 to both releases, this is plenty for an 0.6.0 and a 0.2.0 > > On Wed, Aug 15, 2018 at 11:04 AM Justin Leet > wrote: > > > I just sent a thread about release cadence. Jon, I'd recommend starting

Re: [DISCUSS] Release cadence

2018-08-15 Thread Casey Stella
Strictly selfishly, I'd love for a release to happen quickly enough to have something to announce to the board during the reports. Once every 2 months or when a sufficiently complicated change happens sounds like a sensible cadence. I very much support a "how do we get to 1.0" discussion, maybe

Re: [DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread Casey Stella
+1 to both releases, this is plenty for an 0.6.0 and a 0.2.0 On Wed, Aug 15, 2018 at 11:04 AM Justin Leet wrote: > I just sent a thread about release cadence. Jon, I'd recommend starting a > thread on a 1.0 roadmap. I thought about merging the threads, but I think > that's just going to result

Re: Slack Channel

2018-08-15 Thread Michael Miklavcic
Invite sent On Wed, Aug 15, 2018 at 10:57 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > Hello dev team, may I please join your slack channel :) >

Re: Slack Channel

2018-08-15 Thread Casey Stella
Sorry Simon, I retract the comment! I didn't realize it was possible, but it is possible to invite. On Wed, Aug 15, 2018 at 1:01 PM Casey Stella wrote: > Sadly, it's the ASF slack and I believe it requires an @apache.org email > address. > > On Wed, Aug 15, 2018 at 12:57 PM Simon Elliston Ball

Re: Slack Channel

2018-08-15 Thread Casey Stella
Sadly, it's the ASF slack and I believe it requires an @apache.org email address. On Wed, Aug 15, 2018 at 12:57 PM Simon Elliston Ball < si...@simonellistonball.com> wrote: > Hello dev team, may I please join your slack channel :) >

Slack Channel

2018-08-15 Thread Simon Elliston Ball
Hello dev team, may I please join your slack channel :)

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Michael Miklavcic
+ Metron user list On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Turns out we are able to invite folks on an ad-hoc basis. See instructions > here - > https://cwiki.apache.org/confluence/display/METRON/Community+Resources > > > On Wed, Aug 15, 2018 at

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Michael Miklavcic
Turns out we are able to invite folks on an ad-hoc basis. See instructions here - https://cwiki.apache.org/confluence/display/METRON/Community+Resources On Wed, Aug 15, 2018 at 9:23 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > It's another option with different features. I

Re: [DISCUSS] Release cadence

2018-08-15 Thread zeo...@gmail.com
I'm a fan of a hybrid time/feature-based cadence. Something like "When 3 months has passed since our last release, or a sufficiently complicated change has been introduced to master (like merging a FB), a discuss thread is started". I'm primarily thinking of what the upgrade path looks like

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Michael Miklavcic
It's another option with different features. I imagine many people will use both. On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > Since this is committers only, would it make more sense to stick to IRC? Or > is exclusivity the idea? > > On 15 August 2018

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Simon Elliston Ball
Since this is committers only, would it make more sense to stick to IRC? Or is exclusivity the idea? On 15 August 2018 at 16:09, Nick Allen wrote: > Thanks for the instructions! > > On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > The Metron

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Nick Allen
Thanks for the instructions! On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > The Metron community has a Slack channel available for communication > (similar to the existing IRC channel, only on Slack). > > To join: > >1. Go to slack.com. >2.

Re: [DISCUSS] Metron Parsers in Nifi

2018-08-15 Thread Otto Fowler
On August 15, 2018 at 09:30:47, Justin Leet (justinjl...@gmail.com) wrote: As an exercise, let me summarize the points of contention I've seen and lay out the tradeoffs as I see them. That way we can prioritize what's important to us in a NiFi implementation and better work towards a favorable

Re: [DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread Justin Leet
I just sent a thread about release cadence. Jon, I'd recommend starting a thread on a 1.0 roadmap. I thought about merging the threads, but I think that's just going to result in more crosstalk, so I'll let you start that conversation. On Wed, Aug 15, 2018 at 10:37 AM Nick Allen wrote: > +1 to

[DISCUSS] Release cadence

2018-08-15 Thread Justin Leet
Hi all, In concert with the discuss thread on a potential 0.6.0 release, I'd also like start a discussion about our release cadence. We've generally been pretty relaxed around doing releases, and I'm curious what people's thoughts are on adopting a somewhat more regular schedule. Couple

Re: [DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread Nick Allen
+1 to a 0.6.0 release that includes the Pcap Panel and Solr work. +1 to doing a 0.2.0 release for metron-bro-plugin-kafka. I *think* we need to do the plugin release first, so that the 0.6.0 Metron release will point to plugin 0.2.0. FWIW, here are the changes since the last release. 6 days

[ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Michael Miklavcic
The Metron community has a Slack channel available for communication (similar to the existing IRC channel, only on Slack). To join: 1. Go to slack.com. 2. For organization/group, you'll enter "the-asf" 3. Use your Apache email for your login 4. Click "Channels" and look for #metron

Re: [DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread Justin Leet
I was actually going to kick out another thread in a little bit around our release schedule, it should be out shortly. Good point on the metron-bro-plugin-kafka. I'm in favor of putting out a 0.2.0 release of it simultaneously. On Wed, Aug 15, 2018 at 9:48 AM zeo...@gmail.com wrote: > I agree

Re: [DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread zeo...@gmail.com
I agree - I would love to see a release not long after the PCAP FB gets into master, and 0.6.0 makes sense to me. I'd also like to see a 0.2 release of metron-bro-plugin-kafka. There is one new commit, and I have a PR open which is waiting on some tests before it's ready to be evaluated/merged.

Re: [DISCUSS] Metron Parsers in Nifi

2018-08-15 Thread Justin Leet
As an exercise, let me summarize the points of contention I've seen and lay out the tradeoffs as I see them. That way we can prioritize what's important to us in a NiFi implementation and better work towards a favorable solution (basically, I want to requirements we have for an MVP). My

[DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread Justin Leet
Hi all, It's been a little while since the last release, and a couple major items have gone in since then (or are hopefully close to going in!). In particular, I'd personally like to see a release with our Solr work and the close-to-completion