Re: [DISCUSS] Getting to a 1.0 release

2018-09-07 Thread Nick Allen
We should also publish binaries for each release. That might include any of the following... - Publish the jars to a Maven repository - Publish the RPMs and DEBs to a repository - Host a pre-built "Full Dev" in Vagrant Cloud On Wed, Aug 15, 2018 at 2:49 PM zeo...@gmail.com wrote: >

Re: [DISCUSS] Getting to a 1.0 release

2018-08-27 Thread Otto Fowler
There is a jira to make indexing extensible, such that you could define the indexing topology much as you define the parsing, and can load different indexers. On August 27, 2018 at 02:44:24, Ali Nazemian (alinazem...@gmail.com) wrote: One thing that we could imagine for v1.0 might be an ability

Re: [DISCUSS] Getting to a 1.0 release

2018-08-27 Thread zeo...@gmail.com
I agree with that wholeheartedly. The more integrations in and out of the stack at various points, the better. Jon On Mon, Aug 27, 2018, 02:44 Ali Nazemian wrote: > One thing that we could imagine for v1.0 might be an ability to extend > Metron from adding more pipelines to it. For example,

Re: [DISCUSS] Getting to a 1.0 release

2018-08-27 Thread Ali Nazemian
One thing that we could imagine for v1.0 might be an ability to extend Metron from adding more pipelines to it. For example, being able to extend Metron to be integrated with other endpoints more easily from Storm perspective. For example, what if we would like to create other topologies to write

Re: [DISCUSS] Getting to a 1.0 release

2018-08-20 Thread Casey Stella
I completely agree, Mike. Our docs are either very high level or very low level (and possibly stale) and, worse, aren't aimed at the actors that you've stated. I think that the HBase project does a good job of providing coherent and useable documentation in their "HBase Book" (see

Re: [DISCUSS] Getting to a 1.0 release

2018-08-18 Thread Michael Miklavcic
Apologies for any spelling mishaps as I'm writing from my phone. I'm for improving our docs. I'd like to see us guide our various profiles of user towards the specific documentation for the abstraction levels they'll be most interested in working from. I think we should have platform docs about

Re: [DISCUSS] Getting to a 1.0 release

2018-08-18 Thread Nick Allen
Yes, I imagine just a separate top level directory which would contain the docs. We would need someone to survey what doc tools are out there and provide some advice. Maybe we could look around at other open source projects that have done their docs well and emulate them. On Sat, Aug 18, 2018,

Re: [DISCUSS] Getting to a 1.0 release

2018-08-18 Thread Kyle Richardson
+1 to separating developer docs and user docs. How should we approach that. Have a separate doc book? I haven’t had a ton of time to contribute to code lately but I’d be happy to help write some of these. On Sat, Aug 18, 2018 at 9:48 AM Nick Allen wrote: > Personally, I think the state of our

Re: [DISCUSS] Getting to a 1.0 release

2018-08-18 Thread Nick Allen
Personally, I think the state of our docs and web presence is an inhibitor to growing the Metron community. Unless we can offer concise, compelling answers to the basic questions (What can I do with Metron? Who does it help? How do I do that?), potential users and contributors are unable to see

Re: [DISCUSS] Getting to a 1.0 release

2018-08-18 Thread Nick Allen
I'd like to see us focus on improving our docs before a version 1.0. Right now we just stitch together a bunch of READMEs, which is a great stride from where we started, but is not ideal. Our docs should focused on the user and use cases; What can I do with Metron? Who does it help? How do I do

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Simon Elliston Ball
Agreed, should we add TDE by default, and get the ranger policies on by default? That leaves secured in Kafka, which would have to be built into the consumers and producers to encrypt into the on disk Kafka topics. Does that seem necessary to people? It would have performance implications for

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Otto Fowler
Well, I look at it like this. The Secure Vault was part of the original metron pitch, and many may have used that as part of their evaluations. “Look, it is going to have a security vault type thing, it is on the roadmap”. Regardless of the implementation, conceptually, security of data at rest

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Simon Elliston Ball
+1 to that. That’s more the TDE bit, we would also need Kafka SSL, and the Knox stuff (METRON-1663 adds SSL to all the UI and rest api stuff) Simon > On 15 Aug 2018, at 21:03, Otto Fowler wrote: > > https://issues.apache.org/jira/browse/METRON-106 > At least making sure it is met and closing

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Simon Elliston Ball
That’s going back a way. I always saw that concept as begin about the formats, e.g. Orc, and meta data around it plus the data service api to get at it. I’m all for that too, but think it needs more thought than the ticket captures. Simon > On 15 Aug 2018, at 20:53, Otto Fowler wrote: > >

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Otto Fowler
https://issues.apache.org/jira/browse/METRON-106 At least making sure it is met and closing it On August 15, 2018 at 15:53:02, Otto Fowler (ottobackwa...@gmail.com) wrote: https://issues.apache.org/jira/browse/METRON-343 On August 15, 2018 at 15:47:24, Simon Elliston Ball (

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Otto Fowler
https://issues.apache.org/jira/browse/METRON-343 On August 15, 2018 at 15:47:24, Simon Elliston Ball ( si...@simonellistonball.com) wrote: What would you see as secure? I’ve seen people use TDE for the HDFS store, but it’s harder to encrypt storage with solr / es. Something I was thinking of

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Simon Elliston Ball
What would you see as secure? I’ve seen people use TDE for the HDFS store, but it’s harder to encrypt storage with solr / es. Something I was thinking of doing to follow up on the Knox Feature was to add Ranger integration for securing and auditing configs, and potentially extending to the

Re: [DISCUSS] Getting to a 1.0 release

2018-08-15 Thread Otto Fowler
Secure storage off the top of my head On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote: So, as has been discussed in a few < https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E> other <

[DISCUSS] Getting to a 1.0 release

2018-08-15 Thread zeo...@gmail.com
So, as has been discussed in a few other recent