Personally, I think the state of our docs and web presence is an inhibitor to growing the Metron community. Unless we can offer concise, compelling answers to the basic questions (What can I do with Metron? Who does it help? How do I do that?), potential users and contributors are unable to see the value of Metron.
On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen <n...@nickallen.org> wrote: > I'd like to see us focus on improving our docs before a version 1.0. > Right now we just stitch together a bunch of READMEs, which is a great > stride from where we started, but is not ideal. > > Our docs should focused on the user and use cases; What can I do with > Metron? Who does it help? How do I do that? > > The docs should be separate from the code base to allow for an > organization that is focused on the user rather than the implementation. > This allows the READMEs to focus on the developer and the implementation, > which should make them more digestible too. The docs should be version > controlled and maintained through PRs, just like the code. We should take > just as much pride in our docs as we do in our code. > > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> Agreed, should we add TDE by default, and get the ranger policies on by >> default? That leaves secured in Kafka, which would have to be built into >> the consumers and producers to encrypt into the on disk Kafka topics. Does >> that seem necessary to people? It would have performance implications for >> sure. >> >> Simon >> >> > On 15 Aug 2018, at 21:26, Otto Fowler <ottobackwa...@gmail.com> wrote: >> > >> > Well, I look at it like this. >> > >> > The Secure Vault was part of the original metron pitch, and many may >> have used that as part of their evaluations. >> > “Look, it is going to have a security vault type thing, it is on the >> roadmap”. >> > >> > Regardless of the implementation, conceptually, security of data at >> rest is important, and is a major outstanding item or the core metron >> proposition. >> > >> > >> > >> > >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball ( >> si...@simonellistonball.com) wrote: >> >> >> >> That’s going back a way. I always saw that concept as begin about the >> formats, e.g. Orc, and meta data around it plus the data service api to get >> at it. I’m all for that too, but think it needs more thought than the >> ticket captures. >> >> >> >> Simon >> >> >> >> On 15 Aug 2018, at 20:53, Otto Fowler <ottobackwa...@gmail.com> wrote: >> >> >> >>> https://issues.apache.org/jira/browse/METRON-343 >> >>> >> >>>> On August 15, 2018 at 15:47:24, Simon Elliston Ball ( >> si...@simonellistonball.com) wrote: >> >>>> >> >>>> What would you see as secure? I’ve seen people use TDE for the HDFS >> store, but it’s harder to encrypt storage with solr / es. Something I was >> thinking of doing to follow up on the Knox Feature was to add Ranger >> integration for securing and auditing configs, and potentially extending to >> the index destinations. Do you think that would cover the secure storage >> concept? >> >>>> >> >>>> Simon >> >>>> >> >>>> > On 15 Aug 2018, at 20:39, Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>>> > >> >>>> > Secure storage off the top of my head >> >>>> > >> >>>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) >> wrote: >> >>>> > >> >>>> > So, as has been discussed in a few >> >>>> > < >> >>>> > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E> >> >>>> > >> >>>> > other >> >>>> > < >> >>>> > https://lists.apache.org/thread.html/427a20c22207e84331b94e8 >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E> >> >>>> > >> >>>> > recent dev list threads, I would like to discuss what a Metron 1.0 >> release >> >>>> > looks like. >> >>>> > >> >>>> > In order to kick off the conversation, I would like to make a few >> >>>> > suggestions regarding "what 1.0 means to me," but I'm very >> interested to >> >>>> > hear everybody else's opinions. >> >>>> > >> >>>> > In order to go 1.0 I believe we should have: >> >>>> > 1. A clear, supported method of upgrading from one version of >> Metron to the >> >>>> > next. We have attempted >> >>>> > <https://github.com/apache/metron/blob/master/Upgrading.md> to >> make this >> >>>> > easier in the past, but it is currently not >> >>>> > < >> >>>> > https://github.com/apache/metron/tree/master/metron-deployme >> nt/packaging/ambari/metron-mpack#limitations> >> >>>> > >> >>>> > supported >> >>>> > < >> >>>> > https://github.com/apache/metron/tree/master/metron-deployme >> nt/packaging/ambari/elasticsearch-mpack#limitations> >> >>>> > >> >>>> > . >> >>>> > 2. Authentication for all of the UIs and APIs should be secure and >> support >> >>>> > SSO. I believe this is in progress via METRON-1663 >> >>>> > <https://issues.apache.org/jira/browse/METRON-1663>. >> >>>> > 3. Each of our personas >> >>>> > < >> >>>> > https://cwiki.apache.org/confluence/display/METRON/Metron+ >> User+Personas+And+Benefits> >> >>>> > >> >>>> > should >> >>>> > be well documented, understood, and supported. >> >>>> > - The current state of documentation is, in my opinion, inadequate >> and I >> >>>> > admit I am partially to blame for this. I suggest we define a >> strict >> >>>> > approach for documentation, align to it (such as perhaps migrating >> all >> >>>> > useful wiki documentation to git), and enforce it. >> >>>> > - I would consider METRON-1699 >> >>>> > <https://issues.apache.org/jira/browse/METRON-1699> as a critical >> item for >> >>>> > a Security Data Scientist, but it is currently not clear to me >> where the >> >>>> > line exists between some of the other personas, or that each >> persona has >> >>>> > been sufficiently implemented. >> >>>> > 4. A performance tuning guide should be available for all of the >> main >> >>>> > components, whether as an independent document or as a part of a >> larger >> >>>> > document. >> >>>> > 5. Simple data ingest. >> >>>> > - Similar to the ongoing conversation for NiFi integration >> >>>> > < >> >>>> > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f269 >> 73f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>, >> >>>> > >> >>>> > we should be able to say that we have broken down the barriers to >> getting >> >>>> > data into a Metron cluster in easy and efficient ways. In addition >> to >> >>>> > NiFi, having support for other popular tools such as beats >> >>>> > <https://www.elastic.co/products/beats>, fluentd < >> https://www.fluentd.org/>, >> >>>> > >> >>>> > etc. >> >>>> > - Parsers should be pluggable, with independent tests and the >> ability to >> >>>> > make versioned modifications with roll-backs. >> >>>> > >> >>>> > What else? Are any of these items not necessary for a 1.0? >> >>>> > >> >>>> > Jon >> >>>> > -- >> >>>> > >> >>>> > Jon >> > >
