+1 to separating developer docs and user docs. How should we approach that. Have a separate doc book? I haven’t had a ton of time to contribute to code lately but I’d be happy to help write some of these.
On Sat, Aug 18, 2018 at 9:48 AM Nick Allen <n...@nickallen.org> wrote: > Personally, I think the state of our docs and web presence is an inhibitor > to growing the Metron community. Unless we can offer concise, compelling > answers to the basic questions (What can I do with Metron? Who does it > help? How do I do that?), potential users and contributors are unable to > see the value of Metron. > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen <n...@nickallen.org> wrote: > > > I'd like to see us focus on improving our docs before a version 1.0. > > Right now we just stitch together a bunch of READMEs, which is a great > > stride from where we started, but is not ideal. > > > > Our docs should focused on the user and use cases; What can I do with > > Metron? Who does it help? How do I do that? > > > > The docs should be separate from the code base to allow for an > > organization that is focused on the user rather than the implementation. > > This allows the READMEs to focus on the developer and the implementation, > > which should make them more digestible too. The docs should be version > > controlled and maintained through PRs, just like the code. We should > take > > just as much pride in our docs as we do in our code. > > > > > > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball < > > si...@simonellistonball.com> wrote: > > > >> Agreed, should we add TDE by default, and get the ranger policies on by > >> default? That leaves secured in Kafka, which would have to be built into > >> the consumers and producers to encrypt into the on disk Kafka topics. > Does > >> that seem necessary to people? It would have performance implications > for > >> sure. > >> > >> Simon > >> > >> > On 15 Aug 2018, at 21:26, Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> > > >> > Well, I look at it like this. > >> > > >> > The Secure Vault was part of the original metron pitch, and many may > >> have used that as part of their evaluations. > >> > “Look, it is going to have a security vault type thing, it is on the > >> roadmap”. > >> > > >> > Regardless of the implementation, conceptually, security of data at > >> rest is important, and is a major outstanding item or the core metron > >> proposition. > >> > > >> > > >> > > >> > > >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball ( > >> si...@simonellistonball.com) wrote: > >> >> > >> >> That’s going back a way. I always saw that concept as begin about the > >> formats, e.g. Orc, and meta data around it plus the data service api to > get > >> at it. I’m all for that too, but think it needs more thought than the > >> ticket captures. > >> >> > >> >> Simon > >> >> > >> >> On 15 Aug 2018, at 20:53, Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> >> > >> >>> https://issues.apache.org/jira/browse/METRON-343 > >> >>> > >> >>>> On August 15, 2018 at 15:47:24, Simon Elliston Ball ( > >> si...@simonellistonball.com) wrote: > >> >>>> > >> >>>> What would you see as secure? I’ve seen people use TDE for the HDFS > >> store, but it’s harder to encrypt storage with solr / es. Something I > was > >> thinking of doing to follow up on the Knox Feature was to add Ranger > >> integration for securing and auditing configs, and potentially > extending to > >> the index destinations. Do you think that would cover the secure storage > >> concept? > >> >>>> > >> >>>> Simon > >> >>>> > >> >>>> > On 15 Aug 2018, at 20:39, Otto Fowler <ottobackwa...@gmail.com> > >> wrote: > >> >>>> > > >> >>>> > Secure storage off the top of my head > >> >>>> > > >> >>>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com ( > zeo...@gmail.com) > >> wrote: > >> >>>> > > >> >>>> > So, as has been discussed in a few > >> >>>> > < > >> >>>> > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a > >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E> > >> >>>> > > >> >>>> > other > >> >>>> > < > >> >>>> > https://lists.apache.org/thread.html/427a20c22207e84331b94e8 > >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E> > >> >>>> > > >> >>>> > recent dev list threads, I would like to discuss what a Metron > 1.0 > >> release > >> >>>> > looks like. > >> >>>> > > >> >>>> > In order to kick off the conversation, I would like to make a few > >> >>>> > suggestions regarding "what 1.0 means to me," but I'm very > >> interested to > >> >>>> > hear everybody else's opinions. > >> >>>> > > >> >>>> > In order to go 1.0 I believe we should have: > >> >>>> > 1. A clear, supported method of upgrading from one version of > >> Metron to the > >> >>>> > next. We have attempted > >> >>>> > <https://github.com/apache/metron/blob/master/Upgrading.md> to > >> make this > >> >>>> > easier in the past, but it is currently not > >> >>>> > < > >> >>>> > https://github.com/apache/metron/tree/master/metron-deployme > >> nt/packaging/ambari/metron-mpack#limitations> > >> >>>> > > >> >>>> > supported > >> >>>> > < > >> >>>> > https://github.com/apache/metron/tree/master/metron-deployme > >> nt/packaging/ambari/elasticsearch-mpack#limitations> > >> >>>> > > >> >>>> > . > >> >>>> > 2. Authentication for all of the UIs and APIs should be secure > and > >> support > >> >>>> > SSO. I believe this is in progress via METRON-1663 > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1663>. > >> >>>> > 3. Each of our personas > >> >>>> > < > >> >>>> > https://cwiki.apache.org/confluence/display/METRON/Metron+ > >> User+Personas+And+Benefits> > >> >>>> > > >> >>>> > should > >> >>>> > be well documented, understood, and supported. > >> >>>> > - The current state of documentation is, in my opinion, > inadequate > >> and I > >> >>>> > admit I am partially to blame for this. I suggest we define a > >> strict > >> >>>> > approach for documentation, align to it (such as perhaps > migrating > >> all > >> >>>> > useful wiki documentation to git), and enforce it. > >> >>>> > - I would consider METRON-1699 > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1699> as a > critical > >> item for > >> >>>> > a Security Data Scientist, but it is currently not clear to me > >> where the > >> >>>> > line exists between some of the other personas, or that each > >> persona has > >> >>>> > been sufficiently implemented. > >> >>>> > 4. A performance tuning guide should be available for all of the > >> main > >> >>>> > components, whether as an independent document or as a part of a > >> larger > >> >>>> > document. > >> >>>> > 5. Simple data ingest. > >> >>>> > - Similar to the ongoing conversation for NiFi integration > >> >>>> > < > >> >>>> > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f269 > >> 73f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>, > >> >>>> > > >> >>>> > we should be able to say that we have broken down the barriers to > >> getting > >> >>>> > data into a Metron cluster in easy and efficient ways. In > addition > >> to > >> >>>> > NiFi, having support for other popular tools such as beats > >> >>>> > <https://www.elastic.co/products/beats>, fluentd < > >> https://www.fluentd.org/>, > >> >>>> > > >> >>>> > etc. > >> >>>> > - Parsers should be pluggable, with independent tests and the > >> ability to > >> >>>> > make versioned modifications with roll-backs. > >> >>>> > > >> >>>> > What else? Are any of these items not necessary for a 1.0? > >> >>>> > > >> >>>> > Jon > >> >>>> > -- > >> >>>> > > >> >>>> > Jon > >> > > > > >
