I completely agree, Mike. Our docs are either very high level or very low level (and possibly stale) and, worse, aren't aimed at the actors that you've stated. I think that the HBase project does a good job of providing coherent and useable documentation in their "HBase Book" (see https://hbase.apache.org/book.html). It's not actor-specific, but it is coherent advice for the practical practitioner of HBase (both admin and developer) and speaks with one voice. I think Metron's need is a bit different, but at the minimum some coherent docs that speaks with one voice and has a coherent pitch about what Metron is used for and what it isn't used for is well needed.
On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Apologies for any spelling mishaps as I'm writing from my phone. > > I'm for improving our docs. I'd like to see us guide our various profiles > of user towards the specific documentation for the abstraction levels > they'll be most interested in working from. I think we should have platform > docs about how we're a broadly useful, extensible streaming analytics > platform for cyber security as well as docs that emphasize more narrow and > specific use cases. > > Personally, I think I see 3 potential tiers or classifications of docs. > These are just observations and ideas I had, not necessarily a prescription > for organizing docs: > - Low level tool instructions, eg > - how do I run the pcap toplogy and then query with the CLI and UI? > - Platform docs about building on top of Metron, e.g. > - writing custom parsers, enrichment, and threat Intel (imho we should > start to take a more opinionated view of leveraging Stellar as this > extension point rather than implementing new parser classes in Java) > - using the profiler for constructing outlier analysis use cases > - using MAAS for building and deploying models for use in enrichment > - Docs around more specific use cases that solve specific as opposed to > more general problems, similar to those we have in the use-cases folder. > > I think one of our challenges currently is that our docs could be better > tailored to the "actors" we've talked about in the past. An individual SOC > analyst will have a very different set of interests than would a reseller > that wants to build on top of our platform to expose new modules and > functionality to those SOC analyst. And we can, and do, currently support > both. > > > On Sat, Aug 18, 2018, 9:34 AM Nick Allen <n...@nickallen.org> wrote: > > > Yes, I imagine just a separate top level directory which would contain > the > > docs. > > > > We would need someone to survey what doc tools are out there and provide > > some advice. > > > > Maybe we could look around at other open source projects that have done > > their docs well and emulate them. > > > > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson < > kylerichards...@gmail.com> > > wrote: > > > > > +1 to separating developer docs and user docs. How should we approach > > that. > > > Have a separate doc book? I haven’t had a ton of time to contribute to > > code > > > lately but I’d be happy to help write some of these. > > > > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen <n...@nickallen.org> wrote: > > > > > > > Personally, I think the state of our docs and web presence is an > > > inhibitor > > > > to growing the Metron community. Unless we can offer concise, > > compelling > > > > answers to the basic questions (What can I do with Metron? Who does > it > > > > help? How do I do that?), potential users and contributors are unable > > to > > > > see the value of Metron. > > > > > > > > > > > > > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen <n...@nickallen.org> > > wrote: > > > > > > > > > I'd like to see us focus on improving our docs before a version > 1.0. > > > > > Right now we just stitch together a bunch of READMEs, which is a > > great > > > > > stride from where we started, but is not ideal. > > > > > > > > > > Our docs should focused on the user and use cases; What can I do > with > > > > > Metron? Who does it help? How do I do that? > > > > > > > > > > The docs should be separate from the code base to allow for an > > > > > organization that is focused on the user rather than the > > > implementation. > > > > > This allows the READMEs to focus on the developer and the > > > implementation, > > > > > which should make them more digestible too. The docs should be > > version > > > > > controlled and maintained through PRs, just like the code. We > should > > > > take > > > > > just as much pride in our docs as we do in our code. > > > > > > > > > > > > > > > > > > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball < > > > > > si...@simonellistonball.com> wrote: > > > > > > > > > >> Agreed, should we add TDE by default, and get the ranger policies > on > > > by > > > > >> default? That leaves secured in Kafka, which would have to be > built > > > into > > > > >> the consumers and producers to encrypt into the on disk Kafka > > topics. > > > > Does > > > > >> that seem necessary to people? It would have performance > > implications > > > > for > > > > >> sure. > > > > >> > > > > >> Simon > > > > >> > > > > >> > On 15 Aug 2018, at 21:26, Otto Fowler <ottobackwa...@gmail.com> > > > > wrote: > > > > >> > > > > > >> > Well, I look at it like this. > > > > >> > > > > > >> > The Secure Vault was part of the original metron pitch, and many > > may > > > > >> have used that as part of their evaluations. > > > > >> > “Look, it is going to have a security vault type thing, it is on > > the > > > > >> roadmap”. > > > > >> > > > > > >> > Regardless of the implementation, conceptually, security of data > > at > > > > >> rest is important, and is a major outstanding item or the core > > metron > > > > >> proposition. > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball ( > > > > >> si...@simonellistonball.com) wrote: > > > > >> >> > > > > >> >> That’s going back a way. I always saw that concept as begin > about > > > the > > > > >> formats, e.g. Orc, and meta data around it plus the data service > api > > > to > > > > get > > > > >> at it. I’m all for that too, but think it needs more thought than > > the > > > > >> ticket captures. > > > > >> >> > > > > >> >> Simon > > > > >> >> > > > > >> >> On 15 Aug 2018, at 20:53, Otto Fowler <ottobackwa...@gmail.com > > > > > > wrote: > > > > >> >> > > > > >> >>> https://issues.apache.org/jira/browse/METRON-343 > > > > >> >>> > > > > >> >>>> On August 15, 2018 at 15:47:24, Simon Elliston Ball ( > > > > >> si...@simonellistonball.com) wrote: > > > > >> >>>> > > > > >> >>>> What would you see as secure? I’ve seen people use TDE for > the > > > HDFS > > > > >> store, but it’s harder to encrypt storage with solr / es. > Something > > I > > > > was > > > > >> thinking of doing to follow up on the Knox Feature was to add > Ranger > > > > >> integration for securing and auditing configs, and potentially > > > > extending to > > > > >> the index destinations. Do you think that would cover the secure > > > storage > > > > >> concept? > > > > >> >>>> > > > > >> >>>> Simon > > > > >> >>>> > > > > >> >>>> > On 15 Aug 2018, at 20:39, Otto Fowler < > > ottobackwa...@gmail.com > > > > > > > > >> wrote: > > > > >> >>>> > > > > > >> >>>> > Secure storage off the top of my head > > > > >> >>>> > > > > > >> >>>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com ( > > > > zeo...@gmail.com) > > > > >> wrote: > > > > >> >>>> > > > > > >> >>>> > So, as has been discussed in a few > > > > >> >>>> > < > > > > >> >>>> > > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a > > > > >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E> > > > > >> >>>> > > > > > >> >>>> > other > > > > >> >>>> > < > > > > >> >>>> > > https://lists.apache.org/thread.html/427a20c22207e84331b94e8 > > > > >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E> > > > > >> >>>> > > > > > >> >>>> > recent dev list threads, I would like to discuss what a > > Metron > > > > 1.0 > > > > >> release > > > > >> >>>> > looks like. > > > > >> >>>> > > > > > >> >>>> > In order to kick off the conversation, I would like to > make a > > > few > > > > >> >>>> > suggestions regarding "what 1.0 means to me," but I'm very > > > > >> interested to > > > > >> >>>> > hear everybody else's opinions. > > > > >> >>>> > > > > > >> >>>> > In order to go 1.0 I believe we should have: > > > > >> >>>> > 1. A clear, supported method of upgrading from one version > of > > > > >> Metron to the > > > > >> >>>> > next. We have attempted > > > > >> >>>> > <https://github.com/apache/metron/blob/master/Upgrading.md > > > > to > > > > >> make this > > > > >> >>>> > easier in the past, but it is currently not > > > > >> >>>> > < > > > > >> >>>> > > https://github.com/apache/metron/tree/master/metron-deployme > > > > >> nt/packaging/ambari/metron-mpack#limitations> > > > > >> >>>> > > > > > >> >>>> > supported > > > > >> >>>> > < > > > > >> >>>> > > https://github.com/apache/metron/tree/master/metron-deployme > > > > >> nt/packaging/ambari/elasticsearch-mpack#limitations> > > > > >> >>>> > > > > > >> >>>> > . > > > > >> >>>> > 2. Authentication for all of the UIs and APIs should be > > secure > > > > and > > > > >> support > > > > >> >>>> > SSO. I believe this is in progress via METRON-1663 > > > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1663>. > > > > >> >>>> > 3. Each of our personas > > > > >> >>>> > < > > > > >> >>>> > https://cwiki.apache.org/confluence/display/METRON/Metron+ > > > > >> User+Personas+And+Benefits> > > > > >> >>>> > > > > > >> >>>> > should > > > > >> >>>> > be well documented, understood, and supported. > > > > >> >>>> > - The current state of documentation is, in my opinion, > > > > inadequate > > > > >> and I > > > > >> >>>> > admit I am partially to blame for this. I suggest we > define a > > > > >> strict > > > > >> >>>> > approach for documentation, align to it (such as perhaps > > > > migrating > > > > >> all > > > > >> >>>> > useful wiki documentation to git), and enforce it. > > > > >> >>>> > - I would consider METRON-1699 > > > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1699> as a > > > > critical > > > > >> item for > > > > >> >>>> > a Security Data Scientist, but it is currently not clear to > > me > > > > >> where the > > > > >> >>>> > line exists between some of the other personas, or that > each > > > > >> persona has > > > > >> >>>> > been sufficiently implemented. > > > > >> >>>> > 4. A performance tuning guide should be available for all > of > > > the > > > > >> main > > > > >> >>>> > components, whether as an independent document or as a part > > of > > > a > > > > >> larger > > > > >> >>>> > document. > > > > >> >>>> > 5. Simple data ingest. > > > > >> >>>> > - Similar to the ongoing conversation for NiFi integration > > > > >> >>>> > < > > > > >> >>>> > > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f269 > > > > >> 73f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>, > > > > >> >>>> > > > > > >> >>>> > we should be able to say that we have broken down the > > barriers > > > to > > > > >> getting > > > > >> >>>> > data into a Metron cluster in easy and efficient ways. In > > > > addition > > > > >> to > > > > >> >>>> > NiFi, having support for other popular tools such as beats > > > > >> >>>> > <https://www.elastic.co/products/beats>, fluentd < > > > > >> https://www.fluentd.org/>, > > > > >> >>>> > > > > > >> >>>> > etc. > > > > >> >>>> > - Parsers should be pluggable, with independent tests and > the > > > > >> ability to > > > > >> >>>> > make versioned modifications with roll-backs. > > > > >> >>>> > > > > > >> >>>> > What else? Are any of these items not necessary for a 1.0? > > > > >> >>>> > > > > > >> >>>> > Jon > > > > >> >>>> > -- > > > > >> >>>> > > > > > >> >>>> > Jon > > > > >> > > > > > > > > > > > > > > > > > > > >
