Apologies for any spelling mishaps as I'm writing from my phone.

I'm for improving our docs. I'd like to see us guide our various profiles
of user towards the specific documentation for the abstraction levels
they'll be most interested in working from. I think we should have platform
docs about how we're a broadly useful, extensible streaming analytics
platform for cyber security as well as docs that emphasize more narrow and
specific use cases.

Personally, I think I see 3 potential tiers or classifications of docs.
These are just observations and ideas I had, not necessarily a prescription
for organizing docs:
- Low level tool instructions, eg
    - how do I run the pcap toplogy and then query with the CLI and UI?
- Platform docs about building on top of Metron, e.g.
    - writing custom parsers, enrichment, and threat Intel (imho we should
start to take a more opinionated view of leveraging Stellar as this
extension point rather than implementing new parser classes in Java)
    - using the profiler for constructing outlier analysis use cases
    - using MAAS for building and deploying models for use in enrichment
- Docs around more specific use cases that solve specific as opposed to
more general problems, similar to those we have in the use-cases folder.

I think one of our challenges currently is that our docs could be better
tailored to the "actors" we've talked about in the past. An individual SOC
analyst will have a very different set of interests than would a reseller
that wants to build on top of our platform to expose new modules and
functionality to those SOC analyst. And we can, and do, currently support
both.


On Sat, Aug 18, 2018, 9:34 AM Nick Allen <n...@nickallen.org> wrote:

> Yes, I imagine just a separate top level directory which would contain the
> docs.
>
> We would need someone to survey what doc tools are out there and provide
> some advice.
>
> Maybe we could look around at other open source projects that have done
> their docs well and emulate them.
>
> On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson <kylerichards...@gmail.com>
> wrote:
>
> > +1 to separating developer docs and user docs. How should we approach
> that.
> > Have a separate doc book? I haven’t had a ton of time to contribute to
> code
> > lately but I’d be happy to help write some of these.
> >
> > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen <n...@nickallen.org> wrote:
> >
> > > Personally, I think the state of our docs and web presence is an
> > inhibitor
> > > to growing the Metron community.  Unless we can offer concise,
> compelling
> > > answers to the basic questions (What can I do with Metron?  Who does it
> > > help? How do I do that?), potential users and contributors are unable
> to
> > > see the value of Metron.
> > >
> > >
> > >
> > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen <n...@nickallen.org>
> wrote:
> > >
> > > > I'd like to see us focus on improving our docs before a version 1.0.
> > > > Right now we just stitch together a bunch of READMEs, which is a
> great
> > > > stride from where we started, but is not ideal.
> > > >
> > > > Our docs should focused on the user and use cases; What can I do with
> > > > Metron?  Who does it help? How do I do that?
> > > >
> > > > The docs should be separate from the code base to allow for an
> > > > organization that is focused on the user rather than the
> > implementation.
> > > > This allows the READMEs to focus on the developer and the
> > implementation,
> > > > which should make them more digestible too.  The docs should be
> version
> > > > controlled and maintained through PRs, just like the code.  We should
> > > take
> > > > just as much pride in our docs as we do in our code.
> > > >
> > > >
> > > >
> > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > > > si...@simonellistonball.com> wrote:
> > > >
> > > >> Agreed, should we add TDE by default, and get the ranger policies on
> > by
> > > >> default? That leaves secured in Kafka, which would have to be built
> > into
> > > >> the consumers and producers to encrypt into the on disk Kafka
> topics.
> > > Does
> > > >> that seem necessary to people? It would have performance
> implications
> > > for
> > > >> sure.
> > > >>
> > > >> Simon
> > > >>
> > > >> > On 15 Aug 2018, at 21:26, Otto Fowler <ottobackwa...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > Well, I look at it like this.
> > > >> >
> > > >> > The Secure Vault was part of the original metron pitch, and many
> may
> > > >> have used that as part of their evaluations.
> > > >> > “Look, it is going to have a security vault type thing, it is on
> the
> > > >> roadmap”.
> > > >> >
> > > >> > Regardless of the implementation, conceptually, security of data
> at
> > > >> rest is important, and is a major outstanding item or the core
> metron
> > > >> proposition.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball (
> > > >> si...@simonellistonball.com) wrote:
> > > >> >>
> > > >> >> That’s going back a way. I always saw that concept as begin about
> > the
> > > >> formats, e.g. Orc, and meta data around it plus the data service api
> > to
> > > get
> > > >> at it. I’m all for that too, but think it needs more thought than
> the
> > > >> ticket captures.
> > > >> >>
> > > >> >> Simon
> > > >> >>
> > > >> >> On 15 Aug 2018, at 20:53, Otto Fowler <ottobackwa...@gmail.com>
> > > wrote:
> > > >> >>
> > > >> >>> https://issues.apache.org/jira/browse/METRON-343
> > > >> >>>
> > > >> >>>> On August 15, 2018 at 15:47:24, Simon Elliston Ball (
> > > >> si...@simonellistonball.com) wrote:
> > > >> >>>>
> > > >> >>>> What would you see as secure? I’ve seen people use TDE for the
> > HDFS
> > > >> store, but it’s harder to encrypt storage with solr / es. Something
> I
> > > was
> > > >> thinking of doing to follow up on the Knox Feature was to add Ranger
> > > >> integration for securing and auditing configs, and potentially
> > > extending to
> > > >> the index destinations. Do you think that would cover the secure
> > storage
> > > >> concept?
> > > >> >>>>
> > > >> >>>> Simon
> > > >> >>>>
> > > >> >>>> > On 15 Aug 2018, at 20:39, Otto Fowler <
> ottobackwa...@gmail.com
> > >
> > > >> wrote:
> > > >> >>>> >
> > > >> >>>> > Secure storage off the top of my head
> > > >> >>>> >
> > > >> >>>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com (
> > > zeo...@gmail.com)
> > > >> wrote:
> > > >> >>>> >
> > > >> >>>> > So, as has been discussed in a few
> > > >> >>>> > <
> > > >> >>>> > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a
> > > >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
> > > >> >>>> >
> > > >> >>>> > other
> > > >> >>>> > <
> > > >> >>>> > https://lists.apache.org/thread.html/427a20c22207e84331b94e8
> > > >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
> > > >> >>>> >
> > > >> >>>> > recent dev list threads, I would like to discuss what a
> Metron
> > > 1.0
> > > >> release
> > > >> >>>> > looks like.
> > > >> >>>> >
> > > >> >>>> > In order to kick off the conversation, I would like to make a
> > few
> > > >> >>>> > suggestions regarding "what 1.0 means to me," but I'm very
> > > >> interested to
> > > >> >>>> > hear everybody else's opinions.
> > > >> >>>> >
> > > >> >>>> > In order to go 1.0 I believe we should have:
> > > >> >>>> > 1. A clear, supported method of upgrading from one version of
> > > >> Metron to the
> > > >> >>>> > next. We have attempted
> > > >> >>>> > <https://github.com/apache/metron/blob/master/Upgrading.md>
> to
> > > >> make this
> > > >> >>>> > easier in the past, but it is currently not
> > > >> >>>> > <
> > > >> >>>> > https://github.com/apache/metron/tree/master/metron-deployme
> > > >> nt/packaging/ambari/metron-mpack#limitations>
> > > >> >>>> >
> > > >> >>>> > supported
> > > >> >>>> > <
> > > >> >>>> > https://github.com/apache/metron/tree/master/metron-deployme
> > > >> nt/packaging/ambari/elasticsearch-mpack#limitations>
> > > >> >>>> >
> > > >> >>>> > .
> > > >> >>>> > 2. Authentication for all of the UIs and APIs should be
> secure
> > > and
> > > >> support
> > > >> >>>> > SSO. I believe this is in progress via METRON-1663
> > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1663>.
> > > >> >>>> > 3. Each of our personas
> > > >> >>>> > <
> > > >> >>>> > https://cwiki.apache.org/confluence/display/METRON/Metron+
> > > >> User+Personas+And+Benefits>
> > > >> >>>> >
> > > >> >>>> > should
> > > >> >>>> > be well documented, understood, and supported.
> > > >> >>>> > - The current state of documentation is, in my opinion,
> > > inadequate
> > > >> and I
> > > >> >>>> > admit I am partially to blame for this. I suggest we define a
> > > >> strict
> > > >> >>>> > approach for documentation, align to it (such as perhaps
> > > migrating
> > > >> all
> > > >> >>>> > useful wiki documentation to git), and enforce it.
> > > >> >>>> > - I would consider METRON-1699
> > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1699> as a
> > > critical
> > > >> item for
> > > >> >>>> > a Security Data Scientist, but it is currently not clear to
> me
> > > >> where the
> > > >> >>>> > line exists between some of the other personas, or that each
> > > >> persona has
> > > >> >>>> > been sufficiently implemented.
> > > >> >>>> > 4. A performance tuning guide should be available for all of
> > the
> > > >> main
> > > >> >>>> > components, whether as an independent document or as a part
> of
> > a
> > > >> larger
> > > >> >>>> > document.
> > > >> >>>> > 5. Simple data ingest.
> > > >> >>>> > - Similar to the ongoing conversation for NiFi integration
> > > >> >>>> > <
> > > >> >>>> > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f269
> > > >> 73f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
> > > >> >>>> >
> > > >> >>>> > we should be able to say that we have broken down the
> barriers
> > to
> > > >> getting
> > > >> >>>> > data into a Metron cluster in easy and efficient ways. In
> > > addition
> > > >> to
> > > >> >>>> > NiFi, having support for other popular tools such as beats
> > > >> >>>> > <https://www.elastic.co/products/beats>, fluentd <
> > > >> https://www.fluentd.org/>,
> > > >> >>>> >
> > > >> >>>> > etc.
> > > >> >>>> > - Parsers should be pluggable, with independent tests and the
> > > >> ability to
> > > >> >>>> > make versioned modifications with roll-backs.
> > > >> >>>> >
> > > >> >>>> > What else? Are any of these items not necessary for a 1.0?
> > > >> >>>> >
> > > >> >>>> > Jon
> > > >> >>>> > --
> > > >> >>>> >
> > > >> >>>> > Jon
> > > >>
> > > >
> > > >
> > >
> >
>

Reply via email to