I agree with that wholeheartedly.  The more integrations in and out of the
stack at various points, the better.

Jon

On Mon, Aug 27, 2018, 02:44 Ali Nazemian <alinazem...@gmail.com> wrote:

> One thing that we could imagine for v1.0 might be an ability to extend
> Metron from adding more pipelines to it. For example, being able to extend
> Metron to be integrated with other endpoints more easily from Storm
> perspective. For example, what if we would like to create other topologies
> to write files in ORC directly rather than HDFS or index it to Druid. What
> if we would like to build an automatic security response and move to become
> SOAR. All these integrations can be done even now and there are other users
> that may have done it already. However, providing a clear extension point
> can make it easier to contribute other pipelines to the community. I think
> by adding this level of extendability, Metron community will grow way
> faster by adding more integration that can be available.
>
> Cheers,
> Ali
>
> On Mon, Aug 20, 2018 at 10:50 PM Casey Stella <ceste...@gmail.com> wrote:
>
> > I completely agree, Mike.  Our docs are either very high level or very
> low
> > level (and possibly stale) and, worse, aren't aimed at the actors that
> > you've stated.
> > I think that the HBase project does a good job of providing coherent and
> > useable documentation in their "HBase Book" (see
> > https://hbase.apache.org/book.html).
> > It's not actor-specific, but it is coherent advice for the practical
> > practitioner of HBase (both admin and developer) and speaks with one
> > voice.  I think Metron's need
> > is a bit different, but at the minimum some coherent docs that speaks
> with
> > one voice and has a coherent pitch about what Metron is used for and what
> > it isn't used for
> > is well needed.
> >
> > On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Apologies for any spelling mishaps as I'm writing from my phone.
> > >
> > > I'm for improving our docs. I'd like to see us guide our various
> profiles
> > > of user towards the specific documentation for the abstraction levels
> > > they'll be most interested in working from. I think we should have
> > platform
> > > docs about how we're a broadly useful, extensible streaming analytics
> > > platform for cyber security as well as docs that emphasize more narrow
> > and
> > > specific use cases.
> > >
> > > Personally, I think I see 3 potential tiers or classifications of docs.
> > > These are just observations and ideas I had, not necessarily a
> > prescription
> > > for organizing docs:
> > > - Low level tool instructions, eg
> > >     - how do I run the pcap toplogy and then query with the CLI and UI?
> > > - Platform docs about building on top of Metron, e.g.
> > >     - writing custom parsers, enrichment, and threat Intel (imho we
> > should
> > > start to take a more opinionated view of leveraging Stellar as this
> > > extension point rather than implementing new parser classes in Java)
> > >     - using the profiler for constructing outlier analysis use cases
> > >     - using MAAS for building and deploying models for use in
> enrichment
> > > - Docs around more specific use cases that solve specific as opposed to
> > > more general problems, similar to those we have in the use-cases
> folder.
> > >
> > > I think one of our challenges currently is that our docs could be
> better
> > > tailored to the "actors" we've talked about in the past. An individual
> > SOC
> > > analyst will have a very different set of interests than would a
> reseller
> > > that wants to build on top of our platform to expose new modules and
> > > functionality to those SOC analyst. And we can, and do, currently
> support
> > > both.
> > >
> > >
> > > On Sat, Aug 18, 2018, 9:34 AM Nick Allen <n...@nickallen.org> wrote:
> > >
> > > > Yes, I imagine just a separate top level directory which would
> contain
> > > the
> > > > docs.
> > > >
> > > > We would need someone to survey what doc tools are out there and
> > provide
> > > > some advice.
> > > >
> > > > Maybe we could look around at other open source projects that have
> done
> > > > their docs well and emulate them.
> > > >
> > > > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson <
> > > kylerichards...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 to separating developer docs and user docs. How should we
> approach
> > > > that.
> > > > > Have a separate doc book? I haven’t had a ton of time to contribute
> > to
> > > > code
> > > > > lately but I’d be happy to help write some of these.
> > > > >
> > > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen <n...@nickallen.org>
> > wrote:
> > > > >
> > > > > > Personally, I think the state of our docs and web presence is an
> > > > > inhibitor
> > > > > > to growing the Metron community.  Unless we can offer concise,
> > > > compelling
> > > > > > answers to the basic questions (What can I do with Metron?  Who
> > does
> > > it
> > > > > > help? How do I do that?), potential users and contributors are
> > unable
> > > > to
> > > > > > see the value of Metron.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen <n...@nickallen.org>
> > > > wrote:
> > > > > >
> > > > > > > I'd like to see us focus on improving our docs before a version
> > > 1.0.
> > > > > > > Right now we just stitch together a bunch of READMEs, which is
> a
> > > > great
> > > > > > > stride from where we started, but is not ideal.
> > > > > > >
> > > > > > > Our docs should focused on the user and use cases; What can I
> do
> > > with
> > > > > > > Metron?  Who does it help? How do I do that?
> > > > > > >
> > > > > > > The docs should be separate from the code base to allow for an
> > > > > > > organization that is focused on the user rather than the
> > > > > implementation.
> > > > > > > This allows the READMEs to focus on the developer and the
> > > > > implementation,
> > > > > > > which should make them more digestible too.  The docs should be
> > > > version
> > > > > > > controlled and maintained through PRs, just like the code.  We
> > > should
> > > > > > take
> > > > > > > just as much pride in our docs as we do in our code.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > > > > > > si...@simonellistonball.com> wrote:
> > > > > > >
> > > > > > >> Agreed, should we add TDE by default, and get the ranger
> > policies
> > > on
> > > > > by
> > > > > > >> default? That leaves secured in Kafka, which would have to be
> > > built
> > > > > into
> > > > > > >> the consumers and producers to encrypt into the on disk Kafka
> > > > topics.
> > > > > > Does
> > > > > > >> that seem necessary to people? It would have performance
> > > > implications
> > > > > > for
> > > > > > >> sure.
> > > > > > >>
> > > > > > >> Simon
> > > > > > >>
> > > > > > >> > On 15 Aug 2018, at 21:26, Otto Fowler <
> > ottobackwa...@gmail.com>
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > Well, I look at it like this.
> > > > > > >> >
> > > > > > >> > The Secure Vault was part of the original metron pitch, and
> > many
> > > > may
> > > > > > >> have used that as part of their evaluations.
> > > > > > >> > “Look, it is going to have a security vault type thing, it
> is
> > on
> > > > the
> > > > > > >> roadmap”.
> > > > > > >> >
> > > > > > >> > Regardless of the implementation, conceptually, security of
> > data
> > > > at
> > > > > > >> rest is important, and is a major outstanding item or the core
> > > > metron
> > > > > > >> proposition.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball (
> > > > > > >> si...@simonellistonball.com) wrote:
> > > > > > >> >>
> > > > > > >> >> That’s going back a way. I always saw that concept as begin
> > > about
> > > > > the
> > > > > > >> formats, e.g. Orc, and meta data around it plus the data
> service
> > > api
> > > > > to
> > > > > > get
> > > > > > >> at it. I’m all for that too, but think it needs more thought
> > than
> > > > the
> > > > > > >> ticket captures.
> > > > > > >> >>
> > > > > > >> >> Simon
> > > > > > >> >>
> > > > > > >> >> On 15 Aug 2018, at 20:53, Otto Fowler <
> > ottobackwa...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >> >>
> > > > > > >> >>> https://issues.apache.org/jira/browse/METRON-343
> > > > > > >> >>>
> > > > > > >> >>>> On August 15, 2018 at 15:47:24, Simon Elliston Ball (
> > > > > > >> si...@simonellistonball.com) wrote:
> > > > > > >> >>>>
> > > > > > >> >>>> What would you see as secure? I’ve seen people use TDE
> for
> > > the
> > > > > HDFS
> > > > > > >> store, but it’s harder to encrypt storage with solr / es.
> > > Something
> > > > I
> > > > > > was
> > > > > > >> thinking of doing to follow up on the Knox Feature was to add
> > > Ranger
> > > > > > >> integration for securing and auditing configs, and potentially
> > > > > > extending to
> > > > > > >> the index destinations. Do you think that would cover the
> secure
> > > > > storage
> > > > > > >> concept?
> > > > > > >> >>>>
> > > > > > >> >>>> Simon
> > > > > > >> >>>>
> > > > > > >> >>>> > On 15 Aug 2018, at 20:39, Otto Fowler <
> > > > ottobackwa...@gmail.com
> > > > > >
> > > > > > >> wrote:
> > > > > > >> >>>> >
> > > > > > >> >>>> > Secure storage off the top of my head
> > > > > > >> >>>> >
> > > > > > >> >>>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com (
> > > > > > zeo...@gmail.com)
> > > > > > >> wrote:
> > > > > > >> >>>> >
> > > > > > >> >>>> > So, as has been discussed in a few
> > > > > > >> >>>> > <
> > > > > > >> >>>> >
> > > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a
> > > > > > >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org
> %3E>
> > > > > > >> >>>> >
> > > > > > >> >>>> > other
> > > > > > >> >>>> > <
> > > > > > >> >>>> >
> > > https://lists.apache.org/thread.html/427a20c22207e84331b94e8
> > > > > > >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org
> %3E>
> > > > > > >> >>>> >
> > > > > > >> >>>> > recent dev list threads, I would like to discuss what a
> > > > Metron
> > > > > > 1.0
> > > > > > >> release
> > > > > > >> >>>> > looks like.
> > > > > > >> >>>> >
> > > > > > >> >>>> > In order to kick off the conversation, I would like to
> > > make a
> > > > > few
> > > > > > >> >>>> > suggestions regarding "what 1.0 means to me," but I'm
> > very
> > > > > > >> interested to
> > > > > > >> >>>> > hear everybody else's opinions.
> > > > > > >> >>>> >
> > > > > > >> >>>> > In order to go 1.0 I believe we should have:
> > > > > > >> >>>> > 1. A clear, supported method of upgrading from one
> > version
> > > of
> > > > > > >> Metron to the
> > > > > > >> >>>> > next. We have attempted
> > > > > > >> >>>> > <
> > https://github.com/apache/metron/blob/master/Upgrading.md
> > > >
> > > > to
> > > > > > >> make this
> > > > > > >> >>>> > easier in the past, but it is currently not
> > > > > > >> >>>> > <
> > > > > > >> >>>> >
> > > https://github.com/apache/metron/tree/master/metron-deployme
> > > > > > >> nt/packaging/ambari/metron-mpack#limitations>
> > > > > > >> >>>> >
> > > > > > >> >>>> > supported
> > > > > > >> >>>> > <
> > > > > > >> >>>> >
> > > https://github.com/apache/metron/tree/master/metron-deployme
> > > > > > >> nt/packaging/ambari/elasticsearch-mpack#limitations>
> > > > > > >> >>>> >
> > > > > > >> >>>> > .
> > > > > > >> >>>> > 2. Authentication for all of the UIs and APIs should be
> > > > secure
> > > > > > and
> > > > > > >> support
> > > > > > >> >>>> > SSO. I believe this is in progress via METRON-1663
> > > > > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1663>.
> > > > > > >> >>>> > 3. Each of our personas
> > > > > > >> >>>> > <
> > > > > > >> >>>> >
> > https://cwiki.apache.org/confluence/display/METRON/Metron+
> > > > > > >> User+Personas+And+Benefits>
> > > > > > >> >>>> >
> > > > > > >> >>>> > should
> > > > > > >> >>>> > be well documented, understood, and supported.
> > > > > > >> >>>> > - The current state of documentation is, in my opinion,
> > > > > > inadequate
> > > > > > >> and I
> > > > > > >> >>>> > admit I am partially to blame for this. I suggest we
> > > define a
> > > > > > >> strict
> > > > > > >> >>>> > approach for documentation, align to it (such as
> perhaps
> > > > > > migrating
> > > > > > >> all
> > > > > > >> >>>> > useful wiki documentation to git), and enforce it.
> > > > > > >> >>>> > - I would consider METRON-1699
> > > > > > >> >>>> > <https://issues.apache.org/jira/browse/METRON-1699>
> as a
> > > > > > critical
> > > > > > >> item for
> > > > > > >> >>>> > a Security Data Scientist, but it is currently not
> clear
> > to
> > > > me
> > > > > > >> where the
> > > > > > >> >>>> > line exists between some of the other personas, or that
> > > each
> > > > > > >> persona has
> > > > > > >> >>>> > been sufficiently implemented.
> > > > > > >> >>>> > 4. A performance tuning guide should be available for
> all
> > > of
> > > > > the
> > > > > > >> main
> > > > > > >> >>>> > components, whether as an independent document or as a
> > part
> > > > of
> > > > > a
> > > > > > >> larger
> > > > > > >> >>>> > document.
> > > > > > >> >>>> > 5. Simple data ingest.
> > > > > > >> >>>> > - Similar to the ongoing conversation for NiFi
> > integration
> > > > > > >> >>>> > <
> > > > > > >> >>>> >
> > > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f269
> > > > > > >> 73f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org
> %3E>,
> > > > > > >> >>>> >
> > > > > > >> >>>> > we should be able to say that we have broken down the
> > > > barriers
> > > > > to
> > > > > > >> getting
> > > > > > >> >>>> > data into a Metron cluster in easy and efficient ways.
> In
> > > > > > addition
> > > > > > >> to
> > > > > > >> >>>> > NiFi, having support for other popular tools such as
> > beats
> > > > > > >> >>>> > <https://www.elastic.co/products/beats>, fluentd <
> > > > > > >> https://www.fluentd.org/>,
> > > > > > >> >>>> >
> > > > > > >> >>>> > etc.
> > > > > > >> >>>> > - Parsers should be pluggable, with independent tests
> and
> > > the
> > > > > > >> ability to
> > > > > > >> >>>> > make versioned modifications with roll-backs.
> > > > > > >> >>>> >
> > > > > > >> >>>> > What else? Are any of these items not necessary for a
> > 1.0?
> > > > > > >> >>>> >
> > > > > > >> >>>> > Jon
> > > > > > >> >>>> > --
> > > > > > >> >>>> >
> > > > > > >> >>>> > Jon
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> A.Nazemian
>
-- 

Jon

Reply via email to