Re: [DISCUSS] Getting to a 1.0 release

[Nick Allen]

We should also publish binaries for each release.  That might include any
of the following...

   - Publish the jars to a Maven repository
   - Publish the RPMs and DEBs to a repository
   - Host a pre-built "Full Dev" in Vagrant Cloud



On Wed, Aug 15, 2018 at 2:49 PM zeo...@gmail.com  wrote:

> So, as has been discussed in a few
> <
> https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E
> >
> other
> <
> https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E
> >
> recent dev list threads, I would like to discuss what a Metron 1.0 release
> looks like.
>
> In order to kick off the conversation, I would like to make a few
> suggestions regarding "what 1.0 means to me," but I'm very interested to
> hear everybody else's opinions.
>
> In order to go 1.0 I believe we should have:
> 1. A clear, supported method of upgrading from one version of Metron to the
> next.  We have attempted
>  to make this
> easier in the past, but it is currently not
> <
> https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations
> >
> supported
> <
> https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations
> >
> .
> 2. Authentication for all of the UIs and APIs should be secure and support
> SSO.  I believe this is in progress via METRON-1663
> .
> 3. Each of our personas
> <
> https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits
> >
> should
> be well documented, understood, and supported.
>  - The current state of documentation is, in my opinion, inadequate and I
> admit I am partially to blame for this.  I suggest we define a strict
> approach for documentation, align to it (such as perhaps migrating all
> useful wiki documentation to git), and enforce it.
>  - I would consider METRON-1699
>  as a critical item for
> a Security Data Scientist, but it is currently not clear to me where the
> line exists between some of the other personas, or that each persona has
> been sufficiently implemented.
> 4. A performance tuning guide should be available for all of the main
> components, whether as an independent document or as a part of a larger
> document.
> 5. Simple data ingest.
>  - Similar to the ongoing conversation for NiFi integration
> <
> https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E
> >,
> we should be able to say that we have broken down the barriers to getting
> data into a Metron cluster in easy and efficient ways.  In addition to
> NiFi, having support for other popular tools such as beats
> , fluentd  >,
> etc.
>  - Parsers should be pluggable, with independent tests and the ability to
> make versioned modifications with roll-backs.
>
> What else?  Are any of these items not necessary for a 1.0?
>
> Jon
> --
>
> Jon
>

Re: [DISCUSS] Getting to a 1.0 release

[Otto Fowler]

There is a jira to make indexing extensible, such that you could define the
indexing topology much as you define the
parsing, and can load different indexers.

On August 27, 2018 at 02:44:24, Ali Nazemian (alinazem...@gmail.com) wrote:

One thing that we could imagine for v1.0 might be an ability to extend
Metron from adding more pipelines to it. For example, being able to extend
Metron to be integrated with other endpoints more easily from Storm
perspective. For example, what if we would like to create other topologies
to write files in ORC directly rather than HDFS or index it to Druid. What
if we would like to build an automatic security response and move to become
SOAR. All these integrations can be done even now and there are other users
that may have done it already. However, providing a clear extension point
can make it easier to contribute other pipelines to the community. I think
by adding this level of extendability, Metron community will grow way
faster by adding more integration that can be available.

Cheers,
Ali

On Mon, Aug 20, 2018 at 10:50 PM Casey Stella  wrote:

> I completely agree, Mike. Our docs are either very high level or very low
> level (and possibly stale) and, worse, aren't aimed at the actors that
> you've stated.
> I think that the HBase project does a good job of providing coherent and
> useable documentation in their "HBase Book" (see
> https://hbase.apache.org/book.html).
> It's not actor-specific, but it is coherent advice for the practical
> practitioner of HBase (both admin and developer) and speaks with one
> voice. I think Metron's need
> is a bit different, but at the minimum some coherent docs that speaks
with
> one voice and has a coherent pitch about what Metron is used for and what
> it isn't used for
> is well needed.
>
> On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Apologies for any spelling mishaps as I'm writing from my phone.
> >
> > I'm for improving our docs. I'd like to see us guide our various
profiles
> > of user towards the specific documentation for the abstraction levels
> > they'll be most interested in working from. I think we should have
> platform
> > docs about how we're a broadly useful, extensible streaming analytics
> > platform for cyber security as well as docs that emphasize more narrow
> and
> > specific use cases.
> >
> > Personally, I think I see 3 potential tiers or classifications of docs.
> > These are just observations and ideas I had, not necessarily a
> prescription
> > for organizing docs:
> > - Low level tool instructions, eg
> > - how do I run the pcap toplogy and then query with the CLI and UI?
> > - Platform docs about building on top of Metron, e.g.
> > - writing custom parsers, enrichment, and threat Intel (imho we
> should
> > start to take a more opinionated view of leveraging Stellar as this
> > extension point rather than implementing new parser classes in Java)
> > - using the profiler for constructing outlier analysis use cases
> > - using MAAS for building and deploying models for use in enrichment
> > - Docs around more specific use cases that solve specific as opposed to
> > more general problems, similar to those we have in the use-cases
folder.
> >
> > I think one of our challenges currently is that our docs could be
better
> > tailored to the "actors" we've talked about in the past. An individual
> SOC
> > analyst will have a very different set of interests than would a
reseller
> > that wants to build on top of our platform to expose new modules and
> > functionality to those SOC analyst. And we can, and do, currently
support
> > both.
> >
> >
> > On Sat, Aug 18, 2018, 9:34 AM Nick Allen  wrote:
> >
> > > Yes, I imagine just a separate top level directory which would
contain
> > the
> > > docs.
> > >
> > > We would need someone to survey what doc tools are out there and
> provide
> > > some advice.
> > >
> > > Maybe we could look around at other open source projects that have
done
> > > their docs well and emulate them.
> > >
> > > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson <
> > kylerichards...@gmail.com>
> > > wrote:
> > >
> > > > +1 to separating developer docs and user docs. How should we
approach
> > > that.
> > > > Have a separate doc book? I haven’t had a ton of time to contribute
> to
> > > code
> > > > lately but I’d be happy to help write some of these.
> > > >
> > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen 
> wrote:
> > > >
> > > > > Personally, I think the state of our docs and web presence is an
> > > > inhibitor
> > > > > to growing the Metron community. Unless we can offer concise,
> > > compelling
> > > > > answers to the basic questions (What can I do with Metron? Who
> does
> > it
> > > > > help? How do I do that?), potential users and contributors are
> unable
> > > to
> > > > > see the value of Metron.
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen 
> > > wrote:
> > > > >
> > > > > > I'd like to 

Re: [DISCUSS] Getting to a 1.0 release

[zeo...@gmail.com]

I agree with that wholeheartedly.  The more integrations in and out of the
stack at various points, the better.

Jon

On Mon, Aug 27, 2018, 02:44 Ali Nazemian  wrote:

> One thing that we could imagine for v1.0 might be an ability to extend
> Metron from adding more pipelines to it. For example, being able to extend
> Metron to be integrated with other endpoints more easily from Storm
> perspective. For example, what if we would like to create other topologies
> to write files in ORC directly rather than HDFS or index it to Druid. What
> if we would like to build an automatic security response and move to become
> SOAR. All these integrations can be done even now and there are other users
> that may have done it already. However, providing a clear extension point
> can make it easier to contribute other pipelines to the community. I think
> by adding this level of extendability, Metron community will grow way
> faster by adding more integration that can be available.
>
> Cheers,
> Ali
>
> On Mon, Aug 20, 2018 at 10:50 PM Casey Stella  wrote:
>
> > I completely agree, Mike.  Our docs are either very high level or very
> low
> > level (and possibly stale) and, worse, aren't aimed at the actors that
> > you've stated.
> > I think that the HBase project does a good job of providing coherent and
> > useable documentation in their "HBase Book" (see
> > https://hbase.apache.org/book.html).
> > It's not actor-specific, but it is coherent advice for the practical
> > practitioner of HBase (both admin and developer) and speaks with one
> > voice.  I think Metron's need
> > is a bit different, but at the minimum some coherent docs that speaks
> with
> > one voice and has a coherent pitch about what Metron is used for and what
> > it isn't used for
> > is well needed.
> >
> > On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Apologies for any spelling mishaps as I'm writing from my phone.
> > >
> > > I'm for improving our docs. I'd like to see us guide our various
> profiles
> > > of user towards the specific documentation for the abstraction levels
> > > they'll be most interested in working from. I think we should have
> > platform
> > > docs about how we're a broadly useful, extensible streaming analytics
> > > platform for cyber security as well as docs that emphasize more narrow
> > and
> > > specific use cases.
> > >
> > > Personally, I think I see 3 potential tiers or classifications of docs.
> > > These are just observations and ideas I had, not necessarily a
> > prescription
> > > for organizing docs:
> > > - Low level tool instructions, eg
> > > - how do I run the pcap toplogy and then query with the CLI and UI?
> > > - Platform docs about building on top of Metron, e.g.
> > > - writing custom parsers, enrichment, and threat Intel (imho we
> > should
> > > start to take a more opinionated view of leveraging Stellar as this
> > > extension point rather than implementing new parser classes in Java)
> > > - using the profiler for constructing outlier analysis use cases
> > > - using MAAS for building and deploying models for use in
> enrichment
> > > - Docs around more specific use cases that solve specific as opposed to
> > > more general problems, similar to those we have in the use-cases
> folder.
> > >
> > > I think one of our challenges currently is that our docs could be
> better
> > > tailored to the "actors" we've talked about in the past. An individual
> > SOC
> > > analyst will have a very different set of interests than would a
> reseller
> > > that wants to build on top of our platform to expose new modules and
> > > functionality to those SOC analyst. And we can, and do, currently
> support
> > > both.
> > >
> > >
> > > On Sat, Aug 18, 2018, 9:34 AM Nick Allen  wrote:
> > >
> > > > Yes, I imagine just a separate top level directory which would
> contain
> > > the
> > > > docs.
> > > >
> > > > We would need someone to survey what doc tools are out there and
> > provide
> > > > some advice.
> > > >
> > > > Maybe we could look around at other open source projects that have
> done
> > > > their docs well and emulate them.
> > > >
> > > > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson <
> > > kylerichards...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 to separating developer docs and user docs. How should we
> approach
> > > > that.
> > > > > Have a separate doc book? I haven’t had a ton of time to contribute
> > to
> > > > code
> > > > > lately but I’d be happy to help write some of these.
> > > > >
> > > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen 
> > wrote:
> > > > >
> > > > > > Personally, I think the state of our docs and web presence is an
> > > > > inhibitor
> > > > > > to growing the Metron community.  Unless we can offer concise,
> > > > compelling
> > > > > > answers to the basic questions (What can I do with Metron?  Who
> > does
> > > it
> > > > > > help? How do I do that?), potential users and contributors are
> > 

Re: [DISCUSS] Getting to a 1.0 release

[Ali Nazemian]

One thing that we could imagine for v1.0 might be an ability to extend
Metron from adding more pipelines to it. For example, being able to extend
Metron to be integrated with other endpoints more easily from Storm
perspective. For example, what if we would like to create other topologies
to write files in ORC directly rather than HDFS or index it to Druid. What
if we would like to build an automatic security response and move to become
SOAR. All these integrations can be done even now and there are other users
that may have done it already. However, providing a clear extension point
can make it easier to contribute other pipelines to the community. I think
by adding this level of extendability, Metron community will grow way
faster by adding more integration that can be available.

Cheers,
Ali

On Mon, Aug 20, 2018 at 10:50 PM Casey Stella  wrote:

> I completely agree, Mike.  Our docs are either very high level or very low
> level (and possibly stale) and, worse, aren't aimed at the actors that
> you've stated.
> I think that the HBase project does a good job of providing coherent and
> useable documentation in their "HBase Book" (see
> https://hbase.apache.org/book.html).
> It's not actor-specific, but it is coherent advice for the practical
> practitioner of HBase (both admin and developer) and speaks with one
> voice.  I think Metron's need
> is a bit different, but at the minimum some coherent docs that speaks with
> one voice and has a coherent pitch about what Metron is used for and what
> it isn't used for
> is well needed.
>
> On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Apologies for any spelling mishaps as I'm writing from my phone.
> >
> > I'm for improving our docs. I'd like to see us guide our various profiles
> > of user towards the specific documentation for the abstraction levels
> > they'll be most interested in working from. I think we should have
> platform
> > docs about how we're a broadly useful, extensible streaming analytics
> > platform for cyber security as well as docs that emphasize more narrow
> and
> > specific use cases.
> >
> > Personally, I think I see 3 potential tiers or classifications of docs.
> > These are just observations and ideas I had, not necessarily a
> prescription
> > for organizing docs:
> > - Low level tool instructions, eg
> > - how do I run the pcap toplogy and then query with the CLI and UI?
> > - Platform docs about building on top of Metron, e.g.
> > - writing custom parsers, enrichment, and threat Intel (imho we
> should
> > start to take a more opinionated view of leveraging Stellar as this
> > extension point rather than implementing new parser classes in Java)
> > - using the profiler for constructing outlier analysis use cases
> > - using MAAS for building and deploying models for use in enrichment
> > - Docs around more specific use cases that solve specific as opposed to
> > more general problems, similar to those we have in the use-cases folder.
> >
> > I think one of our challenges currently is that our docs could be better
> > tailored to the "actors" we've talked about in the past. An individual
> SOC
> > analyst will have a very different set of interests than would a reseller
> > that wants to build on top of our platform to expose new modules and
> > functionality to those SOC analyst. And we can, and do, currently support
> > both.
> >
> >
> > On Sat, Aug 18, 2018, 9:34 AM Nick Allen  wrote:
> >
> > > Yes, I imagine just a separate top level directory which would contain
> > the
> > > docs.
> > >
> > > We would need someone to survey what doc tools are out there and
> provide
> > > some advice.
> > >
> > > Maybe we could look around at other open source projects that have done
> > > their docs well and emulate them.
> > >
> > > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson <
> > kylerichards...@gmail.com>
> > > wrote:
> > >
> > > > +1 to separating developer docs and user docs. How should we approach
> > > that.
> > > > Have a separate doc book? I haven’t had a ton of time to contribute
> to
> > > code
> > > > lately but I’d be happy to help write some of these.
> > > >
> > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen 
> wrote:
> > > >
> > > > > Personally, I think the state of our docs and web presence is an
> > > > inhibitor
> > > > > to growing the Metron community.  Unless we can offer concise,
> > > compelling
> > > > > answers to the basic questions (What can I do with Metron?  Who
> does
> > it
> > > > > help? How do I do that?), potential users and contributors are
> unable
> > > to
> > > > > see the value of Metron.
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen 
> > > wrote:
> > > > >
> > > > > > I'd like to see us focus on improving our docs before a version
> > 1.0.
> > > > > > Right now we just stitch together a bunch of READMEs, which is a
> > > great
> > > > > > stride from where we started, but is not ideal.
> > > > 

Re: [DISCUSS] Getting to a 1.0 release

[Casey Stella]

I completely agree, Mike.  Our docs are either very high level or very low
level (and possibly stale) and, worse, aren't aimed at the actors that
you've stated.
I think that the HBase project does a good job of providing coherent and
useable documentation in their "HBase Book" (see
https://hbase.apache.org/book.html).
It's not actor-specific, but it is coherent advice for the practical
practitioner of HBase (both admin and developer) and speaks with one
voice.  I think Metron's need
is a bit different, but at the minimum some coherent docs that speaks with
one voice and has a coherent pitch about what Metron is used for and what
it isn't used for
is well needed.

On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Apologies for any spelling mishaps as I'm writing from my phone.
>
> I'm for improving our docs. I'd like to see us guide our various profiles
> of user towards the specific documentation for the abstraction levels
> they'll be most interested in working from. I think we should have platform
> docs about how we're a broadly useful, extensible streaming analytics
> platform for cyber security as well as docs that emphasize more narrow and
> specific use cases.
>
> Personally, I think I see 3 potential tiers or classifications of docs.
> These are just observations and ideas I had, not necessarily a prescription
> for organizing docs:
> - Low level tool instructions, eg
> - how do I run the pcap toplogy and then query with the CLI and UI?
> - Platform docs about building on top of Metron, e.g.
> - writing custom parsers, enrichment, and threat Intel (imho we should
> start to take a more opinionated view of leveraging Stellar as this
> extension point rather than implementing new parser classes in Java)
> - using the profiler for constructing outlier analysis use cases
> - using MAAS for building and deploying models for use in enrichment
> - Docs around more specific use cases that solve specific as opposed to
> more general problems, similar to those we have in the use-cases folder.
>
> I think one of our challenges currently is that our docs could be better
> tailored to the "actors" we've talked about in the past. An individual SOC
> analyst will have a very different set of interests than would a reseller
> that wants to build on top of our platform to expose new modules and
> functionality to those SOC analyst. And we can, and do, currently support
> both.
>
>
> On Sat, Aug 18, 2018, 9:34 AM Nick Allen  wrote:
>
> > Yes, I imagine just a separate top level directory which would contain
> the
> > docs.
> >
> > We would need someone to survey what doc tools are out there and provide
> > some advice.
> >
> > Maybe we could look around at other open source projects that have done
> > their docs well and emulate them.
> >
> > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson <
> kylerichards...@gmail.com>
> > wrote:
> >
> > > +1 to separating developer docs and user docs. How should we approach
> > that.
> > > Have a separate doc book? I haven’t had a ton of time to contribute to
> > code
> > > lately but I’d be happy to help write some of these.
> > >
> > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen  wrote:
> > >
> > > > Personally, I think the state of our docs and web presence is an
> > > inhibitor
> > > > to growing the Metron community.  Unless we can offer concise,
> > compelling
> > > > answers to the basic questions (What can I do with Metron?  Who does
> it
> > > > help? How do I do that?), potential users and contributors are unable
> > to
> > > > see the value of Metron.
> > > >
> > > >
> > > >
> > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen 
> > wrote:
> > > >
> > > > > I'd like to see us focus on improving our docs before a version
> 1.0.
> > > > > Right now we just stitch together a bunch of READMEs, which is a
> > great
> > > > > stride from where we started, but is not ideal.
> > > > >
> > > > > Our docs should focused on the user and use cases; What can I do
> with
> > > > > Metron?  Who does it help? How do I do that?
> > > > >
> > > > > The docs should be separate from the code base to allow for an
> > > > > organization that is focused on the user rather than the
> > > implementation.
> > > > > This allows the READMEs to focus on the developer and the
> > > implementation,
> > > > > which should make them more digestible too.  The docs should be
> > version
> > > > > controlled and maintained through PRs, just like the code.  We
> should
> > > > take
> > > > > just as much pride in our docs as we do in our code.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > > > > si...@simonellistonball.com> wrote:
> > > > >
> > > > >> Agreed, should we add TDE by default, and get the ranger policies
> on
> > > by
> > > > >> default? That leaves secured in Kafka, which would have to be
> built
> > > into
> > > > >> the consumers and producers to encrypt into the on disk Kafka
> > 

Re: [DISCUSS] Getting to a 1.0 release

[Michael Miklavcic]

Apologies for any spelling mishaps as I'm writing from my phone.

I'm for improving our docs. I'd like to see us guide our various profiles
of user towards the specific documentation for the abstraction levels
they'll be most interested in working from. I think we should have platform
docs about how we're a broadly useful, extensible streaming analytics
platform for cyber security as well as docs that emphasize more narrow and
specific use cases.

Personally, I think I see 3 potential tiers or classifications of docs.
These are just observations and ideas I had, not necessarily a prescription
for organizing docs:
- Low level tool instructions, eg
- how do I run the pcap toplogy and then query with the CLI and UI?
- Platform docs about building on top of Metron, e.g.
- writing custom parsers, enrichment, and threat Intel (imho we should
start to take a more opinionated view of leveraging Stellar as this
extension point rather than implementing new parser classes in Java)
- using the profiler for constructing outlier analysis use cases
- using MAAS for building and deploying models for use in enrichment
- Docs around more specific use cases that solve specific as opposed to
more general problems, similar to those we have in the use-cases folder.

I think one of our challenges currently is that our docs could be better
tailored to the "actors" we've talked about in the past. An individual SOC
analyst will have a very different set of interests than would a reseller
that wants to build on top of our platform to expose new modules and
functionality to those SOC analyst. And we can, and do, currently support
both.


On Sat, Aug 18, 2018, 9:34 AM Nick Allen  wrote:

> Yes, I imagine just a separate top level directory which would contain the
> docs.
>
> We would need someone to survey what doc tools are out there and provide
> some advice.
>
> Maybe we could look around at other open source projects that have done
> their docs well and emulate them.
>
> On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson 
> wrote:
>
> > +1 to separating developer docs and user docs. How should we approach
> that.
> > Have a separate doc book? I haven’t had a ton of time to contribute to
> code
> > lately but I’d be happy to help write some of these.
> >
> > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen  wrote:
> >
> > > Personally, I think the state of our docs and web presence is an
> > inhibitor
> > > to growing the Metron community.  Unless we can offer concise,
> compelling
> > > answers to the basic questions (What can I do with Metron?  Who does it
> > > help? How do I do that?), potential users and contributors are unable
> to
> > > see the value of Metron.
> > >
> > >
> > >
> > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen 
> wrote:
> > >
> > > > I'd like to see us focus on improving our docs before a version 1.0.
> > > > Right now we just stitch together a bunch of READMEs, which is a
> great
> > > > stride from where we started, but is not ideal.
> > > >
> > > > Our docs should focused on the user and use cases; What can I do with
> > > > Metron?  Who does it help? How do I do that?
> > > >
> > > > The docs should be separate from the code base to allow for an
> > > > organization that is focused on the user rather than the
> > implementation.
> > > > This allows the READMEs to focus on the developer and the
> > implementation,
> > > > which should make them more digestible too.  The docs should be
> version
> > > > controlled and maintained through PRs, just like the code.  We should
> > > take
> > > > just as much pride in our docs as we do in our code.
> > > >
> > > >
> > > >
> > > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > > > si...@simonellistonball.com> wrote:
> > > >
> > > >> Agreed, should we add TDE by default, and get the ranger policies on
> > by
> > > >> default? That leaves secured in Kafka, which would have to be built
> > into
> > > >> the consumers and producers to encrypt into the on disk Kafka
> topics.
> > > Does
> > > >> that seem necessary to people? It would have performance
> implications
> > > for
> > > >> sure.
> > > >>
> > > >> Simon
> > > >>
> > > >> > On 15 Aug 2018, at 21:26, Otto Fowler 
> > > wrote:
> > > >> >
> > > >> > Well, I look at it like this.
> > > >> >
> > > >> > The Secure Vault was part of the original metron pitch, and many
> may
> > > >> have used that as part of their evaluations.
> > > >> > “Look, it is going to have a security vault type thing, it is on
> the
> > > >> roadmap”.
> > > >> >
> > > >> > Regardless of the implementation, conceptually, security of data
> at
> > > >> rest is important, and is a major outstanding item or the core
> metron
> > > >> proposition.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball (
> > > >> si...@simonellistonball.com) wrote:
> > > >> >>
> > > >> >> That’s going back a way. I always saw that concept as begin about
> > the
> > > >> formats, e.g. Orc, 

Re: [DISCUSS] Getting to a 1.0 release

[Nick Allen]

Yes, I imagine just a separate top level directory which would contain the
docs.

We would need someone to survey what doc tools are out there and provide
some advice.

Maybe we could look around at other open source projects that have done
their docs well and emulate them.

On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson 
wrote:

> +1 to separating developer docs and user docs. How should we approach that.
> Have a separate doc book? I haven’t had a ton of time to contribute to code
> lately but I’d be happy to help write some of these.
>
> On Sat, Aug 18, 2018 at 9:48 AM Nick Allen  wrote:
>
> > Personally, I think the state of our docs and web presence is an
> inhibitor
> > to growing the Metron community.  Unless we can offer concise, compelling
> > answers to the basic questions (What can I do with Metron?  Who does it
> > help? How do I do that?), potential users and contributors are unable to
> > see the value of Metron.
> >
> >
> >
> > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen  wrote:
> >
> > > I'd like to see us focus on improving our docs before a version 1.0.
> > > Right now we just stitch together a bunch of READMEs, which is a great
> > > stride from where we started, but is not ideal.
> > >
> > > Our docs should focused on the user and use cases; What can I do with
> > > Metron?  Who does it help? How do I do that?
> > >
> > > The docs should be separate from the code base to allow for an
> > > organization that is focused on the user rather than the
> implementation.
> > > This allows the READMEs to focus on the developer and the
> implementation,
> > > which should make them more digestible too.  The docs should be version
> > > controlled and maintained through PRs, just like the code.  We should
> > take
> > > just as much pride in our docs as we do in our code.
> > >
> > >
> > >
> > > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > > si...@simonellistonball.com> wrote:
> > >
> > >> Agreed, should we add TDE by default, and get the ranger policies on
> by
> > >> default? That leaves secured in Kafka, which would have to be built
> into
> > >> the consumers and producers to encrypt into the on disk Kafka topics.
> > Does
> > >> that seem necessary to people? It would have performance implications
> > for
> > >> sure.
> > >>
> > >> Simon
> > >>
> > >> > On 15 Aug 2018, at 21:26, Otto Fowler 
> > wrote:
> > >> >
> > >> > Well, I look at it like this.
> > >> >
> > >> > The Secure Vault was part of the original metron pitch, and many may
> > >> have used that as part of their evaluations.
> > >> > “Look, it is going to have a security vault type thing, it is on the
> > >> roadmap”.
> > >> >
> > >> > Regardless of the implementation, conceptually, security of data at
> > >> rest is important, and is a major outstanding item or the core metron
> > >> proposition.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball (
> > >> si...@simonellistonball.com) wrote:
> > >> >>
> > >> >> That’s going back a way. I always saw that concept as begin about
> the
> > >> formats, e.g. Orc, and meta data around it plus the data service api
> to
> > get
> > >> at it. I’m all for that too, but think it needs more thought than the
> > >> ticket captures.
> > >> >>
> > >> >> Simon
> > >> >>
> > >> >> On 15 Aug 2018, at 20:53, Otto Fowler 
> > wrote:
> > >> >>
> > >> >>> https://issues.apache.org/jira/browse/METRON-343
> > >> >>>
> > >>  On August 15, 2018 at 15:47:24, Simon Elliston Ball (
> > >> si...@simonellistonball.com) wrote:
> > >> 
> > >>  What would you see as secure? I’ve seen people use TDE for the
> HDFS
> > >> store, but it’s harder to encrypt storage with solr / es. Something I
> > was
> > >> thinking of doing to follow up on the Knox Feature was to add Ranger
> > >> integration for securing and auditing configs, and potentially
> > extending to
> > >> the index destinations. Do you think that would cover the secure
> storage
> > >> concept?
> > >> 
> > >>  Simon
> > >> 
> > >>  > On 15 Aug 2018, at 20:39, Otto Fowler  >
> > >> wrote:
> > >>  >
> > >>  > Secure storage off the top of my head
> > >>  >
> > >>  > On August 15, 2018 at 14:49:26, zeo...@gmail.com (
> > zeo...@gmail.com)
> > >> wrote:
> > >>  >
> > >>  > So, as has been discussed in a few
> > >>  > <
> > >>  > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a
> > >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
> > >>  >
> > >>  > other
> > >>  > <
> > >>  > https://lists.apache.org/thread.html/427a20c22207e84331b94e8
> > >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
> > >>  >
> > >>  > recent dev list threads, I would like to discuss what a Metron
> > 1.0
> > >> release
> > >>  > looks like.
> > >>  >
> > >>  > In order to kick off the conversation, I would like to make a
> few
> > >>  > suggestions regarding "what 1.0 means to me," but 

Re: [DISCUSS] Getting to a 1.0 release

[Kyle Richardson]

+1 to separating developer docs and user docs. How should we approach that.
Have a separate doc book? I haven’t had a ton of time to contribute to code
lately but I’d be happy to help write some of these.

On Sat, Aug 18, 2018 at 9:48 AM Nick Allen  wrote:

> Personally, I think the state of our docs and web presence is an inhibitor
> to growing the Metron community.  Unless we can offer concise, compelling
> answers to the basic questions (What can I do with Metron?  Who does it
> help? How do I do that?), potential users and contributors are unable to
> see the value of Metron.
>
>
>
> On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen  wrote:
>
> > I'd like to see us focus on improving our docs before a version 1.0.
> > Right now we just stitch together a bunch of READMEs, which is a great
> > stride from where we started, but is not ideal.
> >
> > Our docs should focused on the user and use cases; What can I do with
> > Metron?  Who does it help? How do I do that?
> >
> > The docs should be separate from the code base to allow for an
> > organization that is focused on the user rather than the implementation.
> > This allows the READMEs to focus on the developer and the implementation,
> > which should make them more digestible too.  The docs should be version
> > controlled and maintained through PRs, just like the code.  We should
> take
> > just as much pride in our docs as we do in our code.
> >
> >
> >
> > On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> Agreed, should we add TDE by default, and get the ranger policies on by
> >> default? That leaves secured in Kafka, which would have to be built into
> >> the consumers and producers to encrypt into the on disk Kafka topics.
> Does
> >> that seem necessary to people? It would have performance implications
> for
> >> sure.
> >>
> >> Simon
> >>
> >> > On 15 Aug 2018, at 21:26, Otto Fowler 
> wrote:
> >> >
> >> > Well, I look at it like this.
> >> >
> >> > The Secure Vault was part of the original metron pitch, and many may
> >> have used that as part of their evaluations.
> >> > “Look, it is going to have a security vault type thing, it is on the
> >> roadmap”.
> >> >
> >> > Regardless of the implementation, conceptually, security of data at
> >> rest is important, and is a major outstanding item or the core metron
> >> proposition.
> >> >
> >> >
> >> >
> >> >
> >> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball (
> >> si...@simonellistonball.com) wrote:
> >> >>
> >> >> That’s going back a way. I always saw that concept as begin about the
> >> formats, e.g. Orc, and meta data around it plus the data service api to
> get
> >> at it. I’m all for that too, but think it needs more thought than the
> >> ticket captures.
> >> >>
> >> >> Simon
> >> >>
> >> >> On 15 Aug 2018, at 20:53, Otto Fowler 
> wrote:
> >> >>
> >> >>> https://issues.apache.org/jira/browse/METRON-343
> >> >>>
> >>  On August 15, 2018 at 15:47:24, Simon Elliston Ball (
> >> si...@simonellistonball.com) wrote:
> >> 
> >>  What would you see as secure? I’ve seen people use TDE for the HDFS
> >> store, but it’s harder to encrypt storage with solr / es. Something I
> was
> >> thinking of doing to follow up on the Knox Feature was to add Ranger
> >> integration for securing and auditing configs, and potentially
> extending to
> >> the index destinations. Do you think that would cover the secure storage
> >> concept?
> >> 
> >>  Simon
> >> 
> >>  > On 15 Aug 2018, at 20:39, Otto Fowler 
> >> wrote:
> >>  >
> >>  > Secure storage off the top of my head
> >>  >
> >>  > On August 15, 2018 at 14:49:26, zeo...@gmail.com (
> zeo...@gmail.com)
> >> wrote:
> >>  >
> >>  > So, as has been discussed in a few
> >>  > <
> >>  > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a
> >> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
> >>  >
> >>  > other
> >>  > <
> >>  > https://lists.apache.org/thread.html/427a20c22207e84331b94e8
> >> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
> >>  >
> >>  > recent dev list threads, I would like to discuss what a Metron
> 1.0
> >> release
> >>  > looks like.
> >>  >
> >>  > In order to kick off the conversation, I would like to make a few
> >>  > suggestions regarding "what 1.0 means to me," but I'm very
> >> interested to
> >>  > hear everybody else's opinions.
> >>  >
> >>  > In order to go 1.0 I believe we should have:
> >>  > 1. A clear, supported method of upgrading from one version of
> >> Metron to the
> >>  > next. We have attempted
> >>  >  to
> >> make this
> >>  > easier in the past, but it is currently not
> >>  > <
> >>  > https://github.com/apache/metron/tree/master/metron-deployme
> >> nt/packaging/ambari/metron-mpack#limitations>
> >>  >
> >>  > supported
> 

Re: [DISCUSS] Getting to a 1.0 release

[Nick Allen]

Personally, I think the state of our docs and web presence is an inhibitor
to growing the Metron community.  Unless we can offer concise, compelling
answers to the basic questions (What can I do with Metron?  Who does it
help? How do I do that?), potential users and contributors are unable to
see the value of Metron.



On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen  wrote:

> I'd like to see us focus on improving our docs before a version 1.0.
> Right now we just stitch together a bunch of READMEs, which is a great
> stride from where we started, but is not ideal.
>
> Our docs should focused on the user and use cases; What can I do with
> Metron?  Who does it help? How do I do that?
>
> The docs should be separate from the code base to allow for an
> organization that is focused on the user rather than the implementation.
> This allows the READMEs to focus on the developer and the implementation,
> which should make them more digestible too.  The docs should be version
> controlled and maintained through PRs, just like the code.  We should take
> just as much pride in our docs as we do in our code.
>
>
>
> On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> Agreed, should we add TDE by default, and get the ranger policies on by
>> default? That leaves secured in Kafka, which would have to be built into
>> the consumers and producers to encrypt into the on disk Kafka topics. Does
>> that seem necessary to people? It would have performance implications for
>> sure.
>>
>> Simon
>>
>> > On 15 Aug 2018, at 21:26, Otto Fowler  wrote:
>> >
>> > Well, I look at it like this.
>> >
>> > The Secure Vault was part of the original metron pitch, and many may
>> have used that as part of their evaluations.
>> > “Look, it is going to have a security vault type thing, it is on the
>> roadmap”.
>> >
>> > Regardless of the implementation, conceptually, security of data at
>> rest is important, and is a major outstanding item or the core metron
>> proposition.
>> >
>> >
>> >
>> >
>> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>> >>
>> >> That’s going back a way. I always saw that concept as begin about the
>> formats, e.g. Orc, and meta data around it plus the data service api to get
>> at it. I’m all for that too, but think it needs more thought than the
>> ticket captures.
>> >>
>> >> Simon
>> >>
>> >> On 15 Aug 2018, at 20:53, Otto Fowler  wrote:
>> >>
>> >>> https://issues.apache.org/jira/browse/METRON-343
>> >>>
>>  On August 15, 2018 at 15:47:24, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>> 
>>  What would you see as secure? I’ve seen people use TDE for the HDFS
>> store, but it’s harder to encrypt storage with solr / es. Something I was
>> thinking of doing to follow up on the Knox Feature was to add Ranger
>> integration for securing and auditing configs, and potentially extending to
>> the index destinations. Do you think that would cover the secure storage
>> concept?
>> 
>>  Simon
>> 
>>  > On 15 Aug 2018, at 20:39, Otto Fowler 
>> wrote:
>>  >
>>  > Secure storage off the top of my head
>>  >
>>  > On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com)
>> wrote:
>>  >
>>  > So, as has been discussed in a few
>>  > <
>>  > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23a
>> c3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
>>  >
>>  > other
>>  > <
>>  > https://lists.apache.org/thread.html/427a20c22207e84331b94e8
>> ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
>>  >
>>  > recent dev list threads, I would like to discuss what a Metron 1.0
>> release
>>  > looks like.
>>  >
>>  > In order to kick off the conversation, I would like to make a few
>>  > suggestions regarding "what 1.0 means to me," but I'm very
>> interested to
>>  > hear everybody else's opinions.
>>  >
>>  > In order to go 1.0 I believe we should have:
>>  > 1. A clear, supported method of upgrading from one version of
>> Metron to the
>>  > next. We have attempted
>>  >  to
>> make this
>>  > easier in the past, but it is currently not
>>  > <
>>  > https://github.com/apache/metron/tree/master/metron-deployme
>> nt/packaging/ambari/metron-mpack#limitations>
>>  >
>>  > supported
>>  > <
>>  > https://github.com/apache/metron/tree/master/metron-deployme
>> nt/packaging/ambari/elasticsearch-mpack#limitations>
>>  >
>>  > .
>>  > 2. Authentication for all of the UIs and APIs should be secure and
>> support
>>  > SSO. I believe this is in progress via METRON-1663
>>  > .
>>  > 3. Each of our personas
>>  > <
>>  > https://cwiki.apache.org/confluence/display/METRON/Metron+
>> 

Re: [DISCUSS] Getting to a 1.0 release

[Nick Allen]

I'd like to see us focus on improving our docs before a version 1.0.  Right
now we just stitch together a bunch of READMEs, which is a great stride
from where we started, but is not ideal.

Our docs should focused on the user and use cases; What can I do with
Metron?  Who does it help? How do I do that?

The docs should be separate from the code base to allow for an organization
that is focused on the user rather than the implementation.  This allows
the READMEs to focus on the developer and the implementation, which should
make them more digestible too.  The docs should be version controlled and
maintained through PRs, just like the code.  We should take just as much
pride in our docs as we do in our code.



On Wed, Aug 15, 2018 at 4:35 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Agreed, should we add TDE by default, and get the ranger policies on by
> default? That leaves secured in Kafka, which would have to be built into
> the consumers and producers to encrypt into the on disk Kafka topics. Does
> that seem necessary to people? It would have performance implications for
> sure.
>
> Simon
>
> > On 15 Aug 2018, at 21:26, Otto Fowler  wrote:
> >
> > Well, I look at it like this.
> >
> > The Secure Vault was part of the original metron pitch, and many may
> have used that as part of their evaluations.
> > “Look, it is going to have a security vault type thing, it is on the
> roadmap”.
> >
> > Regardless of the implementation, conceptually, security of data at rest
> is important, and is a major outstanding item or the core metron
> proposition.
> >
> >
> >
> >
> >> On August 15, 2018 at 16:03:19, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
> >>
> >> That’s going back a way. I always saw that concept as begin about the
> formats, e.g. Orc, and meta data around it plus the data service api to get
> at it. I’m all for that too, but think it needs more thought than the
> ticket captures.
> >>
> >> Simon
> >>
> >> On 15 Aug 2018, at 20:53, Otto Fowler  wrote:
> >>
> >>> https://issues.apache.org/jira/browse/METRON-343
> >>>
>  On August 15, 2018 at 15:47:24, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
> 
>  What would you see as secure? I’ve seen people use TDE for the HDFS
> store, but it’s harder to encrypt storage with solr / es. Something I was
> thinking of doing to follow up on the Knox Feature was to add Ranger
> integration for securing and auditing configs, and potentially extending to
> the index destinations. Do you think that would cover the secure storage
> concept?
> 
>  Simon
> 
>  > On 15 Aug 2018, at 20:39, Otto Fowler 
> wrote:
>  >
>  > Secure storage off the top of my head
>  >
>  > On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>  >
>  > So, as has been discussed in a few
>  > <
>  > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca0
> 4c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
>  >
>  > other
>  > <
>  > https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a41
> 72a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
>  >
>  > recent dev list threads, I would like to discuss what a Metron 1.0
> release
>  > looks like.
>  >
>  > In order to kick off the conversation, I would like to make a few
>  > suggestions regarding "what 1.0 means to me," but I'm very
> interested to
>  > hear everybody else's opinions.
>  >
>  > In order to go 1.0 I believe we should have:
>  > 1. A clear, supported method of upgrading from one version of
> Metron to the
>  > next. We have attempted
>  >  to
> make this
>  > easier in the past, but it is currently not
>  > <
>  > https://github.com/apache/metron/tree/master/metron-
> deployment/packaging/ambari/metron-mpack#limitations>
>  >
>  > supported
>  > <
>  > https://github.com/apache/metron/tree/master/metron-
> deployment/packaging/ambari/elasticsearch-mpack#limitations>
>  >
>  > .
>  > 2. Authentication for all of the UIs and APIs should be secure and
> support
>  > SSO. I believe this is in progress via METRON-1663
>  > .
>  > 3. Each of our personas
>  > <
>  > https://cwiki.apache.org/confluence/display/METRON/
> Metron+User+Personas+And+Benefits>
>  >
>  > should
>  > be well documented, understood, and supported.
>  > - The current state of documentation is, in my opinion, inadequate
> and I
>  > admit I am partially to blame for this. I suggest we define a strict
>  > approach for documentation, align to it (such as perhaps migrating
> all
>  > useful wiki documentation to git), and enforce it.
>  > - I would consider METRON-1699
>  >  as a critical

Re: [DISCUSS] Getting to a 1.0 release

[Simon Elliston Ball]

Agreed, should we add TDE by default, and get the ranger policies on by 
default? That leaves secured in Kafka, which would have to be built into the 
consumers and producers to encrypt into the on disk Kafka topics. Does that 
seem necessary to people? It would have performance implications for sure. 

Simon

> On 15 Aug 2018, at 21:26, Otto Fowler  wrote:
> 
> Well, I look at it like this.
> 
> The Secure Vault was part of the original metron pitch, and many may have 
> used that as part of their evaluations.
> “Look, it is going to have a security vault type thing, it is on the roadmap”.
> 
> Regardless of the implementation, conceptually, security of data at rest is 
> important, and is a major outstanding item or the core metron proposition.
> 
> 
> 
> 
>> On August 15, 2018 at 16:03:19, Simon Elliston Ball 
>> (si...@simonellistonball.com) wrote:
>> 
>> That’s going back a way. I always saw that concept as begin about the 
>> formats, e.g. Orc, and meta data around it plus the data service api to get 
>> at it. I’m all for that too, but think it needs more thought than the ticket 
>> captures. 
>> 
>> Simon
>> 
>> On 15 Aug 2018, at 20:53, Otto Fowler  wrote:
>> 
>>> https://issues.apache.org/jira/browse/METRON-343
>>> 
 On August 15, 2018 at 15:47:24, Simon Elliston Ball 
 (si...@simonellistonball.com) wrote:
 
 What would you see as secure? I’ve seen people use TDE for the HDFS store, 
 but it’s harder to encrypt storage with solr / es. Something I was 
 thinking of doing to follow up on the Knox Feature was to add Ranger 
 integration for securing and auditing configs, and potentially extending 
 to the index destinations. Do you think that would cover the secure 
 storage concept?
 
 Simon
 
 > On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
 >
 > Secure storage off the top of my head
 >
 > On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) 
 > wrote:
 >
 > So, as has been discussed in a few
 > <
 > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
 >
 > other
 > <
 > https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
 >
 > recent dev list threads, I would like to discuss what a Metron 1.0 
 > release
 > looks like.
 >
 > In order to kick off the conversation, I would like to make a few
 > suggestions regarding "what 1.0 means to me," but I'm very interested to
 > hear everybody else's opinions.
 >
 > In order to go 1.0 I believe we should have:
 > 1. A clear, supported method of upgrading from one version of Metron to 
 > the
 > next. We have attempted
 >  to make this
 > easier in the past, but it is currently not
 > <
 > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>
 >
 > supported
 > <
 > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>
 >
 > .
 > 2. Authentication for all of the UIs and APIs should be secure and 
 > support
 > SSO. I believe this is in progress via METRON-1663
 > .
 > 3. Each of our personas
 > <
 > https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>
 >
 > should
 > be well documented, understood, and supported.
 > - The current state of documentation is, in my opinion, inadequate and I
 > admit I am partially to blame for this. I suggest we define a strict
 > approach for documentation, align to it (such as perhaps migrating all
 > useful wiki documentation to git), and enforce it.
 > - I would consider METRON-1699
 >  as a critical item 
 > for
 > a Security Data Scientist, but it is currently not clear to me where the
 > line exists between some of the other personas, or that each persona has
 > been sufficiently implemented.
 > 4. A performance tuning guide should be available for all of the main
 > components, whether as an independent document or as a part of a larger
 > document.
 > 5. Simple data ingest.
 > - Similar to the ongoing conversation for NiFi integration
 > <
 > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
 >
 > we should be able to say that we have broken down the barriers to getting
 > data into a Metron cluster in easy and efficient ways. In addition to
 > NiFi, having support for other popular tools such as beats
 > , fluentd 
 > 

Re: [DISCUSS] Getting to a 1.0 release

[Otto Fowler]

Well, I look at it like this.

The Secure Vault was part of the original metron pitch, and many may have
used that as part of their evaluations.
“Look, it is going to have a security vault type thing, it is on the
roadmap”.

Regardless of the implementation, conceptually, security of data at rest is
important, and is a major outstanding item or the core metron proposition.




On August 15, 2018 at 16:03:19, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

That’s going back a way. I always saw that concept as begin about the
formats, e.g. Orc, and meta data around it plus the data service api to get
at it. I’m all for that too, but think it needs more thought than the
ticket captures.

Simon

On 15 Aug 2018, at 20:53, Otto Fowler  wrote:

https://issues.apache.org/jira/browse/METRON-343

On August 15, 2018 at 15:47:24, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

What would you see as secure? I’ve seen people use TDE for the HDFS store,
but it’s harder to encrypt storage with solr / es. Something I was thinking
of doing to follow up on the Knox Feature was to add Ranger integration for
securing and auditing configs, and potentially extending to the index
destinations. Do you think that would cover the secure storage concept?

Simon

> On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
>
> Secure storage off the top of my head
>
> On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> So, as has been discussed in a few
> <
>
https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E
>
>
> other
> <
>
https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E
>
>
> recent dev list threads, I would like to discuss what a Metron 1.0 release
> looks like.
>
> In order to kick off the conversation, I would like to make a few
> suggestions regarding "what 1.0 means to me," but I'm very interested to
> hear everybody else's opinions.
>
> In order to go 1.0 I believe we should have:
> 1. A clear, supported method of upgrading from one version of Metron to
the
> next. We have attempted
>  to make this
> easier in the past, but it is currently not
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations
>
>
> supported
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations
>
>
> .
> 2. Authentication for all of the UIs and APIs should be secure and support
> SSO. I believe this is in progress via METRON-1663
> .
> 3. Each of our personas
> <
>
https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits
>
>
> should
> be well documented, understood, and supported.
> - The current state of documentation is, in my opinion, inadequate and I
> admit I am partially to blame for this. I suggest we define a strict
> approach for documentation, align to it (such as perhaps migrating all
> useful wiki documentation to git), and enforce it.
> - I would consider METRON-1699
>  as a critical item for
> a Security Data Scientist, but it is currently not clear to me where the
> line exists between some of the other personas, or that each persona has
> been sufficiently implemented.
> 4. A performance tuning guide should be available for all of the main
> components, whether as an independent document or as a part of a larger
> document.
> 5. Simple data ingest.
> - Similar to the ongoing conversation for NiFi integration
> <
>
https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E
>,
>
> we should be able to say that we have broken down the barriers to getting
> data into a Metron cluster in easy and efficient ways. In addition to
> NiFi, having support for other popular tools such as beats
> , fluentd ,
>
> etc.
> - Parsers should be pluggable, with independent tests and the ability to
> make versioned modifications with roll-backs.
>
> What else? Are any of these items not necessary for a 1.0?
>
> Jon
> --
>
> Jon

Re: [DISCUSS] Getting to a 1.0 release

[Simon Elliston Ball]

+1 to that. That’s more the TDE bit, we would also need Kafka SSL, and the Knox 
stuff (METRON-1663 adds SSL to all the UI and rest api stuff)

Simon

> On 15 Aug 2018, at 21:03, Otto Fowler  wrote:
> 
> https://issues.apache.org/jira/browse/METRON-106
> At least making sure it is met and closing it
> 
> 
> 
>> On August 15, 2018 at 15:53:02, Otto Fowler (ottobackwa...@gmail.com) wrote:
>> 
>> https://issues.apache.org/jira/browse/METRON-343
>> 
>>> On August 15, 2018 at 15:47:24, Simon Elliston Ball 
>>> (si...@simonellistonball.com) wrote:
>>> 
>>> What would you see as secure? I’ve seen people use TDE for the HDFS store, 
>>> but it’s harder to encrypt storage with solr / es. Something I was thinking 
>>> of doing to follow up on the Knox Feature was to add Ranger integration for 
>>> securing and auditing configs, and potentially extending to the index 
>>> destinations. Do you think that would cover the secure storage concept?
>>> 
>>> Simon
>>> 
>>> > On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
>>> >
>>> > Secure storage off the top of my head
>>> >
>>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:
>>> >
>>> > So, as has been discussed in a few
>>> > <
>>> > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
>>> >
>>> > other
>>> > <
>>> > https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
>>> >
>>> > recent dev list threads, I would like to discuss what a Metron 1.0 release
>>> > looks like.
>>> >
>>> > In order to kick off the conversation, I would like to make a few
>>> > suggestions regarding "what 1.0 means to me," but I'm very interested to
>>> > hear everybody else's opinions.
>>> >
>>> > In order to go 1.0 I believe we should have:
>>> > 1. A clear, supported method of upgrading from one version of Metron to 
>>> > the
>>> > next. We have attempted
>>> >  to make this
>>> > easier in the past, but it is currently not
>>> > <
>>> > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>
>>> >
>>> > supported
>>> > <
>>> > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>
>>> >
>>> > .
>>> > 2. Authentication for all of the UIs and APIs should be secure and support
>>> > SSO. I believe this is in progress via METRON-1663
>>> > .
>>> > 3. Each of our personas
>>> > <
>>> > https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>
>>> >
>>> > should
>>> > be well documented, understood, and supported.
>>> > - The current state of documentation is, in my opinion, inadequate and I
>>> > admit I am partially to blame for this. I suggest we define a strict
>>> > approach for documentation, align to it (such as perhaps migrating all
>>> > useful wiki documentation to git), and enforce it.
>>> > - I would consider METRON-1699
>>> >  as a critical item for
>>> > a Security Data Scientist, but it is currently not clear to me where the
>>> > line exists between some of the other personas, or that each persona has
>>> > been sufficiently implemented.
>>> > 4. A performance tuning guide should be available for all of the main
>>> > components, whether as an independent document or as a part of a larger
>>> > document.
>>> > 5. Simple data ingest.
>>> > - Similar to the ongoing conversation for NiFi integration
>>> > <
>>> > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
>>> >
>>> > we should be able to say that we have broken down the barriers to getting
>>> > data into a Metron cluster in easy and efficient ways. In addition to
>>> > NiFi, having support for other popular tools such as beats
>>> > , fluentd 
>>> > ,
>>> >
>>> > etc.
>>> > - Parsers should be pluggable, with independent tests and the ability to
>>> > make versioned modifications with roll-backs.
>>> >
>>> > What else? Are any of these items not necessary for a 1.0?
>>> >
>>> > Jon
>>> > --
>>> >
>>> > Jon

Re: [DISCUSS] Getting to a 1.0 release

[Simon Elliston Ball]

That’s going back a way. I always saw that concept as begin about the formats, 
e.g. Orc, and meta data around it plus the data service api to get at it. I’m 
all for that too, but think it needs more thought than the ticket captures. 

Simon

> On 15 Aug 2018, at 20:53, Otto Fowler  wrote:
> 
> https://issues.apache.org/jira/browse/METRON-343
> 
>> On August 15, 2018 at 15:47:24, Simon Elliston Ball 
>> (si...@simonellistonball.com) wrote:
>> 
>> What would you see as secure? I’ve seen people use TDE for the HDFS store, 
>> but it’s harder to encrypt storage with solr / es. Something I was thinking 
>> of doing to follow up on the Knox Feature was to add Ranger integration for 
>> securing and auditing configs, and potentially extending to the index 
>> destinations. Do you think that would cover the secure storage concept? 
>> 
>> Simon 
>> 
>> > On 15 Aug 2018, at 20:39, Otto Fowler  wrote: 
>> > 
>> > Secure storage off the top of my head 
>> > 
>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote: 
>> > 
>> > So, as has been discussed in a few 
>> > < 
>> > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
>> >  
>> > 
>> > other 
>> > < 
>> > https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
>> >  
>> > 
>> > recent dev list threads, I would like to discuss what a Metron 1.0 release 
>> > looks like. 
>> > 
>> > In order to kick off the conversation, I would like to make a few 
>> > suggestions regarding "what 1.0 means to me," but I'm very interested to 
>> > hear everybody else's opinions. 
>> > 
>> > In order to go 1.0 I believe we should have: 
>> > 1. A clear, supported method of upgrading from one version of Metron to 
>> > the 
>> > next. We have attempted 
>> >  to make this 
>> > easier in the past, but it is currently not 
>> > < 
>> > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>
>> >  
>> > 
>> > supported 
>> > < 
>> > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>
>> >  
>> > 
>> > . 
>> > 2. Authentication for all of the UIs and APIs should be secure and support 
>> > SSO. I believe this is in progress via METRON-1663 
>> > . 
>> > 3. Each of our personas 
>> > < 
>> > https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>
>> >  
>> > 
>> > should 
>> > be well documented, understood, and supported. 
>> > - The current state of documentation is, in my opinion, inadequate and I 
>> > admit I am partially to blame for this. I suggest we define a strict 
>> > approach for documentation, align to it (such as perhaps migrating all 
>> > useful wiki documentation to git), and enforce it. 
>> > - I would consider METRON-1699 
>> >  as a critical item for 
>> > a Security Data Scientist, but it is currently not clear to me where the 
>> > line exists between some of the other personas, or that each persona has 
>> > been sufficiently implemented. 
>> > 4. A performance tuning guide should be available for all of the main 
>> > components, whether as an independent document or as a part of a larger 
>> > document. 
>> > 5. Simple data ingest. 
>> > - Similar to the ongoing conversation for NiFi integration 
>> > < 
>> > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
>> >  
>> > 
>> > we should be able to say that we have broken down the barriers to getting 
>> > data into a Metron cluster in easy and efficient ways. In addition to 
>> > NiFi, having support for other popular tools such as beats 
>> > , fluentd 
>> > , 
>> > 
>> > etc. 
>> > - Parsers should be pluggable, with independent tests and the ability to 
>> > make versioned modifications with roll-backs. 
>> > 
>> > What else? Are any of these items not necessary for a 1.0? 
>> > 
>> > Jon 
>> > -- 
>> > 
>> > Jon 

Re: [DISCUSS] Getting to a 1.0 release

[Otto Fowler]

https://issues.apache.org/jira/browse/METRON-106
At least making sure it is met and closing it



On August 15, 2018 at 15:53:02, Otto Fowler (ottobackwa...@gmail.com) wrote:

https://issues.apache.org/jira/browse/METRON-343

On August 15, 2018 at 15:47:24, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

What would you see as secure? I’ve seen people use TDE for the HDFS store,
but it’s harder to encrypt storage with solr / es. Something I was thinking
of doing to follow up on the Knox Feature was to add Ranger integration for
securing and auditing configs, and potentially extending to the index
destinations. Do you think that would cover the secure storage concept?

Simon

> On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
>
> Secure storage off the top of my head
>
> On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> So, as has been discussed in a few
> <
>
https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E
>
>
> other
> <
>
https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E
>
>
> recent dev list threads, I would like to discuss what a Metron 1.0 release
> looks like.
>
> In order to kick off the conversation, I would like to make a few
> suggestions regarding "what 1.0 means to me," but I'm very interested to
> hear everybody else's opinions.
>
> In order to go 1.0 I believe we should have:
> 1. A clear, supported method of upgrading from one version of Metron to
the
> next. We have attempted
>  to make this
> easier in the past, but it is currently not
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations
>
>
> supported
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations
>
>
> .
> 2. Authentication for all of the UIs and APIs should be secure and support
> SSO. I believe this is in progress via METRON-1663
> .
> 3. Each of our personas
> <
>
https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits
>
>
> should
> be well documented, understood, and supported.
> - The current state of documentation is, in my opinion, inadequate and I
> admit I am partially to blame for this. I suggest we define a strict
> approach for documentation, align to it (such as perhaps migrating all
> useful wiki documentation to git), and enforce it.
> - I would consider METRON-1699
>  as a critical item for
> a Security Data Scientist, but it is currently not clear to me where the
> line exists between some of the other personas, or that each persona has
> been sufficiently implemented.
> 4. A performance tuning guide should be available for all of the main
> components, whether as an independent document or as a part of a larger
> document.
> 5. Simple data ingest.
> - Similar to the ongoing conversation for NiFi integration
> <
>
https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E
>,
>
> we should be able to say that we have broken down the barriers to getting
> data into a Metron cluster in easy and efficient ways. In addition to
> NiFi, having support for other popular tools such as beats
> , fluentd ,
>
> etc.
> - Parsers should be pluggable, with independent tests and the ability to
> make versioned modifications with roll-backs.
>
> What else? Are any of these items not necessary for a 1.0?
>
> Jon
> --
>
> Jon

Re: [DISCUSS] Getting to a 1.0 release

[Otto Fowler]

https://issues.apache.org/jira/browse/METRON-343

On August 15, 2018 at 15:47:24, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

What would you see as secure? I’ve seen people use TDE for the HDFS store,
but it’s harder to encrypt storage with solr / es. Something I was thinking
of doing to follow up on the Knox Feature was to add Ranger integration for
securing and auditing configs, and potentially extending to the index
destinations. Do you think that would cover the secure storage concept?

Simon

> On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
>
> Secure storage off the top of my head
>
> On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com)
wrote:
>
> So, as has been discussed in a few
> <
>
https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>

>
> other
> <
>
https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>

>
> recent dev list threads, I would like to discuss what a Metron 1.0
release
> looks like.
>
> In order to kick off the conversation, I would like to make a few
> suggestions regarding "what 1.0 means to me," but I'm very interested to
> hear everybody else's opinions.
>
> In order to go 1.0 I believe we should have:
> 1. A clear, supported method of upgrading from one version of Metron to
the
> next. We have attempted
>  to make this
> easier in the past, but it is currently not
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>

>
> supported
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>

>
> .
> 2. Authentication for all of the UIs and APIs should be secure and
support
> SSO. I believe this is in progress via METRON-1663
> .
> 3. Each of our personas
> <
>
https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>

>
> should
> be well documented, understood, and supported.
> - The current state of documentation is, in my opinion, inadequate and I
> admit I am partially to blame for this. I suggest we define a strict
> approach for documentation, align to it (such as perhaps migrating all
> useful wiki documentation to git), and enforce it.
> - I would consider METRON-1699
>  as a critical item
for
> a Security Data Scientist, but it is currently not clear to me where the
> line exists between some of the other personas, or that each persona has
> been sufficiently implemented.
> 4. A performance tuning guide should be available for all of the main
> components, whether as an independent document or as a part of a larger
> document.
> 5. Simple data ingest.
> - Similar to the ongoing conversation for NiFi integration
> <
>
https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,

>
> we should be able to say that we have broken down the barriers to getting
> data into a Metron cluster in easy and efficient ways. In addition to
> NiFi, having support for other popular tools such as beats
> , fluentd ,

>
> etc.
> - Parsers should be pluggable, with independent tests and the ability to
> make versioned modifications with roll-backs.
>
> What else? Are any of these items not necessary for a 1.0?
>
> Jon
> --
>
> Jon

Re: [DISCUSS] Getting to a 1.0 release

[Simon Elliston Ball]

What would you see as secure? I’ve seen people use TDE for the HDFS store, but 
it’s harder to encrypt storage with solr / es. Something I was thinking of 
doing to follow up on the Knox Feature was to add Ranger integration for 
securing and auditing configs, and potentially extending to the index 
destinations. Do you think that would cover the secure storage concept? 

Simon

> On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
> 
> Secure storage off the top of my head
> 
> On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:
> 
> So, as has been discussed in a few
> <
> https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
> 
> other
> <
> https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
> 
> recent dev list threads, I would like to discuss what a Metron 1.0 release
> looks like.
> 
> In order to kick off the conversation, I would like to make a few
> suggestions regarding "what 1.0 means to me," but I'm very interested to
> hear everybody else's opinions.
> 
> In order to go 1.0 I believe we should have:
> 1. A clear, supported method of upgrading from one version of Metron to the
> next. We have attempted
>  to make this
> easier in the past, but it is currently not
> <
> https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>
> 
> supported
> <
> https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>
> 
> .
> 2. Authentication for all of the UIs and APIs should be secure and support
> SSO. I believe this is in progress via METRON-1663
> .
> 3. Each of our personas
> <
> https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>
> 
> should
> be well documented, understood, and supported.
> - The current state of documentation is, in my opinion, inadequate and I
> admit I am partially to blame for this. I suggest we define a strict
> approach for documentation, align to it (such as perhaps migrating all
> useful wiki documentation to git), and enforce it.
> - I would consider METRON-1699
>  as a critical item for
> a Security Data Scientist, but it is currently not clear to me where the
> line exists between some of the other personas, or that each persona has
> been sufficiently implemented.
> 4. A performance tuning guide should be available for all of the main
> components, whether as an independent document or as a part of a larger
> document.
> 5. Simple data ingest.
> - Similar to the ongoing conversation for NiFi integration
> <
> https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
> 
> we should be able to say that we have broken down the barriers to getting
> data into a Metron cluster in easy and efficient ways. In addition to
> NiFi, having support for other popular tools such as beats
> , fluentd ,
> 
> etc.
> - Parsers should be pluggable, with independent tests and the ability to
> make versioned modifications with roll-backs.
> 
> What else? Are any of these items not necessary for a 1.0?
> 
> Jon
> -- 
> 
> Jon

Re: [DISCUSS] Getting to a 1.0 release

[Otto Fowler]

Secure storage off the top of my head

On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:

So, as has been discussed in a few
<
https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>

other
<
https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>

recent dev list threads, I would like to discuss what a Metron 1.0 release
looks like.

In order to kick off the conversation, I would like to make a few
suggestions regarding "what 1.0 means to me," but I'm very interested to
hear everybody else's opinions.

In order to go 1.0 I believe we should have:
1. A clear, supported method of upgrading from one version of Metron to the
next. We have attempted
 to make this
easier in the past, but it is currently not
<
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>

supported
<
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>

.
2. Authentication for all of the UIs and APIs should be secure and support
SSO. I believe this is in progress via METRON-1663
.
3. Each of our personas
<
https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>

should
be well documented, understood, and supported.
- The current state of documentation is, in my opinion, inadequate and I
admit I am partially to blame for this. I suggest we define a strict
approach for documentation, align to it (such as perhaps migrating all
useful wiki documentation to git), and enforce it.
- I would consider METRON-1699
 as a critical item for
a Security Data Scientist, but it is currently not clear to me where the
line exists between some of the other personas, or that each persona has
been sufficiently implemented.
4. A performance tuning guide should be available for all of the main
components, whether as an independent document or as a part of a larger
document.
5. Simple data ingest.
- Similar to the ongoing conversation for NiFi integration
<
https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,

we should be able to say that we have broken down the barriers to getting
data into a Metron cluster in easy and efficient ways. In addition to
NiFi, having support for other popular tools such as beats
, fluentd ,

etc.
- Parsers should be pluggable, with independent tests and the ability to
make versioned modifications with roll-backs.

What else? Are any of these items not necessary for a 1.0?

Jon
-- 

Jon

[DISCUSS] Getting to a 1.0 release

[zeo...@gmail.com]

So, as has been discussed in a few

other

recent dev list threads, I would like to discuss what a Metron 1.0 release
looks like.

In order to kick off the conversation, I would like to make a few
suggestions regarding "what 1.0 means to me," but I'm very interested to
hear everybody else's opinions.

In order to go 1.0 I believe we should have:
1. A clear, supported method of upgrading from one version of Metron to the
next.  We have attempted
 to make this
easier in the past, but it is currently not

supported

.
2. Authentication for all of the UIs and APIs should be secure and support
SSO.  I believe this is in progress via METRON-1663
.
3. Each of our personas

should
be well documented, understood, and supported.
 - The current state of documentation is, in my opinion, inadequate and I
admit I am partially to blame for this.  I suggest we define a strict
approach for documentation, align to it (such as perhaps migrating all
useful wiki documentation to git), and enforce it.
 - I would consider METRON-1699
 as a critical item for
a Security Data Scientist, but it is currently not clear to me where the
line exists between some of the other personas, or that each persona has
been sufficiently implemented.
4. A performance tuning guide should be available for all of the main
components, whether as an independent document or as a part of a larger
document.
5. Simple data ingest.
 - Similar to the ongoing conversation for NiFi integration
,
we should be able to say that we have broken down the barriers to getting
data into a Metron cluster in easy and efficient ways.  In addition to
NiFi, having support for other popular tools such as beats
, fluentd ,
etc.
 - Parsers should be pluggable, with independent tests and the ability to
make versioned modifications with roll-backs.

What else?  Are any of these items not necessary for a 1.0?

Jon
-- 

Jon