date:20180815

Agreed, should we add TDE by default, and get the ranger policies on by 
default? That leaves secured in Kafka, which would have to be built into the 
consumers and producers to encrypt into the on disk Kafka topics. Does that 
seem necessary to people? It would have performance implications for sure. 

Simon

> On 15 Aug 2018, at 21:26, Otto Fowler  wrote:
> 
> Well, I look at it like this.
> 
> The Secure Vault was part of the original metron pitch, and many may have 
> used that as part of their evaluations.
> “Look, it is going to have a security vault type thing, it is on the roadmap”.
> 
> Regardless of the implementation, conceptually, security of data at rest is 
> important, and is a major outstanding item or the core metron proposition.
> 
> 
> 
> 
>> On August 15, 2018 at 16:03:19, Simon Elliston Ball 
>> (si...@simonellistonball.com) wrote:
>> 
>> That’s going back a way. I always saw that concept as begin about the 
>> formats, e.g. Orc, and meta data around it plus the data service api to get 
>> at it. I’m all for that too, but think it needs more thought than the ticket 
>> captures. 
>> 
>> Simon
>> 
>> On 15 Aug 2018, at 20:53, Otto Fowler  wrote:
>> 
>>> https://issues.apache.org/jira/browse/METRON-343
>>> 
 On August 15, 2018 at 15:47:24, Simon Elliston Ball 
 (si...@simonellistonball.com) wrote:
 
 What would you see as secure? I’ve seen people use TDE for the HDFS store, 
 but it’s harder to encrypt storage with solr / es. Something I was 
 thinking of doing to follow up on the Knox Feature was to add Ranger 
 integration for securing and auditing configs, and potentially extending 
 to the index destinations. Do you think that would cover the secure 
 storage concept?
 
 Simon
 
 > On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
 >
 > Secure storage off the top of my head
 >
 > On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) 
 > wrote:
 >
 > So, as has been discussed in a few
 > <
 > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
 >
 > other
 > <
 > https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
 >
 > recent dev list threads, I would like to discuss what a Metron 1.0 
 > release
 > looks like.
 >
 > In order to kick off the conversation, I would like to make a few
 > suggestions regarding "what 1.0 means to me," but I'm very interested to
 > hear everybody else's opinions.
 >
 > In order to go 1.0 I believe we should have:
 > 1. A clear, supported method of upgrading from one version of Metron to 
 > the
 > next. We have attempted
 >  to make this
 > easier in the past, but it is currently not
 > <
 > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>
 >
 > supported
 > <
 > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>
 >
 > .
 > 2. Authentication for all of the UIs and APIs should be secure and 
 > support
 > SSO. I believe this is in progress via METRON-1663
 > .
 > 3. Each of our personas
 > <
 > https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>
 >
 > should
 > be well documented, understood, and supported.
 > - The current state of documentation is, in my opinion, inadequate and I
 > admit I am partially to blame for this. I suggest we define a strict
 > approach for documentation, align to it (such as perhaps migrating all
 > useful wiki documentation to git), and enforce it.
 > - I would consider METRON-1699
 >  as a critical item 
 > for
 > a Security Data Scientist, but it is currently not clear to me where the
 > line exists between some of the other personas, or that each persona has
 > been sufficiently implemented.
 > 4. A performance tuning guide should be available for all of the main
 > components, whether as an independent document or as a part of a larger
 > document.
 > 5. Simple data ingest.
 > - Similar to the ongoing conversation for NiFi integration
 > <
 > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
 >
 > we should be able to say that we have broken down the barriers to getting
 > data into a Metron cluster in easy and efficient ways. In addition to
 > NiFi, having support for other popular tools such as beats
 > , fluentd 
 >

Re: [DISCUSS] Getting to a 1.0 release

Well, I look at it like this.

The Secure Vault was part of the original metron pitch, and many may have
used that as part of their evaluations.
“Look, it is going to have a security vault type thing, it is on the
roadmap”.

Regardless of the implementation, conceptually, security of data at rest is
important, and is a major outstanding item or the core metron proposition.




On August 15, 2018 at 16:03:19, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

That’s going back a way. I always saw that concept as begin about the
formats, e.g. Orc, and meta data around it plus the data service api to get
at it. I’m all for that too, but think it needs more thought than the
ticket captures.

Simon

On 15 Aug 2018, at 20:53, Otto Fowler  wrote:

https://issues.apache.org/jira/browse/METRON-343

On August 15, 2018 at 15:47:24, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

What would you see as secure? I’ve seen people use TDE for the HDFS store,
but it’s harder to encrypt storage with solr / es. Something I was thinking
of doing to follow up on the Knox Feature was to add Ranger integration for
securing and auditing configs, and potentially extending to the index
destinations. Do you think that would cover the secure storage concept?

Simon

> On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
>
> Secure storage off the top of my head
>
> On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> So, as has been discussed in a few
> <
>
https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E
>
>
> other
> <
>
https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E
>
>
> recent dev list threads, I would like to discuss what a Metron 1.0 release
> looks like.
>
> In order to kick off the conversation, I would like to make a few
> suggestions regarding "what 1.0 means to me," but I'm very interested to
> hear everybody else's opinions.
>
> In order to go 1.0 I believe we should have:
> 1. A clear, supported method of upgrading from one version of Metron to
the
> next. We have attempted
>  to make this
> easier in the past, but it is currently not
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations
>
>
> supported
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations
>
>
> .
> 2. Authentication for all of the UIs and APIs should be secure and support
> SSO. I believe this is in progress via METRON-1663
> .
> 3. Each of our personas
> <
>
https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits
>
>
> should
> be well documented, understood, and supported.
> - The current state of documentation is, in my opinion, inadequate and I
> admit I am partially to blame for this. I suggest we define a strict
> approach for documentation, align to it (such as perhaps migrating all
> useful wiki documentation to git), and enforce it.
> - I would consider METRON-1699
>  as a critical item for
> a Security Data Scientist, but it is currently not clear to me where the
> line exists between some of the other personas, or that each persona has
> been sufficiently implemented.
> 4. A performance tuning guide should be available for all of the main
> components, whether as an independent document or as a part of a larger
> document.
> 5. Simple data ingest.
> - Similar to the ongoing conversation for NiFi integration
> <
>
https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E
>,
>
> we should be able to say that we have broken down the barriers to getting
> data into a Metron cluster in easy and efficient ways. In addition to
> NiFi, having support for other popular tools such as beats
> , fluentd ,
>
> etc.
> - Parsers should be pluggable, with independent tests and the ability to
> make versioned modifications with roll-backs.
>
> What else? Are any of these items not necessary for a 1.0?
>
> Jon
> --
>
> Jon

Re: [DISCUSS] Getting to a 1.0 release

+1 to that. That’s more the TDE bit, we would also need Kafka SSL, and the Knox 
stuff (METRON-1663 adds SSL to all the UI and rest api stuff)

Simon

> On 15 Aug 2018, at 21:03, Otto Fowler  wrote:
> 
> https://issues.apache.org/jira/browse/METRON-106
> At least making sure it is met and closing it
> 
> 
> 
>> On August 15, 2018 at 15:53:02, Otto Fowler (ottobackwa...@gmail.com) wrote:
>> 
>> https://issues.apache.org/jira/browse/METRON-343
>> 
>>> On August 15, 2018 at 15:47:24, Simon Elliston Ball 
>>> (si...@simonellistonball.com) wrote:
>>> 
>>> What would you see as secure? I’ve seen people use TDE for the HDFS store, 
>>> but it’s harder to encrypt storage with solr / es. Something I was thinking 
>>> of doing to follow up on the Knox Feature was to add Ranger integration for 
>>> securing and auditing configs, and potentially extending to the index 
>>> destinations. Do you think that would cover the secure storage concept?
>>> 
>>> Simon
>>> 
>>> > On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
>>> >
>>> > Secure storage off the top of my head
>>> >
>>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:
>>> >
>>> > So, as has been discussed in a few
>>> > <
>>> > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
>>> >
>>> > other
>>> > <
>>> > https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
>>> >
>>> > recent dev list threads, I would like to discuss what a Metron 1.0 release
>>> > looks like.
>>> >
>>> > In order to kick off the conversation, I would like to make a few
>>> > suggestions regarding "what 1.0 means to me," but I'm very interested to
>>> > hear everybody else's opinions.
>>> >
>>> > In order to go 1.0 I believe we should have:
>>> > 1. A clear, supported method of upgrading from one version of Metron to 
>>> > the
>>> > next. We have attempted
>>> >  to make this
>>> > easier in the past, but it is currently not
>>> > <
>>> > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>
>>> >
>>> > supported
>>> > <
>>> > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>
>>> >
>>> > .
>>> > 2. Authentication for all of the UIs and APIs should be secure and support
>>> > SSO. I believe this is in progress via METRON-1663
>>> > .
>>> > 3. Each of our personas
>>> > <
>>> > https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>
>>> >
>>> > should
>>> > be well documented, understood, and supported.
>>> > - The current state of documentation is, in my opinion, inadequate and I
>>> > admit I am partially to blame for this. I suggest we define a strict
>>> > approach for documentation, align to it (such as perhaps migrating all
>>> > useful wiki documentation to git), and enforce it.
>>> > - I would consider METRON-1699
>>> >  as a critical item for
>>> > a Security Data Scientist, but it is currently not clear to me where the
>>> > line exists between some of the other personas, or that each persona has
>>> > been sufficiently implemented.
>>> > 4. A performance tuning guide should be available for all of the main
>>> > components, whether as an independent document or as a part of a larger
>>> > document.
>>> > 5. Simple data ingest.
>>> > - Similar to the ongoing conversation for NiFi integration
>>> > <
>>> > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
>>> >
>>> > we should be able to say that we have broken down the barriers to getting
>>> > data into a Metron cluster in easy and efficient ways. In addition to
>>> > NiFi, having support for other popular tools such as beats
>>> > , fluentd 
>>> > ,
>>> >
>>> > etc.
>>> > - Parsers should be pluggable, with independent tests and the ability to
>>> > make versioned modifications with roll-backs.
>>> >
>>> > What else? Are any of these items not necessary for a 1.0?
>>> >
>>> > Jon
>>> > --
>>> >
>>> > Jon

Re: [DISCUSS] Getting to a 1.0 release

That’s going back a way. I always saw that concept as begin about the formats, 
e.g. Orc, and meta data around it plus the data service api to get at it. I’m 
all for that too, but think it needs more thought than the ticket captures. 

Simon

> On 15 Aug 2018, at 20:53, Otto Fowler  wrote:
> 
> https://issues.apache.org/jira/browse/METRON-343
> 
>> On August 15, 2018 at 15:47:24, Simon Elliston Ball 
>> (si...@simonellistonball.com) wrote:
>> 
>> What would you see as secure? I’ve seen people use TDE for the HDFS store, 
>> but it’s harder to encrypt storage with solr / es. Something I was thinking 
>> of doing to follow up on the Knox Feature was to add Ranger integration for 
>> securing and auditing configs, and potentially extending to the index 
>> destinations. Do you think that would cover the secure storage concept? 
>> 
>> Simon 
>> 
>> > On 15 Aug 2018, at 20:39, Otto Fowler  wrote: 
>> > 
>> > Secure storage off the top of my head 
>> > 
>> > On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote: 
>> > 
>> > So, as has been discussed in a few 
>> > < 
>> > https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
>> >  
>> > 
>> > other 
>> > < 
>> > https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
>> >  
>> > 
>> > recent dev list threads, I would like to discuss what a Metron 1.0 release 
>> > looks like. 
>> > 
>> > In order to kick off the conversation, I would like to make a few 
>> > suggestions regarding "what 1.0 means to me," but I'm very interested to 
>> > hear everybody else's opinions. 
>> > 
>> > In order to go 1.0 I believe we should have: 
>> > 1. A clear, supported method of upgrading from one version of Metron to 
>> > the 
>> > next. We have attempted 
>> >  to make this 
>> > easier in the past, but it is currently not 
>> > < 
>> > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>
>> >  
>> > 
>> > supported 
>> > < 
>> > https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>
>> >  
>> > 
>> > . 
>> > 2. Authentication for all of the UIs and APIs should be secure and support 
>> > SSO. I believe this is in progress via METRON-1663 
>> > . 
>> > 3. Each of our personas 
>> > < 
>> > https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>
>> >  
>> > 
>> > should 
>> > be well documented, understood, and supported. 
>> > - The current state of documentation is, in my opinion, inadequate and I 
>> > admit I am partially to blame for this. I suggest we define a strict 
>> > approach for documentation, align to it (such as perhaps migrating all 
>> > useful wiki documentation to git), and enforce it. 
>> > - I would consider METRON-1699 
>> >  as a critical item for 
>> > a Security Data Scientist, but it is currently not clear to me where the 
>> > line exists between some of the other personas, or that each persona has 
>> > been sufficiently implemented. 
>> > 4. A performance tuning guide should be available for all of the main 
>> > components, whether as an independent document or as a part of a larger 
>> > document. 
>> > 5. Simple data ingest. 
>> > - Similar to the ongoing conversation for NiFi integration 
>> > < 
>> > https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
>> >  
>> > 
>> > we should be able to say that we have broken down the barriers to getting 
>> > data into a Metron cluster in easy and efficient ways. In addition to 
>> > NiFi, having support for other popular tools such as beats 
>> > , fluentd 
>> > , 
>> > 
>> > etc. 
>> > - Parsers should be pluggable, with independent tests and the ability to 
>> > make versioned modifications with roll-backs. 
>> > 
>> > What else? Are any of these items not necessary for a 1.0? 
>> > 
>> > Jon 
>> > -- 
>> > 
>> > Jon

Re: [DISCUSS] Getting to a 1.0 release

https://issues.apache.org/jira/browse/METRON-106
At least making sure it is met and closing it



On August 15, 2018 at 15:53:02, Otto Fowler (ottobackwa...@gmail.com) wrote:

https://issues.apache.org/jira/browse/METRON-343

On August 15, 2018 at 15:47:24, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

What would you see as secure? I’ve seen people use TDE for the HDFS store,
but it’s harder to encrypt storage with solr / es. Something I was thinking
of doing to follow up on the Knox Feature was to add Ranger integration for
securing and auditing configs, and potentially extending to the index
destinations. Do you think that would cover the secure storage concept?

Simon

> On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
>
> Secure storage off the top of my head
>
> On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> So, as has been discussed in a few
> <
>
https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E
>
>
> other
> <
>
https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E
>
>
> recent dev list threads, I would like to discuss what a Metron 1.0 release
> looks like.
>
> In order to kick off the conversation, I would like to make a few
> suggestions regarding "what 1.0 means to me," but I'm very interested to
> hear everybody else's opinions.
>
> In order to go 1.0 I believe we should have:
> 1. A clear, supported method of upgrading from one version of Metron to
the
> next. We have attempted
>  to make this
> easier in the past, but it is currently not
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations
>
>
> supported
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations
>
>
> .
> 2. Authentication for all of the UIs and APIs should be secure and support
> SSO. I believe this is in progress via METRON-1663
> .
> 3. Each of our personas
> <
>
https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits
>
>
> should
> be well documented, understood, and supported.
> - The current state of documentation is, in my opinion, inadequate and I
> admit I am partially to blame for this. I suggest we define a strict
> approach for documentation, align to it (such as perhaps migrating all
> useful wiki documentation to git), and enforce it.
> - I would consider METRON-1699
>  as a critical item for
> a Security Data Scientist, but it is currently not clear to me where the
> line exists between some of the other personas, or that each persona has
> been sufficiently implemented.
> 4. A performance tuning guide should be available for all of the main
> components, whether as an independent document or as a part of a larger
> document.
> 5. Simple data ingest.
> - Similar to the ongoing conversation for NiFi integration
> <
>
https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E
>,
>
> we should be able to say that we have broken down the barriers to getting
> data into a Metron cluster in easy and efficient ways. In addition to
> NiFi, having support for other popular tools such as beats
> , fluentd ,
>
> etc.
> - Parsers should be pluggable, with independent tests and the ability to
> make versioned modifications with roll-backs.
>
> What else? Are any of these items not necessary for a 1.0?
>
> Jon
> --
>
> Jon

Re: [DISCUSS] Getting to a 1.0 release

https://issues.apache.org/jira/browse/METRON-343

On August 15, 2018 at 15:47:24, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

What would you see as secure? I’ve seen people use TDE for the HDFS store,
but it’s harder to encrypt storage with solr / es. Something I was thinking
of doing to follow up on the Knox Feature was to add Ranger integration for
securing and auditing configs, and potentially extending to the index
destinations. Do you think that would cover the secure storage concept?

Simon

> On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
>
> Secure storage off the top of my head
>
> On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com)
wrote:
>
> So, as has been discussed in a few
> <
>
https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>

>
> other
> <
>
https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>

>
> recent dev list threads, I would like to discuss what a Metron 1.0
release
> looks like.
>
> In order to kick off the conversation, I would like to make a few
> suggestions regarding "what 1.0 means to me," but I'm very interested to
> hear everybody else's opinions.
>
> In order to go 1.0 I believe we should have:
> 1. A clear, supported method of upgrading from one version of Metron to
the
> next. We have attempted
>  to make this
> easier in the past, but it is currently not
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>

>
> supported
> <
>
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>

>
> .
> 2. Authentication for all of the UIs and APIs should be secure and
support
> SSO. I believe this is in progress via METRON-1663
> .
> 3. Each of our personas
> <
>
https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>

>
> should
> be well documented, understood, and supported.
> - The current state of documentation is, in my opinion, inadequate and I
> admit I am partially to blame for this. I suggest we define a strict
> approach for documentation, align to it (such as perhaps migrating all
> useful wiki documentation to git), and enforce it.
> - I would consider METRON-1699
>  as a critical item
for
> a Security Data Scientist, but it is currently not clear to me where the
> line exists between some of the other personas, or that each persona has
> been sufficiently implemented.
> 4. A performance tuning guide should be available for all of the main
> components, whether as an independent document or as a part of a larger
> document.
> 5. Simple data ingest.
> - Similar to the ongoing conversation for NiFi integration
> <
>
https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,

>
> we should be able to say that we have broken down the barriers to getting
> data into a Metron cluster in easy and efficient ways. In addition to
> NiFi, having support for other popular tools such as beats
> , fluentd ,

>
> etc.
> - Parsers should be pluggable, with independent tests and the ability to
> make versioned modifications with roll-backs.
>
> What else? Are any of these items not necessary for a 1.0?
>
> Jon
> --
>
> Jon

Re: [DISCUSS] Getting to a 1.0 release

What would you see as secure? I’ve seen people use TDE for the HDFS store, but 
it’s harder to encrypt storage with solr / es. Something I was thinking of 
doing to follow up on the Knox Feature was to add Ranger integration for 
securing and auditing configs, and potentially extending to the index 
destinations. Do you think that would cover the secure storage concept? 

Simon

> On 15 Aug 2018, at 20:39, Otto Fowler  wrote:
> 
> Secure storage off the top of my head
> 
> On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:
> 
> So, as has been discussed in a few
> <
> https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>
> 
> other
> <
> https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>
> 
> recent dev list threads, I would like to discuss what a Metron 1.0 release
> looks like.
> 
> In order to kick off the conversation, I would like to make a few
> suggestions regarding "what 1.0 means to me," but I'm very interested to
> hear everybody else's opinions.
> 
> In order to go 1.0 I believe we should have:
> 1. A clear, supported method of upgrading from one version of Metron to the
> next. We have attempted
>  to make this
> easier in the past, but it is currently not
> <
> https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>
> 
> supported
> <
> https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>
> 
> .
> 2. Authentication for all of the UIs and APIs should be secure and support
> SSO. I believe this is in progress via METRON-1663
> .
> 3. Each of our personas
> <
> https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>
> 
> should
> be well documented, understood, and supported.
> - The current state of documentation is, in my opinion, inadequate and I
> admit I am partially to blame for this. I suggest we define a strict
> approach for documentation, align to it (such as perhaps migrating all
> useful wiki documentation to git), and enforce it.
> - I would consider METRON-1699
>  as a critical item for
> a Security Data Scientist, but it is currently not clear to me where the
> line exists between some of the other personas, or that each persona has
> been sufficiently implemented.
> 4. A performance tuning guide should be available for all of the main
> components, whether as an independent document or as a part of a larger
> document.
> 5. Simple data ingest.
> - Similar to the ongoing conversation for NiFi integration
> <
> https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,
> 
> we should be able to say that we have broken down the barriers to getting
> data into a Metron cluster in easy and efficient ways. In addition to
> NiFi, having support for other popular tools such as beats
> , fluentd ,
> 
> etc.
> - Parsers should be pluggable, with independent tests and the ability to
> make versioned modifications with roll-backs.
> 
> What else? Are any of these items not necessary for a 1.0?
> 
> Jon
> -- 
> 
> Jon

Re: [DISCUSS] Getting to a 1.0 release

Secure storage off the top of my head

On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote:

So, as has been discussed in a few
<
https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E>

other
<
https://lists.apache.org/thread.html/427a20c22207e84331b94e8ead9a4172a22577d26eb581c0e564d0dc@%3Cdev.metron.apache.org%3E>

recent dev list threads, I would like to discuss what a Metron 1.0 release
looks like.

In order to kick off the conversation, I would like to make a few
suggestions regarding "what 1.0 means to me," but I'm very interested to
hear everybody else's opinions.

In order to go 1.0 I believe we should have:
1. A clear, supported method of upgrading from one version of Metron to the
next. We have attempted
to make this
easier in the past, but it is currently not
<
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/metron-mpack#limitations>

supported
<
https://github.com/apache/metron/tree/master/metron-deployment/packaging/ambari/elasticsearch-mpack#limitations>

.
2. Authentication for all of the UIs and APIs should be secure and support
SSO. I believe this is in progress via METRON-1663
.
3. Each of our personas
<
https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits>

should
be well documented, understood, and supported.
- The current state of documentation is, in my opinion, inadequate and I
admit I am partially to blame for this. I suggest we define a strict
approach for documentation, align to it (such as perhaps migrating all
useful wiki documentation to git), and enforce it.
- I would consider METRON-1699
as a critical item for
a Security Data Scientist, but it is currently not clear to me where the
line exists between some of the other personas, or that each persona has
been sufficiently implemented.
4. A performance tuning guide should be available for all of the main
components, whether as an independent document or as a part of a larger
document.
5. Simple data ingest.
- Similar to the ongoing conversation for NiFi integration
<
https://lists.apache.org/thread.html/d7bb4d32c8c42bd40b2f26973f989bcba16010a672fd8a533a5544bf@%3Cdev.metron.apache.org%3E>,

we should be able to say that we have broken down the barriers to getting
data into a Metron cluster in easy and efficient ways. In addition to
NiFi, having support for other popular tools such as beats
, fluentd ,

etc.
- Parsers should be pluggable, with independent tests and the ability to
make versioned modifications with roll-backs.

What else? Are any of these items not necessary for a 1.0?

Jon
--

Jon

Re: [DISCUSS] Release cadence

Works for me, that would be great.

On Wed, Aug 15, 2018 at 12:22 PM Casey Stella  wrote:

> If you like, I can volunteer to kick off a discuss thread when I submit the
> board report.
>
> On Wed, Aug 15, 2018 at 2:21 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I'm also a fan of the 2-3 month time frame for releases. And I agree it
> > fits nicely with our board report. That said, I think we should minimally
> > kick off a DISCUSS at least every 2 months per the recommendations above.
> > If it's warranted, great. If not, then we bring it up at a stated later
> > time for re-evaluation.
> >
> > Fwiw, some upcoming features post-0.6.0 that I'm seeing which are also
> > large-ish and will fit nicely into the next cycle (pending completion, of
> > course):
> >
> >1. NiFi Metron parsers
> >2. Profiler enhancements - bootstrapping, etc.
> >3. Knox SSO
> >
> >
> >
> > On Wed, Aug 15, 2018 at 11:10 AM Casey Stella 
> wrote:
> >
> > > Strictly selfishly, I'd love for a release to happen quickly enough to
> > have
> > > something to announce to the board during the reports.  Once every 2
> > months
> > > or when a sufficiently complicated change happens sounds like a
> sensible
> > > cadence.
> > >
> > > I very much support a "how do we get to 1.0" discussion, maybe as a
> > > separate thread?
> > >
> > > On Wed, Aug 15, 2018 at 11:56 AM zeo...@gmail.com 
> > > wrote:
> > >
> > > > I'm a fan of a hybrid time/feature-based cadence.  Something like
> > "When 3
> > > > months has passed since our last release, or a sufficiently
> complicated
> > > > change has been introduced to master (like merging a FB), a discuss
> > > thread
> > > > is started".  I'm primarily thinking of what the upgrade path looks
> > like
> > > > (more on that in a "how do we get to 1.0" discuss).
> > > >
> > > > Jon
> > > >
> > > > On Wed, Aug 15, 2018 at 11:02 AM Justin Leet 
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > In concert with the discuss thread on a potential 0.6.0 release,
> I'd
> > > also
> > > > > like start a discussion about our release cadence.  We've generally
> > > been
> > > > > pretty relaxed around doing releases, and I'm curious what people's
> > > > > thoughts are on adopting a somewhat more regular schedule.
> > > > >
> > > > > Couple questions I think are relevant
> > > > > 1. Is this something we should work towards and, if we do, how do
> we
> > > want
> > > > > to go about it?
> > > > >
> > > > >- "Whenever someone feels like pushing out a discuss thread"?
> > > > >- "Let's just start a discuss thread every X and if we want to
> > > release
> > > > >we release"?
> > > > >- "let's try to get a release out every X and what's on the bus
> is
> > > on
> > > > >the bus"?
> > > > >- Something else?
> > > > >
> > > > > 2. Assuming we do want to do more regular releases, what's the
> > > timeframe
> > > > > we'd like to shoot for?
> > > > >
> > > > > Personally, I'd like to just start a discuss thread regularly, with
> > the
> > > > > built-in expectation that not every thread should necessarily lead
> > to a
> > > > > release. I don't want to be forcing release overhead when there's
> not
> > > > > enough to merit a release, but releasing more often than we often
> do
> > > now
> > > > > would provide a lot of values to users.
> > > > >
> > > > > In terms of timeframe, I tend to think a 2-3 month cadence for the
> > > > threads
> > > > > is reasonable. It's long enough to potentially accrue enough
> features
> > > to
> > > > > merit a release, but short enough that when we pass on a release
> > we're
> > > > > probably fine just waiting for another cycle to come around.  The
> > last
> > > > > release was ~2 months ago and we have a good amount of stuff here,
> > but
> > > I
> > > > > also don't expect two feature branches going in to be the norm.
> > > > >
> > > > > I'd expect whatever comes out of this thread to also be relatively
> > > > > informal. At least right now, I don't feel like we need a rigid
> > > schedule,
> > > > > and I'd still like people to feel encouraged to propose a release,
> > > > > particularly when there are a couple major features or critical
> > fixes.
> > > > > Alternatively, I would expect some of these discuss threads to
> > > conclude,
> > > > > "We should do a release, but let's wait a couple waits for these
> > > tickets
> > > > to
> > > > > finish up" (e.g. like the Pcap query panel).
> > > > >
> > > > > Justin
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > >
> >
>

[DISCUSS] Getting to a 1.0 release

2018-08-15 Thread zeo...@gmail.com

So, as has been discussed in a few

other

recent dev list threads, I would like to discuss what a Metron 1.0 release
looks like.

In order to kick off the conversation, I would like to make a few
suggestions regarding "what 1.0 means to me," but I'm very interested to
hear everybody else's opinions.

In order to go 1.0 I believe we should have:
1. A clear, supported method of upgrading from one version of Metron to the
next.  We have attempted
 to make this
easier in the past, but it is currently not

supported

.
2. Authentication for all of the UIs and APIs should be secure and support
SSO.  I believe this is in progress via METRON-1663
.
3. Each of our personas

should
be well documented, understood, and supported.
 - The current state of documentation is, in my opinion, inadequate and I
admit I am partially to blame for this.  I suggest we define a strict
approach for documentation, align to it (such as perhaps migrating all
useful wiki documentation to git), and enforce it.
 - I would consider METRON-1699
 as a critical item for
a Security Data Scientist, but it is currently not clear to me where the
line exists between some of the other personas, or that each persona has
been sufficiently implemented.
4. A performance tuning guide should be available for all of the main
components, whether as an independent document or as a part of a larger
document.
5. Simple data ingest.
 - Similar to the ongoing conversation for NiFi integration
,
we should be able to say that we have broken down the barriers to getting
data into a Metron cluster in easy and efficient ways.  In addition to
NiFi, having support for other popular tools such as beats
, fluentd ,
etc.
 - Parsers should be pluggable, with independent tests and the ability to
make versioned modifications with roll-backs.

What else?  Are any of these items not necessary for a 1.0?

Jon
-- 

Jon

Re: [ANNOUNCE] - Apache Metron Slack channel

Done


On August 15, 2018 at 14:22:45, Vets, Laurens (laur...@daemon.be) wrote:

Could I be invited?

On 15-Aug-18 09:48, Michael Miklavcic wrote:
> + Metron user list
>
> On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Turns out we are able to invite folks on an ad-hoc basis. See
instructions
>> here -
>> https://cwiki.apache.org/confluence/display/METRON/Community+Resources
>>
>>
>> On Wed, Aug 15, 2018 at 9:23 AM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> It's another option with different features. I imagine many people will
>>> use both.
>>>
>>> On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball <
>>> si...@simonellistonball.com> wrote:
>>>
 Since this is committers only, would it make more sense to stick to
IRC?
 Or
 is exclusivity the idea?

 On 15 August 2018 at 16:09, Nick Allen  wrote:

> Thanks for the instructions!
>
> On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> The Metron community has a Slack channel available for communication
>> (similar to the existing IRC channel, only on Slack).
>>
>> To join:
>>
>> 1. Go to slack.com.
>> 2. For organization/group, you'll enter "the-asf"
>> 3. Use your Apache email for your login
>> 4. Click "Channels" and look for #metron (Created by ottO June 15,
> 2018)
>> Best
>> Mike Miklavcic
>>


 --
 --
 simon elliston ball
 @sireb

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Vets, Laurens

Could I be invited?

On 15-Aug-18 09:48, Michael Miklavcic wrote:
> + Metron user list
>
> On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Turns out we are able to invite folks on an ad-hoc basis. See instructions
>> here -
>> https://cwiki.apache.org/confluence/display/METRON/Community+Resources
>>
>>
>> On Wed, Aug 15, 2018 at 9:23 AM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> It's another option with different features. I imagine many people will
>>> use both.
>>>
>>> On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball <
>>> si...@simonellistonball.com> wrote:
>>>
 Since this is committers only, would it make more sense to stick to IRC?
 Or
 is exclusivity the idea?

 On 15 August 2018 at 16:09, Nick Allen  wrote:

> Thanks for the instructions!
>
> On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> The Metron community has a Slack channel available for communication
>> (similar to the existing IRC channel, only on Slack).
>>
>> To join:
>>
>>1. Go to slack.com.
>>2. For organization/group, you'll enter "the-asf"
>>3. Use your Apache email for your login
>>4. Click "Channels" and look for #metron (Created by ottO June 15,
> 2018)
>> Best
>> Mike Miklavcic
>>


 --
 --
 simon elliston ball
 @sireb

Re: [DISCUSS] Release cadence

If you like, I can volunteer to kick off a discuss thread when I submit the
board report.

On Wed, Aug 15, 2018 at 2:21 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I'm also a fan of the 2-3 month time frame for releases. And I agree it
> fits nicely with our board report. That said, I think we should minimally
> kick off a DISCUSS at least every 2 months per the recommendations above.
> If it's warranted, great. If not, then we bring it up at a stated later
> time for re-evaluation.
>
> Fwiw, some upcoming features post-0.6.0 that I'm seeing which are also
> large-ish and will fit nicely into the next cycle (pending completion, of
> course):
>
>1. NiFi Metron parsers
>2. Profiler enhancements - bootstrapping, etc.
>3. Knox SSO
>
>
>
> On Wed, Aug 15, 2018 at 11:10 AM Casey Stella  wrote:
>
> > Strictly selfishly, I'd love for a release to happen quickly enough to
> have
> > something to announce to the board during the reports.  Once every 2
> months
> > or when a sufficiently complicated change happens sounds like a sensible
> > cadence.
> >
> > I very much support a "how do we get to 1.0" discussion, maybe as a
> > separate thread?
> >
> > On Wed, Aug 15, 2018 at 11:56 AM zeo...@gmail.com 
> > wrote:
> >
> > > I'm a fan of a hybrid time/feature-based cadence.  Something like
> "When 3
> > > months has passed since our last release, or a sufficiently complicated
> > > change has been introduced to master (like merging a FB), a discuss
> > thread
> > > is started".  I'm primarily thinking of what the upgrade path looks
> like
> > > (more on that in a "how do we get to 1.0" discuss).
> > >
> > > Jon
> > >
> > > On Wed, Aug 15, 2018 at 11:02 AM Justin Leet 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > In concert with the discuss thread on a potential 0.6.0 release, I'd
> > also
> > > > like start a discussion about our release cadence.  We've generally
> > been
> > > > pretty relaxed around doing releases, and I'm curious what people's
> > > > thoughts are on adopting a somewhat more regular schedule.
> > > >
> > > > Couple questions I think are relevant
> > > > 1. Is this something we should work towards and, if we do, how do we
> > want
> > > > to go about it?
> > > >
> > > >- "Whenever someone feels like pushing out a discuss thread"?
> > > >- "Let's just start a discuss thread every X and if we want to
> > release
> > > >we release"?
> > > >- "let's try to get a release out every X and what's on the bus is
> > on
> > > >the bus"?
> > > >- Something else?
> > > >
> > > > 2. Assuming we do want to do more regular releases, what's the
> > timeframe
> > > > we'd like to shoot for?
> > > >
> > > > Personally, I'd like to just start a discuss thread regularly, with
> the
> > > > built-in expectation that not every thread should necessarily lead
> to a
> > > > release. I don't want to be forcing release overhead when there's not
> > > > enough to merit a release, but releasing more often than we often do
> > now
> > > > would provide a lot of values to users.
> > > >
> > > > In terms of timeframe, I tend to think a 2-3 month cadence for the
> > > threads
> > > > is reasonable. It's long enough to potentially accrue enough features
> > to
> > > > merit a release, but short enough that when we pass on a release
> we're
> > > > probably fine just waiting for another cycle to come around.  The
> last
> > > > release was ~2 months ago and we have a good amount of stuff here,
> but
> > I
> > > > also don't expect two feature branches going in to be the norm.
> > > >
> > > > I'd expect whatever comes out of this thread to also be relatively
> > > > informal. At least right now, I don't feel like we need a rigid
> > schedule,
> > > > and I'd still like people to feel encouraged to propose a release,
> > > > particularly when there are a couple major features or critical
> fixes.
> > > > Alternatively, I would expect some of these discuss threads to
> > conclude,
> > > > "We should do a release, but let's wait a couple waits for these
> > tickets
> > > to
> > > > finish up" (e.g. like the Pcap query panel).
> > > >
> > > > Justin
> > > >
> > > --
> > >
> > > Jon
> > >
> >
>

Re: [DISCUSS] Release cadence

I'm also a fan of the 2-3 month time frame for releases. And I agree it
fits nicely with our board report. That said, I think we should minimally
kick off a DISCUSS at least every 2 months per the recommendations above.
If it's warranted, great. If not, then we bring it up at a stated later
time for re-evaluation.

Fwiw, some upcoming features post-0.6.0 that I'm seeing which are also
large-ish and will fit nicely into the next cycle (pending completion, of
course):

   1. NiFi Metron parsers
   2. Profiler enhancements - bootstrapping, etc.
   3. Knox SSO



On Wed, Aug 15, 2018 at 11:10 AM Casey Stella  wrote:

> Strictly selfishly, I'd love for a release to happen quickly enough to have
> something to announce to the board during the reports.  Once every 2 months
> or when a sufficiently complicated change happens sounds like a sensible
> cadence.
>
> I very much support a "how do we get to 1.0" discussion, maybe as a
> separate thread?
>
> On Wed, Aug 15, 2018 at 11:56 AM zeo...@gmail.com 
> wrote:
>
> > I'm a fan of a hybrid time/feature-based cadence.  Something like "When 3
> > months has passed since our last release, or a sufficiently complicated
> > change has been introduced to master (like merging a FB), a discuss
> thread
> > is started".  I'm primarily thinking of what the upgrade path looks like
> > (more on that in a "how do we get to 1.0" discuss).
> >
> > Jon
> >
> > On Wed, Aug 15, 2018 at 11:02 AM Justin Leet 
> > wrote:
> >
> > > Hi all,
> > >
> > > In concert with the discuss thread on a potential 0.6.0 release, I'd
> also
> > > like start a discussion about our release cadence.  We've generally
> been
> > > pretty relaxed around doing releases, and I'm curious what people's
> > > thoughts are on adopting a somewhat more regular schedule.
> > >
> > > Couple questions I think are relevant
> > > 1. Is this something we should work towards and, if we do, how do we
> want
> > > to go about it?
> > >
> > >- "Whenever someone feels like pushing out a discuss thread"?
> > >- "Let's just start a discuss thread every X and if we want to
> release
> > >we release"?
> > >- "let's try to get a release out every X and what's on the bus is
> on
> > >the bus"?
> > >- Something else?
> > >
> > > 2. Assuming we do want to do more regular releases, what's the
> timeframe
> > > we'd like to shoot for?
> > >
> > > Personally, I'd like to just start a discuss thread regularly, with the
> > > built-in expectation that not every thread should necessarily lead to a
> > > release. I don't want to be forcing release overhead when there's not
> > > enough to merit a release, but releasing more often than we often do
> now
> > > would provide a lot of values to users.
> > >
> > > In terms of timeframe, I tend to think a 2-3 month cadence for the
> > threads
> > > is reasonable. It's long enough to potentially accrue enough features
> to
> > > merit a release, but short enough that when we pass on a release we're
> > > probably fine just waiting for another cycle to come around.  The last
> > > release was ~2 months ago and we have a good amount of stuff here, but
> I
> > > also don't expect two feature branches going in to be the norm.
> > >
> > > I'd expect whatever comes out of this thread to also be relatively
> > > informal. At least right now, I don't feel like we need a rigid
> schedule,
> > > and I'd still like people to feel encouraged to propose a release,
> > > particularly when there are a couple major features or critical fixes.
> > > Alternatively, I would expect some of these discuss threads to
> conclude,
> > > "We should do a release, but let's wait a couple waits for these
> tickets
> > to
> > > finish up" (e.g. like the Pcap query panel).
> > >
> > > Justin
> > >
> > --
> >
> > Jon
> >
>

Re: [DISCUSS] Metron Release 0.6.0?

+1 here as well to the proposed releases.

On Wed, Aug 15, 2018 at 11:06 AM Casey Stella  wrote:

> +1 to both releases, this is plenty for an 0.6.0 and a 0.2.0
>
> On Wed, Aug 15, 2018 at 11:04 AM Justin Leet 
> wrote:
>
> > I just sent a thread about release cadence. Jon, I'd recommend starting a
> > thread on a 1.0 roadmap.  I thought about merging the threads, but I
> think
> > that's just going to result in more crosstalk, so I'll let you start that
> > conversation.
> >
> > On Wed, Aug 15, 2018 at 10:37 AM Nick Allen  wrote:
> >
> > > +1 to a 0.6.0 release that includes the Pcap Panel and Solr work.
> > >
> > > +1 to doing a 0.2.0 release for metron-bro-plugin-kafka.  I *think* we
> > need
> > > to do the plugin release first, so that the 0.6.0 Metron release will
> > point
> > > to plugin 0.2.0.
> > >
> > > FWIW, here are the changes since the last release.
> > >
> > > 6 days ago METRON-1730: Update steps to run pycapa on Centos 6
> (mmiklavc
> > > via mmiklavc) closes apache/metron#1152
> > > 2 weeks ago METRON-1701 Update General notes on the installation of
> > Pycapa
> > > on Kerberized cluster (MohanDV via nickwallen) closes
> apache/metron#1136
> > > 3 weeks ago METRON-1650 Packaging docker containers are too large
> > > (jameslamb via merrimanr) closes apache/metron#1091
> > > 3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
> > > management pack repo info closes apache/incubator-metron#1052
> > > 3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
> > > apache/incubator-metron#1126
> > > 3 weeks ago METRON-1694: Clean up Metron REST docs closes
> > > apache/incubator-metron#1131
> > > 4 weeks ago METRON-1606 Add a 'wrap' to incoming messages in
> > the
> > > metron json parser (ottobackwards) closes apache/metron#1054
> > > 4 weeks ago METRON-1672 Add metron-alerts's UI unit tests to
> travis
> > > build process (justinleet) closes apache/metron#1106
> > > 4 weeks ago METRON-1684 Fix Markdown problems in 3rdPartyParser.md
> > > (justinleet) closes apache/metron#1110
> > > 4 weeks ago METRON-1657 Parser aggregation in storm (justinleet) closes
> > > apache/metron#1099
> > > 4 weeks ago METRON-1651 Fixing failing protractor e2e test (tiborm via
> > > merrimanr) closes apache/metron#1095
> > > 4 weeks ago METRON-1673 Fix Javadoc errors (justinleet) closes
> > > apache/metron#1107
> > > 4 weeks ago METRON-1620: Fixes for forensic clustering use case example
> > > (mmiklavc via mmiklavc) closes apache/metron#1065
> > > 4 weeks ago METRON-1659: The platform-info.sh should check for the
> > vagrant
> > > hostmanager plugin closes apache/incubator-metron#1100
> > > 4 weeks ago METRON-1658: Upgrade bro to 2.5.4 closes
> > > apache/incubator-metron#1101
> > > 4 weeks ago METRON-1236 Add start/stop/restart commands that execute
> > > successfully, when ambari agents run as non-root user closes
> > > apache/incubator-metron#1105
> > > 4 weeks ago METRON-1670: Stellar WEEK_OF_YEAR test is locale sensitive
> > > closes apache/incubator-metron#1104
> > > 5 weeks ago METRON-1660 On Solr, sorting by threat score fails
> > (justinleet)
> > > closes apache/metron#1102
> > > 5 weeks ago METRON-1656 Create KAKFA_SEEK function (nickwallen) closes
> > > apache/metron#1097
> > > 5 weeks ago METRON-1644: Support parser chaining closes
> > > apache/incubator-metron#1084
> > > 5 weeks ago METRON-1655 Make REGEXP_MATCH take multiple regexs in the
> 2nd
> > > arg (ottobackwards) closes apache/metron#1098
> > > 6 weeks ago METRON-1643: Create a REGEX_ROUTING field transformation
> > closes
> > > apache/incubator-metron#1083
> > > 6 weeks ago METRON-1652 Document X-Pack Common Problem (nickwallen)
> > closes
> > > apache/metron#1092
> > > 6 weeks ago METRON-1649 Intermittent Test Failure
> > > ProfileBuilderBoltTest#testFlushExpiredProfiles
> > > (nickwallen) closes apache/metron#1090
> > > 6 weeks ago METRON-1635 Alerts UI status update doesn't
> immediately
> > > show up (merrimanr) closes apache/metron#1080
> > > 6 weeks ago METRON-1642: KafkaWriter should be able choose the topic
> > from a
> > > field in addition to topology construction time closes
> > > apache/incubator-metron#1082
> > > 6 weeks ago METRON-1636: Fix broken unit test setup in metron-alerts
> > closes
> > > apache/incubator-metron#1085
> > > 7 weeks ago METRON-1631 Alerts UI: Dash score does not show if only
> > > filtering by one group (sardell via merrimanr) closes
> apache/metron#1079
> > > 7 weeks ago METRON-1647 Fix logging level score closes
> > > apache/incubator-metron#1089
> > > 7 weeks ago METRON-1621: Sorting alerts table by score closes
> > > apache/incubator-metron#1088
> > > 7 weeks ago METRON-1619: Stellar empty collections should be considered
> > > false in boolean expressions closes apache/incubator-metron#1064
> > > 7 weeks ago METRON-1646 Sensor Stubs should work when kerberized
> > > (nickwallen) closes apache/metron#1087
> > > 7 weeks ago METRON-1645: Check wether the Solr management pac

Re: [DISCUSS] Release cadence

Strictly selfishly, I'd love for a release to happen quickly enough to have
something to announce to the board during the reports.  Once every 2 months
or when a sufficiently complicated change happens sounds like a sensible
cadence.

I very much support a "how do we get to 1.0" discussion, maybe as a
separate thread?

On Wed, Aug 15, 2018 at 11:56 AM zeo...@gmail.com  wrote:

> I'm a fan of a hybrid time/feature-based cadence.  Something like "When 3
> months has passed since our last release, or a sufficiently complicated
> change has been introduced to master (like merging a FB), a discuss thread
> is started".  I'm primarily thinking of what the upgrade path looks like
> (more on that in a "how do we get to 1.0" discuss).
>
> Jon
>
> On Wed, Aug 15, 2018 at 11:02 AM Justin Leet 
> wrote:
>
> > Hi all,
> >
> > In concert with the discuss thread on a potential 0.6.0 release, I'd also
> > like start a discussion about our release cadence.  We've generally been
> > pretty relaxed around doing releases, and I'm curious what people's
> > thoughts are on adopting a somewhat more regular schedule.
> >
> > Couple questions I think are relevant
> > 1. Is this something we should work towards and, if we do, how do we want
> > to go about it?
> >
> >- "Whenever someone feels like pushing out a discuss thread"?
> >- "Let's just start a discuss thread every X and if we want to release
> >we release"?
> >- "let's try to get a release out every X and what's on the bus is on
> >the bus"?
> >- Something else?
> >
> > 2. Assuming we do want to do more regular releases, what's the timeframe
> > we'd like to shoot for?
> >
> > Personally, I'd like to just start a discuss thread regularly, with the
> > built-in expectation that not every thread should necessarily lead to a
> > release. I don't want to be forcing release overhead when there's not
> > enough to merit a release, but releasing more often than we often do now
> > would provide a lot of values to users.
> >
> > In terms of timeframe, I tend to think a 2-3 month cadence for the
> threads
> > is reasonable. It's long enough to potentially accrue enough features to
> > merit a release, but short enough that when we pass on a release we're
> > probably fine just waiting for another cycle to come around.  The last
> > release was ~2 months ago and we have a good amount of stuff here, but I
> > also don't expect two feature branches going in to be the norm.
> >
> > I'd expect whatever comes out of this thread to also be relatively
> > informal. At least right now, I don't feel like we need a rigid schedule,
> > and I'd still like people to feel encouraged to propose a release,
> > particularly when there are a couple major features or critical fixes.
> > Alternatively, I would expect some of these discuss threads to conclude,
> > "We should do a release, but let's wait a couple waits for these tickets
> to
> > finish up" (e.g. like the Pcap query panel).
> >
> > Justin
> >
> --
>
> Jon
>

Re: [DISCUSS] Metron Release 0.6.0?

+1 to both releases, this is plenty for an 0.6.0 and a 0.2.0

On Wed, Aug 15, 2018 at 11:04 AM Justin Leet  wrote:

> I just sent a thread about release cadence. Jon, I'd recommend starting a
> thread on a 1.0 roadmap.  I thought about merging the threads, but I think
> that's just going to result in more crosstalk, so I'll let you start that
> conversation.
>
> On Wed, Aug 15, 2018 at 10:37 AM Nick Allen  wrote:
>
> > +1 to a 0.6.0 release that includes the Pcap Panel and Solr work.
> >
> > +1 to doing a 0.2.0 release for metron-bro-plugin-kafka.  I *think* we
> need
> > to do the plugin release first, so that the 0.6.0 Metron release will
> point
> > to plugin 0.2.0.
> >
> > FWIW, here are the changes since the last release.
> >
> > 6 days ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
> > via mmiklavc) closes apache/metron#1152
> > 2 weeks ago METRON-1701 Update General notes on the installation of
> Pycapa
> > on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
> > 3 weeks ago METRON-1650 Packaging docker containers are too large
> > (jameslamb via merrimanr) closes apache/metron#1091
> > 3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
> > management pack repo info closes apache/incubator-metron#1052
> > 3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
> > apache/incubator-metron#1126
> > 3 weeks ago METRON-1694: Clean up Metron REST docs closes
> > apache/incubator-metron#1131
> > 4 weeks ago METRON-1606 Add a 'wrap' to incoming messages in
> the
> > metron json parser (ottobackwards) closes apache/metron#1054
> > 4 weeks ago METRON-1672 Add metron-alerts's UI unit tests to travis
> > build process (justinleet) closes apache/metron#1106
> > 4 weeks ago METRON-1684 Fix Markdown problems in 3rdPartyParser.md
> > (justinleet) closes apache/metron#1110
> > 4 weeks ago METRON-1657 Parser aggregation in storm (justinleet) closes
> > apache/metron#1099
> > 4 weeks ago METRON-1651 Fixing failing protractor e2e test (tiborm via
> > merrimanr) closes apache/metron#1095
> > 4 weeks ago METRON-1673 Fix Javadoc errors (justinleet) closes
> > apache/metron#1107
> > 4 weeks ago METRON-1620: Fixes for forensic clustering use case example
> > (mmiklavc via mmiklavc) closes apache/metron#1065
> > 4 weeks ago METRON-1659: The platform-info.sh should check for the
> vagrant
> > hostmanager plugin closes apache/incubator-metron#1100
> > 4 weeks ago METRON-1658: Upgrade bro to 2.5.4 closes
> > apache/incubator-metron#1101
> > 4 weeks ago METRON-1236 Add start/stop/restart commands that execute
> > successfully, when ambari agents run as non-root user closes
> > apache/incubator-metron#1105
> > 4 weeks ago METRON-1670: Stellar WEEK_OF_YEAR test is locale sensitive
> > closes apache/incubator-metron#1104
> > 5 weeks ago METRON-1660 On Solr, sorting by threat score fails
> (justinleet)
> > closes apache/metron#1102
> > 5 weeks ago METRON-1656 Create KAKFA_SEEK function (nickwallen) closes
> > apache/metron#1097
> > 5 weeks ago METRON-1644: Support parser chaining closes
> > apache/incubator-metron#1084
> > 5 weeks ago METRON-1655 Make REGEXP_MATCH take multiple regexs in the 2nd
> > arg (ottobackwards) closes apache/metron#1098
> > 6 weeks ago METRON-1643: Create a REGEX_ROUTING field transformation
> closes
> > apache/incubator-metron#1083
> > 6 weeks ago METRON-1652 Document X-Pack Common Problem (nickwallen)
> closes
> > apache/metron#1092
> > 6 weeks ago METRON-1649 Intermittent Test Failure
> > ProfileBuilderBoltTest#testFlushExpiredProfiles
> > (nickwallen) closes apache/metron#1090
> > 6 weeks ago METRON-1635 Alerts UI status update doesn't immediately
> > show up (merrimanr) closes apache/metron#1080
> > 6 weeks ago METRON-1642: KafkaWriter should be able choose the topic
> from a
> > field in addition to topology construction time closes
> > apache/incubator-metron#1082
> > 6 weeks ago METRON-1636: Fix broken unit test setup in metron-alerts
> closes
> > apache/incubator-metron#1085
> > 7 weeks ago METRON-1631 Alerts UI: Dash score does not show if only
> > filtering by one group (sardell via merrimanr) closes apache/metron#1079
> > 7 weeks ago METRON-1647 Fix logging level score closes
> > apache/incubator-metron#1089
> > 7 weeks ago METRON-1621: Sorting alerts table by score closes
> > apache/incubator-metron#1088
> > 7 weeks ago METRON-1619: Stellar empty collections should be considered
> > false in boolean expressions closes apache/incubator-metron#1064
> > 7 weeks ago METRON-1646 Sensor Stubs should work when kerberized
> > (nickwallen) closes apache/metron#1087
> > 7 weeks ago METRON-1645: Check wether the Solr management pack is
> installed
> > before configuring the solr principal name. closes
> > apache/incubator-metron#1086
> > 7 weeks ago Merge branch 'master' into feature/METRON-1416-upgrade-solr
> > 7 weeks ago METRON-1634 Alerts UI add comment doesn't immediately
> show
> > up. (merrimanr) closes apache/metron#1077
>

Re: Slack Channel

Invite sent

On Wed, Aug 15, 2018 at 10:57 AM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Hello dev team, may I please join your slack channel :)
>

Re: Slack Channel

Sorry Simon, I retract the comment!  I didn't realize it was possible, but
it is possible to invite.

On Wed, Aug 15, 2018 at 1:01 PM Casey Stella  wrote:

> Sadly, it's the ASF slack and I believe it requires an @apache.org email
> address.
>
> On Wed, Aug 15, 2018 at 12:57 PM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> Hello dev team, may I please join your slack channel :)
>>
>

Re: Slack Channel

Sadly, it's the ASF slack and I believe it requires an @apache.org email
address.

On Wed, Aug 15, 2018 at 12:57 PM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Hello dev team, may I please join your slack channel :)
>

Slack Channel

Hello dev team, may I please join your slack channel :)

Re: [ANNOUNCE] - Apache Metron Slack channel

+ Metron user list

On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Turns out we are able to invite folks on an ad-hoc basis. See instructions
> here -
> https://cwiki.apache.org/confluence/display/METRON/Community+Resources
>
>
> On Wed, Aug 15, 2018 at 9:23 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> It's another option with different features. I imagine many people will
>> use both.
>>
>> On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball <
>> si...@simonellistonball.com> wrote:
>>
>>> Since this is committers only, would it make more sense to stick to IRC?
>>> Or
>>> is exclusivity the idea?
>>>
>>> On 15 August 2018 at 16:09, Nick Allen  wrote:
>>>
>>> > Thanks for the instructions!
>>> >
>>> > On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic <
>>> > michael.miklav...@gmail.com> wrote:
>>> >
>>> > > The Metron community has a Slack channel available for communication
>>> > > (similar to the existing IRC channel, only on Slack).
>>> > >
>>> > > To join:
>>> > >
>>> > >1. Go to slack.com.
>>> > >2. For organization/group, you'll enter "the-asf"
>>> > >3. Use your Apache email for your login
>>> > >4. Click "Channels" and look for #metron (Created by ottO June 15,
>>> > 2018)
>>> > >
>>> > > Best
>>> > > Mike Miklavcic
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> --
>>> simon elliston ball
>>> @sireb
>>>
>>

Re: [ANNOUNCE] - Apache Metron Slack channel

Turns out we are able to invite folks on an ad-hoc basis. See instructions
here -
https://cwiki.apache.org/confluence/display/METRON/Community+Resources


On Wed, Aug 15, 2018 at 9:23 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> It's another option with different features. I imagine many people will
> use both.
>
> On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> Since this is committers only, would it make more sense to stick to IRC?
>> Or
>> is exclusivity the idea?
>>
>> On 15 August 2018 at 16:09, Nick Allen  wrote:
>>
>> > Thanks for the instructions!
>> >
>> > On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic <
>> > michael.miklav...@gmail.com> wrote:
>> >
>> > > The Metron community has a Slack channel available for communication
>> > > (similar to the existing IRC channel, only on Slack).
>> > >
>> > > To join:
>> > >
>> > >1. Go to slack.com.
>> > >2. For organization/group, you'll enter "the-asf"
>> > >3. Use your Apache email for your login
>> > >4. Click "Channels" and look for #metron (Created by ottO June 15,
>> > 2018)
>> > >
>> > > Best
>> > > Mike Miklavcic
>> > >
>> >
>>
>>
>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
>

Re: [DISCUSS] Release cadence

2018-08-15 Thread zeo...@gmail.com

I'm a fan of a hybrid time/feature-based cadence.  Something like "When 3
months has passed since our last release, or a sufficiently complicated
change has been introduced to master (like merging a FB), a discuss thread
is started".  I'm primarily thinking of what the upgrade path looks like
(more on that in a "how do we get to 1.0" discuss).

Jon

On Wed, Aug 15, 2018 at 11:02 AM Justin Leet  wrote:

> Hi all,
>
> In concert with the discuss thread on a potential 0.6.0 release, I'd also
> like start a discussion about our release cadence.  We've generally been
> pretty relaxed around doing releases, and I'm curious what people's
> thoughts are on adopting a somewhat more regular schedule.
>
> Couple questions I think are relevant
> 1. Is this something we should work towards and, if we do, how do we want
> to go about it?
>
>- "Whenever someone feels like pushing out a discuss thread"?
>- "Let's just start a discuss thread every X and if we want to release
>we release"?
>- "let's try to get a release out every X and what's on the bus is on
>the bus"?
>- Something else?
>
> 2. Assuming we do want to do more regular releases, what's the timeframe
> we'd like to shoot for?
>
> Personally, I'd like to just start a discuss thread regularly, with the
> built-in expectation that not every thread should necessarily lead to a
> release. I don't want to be forcing release overhead when there's not
> enough to merit a release, but releasing more often than we often do now
> would provide a lot of values to users.
>
> In terms of timeframe, I tend to think a 2-3 month cadence for the threads
> is reasonable. It's long enough to potentially accrue enough features to
> merit a release, but short enough that when we pass on a release we're
> probably fine just waiting for another cycle to come around.  The last
> release was ~2 months ago and we have a good amount of stuff here, but I
> also don't expect two feature branches going in to be the norm.
>
> I'd expect whatever comes out of this thread to also be relatively
> informal. At least right now, I don't feel like we need a rigid schedule,
> and I'd still like people to feel encouraged to propose a release,
> particularly when there are a couple major features or critical fixes.
> Alternatively, I would expect some of these discuss threads to conclude,
> "We should do a release, but let's wait a couple waits for these tickets to
> finish up" (e.g. like the Pcap query panel).
>
> Justin
>
-- 

Jon

Re: [ANNOUNCE] - Apache Metron Slack channel

It's another option with different features. I imagine many people will use
both.

On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Since this is committers only, would it make more sense to stick to IRC? Or
> is exclusivity the idea?
>
> On 15 August 2018 at 16:09, Nick Allen  wrote:
>
> > Thanks for the instructions!
> >
> > On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > The Metron community has a Slack channel available for communication
> > > (similar to the existing IRC channel, only on Slack).
> > >
> > > To join:
> > >
> > >1. Go to slack.com.
> > >2. For organization/group, you'll enter "the-asf"
> > >3. Use your Apache email for your login
> > >4. Click "Channels" and look for #metron (Created by ottO June 15,
> > 2018)
> > >
> > > Best
> > > Mike Miklavcic
> > >
> >
>
>
>
> --
> --
> simon elliston ball
> @sireb
>

Re: [ANNOUNCE] - Apache Metron Slack channel

Since this is committers only, would it make more sense to stick to IRC? Or
is exclusivity the idea?

On 15 August 2018 at 16:09, Nick Allen  wrote:

> Thanks for the instructions!
>
> On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > The Metron community has a Slack channel available for communication
> > (similar to the existing IRC channel, only on Slack).
> >
> > To join:
> >
> >1. Go to slack.com.
> >2. For organization/group, you'll enter "the-asf"
> >3. Use your Apache email for your login
> >4. Click "Channels" and look for #metron (Created by ottO June 15,
> 2018)
> >
> > Best
> > Mike Miklavcic
> >
>



-- 
--
simon elliston ball
@sireb

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Nick Allen

Thanks for the instructions!

On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> The Metron community has a Slack channel available for communication
> (similar to the existing IRC channel, only on Slack).
>
> To join:
>
>1. Go to slack.com.
>2. For organization/group, you'll enter "the-asf"
>3. Use your Apache email for your login
>4. Click "Channels" and look for #metron (Created by ottO June 15, 2018)
>
> Best
> Mike Miklavcic
>

Re: [DISCUSS] Metron Parsers in Nifi

On August 15, 2018 at 09:30:47, Justin Leet (justinjl...@gmail.com) wrote:

As an exercise, let me summarize the points of contention I've seen and lay
out the tradeoffs as I see them. That way we can prioritize what's
important to us in a NiFi implementation and better work towards a
favorable solution (basically, I want to requirements we have for an MVP).
My opinions/comments/questions are in *bold*.  Feel free (and encouraged to
disagree). Keep in mind, this stuff probably exists on a spectrum (we might
want to pick and choose what we do, and possibly even when we do it).

   - Splitting off fieldTransformations from the parser itself
   -
  - In Nifi, we're chaining processors to do our fieldTransformations.
  This can't be particularly automatic from a definition, to the best of my
  knowledge.
  - Our configuration between NiFi and Storm differs (because NiFi is
  building Processors and Storm is just acting on the transforms).
  - *I'm mostly fine with splitting these, IMO we just need to make
  sure it's documented. The current colocation of them feels
slightly sketchy
  to me in general (it feels like it's merging pure parsing and something
  more enrichment oriented).** I also like the idea of exposing Stellar
  transformations as their own Processor.*
  -
 - *Could anyone refresh my memory on why fieldTransformations are
 bundled with parsing directly?*

Obviously I am for this.  It makes for more flexible composition, the
ability to use stellar in nifi in general etc etc


   -
   NiFi parser configuration
   -
  - We can do it from ZK, but need to make it available in a manner not
  available in ZK
  - If we don't allow ZK, we can potentially have different sources of
  configs.
  -
 - *I personally don't like this very much. I always hated having
 to hop between things in order to manage these sort of things, but I
 consider that more annoying than blocking.*

If you take out the transformations, there is very little to configure for
most of the parsers.   From a NIFI point of view, having to go into metron
to configure a sensor in order to get it into NiFi makes no sense ( unless
I am not understanding what you are saying ). Expecting them to go back and
forth to understand the configurations doesn’t make much sense either.
Having a dual approach ( a sensor configuration controller service if you
want ZK already configured sensors and nifi property only ) would be the
best.

This is why I was asking about the target users.  If someone is going to
use NIFI _INSTEAD_ of metron for the parsers, then why have the
configurations in zookeeper?   What would the case for a hybrid NIFI +
STORM parsers mixed?  Is that likely?



   -
   Specific parsers vs an aggregated parser
   -
  - If they're all specific, it means every user who wants to implement
  a parser (even an existing one) in NiFi, they have to do
additional work to
  make it work in NiFi.
  -
 - *I don't think it's a lot of work on a per parser basis, and we
 might be able to ease this with some clever handling of our
interfaces.
 However, I personally don't like that there's no way to just run an
 existing Metron parser in NiFi without additional
NiFi-specific work.  To
 be clear, I'd prefer to have a quick way for users to take parsers,
 including preexisting parsers, into NiFi.  I don't think this
should be the
 end solution for most parsers, but it does feel like the
minimal viable
 product solution to cover current users mostly as-is.  In my
mind, users
 should be able to take preexisting Storm parsers and be able
to run them up
 and test them in NiFi with minimal involvement, even if the
end state is to
 do a more NiFi-like implementation.*
 - *If we expand this to other platforms (e.g. Spark?), do we
 expect everything to be reimplemented every time?  Or are we
making that
 decision on a platform-by-platform basis?*
 - *I think most parsers, including our own should be optimized as
 needed for NiFi, including whatever Schema work and
versioning we want to
 do, but I don't think that needs to be done right away.
Looking through
 source, our parsers are:*
 -
- *Asa*
- *Bro*
- *CEF*
- *CSV*
- *Fireeye*
- *ISE*
- *JSON*
- *Lancope*
- *Logstash*
- *Palo Alto*
- *Snort*
- *Sourcefire*
 - *I don't know that I want to go through and convert and test all
 of them in the first pass.*

We can have one nar, with a reader per processor for deployment ( similar
to having the one parser jar now ), like the standard services nar now.

The re-implementation is the glue code that

- loads, feeds, and outputs the result of a

Re: [DISCUSS] Metron Release 0.6.0?

I just sent a thread about release cadence. Jon, I'd recommend starting a
thread on a 1.0 roadmap.  I thought about merging the threads, but I think
that's just going to result in more crosstalk, so I'll let you start that
conversation.

On Wed, Aug 15, 2018 at 10:37 AM Nick Allen  wrote:

> +1 to a 0.6.0 release that includes the Pcap Panel and Solr work.
>
> +1 to doing a 0.2.0 release for metron-bro-plugin-kafka.  I *think* we need
> to do the plugin release first, so that the 0.6.0 Metron release will point
> to plugin 0.2.0.
>
> FWIW, here are the changes since the last release.
>
> 6 days ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
> via mmiklavc) closes apache/metron#1152
> 2 weeks ago METRON-1701 Update General notes on the installation of Pycapa
> on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
> 3 weeks ago METRON-1650 Packaging docker containers are too large
> (jameslamb via merrimanr) closes apache/metron#1091
> 3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
> management pack repo info closes apache/incubator-metron#1052
> 3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
> apache/incubator-metron#1126
> 3 weeks ago METRON-1694: Clean up Metron REST docs closes
> apache/incubator-metron#1131
> 4 weeks ago METRON-1606 Add a 'wrap' to incoming messages in the
> metron json parser (ottobackwards) closes apache/metron#1054
> 4 weeks ago METRON-1672 Add metron-alerts's UI unit tests to travis
> build process (justinleet) closes apache/metron#1106
> 4 weeks ago METRON-1684 Fix Markdown problems in 3rdPartyParser.md
> (justinleet) closes apache/metron#1110
> 4 weeks ago METRON-1657 Parser aggregation in storm (justinleet) closes
> apache/metron#1099
> 4 weeks ago METRON-1651 Fixing failing protractor e2e test (tiborm via
> merrimanr) closes apache/metron#1095
> 4 weeks ago METRON-1673 Fix Javadoc errors (justinleet) closes
> apache/metron#1107
> 4 weeks ago METRON-1620: Fixes for forensic clustering use case example
> (mmiklavc via mmiklavc) closes apache/metron#1065
> 4 weeks ago METRON-1659: The platform-info.sh should check for the vagrant
> hostmanager plugin closes apache/incubator-metron#1100
> 4 weeks ago METRON-1658: Upgrade bro to 2.5.4 closes
> apache/incubator-metron#1101
> 4 weeks ago METRON-1236 Add start/stop/restart commands that execute
> successfully, when ambari agents run as non-root user closes
> apache/incubator-metron#1105
> 4 weeks ago METRON-1670: Stellar WEEK_OF_YEAR test is locale sensitive
> closes apache/incubator-metron#1104
> 5 weeks ago METRON-1660 On Solr, sorting by threat score fails (justinleet)
> closes apache/metron#1102
> 5 weeks ago METRON-1656 Create KAKFA_SEEK function (nickwallen) closes
> apache/metron#1097
> 5 weeks ago METRON-1644: Support parser chaining closes
> apache/incubator-metron#1084
> 5 weeks ago METRON-1655 Make REGEXP_MATCH take multiple regexs in the 2nd
> arg (ottobackwards) closes apache/metron#1098
> 6 weeks ago METRON-1643: Create a REGEX_ROUTING field transformation closes
> apache/incubator-metron#1083
> 6 weeks ago METRON-1652 Document X-Pack Common Problem (nickwallen) closes
> apache/metron#1092
> 6 weeks ago METRON-1649 Intermittent Test Failure
> ProfileBuilderBoltTest#testFlushExpiredProfiles
> (nickwallen) closes apache/metron#1090
> 6 weeks ago METRON-1635 Alerts UI status update doesn't immediately
> show up (merrimanr) closes apache/metron#1080
> 6 weeks ago METRON-1642: KafkaWriter should be able choose the topic from a
> field in addition to topology construction time closes
> apache/incubator-metron#1082
> 6 weeks ago METRON-1636: Fix broken unit test setup in metron-alerts closes
> apache/incubator-metron#1085
> 7 weeks ago METRON-1631 Alerts UI: Dash score does not show if only
> filtering by one group (sardell via merrimanr) closes apache/metron#1079
> 7 weeks ago METRON-1647 Fix logging level score closes
> apache/incubator-metron#1089
> 7 weeks ago METRON-1621: Sorting alerts table by score closes
> apache/incubator-metron#1088
> 7 weeks ago METRON-1619: Stellar empty collections should be considered
> false in boolean expressions closes apache/incubator-metron#1064
> 7 weeks ago METRON-1646 Sensor Stubs should work when kerberized
> (nickwallen) closes apache/metron#1087
> 7 weeks ago METRON-1645: Check wether the Solr management pack is installed
> before configuring the solr principal name. closes
> apache/incubator-metron#1086
> 7 weeks ago Merge branch 'master' into feature/METRON-1416-upgrade-solr
> 7 weeks ago METRON-1634 Alerts UI add comment doesn't immediately show
> up. (merrimanr) closes apache/metron#1077
> 7 weeks ago METRON-1489 Retrofit UI tests to run reliably during nightly QE
> runs (sardell via nickwallen) closes apache/metron#1004
> 7 weeks ago METRON-1637 Wrong path to escalate alert REST endpoint
> (merrimanr) closes apache/metron#1078
> 8 weeks ago METRON-1624 Set Profiler and Enrichment batch parameters

[DISCUSS] Release cadence

Hi all,

In concert with the discuss thread on a potential 0.6.0 release, I'd also
like start a discussion about our release cadence.  We've generally been
pretty relaxed around doing releases, and I'm curious what people's
thoughts are on adopting a somewhat more regular schedule.

Couple questions I think are relevant
1. Is this something we should work towards and, if we do, how do we want
to go about it?

   - "Whenever someone feels like pushing out a discuss thread"?
   - "Let's just start a discuss thread every X and if we want to release
   we release"?
   - "let's try to get a release out every X and what's on the bus is on
   the bus"?
   - Something else?

2. Assuming we do want to do more regular releases, what's the timeframe
we'd like to shoot for?

Personally, I'd like to just start a discuss thread regularly, with the
built-in expectation that not every thread should necessarily lead to a
release. I don't want to be forcing release overhead when there's not
enough to merit a release, but releasing more often than we often do now
would provide a lot of values to users.

In terms of timeframe, I tend to think a 2-3 month cadence for the threads
is reasonable. It's long enough to potentially accrue enough features to
merit a release, but short enough that when we pass on a release we're
probably fine just waiting for another cycle to come around.  The last
release was ~2 months ago and we have a good amount of stuff here, but I
also don't expect two feature branches going in to be the norm.

I'd expect whatever comes out of this thread to also be relatively
informal. At least right now, I don't feel like we need a rigid schedule,
and I'd still like people to feel encouraged to propose a release,
particularly when there are a couple major features or critical fixes.
Alternatively, I would expect some of these discuss threads to conclude,
"We should do a release, but let's wait a couple waits for these tickets to
finish up" (e.g. like the Pcap query panel).

Justin

Re: [DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread Nick Allen

+1 to a 0.6.0 release that includes the Pcap Panel and Solr work.

+1 to doing a 0.2.0 release for metron-bro-plugin-kafka.  I *think* we need
to do the plugin release first, so that the 0.6.0 Metron release will point
to plugin 0.2.0.

FWIW, here are the changes since the last release.

6 days ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
via mmiklavc) closes apache/metron#1152
2 weeks ago METRON-1701 Update General notes on the installation of Pycapa
on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
3 weeks ago METRON-1650 Packaging docker containers are too large
(jameslamb via merrimanr) closes apache/metron#1091
3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
management pack repo info closes apache/incubator-metron#1052
3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
apache/incubator-metron#1126
3 weeks ago METRON-1694: Clean up Metron REST docs closes
apache/incubator-metron#1131
4 weeks ago METRON-1606 Add a 'wrap' to incoming messages in the
metron json parser (ottobackwards) closes apache/metron#1054
4 weeks ago METRON-1672 Add metron-alerts's UI unit tests to travis
build process (justinleet) closes apache/metron#1106
4 weeks ago METRON-1684 Fix Markdown problems in 3rdPartyParser.md
(justinleet) closes apache/metron#1110
4 weeks ago METRON-1657 Parser aggregation in storm (justinleet) closes
apache/metron#1099
4 weeks ago METRON-1651 Fixing failing protractor e2e test (tiborm via
merrimanr) closes apache/metron#1095
4 weeks ago METRON-1673 Fix Javadoc errors (justinleet) closes
apache/metron#1107
4 weeks ago METRON-1620: Fixes for forensic clustering use case example
(mmiklavc via mmiklavc) closes apache/metron#1065
4 weeks ago METRON-1659: The platform-info.sh should check for the vagrant
hostmanager plugin closes apache/incubator-metron#1100
4 weeks ago METRON-1658: Upgrade bro to 2.5.4 closes
apache/incubator-metron#1101
4 weeks ago METRON-1236 Add start/stop/restart commands that execute
successfully, when ambari agents run as non-root user closes
apache/incubator-metron#1105
4 weeks ago METRON-1670: Stellar WEEK_OF_YEAR test is locale sensitive
closes apache/incubator-metron#1104
5 weeks ago METRON-1660 On Solr, sorting by threat score fails (justinleet)
closes apache/metron#1102
5 weeks ago METRON-1656 Create KAKFA_SEEK function (nickwallen) closes
apache/metron#1097
5 weeks ago METRON-1644: Support parser chaining closes
apache/incubator-metron#1084
5 weeks ago METRON-1655 Make REGEXP_MATCH take multiple regexs in the 2nd
arg (ottobackwards) closes apache/metron#1098
6 weeks ago METRON-1643: Create a REGEX_ROUTING field transformation closes
apache/incubator-metron#1083
6 weeks ago METRON-1652 Document X-Pack Common Problem (nickwallen) closes
apache/metron#1092
6 weeks ago METRON-1649 Intermittent Test Failure
ProfileBuilderBoltTest#testFlushExpiredProfiles
(nickwallen) closes apache/metron#1090
6 weeks ago METRON-1635 Alerts UI status update doesn't immediately
show up (merrimanr) closes apache/metron#1080
6 weeks ago METRON-1642: KafkaWriter should be able choose the topic from a
field in addition to topology construction time closes
apache/incubator-metron#1082
6 weeks ago METRON-1636: Fix broken unit test setup in metron-alerts closes
apache/incubator-metron#1085
7 weeks ago METRON-1631 Alerts UI: Dash score does not show if only
filtering by one group (sardell via merrimanr) closes apache/metron#1079
7 weeks ago METRON-1647 Fix logging level score closes
apache/incubator-metron#1089
7 weeks ago METRON-1621: Sorting alerts table by score closes
apache/incubator-metron#1088
7 weeks ago METRON-1619: Stellar empty collections should be considered
false in boolean expressions closes apache/incubator-metron#1064
7 weeks ago METRON-1646 Sensor Stubs should work when kerberized
(nickwallen) closes apache/metron#1087
7 weeks ago METRON-1645: Check wether the Solr management pack is installed
before configuring the solr principal name. closes
apache/incubator-metron#1086
7 weeks ago Merge branch 'master' into feature/METRON-1416-upgrade-solr
7 weeks ago METRON-1634 Alerts UI add comment doesn't immediately show
up. (merrimanr) closes apache/metron#1077
7 weeks ago METRON-1489 Retrofit UI tests to run reliably during nightly QE
runs (sardell via nickwallen) closes apache/metron#1004
7 weeks ago METRON-1637 Wrong path to escalate alert REST endpoint
(merrimanr) closes apache/metron#1078
8 weeks ago METRON-1624 Set Profiler and Enrichment batch parameters in
Ambari (nickwallen) closes apache/metron#1069
8 weeks ago Merge remote-tracking branch 'origin/master' into
feature/METRON-1416-upgrade-solr
8 weeks ago Merge branch 'master' into feature/METRON-1416-upgrade-solr
(nickwallen) closes apache/metron#1075
8 weeks ago METRON-1629 Update Solr documentation (merrimanr via
justinleet) closes apache/metron#1072
8 weeks ago METRON-1633 Incorrect instructions when merging PR into feature
branch (nickwallen) closes

[ANNOUNCE] - Apache Metron Slack channel

The Metron community has a Slack channel available for communication
(similar to the existing IRC channel, only on Slack).

To join:

   1. Go to slack.com.
   2. For organization/group, you'll enter "the-asf"
   3. Use your Apache email for your login
   4. Click "Channels" and look for #metron (Created by ottO June 15, 2018)

Best
Mike Miklavcic

Re: [DISCUSS] Metron Release 0.6.0?

I was actually going to kick out another thread in a little bit around our
release schedule, it should be out shortly.

Good point on the metron-bro-plugin-kafka.  I'm in favor of putting out a
0.2.0 release of it simultaneously.

On Wed, Aug 15, 2018 at 9:48 AM zeo...@gmail.com  wrote:

> I agree - I would love to see a release not long after the PCAP FB gets
> into master, and 0.6.0 makes sense to me.
>
> I'd also like to see a 0.2 release of metron-bro-plugin-kafka.  There is
> one new commit, and I have a PR open which is waiting on some tests before
> it's ready to be evaluated/merged.  I will try to get that work done asap.
> As of right now metron's dev ansible scripts pin to a specific release of
> metron-bro-plugin-kafka (0.1
> <0.1
> https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml
> >),
> and I'm fine leaving that as is until after the coming release, but we
> could also do a metron-bro-plugin-kafka release first and then update
> metron to point the dev environment to the new package prior to the
> upcoming RC.
>
> I would also like to discuss what the roadmap looks like for a 1.0 release
> and perhaps a more regular release schedule.  I have some thoughts but
> don't want to hijack this thread.
>
> Jon
>
> On Wed, Aug 15, 2018 at 9:11 AM Justin Leet  wrote:
>
> > Hi all,
> >
> > It's been a little while since the last release, and a couple major items
> > have gone in since then (or are hopefully close to going in!).  In
> > particular, I'd personally like to see a release with our Solr work
> >  and the
> > close-to-completion PCAP Query Panel
> > .  There is a thread
> > <
> >
> https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E
> > >
> > around what's left before merging the PCAP feature branch, I encourage
> you
> > to take a look. There are also some nice-to-haves as well as some Apache
> > cleanup around the RAT tool and typescript files
> > .
> >
> > Version Number
> > I'm proposing bumping to 0.6.0, in particular because of the Solr and
> PCAP
> > efforts. We can adjust that as necessary.
> >
> > I'm proposing we release this from the Metron master branch, plus any
> > commits the community considers necessary.  Note that I'm proposing that
> > this release occur after the PCAP feature branch is merged into master.
> >
> > Proposed Timeframe
> > I would tentatively like to start work on the RC Wednesday, September
> 5th.
> > It's a little further out than usual, but I wanted to kick off the
> > discussion before Labor Day and to give ongoing  time to settle. And also
> > because I'll be unavailable around Labor Day.
> >
> > JIRA Status
> > There are 31 open PRs at https://github.com/apache/metron/pulls. We
> should
> > work on getting anything we feel merits inclusion closed out. Please
> > respond with any tickets we'd like included.
> >
> > A couple of these are for the PCAP feature branch, and there will be at
> > least one more for documentation.
> >
> > There will be updates necessary to get our Jira up to date.  I'll follow
> up
> > on that, and ask that everyone double check their tickets.
> >
> > There have been 106 commits since the 0.5.0 release (listed at the end of
> > message). There will be a few more when we pull in the PCAP feature
> branch.
> >
> > Completed PRs as of Aug 15 as generated by git log --pretty="%cr %s"
> > tags/apache-metron-0.5.0-release..HEAD.
> >
> > 5 days ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
> > via mmiklavc) closes apache/metron#1152
> > 13 days ago METRON-1701 Update General notes on the installation of
> Pycapa
> > on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
> > 3 weeks ago METRON-1650 Packaging docker containers are too large
> > (jameslamb via merrimanr) closes apache/metron#1091
> > 3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
> > management pack repo info closes apache/incubator-metron#1052
> > 3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
> > apache/incubator-metron#1126
> > 3 weeks ago METRON-1694: Clean up Metron REST docs closes
> > apache/incubator-metron#1131
> > 4 weeks ago METRON-1606 Add a 'wrap' to incoming messages in
> the
> > metron json parser (ottobackwards) closes apache/metron#1054
> > 4 weeks ago METRON-1672 Add metron-alerts's UI unit tests to travis
> > build process (justinleet) closes apache/metron#1106
> > 4 weeks ago METRON-1684 Fix Markdown problems in 3rdPartyParser.md
> > (justinleet) closes apache/metron#1110
> > 4 weeks ago METRON-1657 Parser aggregation in storm (justinleet) closes
> > apache/metron#1099
> > 4 weeks ago METRON-1651 Fixing failing protractor e2e test (tiborm via
> > merrimanr) closes apache/metron#1095
> > 4 weeks ago METRON-1673 Fi

Re: [DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread zeo...@gmail.com

I agree - I would love to see a release not long after the PCAP FB gets
into master, and 0.6.0 makes sense to me.

I'd also like to see a 0.2 release of metron-bro-plugin-kafka.  There is
one new commit, and I have a PR open which is waiting on some tests before
it's ready to be evaluated/merged.  I will try to get that work done asap.
As of right now metron's dev ansible scripts pin to a specific release of
metron-bro-plugin-kafka (0.1
<0.1https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml>),
and I'm fine leaving that as is until after the coming release, but we
could also do a metron-bro-plugin-kafka release first and then update
metron to point the dev environment to the new package prior to the
upcoming RC.

I would also like to discuss what the roadmap looks like for a 1.0 release
and perhaps a more regular release schedule.  I have some thoughts but
don't want to hijack this thread.

Jon

On Wed, Aug 15, 2018 at 9:11 AM Justin Leet  wrote:

> Hi all,
>
> It's been a little while since the last release, and a couple major items
> have gone in since then (or are hopefully close to going in!).  In
> particular, I'd personally like to see a release with our Solr work
>  and the
> close-to-completion PCAP Query Panel
> .  There is a thread
> <
> https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E
> >
> around what's left before merging the PCAP feature branch, I encourage you
> to take a look. There are also some nice-to-haves as well as some Apache
> cleanup around the RAT tool and typescript files
> .
>
> Version Number
> I'm proposing bumping to 0.6.0, in particular because of the Solr and PCAP
> efforts. We can adjust that as necessary.
>
> I'm proposing we release this from the Metron master branch, plus any
> commits the community considers necessary.  Note that I'm proposing that
> this release occur after the PCAP feature branch is merged into master.
>
> Proposed Timeframe
> I would tentatively like to start work on the RC Wednesday, September 5th.
> It's a little further out than usual, but I wanted to kick off the
> discussion before Labor Day and to give ongoing  time to settle. And also
> because I'll be unavailable around Labor Day.
>
> JIRA Status
> There are 31 open PRs at https://github.com/apache/metron/pulls. We should
> work on getting anything we feel merits inclusion closed out. Please
> respond with any tickets we'd like included.
>
> A couple of these are for the PCAP feature branch, and there will be at
> least one more for documentation.
>
> There will be updates necessary to get our Jira up to date.  I'll follow up
> on that, and ask that everyone double check their tickets.
>
> There have been 106 commits since the 0.5.0 release (listed at the end of
> message). There will be a few more when we pull in the PCAP feature branch.
>
> Completed PRs as of Aug 15 as generated by git log --pretty="%cr %s"
> tags/apache-metron-0.5.0-release..HEAD.
>
> 5 days ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
> via mmiklavc) closes apache/metron#1152
> 13 days ago METRON-1701 Update General notes on the installation of Pycapa
> on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
> 3 weeks ago METRON-1650 Packaging docker containers are too large
> (jameslamb via merrimanr) closes apache/metron#1091
> 3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
> management pack repo info closes apache/incubator-metron#1052
> 3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
> apache/incubator-metron#1126
> 3 weeks ago METRON-1694: Clean up Metron REST docs closes
> apache/incubator-metron#1131
> 4 weeks ago METRON-1606 Add a 'wrap' to incoming messages in the
> metron json parser (ottobackwards) closes apache/metron#1054
> 4 weeks ago METRON-1672 Add metron-alerts's UI unit tests to travis
> build process (justinleet) closes apache/metron#1106
> 4 weeks ago METRON-1684 Fix Markdown problems in 3rdPartyParser.md
> (justinleet) closes apache/metron#1110
> 4 weeks ago METRON-1657 Parser aggregation in storm (justinleet) closes
> apache/metron#1099
> 4 weeks ago METRON-1651 Fixing failing protractor e2e test (tiborm via
> merrimanr) closes apache/metron#1095
> 4 weeks ago METRON-1673 Fix Javadoc errors (justinleet) closes
> apache/metron#1107
> 4 weeks ago METRON-1620: Fixes for forensic clustering use case example
> (mmiklavc via mmiklavc) closes apache/metron#1065
> 4 weeks ago METRON-1659: The platform-info.sh should check for the vagrant
> hostmanager plugin closes apache/incubator-metron#1100
> 4 weeks ago METRON-1658: Upgrade bro to 2.5.4 closes
> apache/incubator-metron#1101
> 4 weeks ago METRON-1236 Add start/stop/restart commands that execute
> successfully, w

Re: [DISCUSS] Metron Parsers in Nifi

As an exercise, let me summarize the points of contention I've seen and lay
out the tradeoffs as I see them. That way we can prioritize what's
important to us in a NiFi implementation and better work towards a
favorable solution (basically, I want to requirements we have for an MVP).
My opinions/comments/questions are in *bold*.  Feel free (and encouraged to
disagree). Keep in mind, this stuff probably exists on a spectrum (we might
want to pick and choose what we do, and possibly even when we do it).

   - Splitting off fieldTransformations from the parser itself
  - In Nifi, we're chaining processors to do our fieldTransformations.
  This can't be particularly automatic from a definition, to the best of my
  knowledge.
  - Our configuration between NiFi and Storm differs (because NiFi is
  building Processors and Storm is just acting on the transforms).
  - *I'm mostly fine with splitting these, IMO we just need to make
  sure it's documented. The current colocation of them feels
slightly sketchy
  to me in general (it feels like it's merging pure parsing and something
  more enrichment oriented).** I also like the idea of exposing Stellar
  transformations as their own Processor.*
 - *Could anyone refresh my memory on why fieldTransformations are
 bundled with parsing directly?*
  - NiFi parser configuration
   - We can do it from ZK, but need to make it available in a manner not
  available in ZK
  - If we don't allow ZK, we can potentially have different sources of
  configs.
 - *I personally don't like this very much. I always hated having
 to hop between things in order to manage these sort of things, but I
 consider that more annoying than blocking.*
  - Specific parsers vs an aggregated parser
  - If they're all specific, it means every user who wants to implement
  a parser (even an existing one) in NiFi, they have to do
additional work to
  make it work in NiFi.
 - *I don't think it's a lot of work on a per parser basis, and we
 might be able to ease this with some clever handling of our
interfaces.
 However, I personally don't like that there's no way to just run an
 existing Metron parser in NiFi without additional
NiFi-specific work.  To
 be clear, I'd prefer to have a quick way for users to take parsers,
 including preexisting parsers, into NiFi.  I don't think this
should be the
 end solution for most parsers, but it does feel like the
minimal viable
 product solution to cover current users mostly as-is.  In my
mind, users
 should be able to take preexisting Storm parsers and be able
to run them up
 and test them in NiFi with minimal involvement, even if the
end state is to
 do a more NiFi-like implementation.*
 - *If we expand this to other platforms (e.g. Spark?), do we
 expect everything to be reimplemented every time?  Or are we
making that
 decision on a platform-by-platform basis?*
 - *I think most parsers, including our own should be optimized as
 needed for NiFi, including whatever Schema work and
versioning we want to
 do, but I don't think that needs to be done right away.
Looking through
 source, our parsers are:*
- *Asa*
- *Bro*
- *CEF*
- *CSV*
- *Fireeye*
- *ISE*
- *JSON*
- *Lancope*
- *Logstash*
- *Palo Alto*
- *Snort*
- *Sourcefire*
 - *I don't know that I want to go through and convert and test all
 of them in the first pass.*
  - Processor vs. RecordReader
  - RecordReader is the NiFi hotness.  Sounds like the interface
  actually is stable, which was really my primary concern with it
(Thanks for
  following up Otto!).
 - *RecordReaders seem like they have positive performance
 implications to them, which I'm definitely in favor of.  The Processor
 approach would work, but given the rates of flow we see, it'd
be extremely
 nice to get the RecordReader benefits.  The schema benefits in
 RecordReaders are more clear if we split fieldTransformations
from parsing
 in NiFi, but that split might be more work (although result in a
 potentially much cleaner implementation of RecordReaders).
This would mean
 we have to do at least some upgrading for every parser we
want to be able
 to run in NiFi.*
 - *How much schema versioning do we need to support as part of a
 first cut? How much of this needs to be managed by NiFi
specific features?*
  - *I'm curious on people's thoughts on if we can do some unification
  on some of our parsers against RecordReader as Simon mentioned.  If we do
  that, do we then need to start wrapping NARs around everything as part of

[DISCUSS] Metron Release 0.6.0?