Re: [DISCUSS] Deprecating MySQL

2018-11-19 Thread Ali Nazemian
Great feature to move to LDAP integration and hopefully Ranger
integration afterwards. Does it need to support LDAP and AD separately?

Cheers,
Ali

On Sat, Nov 17, 2018 at 3:29 AM Otto Fowler  wrote:

> I would like to understand the work required to move our JDBC support ( or
> adapt the current support to the abstraction ) to /contrib.
> We could default and only officially support LDAP, but have the /contrib (
> or /extension_examples ) have a “this is how you would support jdbc for
> auth “ project.
>
>
>
> On November 15, 2018 at 15:01:10, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Yes, makes sense. +1 to that.
>
> On Thu, Nov 15, 2018 at 12:54 PM James Sirota  wrote:
>
> > To clarify my position, I don't have a problem with mySql or any other
> > projects relying on it. mySql in itself is not an issue. What I don't
> > want is for a customer to be presented with an option to chose and
> > configure two options for authenticating the UI, which I think is
> > needless. It adds complexity for not much value. Since LDAP is clearly
> > the better way to go that should be what we support without explicitly
> > giving a user an option to switch to JDBC. A user can still do so by
> > extending our abstractions if that is what they chose to do, but this
> would
> > not be officially supported by us. We would not be providing a config or
> > an mPack to do this. A user would have to do it on their own.
> >
> > James
> >
> >
> >
> > 15.11.2018, 12:15, "Michael Miklavcic" :
> > > Incidentally, even without the Metron piece in the picture, what is the
> > > answer for Ambari's database dependency? Which uses a SQL data store.
> > Does
> > > this actually solve the problem of "customers won't install Metron bc
> SQL
> > > store?" or are there other issues we need to address?
> > >
> > > On Thu, Nov 15, 2018 at 9:30 AM James Sirota 
> wrote:
> > >
> > >> Hi Guys,
> > >>
> > >> My opinion on this, as is with Knox SSO, is that the code should be
> > >> pluggable to support JDBC, but we should not continue to support the
> > >> concrete implementation and expose it to users via a setting. This is
> a
> > >> fairly minor feature and the added complexity of supporting switching
> > >> between JDBC and LDAP is simply not worth it. We need to strike a
> > balance
> > >> between ease of use and capabilities/extensibility. For features that
> > are
> > >> worth it such as with analytics and stream processing, the extra
> > capability
> > >> is worth the added complexity in configuration. But for this, it is
> > not.
> > >> So let's keep JDBC around for a release to allow users to migrate to
> > LDAP,
> > >> deprecate it, and move on.
> > >>
> > >> Thanks,
> > >> James
> > >>
> > >> 13.11.2018, 16:03, "Simon Elliston Ball"  > >:
> > >> > We went over the hbase user settings thing on extensive discussions
> > at
> > >> the time. Storing an arbitrary blob of JSON which is only ever
> > accessed by
> > >> a single key (username) was concluded to be a key value problem, not a
> > >> relational problem. Hbase was concluded to be massive overkill as a
> key
> > >> value store in this usecase, unless it was already there and ready to
> > go,
> > >> which in the case of Metron, it is, for enrichments, threat intel and
> > >> profiles. Hence it ended up in Hbase, as a conveniently present data
> > store
> > >> that matched the usage patterns. See
> > >>
> >
>
> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
> > >> and METRON-1337 for discussion.
> > >> >
> > >> > Simon
> > >> >
> > >> >> On 13 Nov 2018, at 18:50, Michael Miklavcic <
> > >> michael.miklav...@gmail.com> wrote:
> > >> >>
> > >> >> Thanks for the write up Simon. I don't think I see any major
> > problems
> > >> with
> > >> >> deprecating the general sql store. However, just to clarify, Metron
> > >> does
> > >> >> NOT require any specific backing store. It's 100% JPA, which means
> > >> anything
> > >> >> that can be configured with the Spring properties we expose. I
> think
> > >> the
> > >> >> most opinionated thing we do there is ship an extremely basic table
> > >> >> creation script for h2 and mysql as a simple example for schema. As
> > an
> > >> >> example, we simply use H2 in full dev, which is entirely in-memory
> > and
> > >> spun
> > >> >> up automatically from configuration. The recent work by Justin Leet
> > >> removes
> > >> >> the need to use a SQL store at all if you choose LDAP -
> > >> >> https://github.com/apache/metron/pull/1246. I'll let him comment
> > >> further on
> > >> >> this, but I think there is one small change that could be made via
> a
> > >> toggle
> > >> >> in Ambari that would even eliminate the user from seeing JDBC
> > settings
> > >> >> altogether during install if they choose LDAP. Again, I think I'm
> on
> > >> board
> > >> >> with deprecating the SQL backing store as I pointed this out on the
> > >> Knox
> > >> >> thread as well, but I 

Re: [DISCUSS] Deprecating MySQL

2018-11-16 Thread Otto Fowler
I would like to understand the work required to move our JDBC support ( or
adapt the current support to the abstraction ) to /contrib.
We could default and only officially support LDAP, but have the /contrib (
or /extension_examples ) have a “this is how you would support jdbc for
auth “ project.



On November 15, 2018 at 15:01:10, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Yes, makes sense. +1 to that.

On Thu, Nov 15, 2018 at 12:54 PM James Sirota  wrote:

> To clarify my position, I don't have a problem with mySql or any other
> projects relying on it. mySql in itself is not an issue. What I don't
> want is for a customer to be presented with an option to chose and
> configure two options for authenticating the UI, which I think is
> needless. It adds complexity for not much value. Since LDAP is clearly
> the better way to go that should be what we support without explicitly
> giving a user an option to switch to JDBC. A user can still do so by
> extending our abstractions if that is what they chose to do, but this
would
> not be officially supported by us. We would not be providing a config or
> an mPack to do this. A user would have to do it on their own.
>
> James
>
>
>
> 15.11.2018, 12:15, "Michael Miklavcic" :
> > Incidentally, even without the Metron piece in the picture, what is the
> > answer for Ambari's database dependency? Which uses a SQL data store.
> Does
> > this actually solve the problem of "customers won't install Metron bc
SQL
> > store?" or are there other issues we need to address?
> >
> > On Thu, Nov 15, 2018 at 9:30 AM James Sirota 
wrote:
> >
> >> Hi Guys,
> >>
> >> My opinion on this, as is with Knox SSO, is that the code should be
> >> pluggable to support JDBC, but we should not continue to support the
> >> concrete implementation and expose it to users via a setting. This is
a
> >> fairly minor feature and the added complexity of supporting switching
> >> between JDBC and LDAP is simply not worth it. We need to strike a
> balance
> >> between ease of use and capabilities/extensibility. For features that
> are
> >> worth it such as with analytics and stream processing, the extra
> capability
> >> is worth the added complexity in configuration. But for this, it is
> not.
> >> So let's keep JDBC around for a release to allow users to migrate to
> LDAP,
> >> deprecate it, and move on.
> >>
> >> Thanks,
> >> James
> >>
> >> 13.11.2018, 16:03, "Simon Elliston Ball"  >:
> >> > We went over the hbase user settings thing on extensive discussions
> at
> >> the time. Storing an arbitrary blob of JSON which is only ever
> accessed by
> >> a single key (username) was concluded to be a key value problem, not a
> >> relational problem. Hbase was concluded to be massive overkill as a
key
> >> value store in this usecase, unless it was already there and ready to
> go,
> >> which in the case of Metron, it is, for enrichments, threat intel and
> >> profiles. Hence it ended up in Hbase, as a conveniently present data
> store
> >> that matched the usage patterns. See
> >>
>
https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
> >> and METRON-1337 for discussion.
> >> >
> >> > Simon
> >> >
> >> >> On 13 Nov 2018, at 18:50, Michael Miklavcic <
> >> michael.miklav...@gmail.com> wrote:
> >> >>
> >> >> Thanks for the write up Simon. I don't think I see any major
> problems
> >> with
> >> >> deprecating the general sql store. However, just to clarify, Metron
> >> does
> >> >> NOT require any specific backing store. It's 100% JPA, which means
> >> anything
> >> >> that can be configured with the Spring properties we expose. I
think
> >> the
> >> >> most opinionated thing we do there is ship an extremely basic table
> >> >> creation script for h2 and mysql as a simple example for schema. As
> an
> >> >> example, we simply use H2 in full dev, which is entirely in-memory
> and
> >> spun
> >> >> up automatically from configuration. The recent work by Justin Leet
> >> removes
> >> >> the need to use a SQL store at all if you choose LDAP -
> >> >> https://github.com/apache/metron/pull/1246. I'll let him comment
> >> further on
> >> >> this, but I think there is one small change that could be made via
a
> >> toggle
> >> >> in Ambari that would even eliminate the user from seeing JDBC
> settings
> >> >> altogether during install if they choose LDAP. Again, I think I'm
on
> >> board
> >> >> with deprecating the SQL backing store as I pointed this out on the
> >> Knox
> >> >> thread as well, but I just wanted to make sure everyone has an
> accurate
> >> >> picture of the current state.
> >> >>
> >> >> I had to double check on the HBase config you mentioned, but it
does
> >> appear
> >> >> that we use it for the Alerts UI. I don't think I realized we were
> >> storing
> >> >> config there instead of the Zookeeper store we use for other system
> >> >> configuration. Ironically enough, I think that it probably makes
> more
> >> 

Re: [DISCUSS] Deprecating MySQL

2018-11-15 Thread Michael Miklavcic
Yes, makes sense. +1 to that.

On Thu, Nov 15, 2018 at 12:54 PM James Sirota  wrote:

> To clarify my position, I don't have a problem with mySql or any other
> projects relying on it.  mySql in itself is not an issue.  What I don't
> want is for a customer to be presented with an option to chose and
> configure two options for authenticating the UI, which I think is
> needless.  It adds complexity for not much value.  Since LDAP is clearly
> the better way to go that should be what we support without explicitly
> giving a user an option to switch to JDBC.  A user can still do so by
> extending our abstractions if that is what they chose to do, but this would
> not be officially supported by us.  We would not be providing a config or
> an mPack to do this.  A user would have to do it on their own.
>
> James
>
>
>
> 15.11.2018, 12:15, "Michael Miklavcic" :
> > Incidentally, even without the Metron piece in the picture, what is the
> > answer for Ambari's database dependency? Which uses a SQL data store.
> Does
> > this actually solve the problem of "customers won't install Metron bc SQL
> > store?" or are there other issues we need to address?
> >
> > On Thu, Nov 15, 2018 at 9:30 AM James Sirota  wrote:
> >
> >>  Hi Guys,
> >>
> >>  My opinion on this, as is with Knox SSO, is that the code should be
> >>  pluggable to support JDBC, but we should not continue to support the
> >>  concrete implementation and expose it to users via a setting. This is a
> >>  fairly minor feature and the added complexity of supporting switching
> >>  between JDBC and LDAP is simply not worth it. We need to strike a
> balance
> >>  between ease of use and capabilities/extensibility. For features that
> are
> >>  worth it such as with analytics and stream processing, the extra
> capability
> >>  is worth the added complexity in configuration. But for this, it is
> not.
> >>  So let's keep JDBC around for a release to allow users to migrate to
> LDAP,
> >>  deprecate it, and move on.
> >>
> >>  Thanks,
> >>  James
> >>
> >>  13.11.2018, 16:03, "Simon Elliston Ball"  >:
> >>  > We went over the hbase user settings thing on extensive discussions
> at
> >>  the time. Storing an arbitrary blob of JSON which is only ever
> accessed by
> >>  a single key (username) was concluded to be a key value problem, not a
> >>  relational problem. Hbase was concluded to be massive overkill as a key
> >>  value store in this usecase, unless it was already there and ready to
> go,
> >>  which in the case of Metron, it is, for enrichments, threat intel and
> >>  profiles. Hence it ended up in Hbase, as a conveniently present data
> store
> >>  that matched the usage patterns. See
> >>
> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
> >>  and METRON-1337 for discussion.
> >>  >
> >>  > Simon
> >>  >
> >>  >> On 13 Nov 2018, at 18:50, Michael Miklavcic <
> >>  michael.miklav...@gmail.com> wrote:
> >>  >>
> >>  >> Thanks for the write up Simon. I don't think I see any major
> problems
> >>  with
> >>  >> deprecating the general sql store. However, just to clarify, Metron
> >>  does
> >>  >> NOT require any specific backing store. It's 100% JPA, which means
> >>  anything
> >>  >> that can be configured with the Spring properties we expose. I think
> >>  the
> >>  >> most opinionated thing we do there is ship an extremely basic table
> >>  >> creation script for h2 and mysql as a simple example for schema. As
> an
> >>  >> example, we simply use H2 in full dev, which is entirely in-memory
> and
> >>  spun
> >>  >> up automatically from configuration. The recent work by Justin Leet
> >>  removes
> >>  >> the need to use a SQL store at all if you choose LDAP -
> >>  >> https://github.com/apache/metron/pull/1246. I'll let him comment
> >>  further on
> >>  >> this, but I think there is one small change that could be made via a
> >>  toggle
> >>  >> in Ambari that would even eliminate the user from seeing JDBC
> settings
> >>  >> altogether during install if they choose LDAP. Again, I think I'm on
> >>  board
> >>  >> with deprecating the SQL backing store as I pointed this out on the
> >>  Knox
> >>  >> thread as well, but I just wanted to make sure everyone has an
> accurate
> >>  >> picture of the current state.
> >>  >>
> >>  >> I had to double check on the HBase config you mentioned, but it does
> >>  appear
> >>  >> that we use it for the Alerts UI. I don't think I realized we were
> >>  storing
> >>  >> config there instead of the Zookeeper store we use for other system
> >>  >> configuration. Ironically enough, I think that it probably makes
> more
> >>  sense
> >>  >> than the current auth info to store in a traditional sql store,
> however
> >>  >> it's in HBase currently so it's a non-issue wrt SQL/JPA either way,
> as
> >>  you
> >>  >> pointed out.
> >>  >>
> >>  >> Whatever architectural changes we choose to add here, I think we
> need
> >>  to
> >>  >> emphasize 

Re: [DISCUSS] Deprecating MySQL

2018-11-15 Thread James Sirota
To clarify my position, I don't have a problem with mySql or any other projects 
relying on it.  mySql in itself is not an issue.  What I don't want is for a 
customer to be presented with an option to chose and configure two options for 
authenticating the UI, which I think is needless.  It adds complexity for not 
much value.  Since LDAP is clearly the better way to go that should be what we 
support without explicitly giving a user an option to switch to JDBC.  A user 
can still do so by extending our abstractions if that is what they chose to do, 
but this would not be officially supported by us.  We would not be providing a 
config or an mPack to do this.  A user would have to do it on their own. 

James



15.11.2018, 12:15, "Michael Miklavcic" :
> Incidentally, even without the Metron piece in the picture, what is the
> answer for Ambari's database dependency? Which uses a SQL data store. Does
> this actually solve the problem of "customers won't install Metron bc SQL
> store?" or are there other issues we need to address?
>
> On Thu, Nov 15, 2018 at 9:30 AM James Sirota  wrote:
>
>>  Hi Guys,
>>
>>  My opinion on this, as is with Knox SSO, is that the code should be
>>  pluggable to support JDBC, but we should not continue to support the
>>  concrete implementation and expose it to users via a setting. This is a
>>  fairly minor feature and the added complexity of supporting switching
>>  between JDBC and LDAP is simply not worth it. We need to strike a balance
>>  between ease of use and capabilities/extensibility. For features that are
>>  worth it such as with analytics and stream processing, the extra capability
>>  is worth the added complexity in configuration. But for this, it is not.
>>  So let's keep JDBC around for a release to allow users to migrate to LDAP,
>>  deprecate it, and move on.
>>
>>  Thanks,
>>  James
>>
>>  13.11.2018, 16:03, "Simon Elliston Ball" :
>>  > We went over the hbase user settings thing on extensive discussions at
>>  the time. Storing an arbitrary blob of JSON which is only ever accessed by
>>  a single key (username) was concluded to be a key value problem, not a
>>  relational problem. Hbase was concluded to be massive overkill as a key
>>  value store in this usecase, unless it was already there and ready to go,
>>  which in the case of Metron, it is, for enrichments, threat intel and
>>  profiles. Hence it ended up in Hbase, as a conveniently present data store
>>  that matched the usage patterns. See
>>  
>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>>  and METRON-1337 for discussion.
>>  >
>>  > Simon
>>  >
>>  >> On 13 Nov 2018, at 18:50, Michael Miklavcic <
>>  michael.miklav...@gmail.com> wrote:
>>  >>
>>  >> Thanks for the write up Simon. I don't think I see any major problems
>>  with
>>  >> deprecating the general sql store. However, just to clarify, Metron
>>  does
>>  >> NOT require any specific backing store. It's 100% JPA, which means
>>  anything
>>  >> that can be configured with the Spring properties we expose. I think
>>  the
>>  >> most opinionated thing we do there is ship an extremely basic table
>>  >> creation script for h2 and mysql as a simple example for schema. As an
>>  >> example, we simply use H2 in full dev, which is entirely in-memory and
>>  spun
>>  >> up automatically from configuration. The recent work by Justin Leet
>>  removes
>>  >> the need to use a SQL store at all if you choose LDAP -
>>  >> https://github.com/apache/metron/pull/1246. I'll let him comment
>>  further on
>>  >> this, but I think there is one small change that could be made via a
>>  toggle
>>  >> in Ambari that would even eliminate the user from seeing JDBC settings
>>  >> altogether during install if they choose LDAP. Again, I think I'm on
>>  board
>>  >> with deprecating the SQL backing store as I pointed this out on the
>>  Knox
>>  >> thread as well, but I just wanted to make sure everyone has an accurate
>>  >> picture of the current state.
>>  >>
>>  >> I had to double check on the HBase config you mentioned, but it does
>>  appear
>>  >> that we use it for the Alerts UI. I don't think I realized we were
>>  storing
>>  >> config there instead of the Zookeeper store we use for other system
>>  >> configuration. Ironically enough, I think that it probably makes more
>>  sense
>>  >> than the current auth info to store in a traditional sql store, however
>>  >> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as
>>  you
>>  >> pointed out.
>>  >>
>>  >> Whatever architectural changes we choose to add here, I think we need
>>  to
>>  >> emphasize pluggability regardless of the specific implementation. That
>>  is
>>  >> to say, I don't think we should make a hard requirement on Knox, in
>>  order
>>  >> to get LDAP, in order to deprecate an optional general SQL backing
>>  store.
>>  >> It makes sensible defaults if that's where we want to go, which is 

Re: [DISCUSS] Deprecating MySQL

2018-11-15 Thread Michael Miklavcic
Just to clarify this point, I am not proposing we alter any interactions
with Ambari and backing stores. I want to make sure we cover all the
problems and provide good solutions where we're able. Again, just to
reiterate my earlier point, I agree with the move to deprecate SQL/JPA for
the UI. And I agree with James about pluggability and simplifying the
surface area of options we expose to users via Ambari config.

Best,
Mike

On Thu, Nov 15, 2018 at 12:14 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Incidentally, even without the Metron piece in the picture, what is the
> answer for Ambari's database dependency? Which uses a SQL data store. Does
> this actually solve the problem of "customers won't install Metron bc SQL
> store?" or are there other issues we need to address?
>
> On Thu, Nov 15, 2018 at 9:30 AM James Sirota  wrote:
>
>> Hi Guys,
>>
>> My opinion on this, as is with Knox SSO, is that the code should be
>> pluggable to support JDBC, but we should not continue to support the
>> concrete implementation and expose it to users via a setting.  This is a
>> fairly minor feature and the added complexity of supporting switching
>> between JDBC and LDAP is simply not worth it.  We need to strike a balance
>> between ease of use and capabilities/extensibility.  For features that are
>> worth it such as with analytics and stream processing, the extra capability
>> is worth the added complexity in configuration.  But for this, it is not.
>> So let's keep JDBC around for a release to allow users to migrate to LDAP,
>> deprecate it, and move on.
>>
>> Thanks,
>> James
>>
>> 13.11.2018, 16:03, "Simon Elliston Ball" :
>> > We went over the hbase user settings thing on extensive discussions at
>> the time. Storing an arbitrary blob of JSON which is only ever accessed by
>> a single key (username) was concluded to be a key value problem, not a
>> relational problem. Hbase was concluded to be massive overkill as a key
>> value store in this usecase, unless it was already there and ready to go,
>> which in the case of Metron, it is, for enrichments, threat intel and
>> profiles. Hence it ended up in Hbase, as a conveniently present data store
>> that matched the usage patterns. See
>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>> and METRON-1337 for discussion.
>> >
>> > Simon
>> >
>> >>  On 13 Nov 2018, at 18:50, Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>> >>
>> >>  Thanks for the write up Simon. I don't think I see any major problems
>> with
>> >>  deprecating the general sql store. However, just to clarify, Metron
>> does
>> >>  NOT require any specific backing store. It's 100% JPA, which means
>> anything
>> >>  that can be configured with the Spring properties we expose. I think
>> the
>> >>  most opinionated thing we do there is ship an extremely basic table
>> >>  creation script for h2 and mysql as a simple example for schema. As an
>> >>  example, we simply use H2 in full dev, which is entirely in-memory
>> and spun
>> >>  up automatically from configuration. The recent work by Justin Leet
>> removes
>> >>  the need to use a SQL store at all if you choose LDAP -
>> >>  https://github.com/apache/metron/pull/1246. I'll let him comment
>> further on
>> >>  this, but I think there is one small change that could be made via a
>> toggle
>> >>  in Ambari that would even eliminate the user from seeing JDBC settings
>> >>  altogether during install if they choose LDAP. Again, I think I'm on
>> board
>> >>  with deprecating the SQL backing store as I pointed this out on the
>> Knox
>> >>  thread as well, but I just wanted to make sure everyone has an
>> accurate
>> >>  picture of the current state.
>> >>
>> >>  I had to double check on the HBase config you mentioned, but it does
>> appear
>> >>  that we use it for the Alerts UI. I don't think I realized we were
>> storing
>> >>  config there instead of the Zookeeper store we use for other system
>> >>  configuration. Ironically enough, I think that it probably makes more
>> sense
>> >>  than the current auth info to store in a traditional sql store,
>> however
>> >>  it's in HBase currently so it's a non-issue wrt SQL/JPA either way,
>> as you
>> >>  pointed out.
>> >>
>> >>  Whatever architectural changes we choose to add here, I think we need
>> to
>> >>  emphasize pluggability regardless of the specific implementation.
>> That is
>> >>  to say, I don't think we should make a hard requirement on Knox, in
>> order
>> >>  to get LDAP, in order to deprecate an optional general SQL backing
>> store.
>> >>  It makes sensible defaults if that's where we want to go, which is
>> the way
>> >>  we have done things for most of the successful features I've seen in
>> >>  Metron. Provide all the options should a user desire them, but
>> abstract
>> >>  away the complexity in the UIs.
>> >>
>> >>  Best,
>> >>  Mike
>> >>
>> >>  On Tue, Nov 13, 2018 at 5:42 AM 

Re: [DISCUSS] Deprecating MySQL

2018-11-15 Thread Michael Miklavcic
Incidentally, even without the Metron piece in the picture, what is the
answer for Ambari's database dependency? Which uses a SQL data store. Does
this actually solve the problem of "customers won't install Metron bc SQL
store?" or are there other issues we need to address?

On Thu, Nov 15, 2018 at 9:30 AM James Sirota  wrote:

> Hi Guys,
>
> My opinion on this, as is with Knox SSO, is that the code should be
> pluggable to support JDBC, but we should not continue to support the
> concrete implementation and expose it to users via a setting.  This is a
> fairly minor feature and the added complexity of supporting switching
> between JDBC and LDAP is simply not worth it.  We need to strike a balance
> between ease of use and capabilities/extensibility.  For features that are
> worth it such as with analytics and stream processing, the extra capability
> is worth the added complexity in configuration.  But for this, it is not.
> So let's keep JDBC around for a release to allow users to migrate to LDAP,
> deprecate it, and move on.
>
> Thanks,
> James
>
> 13.11.2018, 16:03, "Simon Elliston Ball" :
> > We went over the hbase user settings thing on extensive discussions at
> the time. Storing an arbitrary blob of JSON which is only ever accessed by
> a single key (username) was concluded to be a key value problem, not a
> relational problem. Hbase was concluded to be massive overkill as a key
> value store in this usecase, unless it was already there and ready to go,
> which in the case of Metron, it is, for enrichments, threat intel and
> profiles. Hence it ended up in Hbase, as a conveniently present data store
> that matched the usage patterns. See
> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
> and METRON-1337 for discussion.
> >
> > Simon
> >
> >>  On 13 Nov 2018, at 18:50, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
> >>
> >>  Thanks for the write up Simon. I don't think I see any major problems
> with
> >>  deprecating the general sql store. However, just to clarify, Metron
> does
> >>  NOT require any specific backing store. It's 100% JPA, which means
> anything
> >>  that can be configured with the Spring properties we expose. I think
> the
> >>  most opinionated thing we do there is ship an extremely basic table
> >>  creation script for h2 and mysql as a simple example for schema. As an
> >>  example, we simply use H2 in full dev, which is entirely in-memory and
> spun
> >>  up automatically from configuration. The recent work by Justin Leet
> removes
> >>  the need to use a SQL store at all if you choose LDAP -
> >>  https://github.com/apache/metron/pull/1246. I'll let him comment
> further on
> >>  this, but I think there is one small change that could be made via a
> toggle
> >>  in Ambari that would even eliminate the user from seeing JDBC settings
> >>  altogether during install if they choose LDAP. Again, I think I'm on
> board
> >>  with deprecating the SQL backing store as I pointed this out on the
> Knox
> >>  thread as well, but I just wanted to make sure everyone has an accurate
> >>  picture of the current state.
> >>
> >>  I had to double check on the HBase config you mentioned, but it does
> appear
> >>  that we use it for the Alerts UI. I don't think I realized we were
> storing
> >>  config there instead of the Zookeeper store we use for other system
> >>  configuration. Ironically enough, I think that it probably makes more
> sense
> >>  than the current auth info to store in a traditional sql store, however
> >>  it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as
> you
> >>  pointed out.
> >>
> >>  Whatever architectural changes we choose to add here, I think we need
> to
> >>  emphasize pluggability regardless of the specific implementation. That
> is
> >>  to say, I don't think we should make a hard requirement on Knox, in
> order
> >>  to get LDAP, in order to deprecate an optional general SQL backing
> store.
> >>  It makes sensible defaults if that's where we want to go, which is the
> way
> >>  we have done things for most of the successful features I've seen in
> >>  Metron. Provide all the options should a user desire them, but abstract
> >>  away the complexity in the UIs.
> >>
> >>  Best,
> >>  Mike
> >>
> >>  On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
> >>  si...@simonellistonball.com> wrote:
> >>
> >>>  I've been coming across a number of organisations who are blocked from
> >>>  installing Metron by the MySQL auth database.
> >>>
> >>>  The main problems with our MySQL default are:
> >>>
> >>>  * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
> >>>  security platform and usually where the deployment conversation ends
> for me
> >>>  * MySQL install varies from platform to platform
> >>>  * An additional database to manage, backup, etc. so now I have to
> talk to a
> >>>  DBA
> >>>  * Harder to maintain HA for this without 

Re: [DISCUSS] Deprecating MySQL

2018-11-15 Thread Scott C. Cote
I’m certain that I can make the “adapter” for elastic too.   and the adapter 
compatible with HBase for those that want hbase (for other reasons).  Sorry 
that I was not clear about that.

SCott
Scott C. Cote
scottcc...@gmail.com
972.900.1561

twitter: @scottccote



> On Nov 13, 2018, at 5:51 PM, Michael Miklavcic  
> wrote:
> 
> Hey Scott, Solr is an install option, not a hard requirement. Users can
> also choose Elasticsearch currently. HBase is the only option we currently
> provide for streaming enrichments, ie we can depend on it being there for
> every install.
> 
> On Tue, Nov 13, 2018, 4:31 PM Scott C. Cote  
>> Simon,
>> 
>> Since ya’ll are going to have SOLR in your installation (is this still
>> true?), I could make the profile system rely upon SOLR instead of HBASE.
>> At Lucidworks, I did this very thing with proprietary code, but I can make
>> an adapter so the binaries can be stored in SOLR of arbitrary size.  We
>> have chatted briefly about this in Slack.   Am trying to stabilize the
>> employment situation, so that’s why I have not made any progress.
>> 
>> Is there still interest here?
>> 
>> SCott
>> Scott C. Cote
>> scottcc...@gmail.com
>> 972.900.1561
>> 
>> twitter: @scottccote
>> 
>> 
>> 
>>> On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball <
>> si...@simonellistonball.com> wrote:
>>> 
>>> We went over the hbase user settings thing on extensive discussions at
>> the time. Storing an arbitrary blob of JSON which is only ever accessed by
>> a single key (username) was concluded to be a key value problem, not a
>> relational problem. Hbase was concluded to be massive overkill as a key
>> value store in this usecase, unless it was already there and ready to go,
>> which in the case of Metron, it is, for enrichments, threat intel and
>> profiles. Hence it ended up in Hbase, as a conveniently present data store
>> that matched the usage patterns. See
>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>> and METRON-1337 for discussion.
>>> 
>>> Simon
>>> 
 On 13 Nov 2018, at 18:50, Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
 
 Thanks for the write up Simon. I don't think I see any major problems
>> with
 deprecating the general sql store. However, just to clarify, Metron does
 NOT require any specific backing store. It's 100% JPA, which means
>> anything
 that can be configured with the Spring properties we expose. I think the
 most opinionated thing we do there is ship an extremely basic table
 creation script for h2 and mysql as a simple example for schema. As an
 example, we simply use H2 in full dev, which is entirely in-memory and
>> spun
 up automatically from configuration. The recent work by Justin Leet
>> removes
 the need to use a SQL store at all if you choose LDAP -
 https://github.com/apache/metron/pull/1246. I'll let him comment
>> further on
 this, but I think there is one small change that could be made via a
>> toggle
 in Ambari that would even eliminate the user from seeing JDBC settings
 altogether during install if they choose LDAP. Again, I think I'm on
>> board
 with deprecating the SQL backing store as I pointed this out on the Knox
 thread as well, but I just wanted to make sure everyone has an accurate
 picture of the current state.
 
 I had to double check on the HBase config you mentioned, but it does
>> appear
 that we use it for the Alerts UI. I don't think I realized we were
>> storing
 config there instead of the Zookeeper store we use for other system
 configuration. Ironically enough, I think that it probably makes more
>> sense
 than the current auth info to store in a traditional sql store, however
 it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as
>> you
 pointed out.
 
 Whatever architectural changes we choose to add here, I think we need to
 emphasize pluggability regardless of the specific implementation. That
>> is
 to say, I don't think we should make a hard requirement on Knox, in
>> order
 to get LDAP, in order to deprecate an optional general SQL backing
>> store.
 It makes sensible defaults if that's where we want to go, which is the
>> way
 we have done things for most of the successful features I've seen in
 Metron. Provide all the options should a user desire them, but abstract
 away the complexity in the UIs.
 
 Best,
 Mike
 
 
 On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
 si...@simonellistonball.com> wrote:
 
> I've been coming across a number of organisations who are blocked from
> installing Metron by the MySQL auth database.
> 
> The main problems with our MySQL default are:
> 
> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
> security platform and usually where the deployment conversation ends
>> 

Re: [DISCUSS] Deprecating MySQL

2018-11-15 Thread James Sirota
Hi Guys,

My opinion on this, as is with Knox SSO, is that the code should be pluggable 
to support JDBC, but we should not continue to support the concrete 
implementation and expose it to users via a setting.  This is a fairly minor 
feature and the added complexity of supporting switching between JDBC and LDAP 
is simply not worth it.  We need to strike a balance between ease of use and 
capabilities/extensibility.  For features that are worth it such as with 
analytics and stream processing, the extra capability is worth the added 
complexity in configuration.  But for this, it is not.  So let's keep JDBC 
around for a release to allow users to migrate to LDAP, deprecate it, and move 
on.

Thanks,
James 

13.11.2018, 16:03, "Simon Elliston Ball" :
> We went over the hbase user settings thing on extensive discussions at the 
> time. Storing an arbitrary blob of JSON which is only ever accessed by a 
> single key (username) was concluded to be a key value problem, not a 
> relational problem. Hbase was concluded to be massive overkill as a key value 
> store in this usecase, unless it was already there and ready to go, which in 
> the case of Metron, it is, for enrichments, threat intel and profiles. Hence 
> it ended up in Hbase, as a conveniently present data store that matched the 
> usage patterns. See 
> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>  and METRON-1337 for discussion.
>
> Simon
>
>>  On 13 Nov 2018, at 18:50, Michael Miklavcic  
>> wrote:
>>
>>  Thanks for the write up Simon. I don't think I see any major problems with
>>  deprecating the general sql store. However, just to clarify, Metron does
>>  NOT require any specific backing store. It's 100% JPA, which means anything
>>  that can be configured with the Spring properties we expose. I think the
>>  most opinionated thing we do there is ship an extremely basic table
>>  creation script for h2 and mysql as a simple example for schema. As an
>>  example, we simply use H2 in full dev, which is entirely in-memory and spun
>>  up automatically from configuration. The recent work by Justin Leet removes
>>  the need to use a SQL store at all if you choose LDAP -
>>  https://github.com/apache/metron/pull/1246. I'll let him comment further on
>>  this, but I think there is one small change that could be made via a toggle
>>  in Ambari that would even eliminate the user from seeing JDBC settings
>>  altogether during install if they choose LDAP. Again, I think I'm on board
>>  with deprecating the SQL backing store as I pointed this out on the Knox
>>  thread as well, but I just wanted to make sure everyone has an accurate
>>  picture of the current state.
>>
>>  I had to double check on the HBase config you mentioned, but it does appear
>>  that we use it for the Alerts UI. I don't think I realized we were storing
>>  config there instead of the Zookeeper store we use for other system
>>  configuration. Ironically enough, I think that it probably makes more sense
>>  than the current auth info to store in a traditional sql store, however
>>  it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
>>  pointed out.
>>
>>  Whatever architectural changes we choose to add here, I think we need to
>>  emphasize pluggability regardless of the specific implementation. That is
>>  to say, I don't think we should make a hard requirement on Knox, in order
>>  to get LDAP, in order to deprecate an optional general SQL backing store.
>>  It makes sensible defaults if that's where we want to go, which is the way
>>  we have done things for most of the successful features I've seen in
>>  Metron. Provide all the options should a user desire them, but abstract
>>  away the complexity in the UIs.
>>
>>  Best,
>>  Mike
>>
>>  On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
>>  si...@simonellistonball.com> wrote:
>>
>>>  I've been coming across a number of organisations who are blocked from
>>>  installing Metron by the MySQL auth database.
>>>
>>>  The main problems with our MySQL default are:
>>>
>>>  * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
>>>  security platform and usually where the deployment conversation ends for me
>>>  * MySQL install varies from platform to platform
>>>  * An additional database to manage, backup, etc. so now I have to talk to a
>>>  DBA
>>>  * Harder to maintain HA for this without externalising and fighting against
>>>  our defaults
>>>  * There are a lot of dependencies for just storing a table of users
>>>  (Eclipse Link, JPA, the MySQL server and the need to get clients installed
>>>  and pushed separately because of licence requirements)
>>>  * Organisations don't want to have to manage yet another user source of
>>>  truth since this leads to operational complexity.
>>>
>>>  In short, managing our own user store makes very little sense to operations
>>>  users.
>>>
>>>  Some of these (licence and 

Re: [DISCUSS] Deprecating MySQL

2018-11-14 Thread Scott Cote
Allow someone to remove the dependence on hbase and only use Solr.  For the way 
you use hbase for profiles, I can make Solr fulfill the same role.

Sent from my iPhone

> On Nov 13, 2018, at 5:41 PM, James Sirota  wrote:
> 
> Metron already works with Solr.  It's not default yet, but all the internals 
> are in.  The profiler currently works with Hbase and I don't think we were 
> ever planning on switching it to Solr.  Just to clarify - metron can index 
> telemetry into Solr.  However, profiling of that telemetry is supported via 
> Hbase.  What exactly are you planning on doing?
> 
> Thanks,
> James 
> 
> 13.11.2018, 16:31, "Scott C. Cote" :
>> Simon,
>> 
>> Since ya’ll are going to have SOLR in your installation (is this still 
>> true?), I could make the profile system rely upon SOLR instead of HBASE. At 
>> Lucidworks, I did this very thing with proprietary code, but I can make an 
>> adapter so the binaries can be stored in SOLR of arbitrary size. We have 
>> chatted briefly about this in Slack. Am trying to stabilize the employment 
>> situation, so that’s why I have not made any progress.
>> 
>> Is there still interest here?
>> 
>> SCott
>> Scott C. Cote
>> scottcc...@gmail.com
>> 972.900.1561
>> 
>> twitter: @scottccote
>> 
>>>  On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball 
>>>  wrote:
>>> 
>>>  We went over the hbase user settings thing on extensive discussions at the 
>>> time. Storing an arbitrary blob of JSON which is only ever accessed by a 
>>> single key (username) was concluded to be a key value problem, not a 
>>> relational problem. Hbase was concluded to be massive overkill as a key 
>>> value store in this usecase, unless it was already there and ready to go, 
>>> which in the case of Metron, it is, for enrichments, threat intel and 
>>> profiles. Hence it ended up in Hbase, as a conveniently present data store 
>>> that matched the usage patterns. See 
>>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>>>  and METRON-1337 for discussion.
>>> 
>>>  Simon
>>> 
  On 13 Nov 2018, at 18:50, Michael Miklavcic  
 wrote:
 
  Thanks for the write up Simon. I don't think I see any major problems with
  deprecating the general sql store. However, just to clarify, Metron does
  NOT require any specific backing store. It's 100% JPA, which means 
 anything
  that can be configured with the Spring properties we expose. I think the
  most opinionated thing we do there is ship an extremely basic table
  creation script for h2 and mysql as a simple example for schema. As an
  example, we simply use H2 in full dev, which is entirely in-memory and 
 spun
  up automatically from configuration. The recent work by Justin Leet 
 removes
  the need to use a SQL store at all if you choose LDAP -
  https://github.com/apache/metron/pull/1246. I'll let him comment further 
 on
  this, but I think there is one small change that could be made via a 
 toggle
  in Ambari that would even eliminate the user from seeing JDBC settings
  altogether during install if they choose LDAP. Again, I think I'm on board
  with deprecating the SQL backing store as I pointed this out on the Knox
  thread as well, but I just wanted to make sure everyone has an accurate
  picture of the current state.
 
  I had to double check on the HBase config you mentioned, but it does 
 appear
  that we use it for the Alerts UI. I don't think I realized we were storing
  config there instead of the Zookeeper store we use for other system
  configuration. Ironically enough, I think that it probably makes more 
 sense
  than the current auth info to store in a traditional sql store, however
  it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
  pointed out.
 
  Whatever architectural changes we choose to add here, I think we need to
  emphasize pluggability regardless of the specific implementation. That is
  to say, I don't think we should make a hard requirement on Knox, in order
  to get LDAP, in order to deprecate an optional general SQL backing store.
  It makes sensible defaults if that's where we want to go, which is the way
  we have done things for most of the successful features I've seen in
  Metron. Provide all the options should a user desire them, but abstract
  away the complexity in the UIs.
 
  Best,
  Mike
 
  On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
  si...@simonellistonball.com> wrote:
 
>  I've been coming across a number of organisations who are blocked from
>  installing Metron by the MySQL auth database.
> 
>  The main problems with our MySQL default are:
> 
>  * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
>  security platform and usually where the deployment 

Re: [DISCUSS] Deprecating MySQL

2018-11-14 Thread Vets, Laurens
Which option would there be for people running installs not tied to an
already existing LDAP or other authentication layer?

Would that be where the local LDAP comes into play?

On 13-Nov-18 04:42, Simon Elliston Ball wrote:
> I've been coming across a number of organisations who are blocked from
> installing Metron by the MySQL auth database.
>
> The main problems with our MySQL default are:
>
> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
> security platform and usually where the deployment conversation ends for me
> * MySQL install varies from platform to platform
> * An additional database to manage, backup, etc. so now I have to talk to a
> DBA
> * Harder to maintain HA for this without externalising and fighting against
> our defaults
> * There are a lot of dependencies for just storing a table of users
> (Eclipse Link, JPA, the MySQL server and the need to get clients installed
> and pushed separately because of licence requirements)
> * Organisations don't want to have to manage yet another user source of
> truth since this leads to operational complexity.
>
> In short, managing our own user store makes very little sense to operations
> users.
>
> Some of these (licence and inconsistency for example) could be solved by
> changing our default DB to something like Postgres, which has easier terms
> to deal with. We could start encrypting passwords, but there would still be
> a lot of dependencies to store users, which is a problem much better solved
> by LDAP.
>
> Now that we have the option to use LDAP for user storage, I would suggest
> that we deprecate and ultimately remove all the RDBMS and ORM dependencies,
> which significantly reduces our dependencies and simplifies deployment and
> long term management of Metron clusters.
>
> So I propose that we deprecate the RDBMS use in the next Apache release,
> and then strip out the RDBMS stuff in the following. We would continue to
> use LDAP for users and HBase for non-LDAPy user settings (as we currently
> do). We should also provide a small demo LDAP for full dev. Since we are
> looking at adding Knox into the stack, that project provides a convenient
> mini-LDAP demo service which would do this job without the need to add
> additional components.
>
> Thoughts? Anyone relying on MySQL for users (if so, are you aware that your
> passwords are all plaintext? How do you currently handle the shortcomings
> and admin overhead?) Any objections?
>
> Simon
>


Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Michael Miklavcic
Hey Scott, Solr is an install option, not a hard requirement. Users can
also choose Elasticsearch currently. HBase is the only option we currently
provide for streaming enrichments, ie we can depend on it being there for
every install.

On Tue, Nov 13, 2018, 4:31 PM Scott C. Cote  Simon,
>
> Since ya’ll are going to have SOLR in your installation (is this still
> true?), I could make the profile system rely upon SOLR instead of HBASE.
> At Lucidworks, I did this very thing with proprietary code, but I can make
> an adapter so the binaries can be stored in SOLR of arbitrary size.  We
> have chatted briefly about this in Slack.   Am trying to stabilize the
> employment situation, so that’s why I have not made any progress.
>
> Is there still interest here?
>
> SCott
> Scott C. Cote
> scottcc...@gmail.com
> 972.900.1561
>
> twitter: @scottccote
>
>
>
> > On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
> >
> > We went over the hbase user settings thing on extensive discussions at
> the time. Storing an arbitrary blob of JSON which is only ever accessed by
> a single key (username) was concluded to be a key value problem, not a
> relational problem. Hbase was concluded to be massive overkill as a key
> value store in this usecase, unless it was already there and ready to go,
> which in the case of Metron, it is, for enrichments, threat intel and
> profiles. Hence it ended up in Hbase, as a conveniently present data store
> that matched the usage patterns. See
> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
> and METRON-1337 for discussion.
> >
> > Simon
> >
> >> On 13 Nov 2018, at 18:50, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
> >>
> >> Thanks for the write up Simon. I don't think I see any major problems
> with
> >> deprecating the general sql store. However, just to clarify, Metron does
> >> NOT require any specific backing store. It's 100% JPA, which means
> anything
> >> that can be configured with the Spring properties we expose. I think the
> >> most opinionated thing we do there is ship an extremely basic table
> >> creation script for h2 and mysql as a simple example for schema. As an
> >> example, we simply use H2 in full dev, which is entirely in-memory and
> spun
> >> up automatically from configuration. The recent work by Justin Leet
> removes
> >> the need to use a SQL store at all if you choose LDAP -
> >> https://github.com/apache/metron/pull/1246. I'll let him comment
> further on
> >> this, but I think there is one small change that could be made via a
> toggle
> >> in Ambari that would even eliminate the user from seeing JDBC settings
> >> altogether during install if they choose LDAP. Again, I think I'm on
> board
> >> with deprecating the SQL backing store as I pointed this out on the Knox
> >> thread as well, but I just wanted to make sure everyone has an accurate
> >> picture of the current state.
> >>
> >> I had to double check on the HBase config you mentioned, but it does
> appear
> >> that we use it for the Alerts UI. I don't think I realized we were
> storing
> >> config there instead of the Zookeeper store we use for other system
> >> configuration. Ironically enough, I think that it probably makes more
> sense
> >> than the current auth info to store in a traditional sql store, however
> >> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as
> you
> >> pointed out.
> >>
> >> Whatever architectural changes we choose to add here, I think we need to
> >> emphasize pluggability regardless of the specific implementation. That
> is
> >> to say, I don't think we should make a hard requirement on Knox, in
> order
> >> to get LDAP, in order to deprecate an optional general SQL backing
> store.
> >> It makes sensible defaults if that's where we want to go, which is the
> way
> >> we have done things for most of the successful features I've seen in
> >> Metron. Provide all the options should a user desire them, but abstract
> >> away the complexity in the UIs.
> >>
> >> Best,
> >> Mike
> >>
> >>
> >> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
> >> si...@simonellistonball.com> wrote:
> >>
> >>> I've been coming across a number of organisations who are blocked from
> >>> installing Metron by the MySQL auth database.
> >>>
> >>> The main problems with our MySQL default are:
> >>>
> >>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
> >>> security platform and usually where the deployment conversation ends
> for me
> >>> * MySQL install varies from platform to platform
> >>> * An additional database to manage, backup, etc. so now I have to talk
> to a
> >>> DBA
> >>> * Harder to maintain HA for this without externalising and fighting
> against
> >>> our defaults
> >>> * There are a lot of dependencies for just storing a table of users
> >>> (Eclipse Link, JPA, the MySQL server and the need to get clients
> 

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Michael Miklavcic
Haha, yeah I had to dredge that up from February as well to remember what
ultimately ended up going into HBase.  Before you get your hackles up, I
think you misunderstood me - I believe we're on the same page. I am saying
that the SQL store would have made sense just fine in that case. Not that
we should actually change it now or use it as a reason to architect things
differently.

On Tue, Nov 13, 2018, 4:03 PM Simon Elliston Ball <
si...@simonellistonball.com wrote:

> We went over the hbase user settings thing on extensive discussions at the
> time. Storing an arbitrary blob of JSON which is only ever accessed by a
> single key (username) was concluded to be a key value problem, not a
> relational problem. Hbase was concluded to be massive overkill as a key
> value store in this usecase, unless it was already there and ready to go,
> which in the case of Metron, it is, for enrichments, threat intel and
> profiles. Hence it ended up in Hbase, as a conveniently present data store
> that matched the usage patterns. See
> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
> and METRON-1337 for discussion.
>
> Simon
>
> > On 13 Nov 2018, at 18:50, Michael Miklavcic 
> wrote:
> >
> > Thanks for the write up Simon. I don't think I see any major problems
> with
> > deprecating the general sql store. However, just to clarify, Metron does
> > NOT require any specific backing store. It's 100% JPA, which means
> anything
> > that can be configured with the Spring properties we expose. I think the
> > most opinionated thing we do there is ship an extremely basic table
> > creation script for h2 and mysql as a simple example for schema. As an
> > example, we simply use H2 in full dev, which is entirely in-memory and
> spun
> > up automatically from configuration. The recent work by Justin Leet
> removes
> > the need to use a SQL store at all if you choose LDAP -
> > https://github.com/apache/metron/pull/1246. I'll let him comment
> further on
> > this, but I think there is one small change that could be made via a
> toggle
> > in Ambari that would even eliminate the user from seeing JDBC settings
> > altogether during install if they choose LDAP. Again, I think I'm on
> board
> > with deprecating the SQL backing store as I pointed this out on the Knox
> > thread as well, but I just wanted to make sure everyone has an accurate
> > picture of the current state.
> >
> > I had to double check on the HBase config you mentioned, but it does
> appear
> > that we use it for the Alerts UI. I don't think I realized we were
> storing
> > config there instead of the Zookeeper store we use for other system
> > configuration. Ironically enough, I think that it probably makes more
> sense
> > than the current auth info to store in a traditional sql store, however
> > it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as
> you
> > pointed out.
> >
> > Whatever architectural changes we choose to add here, I think we need to
> > emphasize pluggability regardless of the specific implementation. That is
> > to say, I don't think we should make a hard requirement on Knox, in order
> > to get LDAP, in order to deprecate an optional general SQL backing store.
> > It makes sensible defaults if that's where we want to go, which is the
> way
> > we have done things for most of the successful features I've seen in
> > Metron. Provide all the options should a user desire them, but abstract
> > away the complexity in the UIs.
> >
> > Best,
> > Mike
> >
> >
> > On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> I've been coming across a number of organisations who are blocked from
> >> installing Metron by the MySQL auth database.
> >>
> >> The main problems with our MySQL default are:
> >>
> >> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
> >> security platform and usually where the deployment conversation ends
> for me
> >> * MySQL install varies from platform to platform
> >> * An additional database to manage, backup, etc. so now I have to talk
> to a
> >> DBA
> >> * Harder to maintain HA for this without externalising and fighting
> against
> >> our defaults
> >> * There are a lot of dependencies for just storing a table of users
> >> (Eclipse Link, JPA, the MySQL server and the need to get clients
> installed
> >> and pushed separately because of licence requirements)
> >> * Organisations don't want to have to manage yet another user source of
> >> truth since this leads to operational complexity.
> >>
> >> In short, managing our own user store makes very little sense to
> operations
> >> users.
> >>
> >> Some of these (licence and inconsistency for example) could be solved by
> >> changing our default DB to something like Postgres, which has easier
> terms
> >> to deal with. We could start encrypting passwords, but there would
> still be
> >> a lot of dependencies to store 

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread James Sirota
Metron already works with Solr.  It's not default yet, but all the internals 
are in.  The profiler currently works with Hbase and I don't think we were ever 
planning on switching it to Solr.  Just to clarify - metron can index telemetry 
into Solr.  However, profiling of that telemetry is supported via Hbase.  What 
exactly are you planning on doing?

Thanks,
James 

13.11.2018, 16:31, "Scott C. Cote" :
> Simon,
>
> Since ya’ll are going to have SOLR in your installation (is this still 
> true?), I could make the profile system rely upon SOLR instead of HBASE. At 
> Lucidworks, I did this very thing with proprietary code, but I can make an 
> adapter so the binaries can be stored in SOLR of arbitrary size. We have 
> chatted briefly about this in Slack. Am trying to stabilize the employment 
> situation, so that’s why I have not made any progress.
>
> Is there still interest here?
>
> SCott
> Scott C. Cote
> scottcc...@gmail.com
> 972.900.1561
>
> twitter: @scottccote
>
>>  On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball 
>>  wrote:
>>
>>  We went over the hbase user settings thing on extensive discussions at the 
>> time. Storing an arbitrary blob of JSON which is only ever accessed by a 
>> single key (username) was concluded to be a key value problem, not a 
>> relational problem. Hbase was concluded to be massive overkill as a key 
>> value store in this usecase, unless it was already there and ready to go, 
>> which in the case of Metron, it is, for enrichments, threat intel and 
>> profiles. Hence it ended up in Hbase, as a conveniently present data store 
>> that matched the usage patterns. See 
>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>>  and METRON-1337 for discussion.
>>
>>  Simon
>>
>>>  On 13 Nov 2018, at 18:50, Michael Miklavcic  
>>> wrote:
>>>
>>>  Thanks for the write up Simon. I don't think I see any major problems with
>>>  deprecating the general sql store. However, just to clarify, Metron does
>>>  NOT require any specific backing store. It's 100% JPA, which means anything
>>>  that can be configured with the Spring properties we expose. I think the
>>>  most opinionated thing we do there is ship an extremely basic table
>>>  creation script for h2 and mysql as a simple example for schema. As an
>>>  example, we simply use H2 in full dev, which is entirely in-memory and spun
>>>  up automatically from configuration. The recent work by Justin Leet removes
>>>  the need to use a SQL store at all if you choose LDAP -
>>>  https://github.com/apache/metron/pull/1246. I'll let him comment further on
>>>  this, but I think there is one small change that could be made via a toggle
>>>  in Ambari that would even eliminate the user from seeing JDBC settings
>>>  altogether during install if they choose LDAP. Again, I think I'm on board
>>>  with deprecating the SQL backing store as I pointed this out on the Knox
>>>  thread as well, but I just wanted to make sure everyone has an accurate
>>>  picture of the current state.
>>>
>>>  I had to double check on the HBase config you mentioned, but it does appear
>>>  that we use it for the Alerts UI. I don't think I realized we were storing
>>>  config there instead of the Zookeeper store we use for other system
>>>  configuration. Ironically enough, I think that it probably makes more sense
>>>  than the current auth info to store in a traditional sql store, however
>>>  it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
>>>  pointed out.
>>>
>>>  Whatever architectural changes we choose to add here, I think we need to
>>>  emphasize pluggability regardless of the specific implementation. That is
>>>  to say, I don't think we should make a hard requirement on Knox, in order
>>>  to get LDAP, in order to deprecate an optional general SQL backing store.
>>>  It makes sensible defaults if that's where we want to go, which is the way
>>>  we have done things for most of the successful features I've seen in
>>>  Metron. Provide all the options should a user desire them, but abstract
>>>  away the complexity in the UIs.
>>>
>>>  Best,
>>>  Mike
>>>
>>>  On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
>>>  si...@simonellistonball.com> wrote:
>>>
  I've been coming across a number of organisations who are blocked from
  installing Metron by the MySQL auth database.

  The main problems with our MySQL default are:

  * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
  security platform and usually where the deployment conversation ends for 
 me
  * MySQL install varies from platform to platform
  * An additional database to manage, backup, etc. so now I have to talk to 
 a
  DBA
  * Harder to maintain HA for this without externalising and fighting 
 against
  our defaults
  * There are a lot of dependencies for just storing a table of users
  (Eclipse Link, 

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Scott C. Cote
Simon,

Since ya’ll are going to have SOLR in your installation (is this still true?), 
I could make the profile system rely upon SOLR instead of HBASE.  At 
Lucidworks, I did this very thing with proprietary code, but I can make an 
adapter so the binaries can be stored in SOLR of arbitrary size.  We have 
chatted briefly about this in Slack.   Am trying to stabilize the employment 
situation, so that’s why I have not made any progress.  

Is there still interest here?

SCott
Scott C. Cote
scottcc...@gmail.com
972.900.1561

twitter: @scottccote



> On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball 
>  wrote:
> 
> We went over the hbase user settings thing on extensive discussions at the 
> time. Storing an arbitrary blob of JSON which is only ever accessed by a 
> single key (username) was concluded to be a key value problem, not a 
> relational problem. Hbase was concluded to be massive overkill as a key value 
> store in this usecase, unless it was already there and ready to go, which in 
> the case of Metron, it is, for enrichments, threat intel and profiles. Hence 
> it ended up in Hbase, as a conveniently present data store that matched the 
> usage patterns. See 
> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>  and METRON-1337 for discussion.
> 
> Simon
> 
>> On 13 Nov 2018, at 18:50, Michael Miklavcic  
>> wrote:
>> 
>> Thanks for the write up Simon. I don't think I see any major problems with
>> deprecating the general sql store. However, just to clarify, Metron does
>> NOT require any specific backing store. It's 100% JPA, which means anything
>> that can be configured with the Spring properties we expose. I think the
>> most opinionated thing we do there is ship an extremely basic table
>> creation script for h2 and mysql as a simple example for schema. As an
>> example, we simply use H2 in full dev, which is entirely in-memory and spun
>> up automatically from configuration. The recent work by Justin Leet removes
>> the need to use a SQL store at all if you choose LDAP -
>> https://github.com/apache/metron/pull/1246. I'll let him comment further on
>> this, but I think there is one small change that could be made via a toggle
>> in Ambari that would even eliminate the user from seeing JDBC settings
>> altogether during install if they choose LDAP. Again, I think I'm on board
>> with deprecating the SQL backing store as I pointed this out on the Knox
>> thread as well, but I just wanted to make sure everyone has an accurate
>> picture of the current state.
>> 
>> I had to double check on the HBase config you mentioned, but it does appear
>> that we use it for the Alerts UI. I don't think I realized we were storing
>> config there instead of the Zookeeper store we use for other system
>> configuration. Ironically enough, I think that it probably makes more sense
>> than the current auth info to store in a traditional sql store, however
>> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
>> pointed out.
>> 
>> Whatever architectural changes we choose to add here, I think we need to
>> emphasize pluggability regardless of the specific implementation. That is
>> to say, I don't think we should make a hard requirement on Knox, in order
>> to get LDAP, in order to deprecate an optional general SQL backing store.
>> It makes sensible defaults if that's where we want to go, which is the way
>> we have done things for most of the successful features I've seen in
>> Metron. Provide all the options should a user desire them, but abstract
>> away the complexity in the UIs.
>> 
>> Best,
>> Mike
>> 
>> 
>> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
>> si...@simonellistonball.com> wrote:
>> 
>>> I've been coming across a number of organisations who are blocked from
>>> installing Metron by the MySQL auth database.
>>> 
>>> The main problems with our MySQL default are:
>>> 
>>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
>>> security platform and usually where the deployment conversation ends for me
>>> * MySQL install varies from platform to platform
>>> * An additional database to manage, backup, etc. so now I have to talk to a
>>> DBA
>>> * Harder to maintain HA for this without externalising and fighting against
>>> our defaults
>>> * There are a lot of dependencies for just storing a table of users
>>> (Eclipse Link, JPA, the MySQL server and the need to get clients installed
>>> and pushed separately because of licence requirements)
>>> * Organisations don't want to have to manage yet another user source of
>>> truth since this leads to operational complexity.
>>> 
>>> In short, managing our own user store makes very little sense to operations
>>> users.
>>> 
>>> Some of these (licence and inconsistency for example) could be solved by
>>> changing our default DB to something like Postgres, which has easier terms
>>> to deal with. We could start encrypting passwords, 

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Simon Elliston Ball
We went over the hbase user settings thing on extensive discussions at the 
time. Storing an arbitrary blob of JSON which is only ever accessed by a single 
key (username) was concluded to be a key value problem, not a relational 
problem. Hbase was concluded to be massive overkill as a key value store in 
this usecase, unless it was already there and ready to go, which in the case of 
Metron, it is, for enrichments, threat intel and profiles. Hence it ended up in 
Hbase, as a conveniently present data store that matched the usage patterns. 
See 
https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
 and METRON-1337 for discussion.

Simon

> On 13 Nov 2018, at 18:50, Michael Miklavcic  
> wrote:
> 
> Thanks for the write up Simon. I don't think I see any major problems with
> deprecating the general sql store. However, just to clarify, Metron does
> NOT require any specific backing store. It's 100% JPA, which means anything
> that can be configured with the Spring properties we expose. I think the
> most opinionated thing we do there is ship an extremely basic table
> creation script for h2 and mysql as a simple example for schema. As an
> example, we simply use H2 in full dev, which is entirely in-memory and spun
> up automatically from configuration. The recent work by Justin Leet removes
> the need to use a SQL store at all if you choose LDAP -
> https://github.com/apache/metron/pull/1246. I'll let him comment further on
> this, but I think there is one small change that could be made via a toggle
> in Ambari that would even eliminate the user from seeing JDBC settings
> altogether during install if they choose LDAP. Again, I think I'm on board
> with deprecating the SQL backing store as I pointed this out on the Knox
> thread as well, but I just wanted to make sure everyone has an accurate
> picture of the current state.
> 
> I had to double check on the HBase config you mentioned, but it does appear
> that we use it for the Alerts UI. I don't think I realized we were storing
> config there instead of the Zookeeper store we use for other system
> configuration. Ironically enough, I think that it probably makes more sense
> than the current auth info to store in a traditional sql store, however
> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
> pointed out.
> 
> Whatever architectural changes we choose to add here, I think we need to
> emphasize pluggability regardless of the specific implementation. That is
> to say, I don't think we should make a hard requirement on Knox, in order
> to get LDAP, in order to deprecate an optional general SQL backing store.
> It makes sensible defaults if that's where we want to go, which is the way
> we have done things for most of the successful features I've seen in
> Metron. Provide all the options should a user desire them, but abstract
> away the complexity in the UIs.
> 
> Best,
> Mike
> 
> 
> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
> 
>> I've been coming across a number of organisations who are blocked from
>> installing Metron by the MySQL auth database.
>> 
>> The main problems with our MySQL default are:
>> 
>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
>> security platform and usually where the deployment conversation ends for me
>> * MySQL install varies from platform to platform
>> * An additional database to manage, backup, etc. so now I have to talk to a
>> DBA
>> * Harder to maintain HA for this without externalising and fighting against
>> our defaults
>> * There are a lot of dependencies for just storing a table of users
>> (Eclipse Link, JPA, the MySQL server and the need to get clients installed
>> and pushed separately because of licence requirements)
>> * Organisations don't want to have to manage yet another user source of
>> truth since this leads to operational complexity.
>> 
>> In short, managing our own user store makes very little sense to operations
>> users.
>> 
>> Some of these (licence and inconsistency for example) could be solved by
>> changing our default DB to something like Postgres, which has easier terms
>> to deal with. We could start encrypting passwords, but there would still be
>> a lot of dependencies to store users, which is a problem much better solved
>> by LDAP.
>> 
>> Now that we have the option to use LDAP for user storage, I would suggest
>> that we deprecate and ultimately remove all the RDBMS and ORM dependencies,
>> which significantly reduces our dependencies and simplifies deployment and
>> long term management of Metron clusters.
>> 
>> So I propose that we deprecate the RDBMS use in the next Apache release,
>> and then strip out the RDBMS stuff in the following. We would continue to
>> use LDAP for users and HBase for non-LDAPy user settings (as we currently
>> do). We should also provide a small demo LDAP for full dev. Since we are
>> 

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Michael Miklavcic
Thanks for the write up Simon. I don't think I see any major problems with
deprecating the general sql store. However, just to clarify, Metron does
NOT require any specific backing store. It's 100% JPA, which means anything
that can be configured with the Spring properties we expose. I think the
most opinionated thing we do there is ship an extremely basic table
creation script for h2 and mysql as a simple example for schema. As an
example, we simply use H2 in full dev, which is entirely in-memory and spun
up automatically from configuration. The recent work by Justin Leet removes
the need to use a SQL store at all if you choose LDAP -
https://github.com/apache/metron/pull/1246. I'll let him comment further on
this, but I think there is one small change that could be made via a toggle
in Ambari that would even eliminate the user from seeing JDBC settings
altogether during install if they choose LDAP. Again, I think I'm on board
with deprecating the SQL backing store as I pointed this out on the Knox
thread as well, but I just wanted to make sure everyone has an accurate
picture of the current state.

I had to double check on the HBase config you mentioned, but it does appear
that we use it for the Alerts UI. I don't think I realized we were storing
config there instead of the Zookeeper store we use for other system
configuration. Ironically enough, I think that it probably makes more sense
than the current auth info to store in a traditional sql store, however
it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
pointed out.

Whatever architectural changes we choose to add here, I think we need to
emphasize pluggability regardless of the specific implementation. That is
to say, I don't think we should make a hard requirement on Knox, in order
to get LDAP, in order to deprecate an optional general SQL backing store.
It makes sensible defaults if that's where we want to go, which is the way
we have done things for most of the successful features I've seen in
Metron. Provide all the options should a user desire them, but abstract
away the complexity in the UIs.

Best,
Mike


On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> I've been coming across a number of organisations who are blocked from
> installing Metron by the MySQL auth database.
>
> The main problems with our MySQL default are:
>
> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
> security platform and usually where the deployment conversation ends for me
> * MySQL install varies from platform to platform
> * An additional database to manage, backup, etc. so now I have to talk to a
> DBA
> * Harder to maintain HA for this without externalising and fighting against
> our defaults
> * There are a lot of dependencies for just storing a table of users
> (Eclipse Link, JPA, the MySQL server and the need to get clients installed
> and pushed separately because of licence requirements)
> * Organisations don't want to have to manage yet another user source of
> truth since this leads to operational complexity.
>
> In short, managing our own user store makes very little sense to operations
> users.
>
> Some of these (licence and inconsistency for example) could be solved by
> changing our default DB to something like Postgres, which has easier terms
> to deal with. We could start encrypting passwords, but there would still be
> a lot of dependencies to store users, which is a problem much better solved
> by LDAP.
>
> Now that we have the option to use LDAP for user storage, I would suggest
> that we deprecate and ultimately remove all the RDBMS and ORM dependencies,
> which significantly reduces our dependencies and simplifies deployment and
> long term management of Metron clusters.
>
> So I propose that we deprecate the RDBMS use in the next Apache release,
> and then strip out the RDBMS stuff in the following. We would continue to
> use LDAP for users and HBase for non-LDAPy user settings (as we currently
> do). We should also provide a small demo LDAP for full dev. Since we are
> looking at adding Knox into the stack, that project provides a convenient
> mini-LDAP demo service which would do this job without the need to add
> additional components.
>
> Thoughts? Anyone relying on MySQL for users (if so, are you aware that your
> passwords are all plaintext? How do you currently handle the shortcomings
> and admin overhead?) Any objections?
>
> Simon
>