Yes, makes sense. +1 to that.

On Thu, Nov 15, 2018 at 12:54 PM James Sirota <jsir...@apache.org> wrote:

> To clarify my position, I don't have a problem with mySql or any other
> projects relying on it.  mySql in itself is not an issue.  What I don't
> want is for a customer to be presented with an option to chose and
> configure two options for authenticating the UI, which I think is
> needless.  It adds complexity for not much value.  Since LDAP is clearly
> the better way to go that should be what we support without explicitly
> giving a user an option to switch to JDBC.  A user can still do so by
> extending our abstractions if that is what they chose to do, but this would
> not be officially supported by us.  We would not be providing a config or
> an mPack to do this.  A user would have to do it on their own.
>
> James
>
>
>
> 15.11.2018, 12:15, "Michael Miklavcic" <michael.miklav...@gmail.com>:
> > Incidentally, even without the Metron piece in the picture, what is the
> > answer for Ambari's database dependency? Which uses a SQL data store.
> Does
> > this actually solve the problem of "customers won't install Metron bc SQL
> > store?" or are there other issues we need to address?
> >
> > On Thu, Nov 15, 2018 at 9:30 AM James Sirota <jsir...@apache.org> wrote:
> >
> >>  Hi Guys,
> >>
> >>  My opinion on this, as is with Knox SSO, is that the code should be
> >>  pluggable to support JDBC, but we should not continue to support the
> >>  concrete implementation and expose it to users via a setting. This is a
> >>  fairly minor feature and the added complexity of supporting switching
> >>  between JDBC and LDAP is simply not worth it. We need to strike a
> balance
> >>  between ease of use and capabilities/extensibility. For features that
> are
> >>  worth it such as with analytics and stream processing, the extra
> capability
> >>  is worth the added complexity in configuration. But for this, it is
> not.
> >>  So let's keep JDBC around for a release to allow users to migrate to
> LDAP,
> >>  deprecate it, and move on.
> >>
> >>  Thanks,
> >>  James
> >>
> >>  13.11.2018, 16:03, "Simon Elliston Ball" <si...@simonellistonball.com
> >:
> >>  > We went over the hbase user settings thing on extensive discussions
> at
> >>  the time. Storing an arbitrary blob of JSON which is only ever
> accessed by
> >>  a single key (username) was concluded to be a key value problem, not a
> >>  relational problem. Hbase was concluded to be massive overkill as a key
> >>  value store in this usecase, unless it was already there and ready to
> go,
> >>  which in the case of Metron, it is, for enrichments, threat intel and
> >>  profiles. Hence it ended up in Hbase, as a conveniently present data
> store
> >>  that matched the usage patterns. See
> >>
> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
> >>  and METRON-1337 for discussion.
> >>  >
> >>  > Simon
> >>  >
> >>  >> On 13 Nov 2018, at 18:50, Michael Miklavcic <
> >>  michael.miklav...@gmail.com> wrote:
> >>  >>
> >>  >> Thanks for the write up Simon. I don't think I see any major
> problems
> >>  with
> >>  >> deprecating the general sql store. However, just to clarify, Metron
> >>  does
> >>  >> NOT require any specific backing store. It's 100% JPA, which means
> >>  anything
> >>  >> that can be configured with the Spring properties we expose. I think
> >>  the
> >>  >> most opinionated thing we do there is ship an extremely basic table
> >>  >> creation script for h2 and mysql as a simple example for schema. As
> an
> >>  >> example, we simply use H2 in full dev, which is entirely in-memory
> and
> >>  spun
> >>  >> up automatically from configuration. The recent work by Justin Leet
> >>  removes
> >>  >> the need to use a SQL store at all if you choose LDAP -
> >>  >> https://github.com/apache/metron/pull/1246. I'll let him comment
> >>  further on
> >>  >> this, but I think there is one small change that could be made via a
> >>  toggle
> >>  >> in Ambari that would even eliminate the user from seeing JDBC
> settings
> >>  >> altogether during install if they choose LDAP. Again, I think I'm on
> >>  board
> >>  >> with deprecating the SQL backing store as I pointed this out on the
> >>  Knox
> >>  >> thread as well, but I just wanted to make sure everyone has an
> accurate
> >>  >> picture of the current state.
> >>  >>
> >>  >> I had to double check on the HBase config you mentioned, but it does
> >>  appear
> >>  >> that we use it for the Alerts UI. I don't think I realized we were
> >>  storing
> >>  >> config there instead of the Zookeeper store we use for other system
> >>  >> configuration. Ironically enough, I think that it probably makes
> more
> >>  sense
> >>  >> than the current auth info to store in a traditional sql store,
> however
> >>  >> it's in HBase currently so it's a non-issue wrt SQL/JPA either way,
> as
> >>  you
> >>  >> pointed out.
> >>  >>
> >>  >> Whatever architectural changes we choose to add here, I think we
> need
> >>  to
> >>  >> emphasize pluggability regardless of the specific implementation.
> That
> >>  is
> >>  >> to say, I don't think we should make a hard requirement on Knox, in
> >>  order
> >>  >> to get LDAP, in order to deprecate an optional general SQL backing
> >>  store.
> >>  >> It makes sensible defaults if that's where we want to go, which is
> the
> >>  way
> >>  >> we have done things for most of the successful features I've seen in
> >>  >> Metron. Provide all the options should a user desire them, but
> abstract
> >>  >> away the complexity in the UIs.
> >>  >>
> >>  >> Best,
> >>  >> Mike
> >>  >>
> >>  >> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
> >>  >> si...@simonellistonball.com> wrote:
> >>  >>
> >>  >>> I've been coming across a number of organisations who are blocked
> from
> >>  >>> installing Metron by the MySQL auth database.
> >>  >>>
> >>  >>> The main problems with our MySQL default are:
> >>  >>>
> >>  >>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing
> in a
> >>  >>> security platform and usually where the deployment conversation
> ends
> >>  for me
> >>  >>> * MySQL install varies from platform to platform
> >>  >>> * An additional database to manage, backup, etc. so now I have to
> >>  talk to a
> >>  >>> DBA
> >>  >>> * Harder to maintain HA for this without externalising and fighting
> >>  against
> >>  >>> our defaults
> >>  >>> * There are a lot of dependencies for just storing a table of users
> >>  >>> (Eclipse Link, JPA, the MySQL server and the need to get clients
> >>  installed
> >>  >>> and pushed separately because of licence requirements)
> >>  >>> * Organisations don't want to have to manage yet another user
> source
> >>  of
> >>  >>> truth since this leads to operational complexity.
> >>  >>>
> >>  >>> In short, managing our own user store makes very little sense to
> >>  operations
> >>  >>> users.
> >>  >>>
> >>  >>> Some of these (licence and inconsistency for example) could be
> solved
> >>  by
> >>  >>> changing our default DB to something like Postgres, which has
> easier
> >>  terms
> >>  >>> to deal with. We could start encrypting passwords, but there would
> >>  still be
> >>  >>> a lot of dependencies to store users, which is a problem much
> better
> >>  solved
> >>  >>> by LDAP.
> >>  >>>
> >>  >>> Now that we have the option to use LDAP for user storage, I would
> >>  suggest
> >>  >>> that we deprecate and ultimately remove all the RDBMS and ORM
> >>  dependencies,
> >>  >>> which significantly reduces our dependencies and simplifies
> >>  deployment and
> >>  >>> long term management of Metron clusters.
> >>  >>>
> >>  >>> So I propose that we deprecate the RDBMS use in the next Apache
> >>  release,
> >>  >>> and then strip out the RDBMS stuff in the following. We would
> >>  continue to
> >>  >>> use LDAP for users and HBase for non-LDAPy user settings (as we
> >>  currently
> >>  >>> do). We should also provide a small demo LDAP for full dev. Since
> we
> >>  are
> >>  >>> looking at adding Knox into the stack, that project provides a
> >>  convenient
> >>  >>> mini-LDAP demo service which would do this job without the need to
> add
> >>  >>> additional components.
> >>  >>>
> >>  >>> Thoughts? Anyone relying on MySQL for users (if so, are you aware
> >>  that your
> >>  >>> passwords are all plaintext? How do you currently handle the
> >>  shortcomings
> >>  >>> and admin overhead?) Any objections?
> >>  >>>
> >>  >>> Simon
> >>
> >>  -------------------
> >>  Thank you,
> >>
> >>  James Sirota
> >>  PMC- Apache Metron
> >>  jsirota AT apache DOT org
>
> -------------------
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Reply via email to