Thanks for the write up Simon. I don't think I see any major problems with
deprecating the general sql store. However, just to clarify, Metron does
NOT require any specific backing store. It's 100% JPA, which means anything
that can be configured with the Spring properties we expose. I think the
most opinionated thing we do there is ship an extremely basic table
creation script for h2 and mysql as a simple example for schema. As an
example, we simply use H2 in full dev, which is entirely in-memory and spun
up automatically from configuration. The recent work by Justin Leet removes
the need to use a SQL store at all if you choose LDAP - I'll let him comment further on
this, but I think there is one small change that could be made via a toggle
in Ambari that would even eliminate the user from seeing JDBC settings
altogether during install if they choose LDAP. Again, I think I'm on board
with deprecating the SQL backing store as I pointed this out on the Knox
thread as well, but I just wanted to make sure everyone has an accurate
picture of the current state.

I had to double check on the HBase config you mentioned, but it does appear
that we use it for the Alerts UI. I don't think I realized we were storing
config there instead of the Zookeeper store we use for other system
configuration. Ironically enough, I think that it probably makes more sense
than the current auth info to store in a traditional sql store, however
it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
pointed out.

Whatever architectural changes we choose to add here, I think we need to
emphasize pluggability regardless of the specific implementation. That is
to say, I don't think we should make a hard requirement on Knox, in order
to get LDAP, in order to deprecate an optional general SQL backing store.
It makes sensible defaults if that's where we want to go, which is the way
we have done things for most of the successful features I've seen in
Metron. Provide all the options should a user desire them, but abstract
away the complexity in the UIs.


On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <> wrote:

> I've been coming across a number of organisations who are blocked from
> installing Metron by the MySQL auth database.
> The main problems with our MySQL default are:
> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
> security platform and usually where the deployment conversation ends for me
> * MySQL install varies from platform to platform
> * An additional database to manage, backup, etc. so now I have to talk to a
> * Harder to maintain HA for this without externalising and fighting against
> our defaults
> * There are a lot of dependencies for just storing a table of users
> (Eclipse Link, JPA, the MySQL server and the need to get clients installed
> and pushed separately because of licence requirements)
> * Organisations don't want to have to manage yet another user source of
> truth since this leads to operational complexity.
> In short, managing our own user store makes very little sense to operations
> users.
> Some of these (licence and inconsistency for example) could be solved by
> changing our default DB to something like Postgres, which has easier terms
> to deal with. We could start encrypting passwords, but there would still be
> a lot of dependencies to store users, which is a problem much better solved
> by LDAP.
> Now that we have the option to use LDAP for user storage, I would suggest
> that we deprecate and ultimately remove all the RDBMS and ORM dependencies,
> which significantly reduces our dependencies and simplifies deployment and
> long term management of Metron clusters.
> So I propose that we deprecate the RDBMS use in the next Apache release,
> and then strip out the RDBMS stuff in the following. We would continue to
> use LDAP for users and HBase for non-LDAPy user settings (as we currently
> do). We should also provide a small demo LDAP for full dev. Since we are
> looking at adding Knox into the stack, that project provides a convenient
> mini-LDAP demo service which would do this job without the need to add
> additional components.
> Thoughts? Anyone relying on MySQL for users (if so, are you aware that your
> passwords are all plaintext? How do you currently handle the shortcomings
> and admin overhead?) Any objections?
> Simon

Reply via email to