Thanks for the write up Simon. I don't think I see any major problems with deprecating the general sql store. However, just to clarify, Metron does NOT require any specific backing store. It's 100% JPA, which means anything that can be configured with the Spring properties we expose. I think the most opinionated thing we do there is ship an extremely basic table creation script for h2 and mysql as a simple example for schema. As an example, we simply use H2 in full dev, which is entirely in-memory and spun up automatically from configuration. The recent work by Justin Leet removes the need to use a SQL store at all if you choose LDAP - https://github.com/apache/metron/pull/1246. I'll let him comment further on this, but I think there is one small change that could be made via a toggle in Ambari that would even eliminate the user from seeing JDBC settings altogether during install if they choose LDAP. Again, I think I'm on board with deprecating the SQL backing store as I pointed this out on the Knox thread as well, but I just wanted to make sure everyone has an accurate picture of the current state.
I had to double check on the HBase config you mentioned, but it does appear that we use it for the Alerts UI. I don't think I realized we were storing config there instead of the Zookeeper store we use for other system configuration. Ironically enough, I think that it probably makes more sense than the current auth info to store in a traditional sql store, however it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you pointed out. Whatever architectural changes we choose to add here, I think we need to emphasize pluggability regardless of the specific implementation. That is to say, I don't think we should make a hard requirement on Knox, in order to get LDAP, in order to deprecate an optional general SQL backing store. It makes sensible defaults if that's where we want to go, which is the way we have done things for most of the successful features I've seen in Metron. Provide all the options should a user desire them, but abstract away the complexity in the UIs. Best, Mike On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > I've been coming across a number of organisations who are blocked from > installing Metron by the MySQL auth database. > > The main problems with our MySQL default are: > > * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a > security platform and usually where the deployment conversation ends for me > * MySQL install varies from platform to platform > * An additional database to manage, backup, etc. so now I have to talk to a > DBA > * Harder to maintain HA for this without externalising and fighting against > our defaults > * There are a lot of dependencies for just storing a table of users > (Eclipse Link, JPA, the MySQL server and the need to get clients installed > and pushed separately because of licence requirements) > * Organisations don't want to have to manage yet another user source of > truth since this leads to operational complexity. > > In short, managing our own user store makes very little sense to operations > users. > > Some of these (licence and inconsistency for example) could be solved by > changing our default DB to something like Postgres, which has easier terms > to deal with. We could start encrypting passwords, but there would still be > a lot of dependencies to store users, which is a problem much better solved > by LDAP. > > Now that we have the option to use LDAP for user storage, I would suggest > that we deprecate and ultimately remove all the RDBMS and ORM dependencies, > which significantly reduces our dependencies and simplifies deployment and > long term management of Metron clusters. > > So I propose that we deprecate the RDBMS use in the next Apache release, > and then strip out the RDBMS stuff in the following. We would continue to > use LDAP for users and HBase for non-LDAPy user settings (as we currently > do). We should also provide a small demo LDAP for full dev. Since we are > looking at adding Knox into the stack, that project provides a convenient > mini-LDAP demo service which would do this job without the need to add > additional components. > > Thoughts? Anyone relying on MySQL for users (if so, are you aware that your > passwords are all plaintext? How do you currently handle the shortcomings > and admin overhead?) Any objections? > > Simon >
