Haha, yeah I had to dredge that up from February as well to remember what ultimately ended up going into HBase. Before you get your hackles up, I think you misunderstood me - I believe we're on the same page. I am saying that the SQL store would have made sense just fine in that case. Not that we should actually change it now or use it as a reason to architect things differently.
On Tue, Nov 13, 2018, 4:03 PM Simon Elliston Ball < si...@simonellistonball.com wrote: > We went over the hbase user settings thing on extensive discussions at the > time. Storing an arbitrary blob of JSON which is only ever accessed by a > single key (username) was concluded to be a key value problem, not a > relational problem. Hbase was concluded to be massive overkill as a key > value store in this usecase, unless it was already there and ready to go, > which in the case of Metron, it is, for enrichments, threat intel and > profiles. Hence it ended up in Hbase, as a conveniently present data store > that matched the usage patterns. See > https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E > and METRON-1337 for discussion. > > Simon > > > On 13 Nov 2018, at 18:50, Michael Miklavcic <michael.miklav...@gmail.com> > wrote: > > > > Thanks for the write up Simon. I don't think I see any major problems > with > > deprecating the general sql store. However, just to clarify, Metron does > > NOT require any specific backing store. It's 100% JPA, which means > anything > > that can be configured with the Spring properties we expose. I think the > > most opinionated thing we do there is ship an extremely basic table > > creation script for h2 and mysql as a simple example for schema. As an > > example, we simply use H2 in full dev, which is entirely in-memory and > spun > > up automatically from configuration. The recent work by Justin Leet > removes > > the need to use a SQL store at all if you choose LDAP - > > https://github.com/apache/metron/pull/1246. I'll let him comment > further on > > this, but I think there is one small change that could be made via a > toggle > > in Ambari that would even eliminate the user from seeing JDBC settings > > altogether during install if they choose LDAP. Again, I think I'm on > board > > with deprecating the SQL backing store as I pointed this out on the Knox > > thread as well, but I just wanted to make sure everyone has an accurate > > picture of the current state. > > > > I had to double check on the HBase config you mentioned, but it does > appear > > that we use it for the Alerts UI. I don't think I realized we were > storing > > config there instead of the Zookeeper store we use for other system > > configuration. Ironically enough, I think that it probably makes more > sense > > than the current auth info to store in a traditional sql store, however > > it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as > you > > pointed out. > > > > Whatever architectural changes we choose to add here, I think we need to > > emphasize pluggability regardless of the specific implementation. That is > > to say, I don't think we should make a hard requirement on Knox, in order > > to get LDAP, in order to deprecate an optional general SQL backing store. > > It makes sensible defaults if that's where we want to go, which is the > way > > we have done things for most of the successful features I've seen in > > Metron. Provide all the options should a user desire them, but abstract > > away the complexity in the UIs. > > > > Best, > > Mike > > > > > > On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < > > si...@simonellistonball.com> wrote: > > > >> I've been coming across a number of organisations who are blocked from > >> installing Metron by the MySQL auth database. > >> > >> The main problems with our MySQL default are: > >> > >> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a > >> security platform and usually where the deployment conversation ends > for me > >> * MySQL install varies from platform to platform > >> * An additional database to manage, backup, etc. so now I have to talk > to a > >> DBA > >> * Harder to maintain HA for this without externalising and fighting > against > >> our defaults > >> * There are a lot of dependencies for just storing a table of users > >> (Eclipse Link, JPA, the MySQL server and the need to get clients > installed > >> and pushed separately because of licence requirements) > >> * Organisations don't want to have to manage yet another user source of > >> truth since this leads to operational complexity. > >> > >> In short, managing our own user store makes very little sense to > operations > >> users. > >> > >> Some of these (licence and inconsistency for example) could be solved by > >> changing our default DB to something like Postgres, which has easier > terms > >> to deal with. We could start encrypting passwords, but there would > still be > >> a lot of dependencies to store users, which is a problem much better > solved > >> by LDAP. > >> > >> Now that we have the option to use LDAP for user storage, I would > suggest > >> that we deprecate and ultimately remove all the RDBMS and ORM > dependencies, > >> which significantly reduces our dependencies and simplifies deployment > and > >> long term management of Metron clusters. > >> > >> So I propose that we deprecate the RDBMS use in the next Apache release, > >> and then strip out the RDBMS stuff in the following. We would continue > to > >> use LDAP for users and HBase for non-LDAPy user settings (as we > currently > >> do). We should also provide a small demo LDAP for full dev. Since we are > >> looking at adding Knox into the stack, that project provides a > convenient > >> mini-LDAP demo service which would do this job without the need to add > >> additional components. > >> > >> Thoughts? Anyone relying on MySQL for users (if so, are you aware that > your > >> passwords are all plaintext? How do you currently handle the > shortcomings > >> and admin overhead?) Any objections? > >> > >> Simon > >> >
