We went over the hbase user settings thing on extensive discussions at the 
time. Storing an arbitrary blob of JSON which is only ever accessed by a single 
key (username) was concluded to be a key value problem, not a relational 
problem. Hbase was concluded to be massive overkill as a key value store in 
this usecase, unless it was already there and ready to go, which in the case of 
Metron, it is, for enrichments, threat intel and profiles. Hence it ended up in 
Hbase, as a conveniently present data store that matched the usage patterns. 
See 
https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
 and METRON-1337 for discussion.

Simon

> On 13 Nov 2018, at 18:50, Michael Miklavcic <michael.miklav...@gmail.com> 
> wrote:
> 
> Thanks for the write up Simon. I don't think I see any major problems with
> deprecating the general sql store. However, just to clarify, Metron does
> NOT require any specific backing store. It's 100% JPA, which means anything
> that can be configured with the Spring properties we expose. I think the
> most opinionated thing we do there is ship an extremely basic table
> creation script for h2 and mysql as a simple example for schema. As an
> example, we simply use H2 in full dev, which is entirely in-memory and spun
> up automatically from configuration. The recent work by Justin Leet removes
> the need to use a SQL store at all if you choose LDAP -
> https://github.com/apache/metron/pull/1246. I'll let him comment further on
> this, but I think there is one small change that could be made via a toggle
> in Ambari that would even eliminate the user from seeing JDBC settings
> altogether during install if they choose LDAP. Again, I think I'm on board
> with deprecating the SQL backing store as I pointed this out on the Knox
> thread as well, but I just wanted to make sure everyone has an accurate
> picture of the current state.
> 
> I had to double check on the HBase config you mentioned, but it does appear
> that we use it for the Alerts UI. I don't think I realized we were storing
> config there instead of the Zookeeper store we use for other system
> configuration. Ironically enough, I think that it probably makes more sense
> than the current auth info to store in a traditional sql store, however
> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
> pointed out.
> 
> Whatever architectural changes we choose to add here, I think we need to
> emphasize pluggability regardless of the specific implementation. That is
> to say, I don't think we should make a hard requirement on Knox, in order
> to get LDAP, in order to deprecate an optional general SQL backing store.
> It makes sensible defaults if that's where we want to go, which is the way
> we have done things for most of the successful features I've seen in
> Metron. Provide all the options should a user desire them, but abstract
> away the complexity in the UIs.
> 
> Best,
> Mike
> 
> 
> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
> 
>> I've been coming across a number of organisations who are blocked from
>> installing Metron by the MySQL auth database.
>> 
>> The main problems with our MySQL default are:
>> 
>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
>> security platform and usually where the deployment conversation ends for me
>> * MySQL install varies from platform to platform
>> * An additional database to manage, backup, etc. so now I have to talk to a
>> DBA
>> * Harder to maintain HA for this without externalising and fighting against
>> our defaults
>> * There are a lot of dependencies for just storing a table of users
>> (Eclipse Link, JPA, the MySQL server and the need to get clients installed
>> and pushed separately because of licence requirements)
>> * Organisations don't want to have to manage yet another user source of
>> truth since this leads to operational complexity.
>> 
>> In short, managing our own user store makes very little sense to operations
>> users.
>> 
>> Some of these (licence and inconsistency for example) could be solved by
>> changing our default DB to something like Postgres, which has easier terms
>> to deal with. We could start encrypting passwords, but there would still be
>> a lot of dependencies to store users, which is a problem much better solved
>> by LDAP.
>> 
>> Now that we have the option to use LDAP for user storage, I would suggest
>> that we deprecate and ultimately remove all the RDBMS and ORM dependencies,
>> which significantly reduces our dependencies and simplifies deployment and
>> long term management of Metron clusters.
>> 
>> So I propose that we deprecate the RDBMS use in the next Apache release,
>> and then strip out the RDBMS stuff in the following. We would continue to
>> use LDAP for users and HBase for non-LDAPy user settings (as we currently
>> do). We should also provide a small demo LDAP for full dev. Since we are
>> looking at adding Knox into the stack, that project provides a convenient
>> mini-LDAP demo service which would do this job without the need to add
>> additional components.
>> 
>> Thoughts? Anyone relying on MySQL for users (if so, are you aware that your
>> passwords are all plaintext? How do you currently handle the shortcomings
>> and admin overhead?) Any objections?
>> 
>> Simon
>> 

Reply via email to