To clarify my position, I don't have a problem with mySql or any other projects relying on it. mySql in itself is not an issue. What I don't want is for a customer to be presented with an option to chose and configure two options for authenticating the UI, which I think is needless. It adds complexity for not much value. Since LDAP is clearly the better way to go that should be what we support without explicitly giving a user an option to switch to JDBC. A user can still do so by extending our abstractions if that is what they chose to do, but this would not be officially supported by us. We would not be providing a config or an mPack to do this. A user would have to do it on their own.
James 15.11.2018, 12:15, "Michael Miklavcic" <michael.miklav...@gmail.com>: > Incidentally, even without the Metron piece in the picture, what is the > answer for Ambari's database dependency? Which uses a SQL data store. Does > this actually solve the problem of "customers won't install Metron bc SQL > store?" or are there other issues we need to address? > > On Thu, Nov 15, 2018 at 9:30 AM James Sirota <jsir...@apache.org> wrote: > >> Hi Guys, >> >> My opinion on this, as is with Knox SSO, is that the code should be >> pluggable to support JDBC, but we should not continue to support the >> concrete implementation and expose it to users via a setting. This is a >> fairly minor feature and the added complexity of supporting switching >> between JDBC and LDAP is simply not worth it. We need to strike a balance >> between ease of use and capabilities/extensibility. For features that are >> worth it such as with analytics and stream processing, the extra capability >> is worth the added complexity in configuration. But for this, it is not. >> So let's keep JDBC around for a release to allow users to migrate to LDAP, >> deprecate it, and move on. >> >> Thanks, >> James >> >> 13.11.2018, 16:03, "Simon Elliston Ball" <si...@simonellistonball.com>: >> > We went over the hbase user settings thing on extensive discussions at >> the time. Storing an arbitrary blob of JSON which is only ever accessed by >> a single key (username) was concluded to be a key value problem, not a >> relational problem. Hbase was concluded to be massive overkill as a key >> value store in this usecase, unless it was already there and ready to go, >> which in the case of Metron, it is, for enrichments, threat intel and >> profiles. Hence it ended up in Hbase, as a conveniently present data store >> that matched the usage patterns. See >> >> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E >> and METRON-1337 for discussion. >> > >> > Simon >> > >> >> On 13 Nov 2018, at 18:50, Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >> >> >> Thanks for the write up Simon. I don't think I see any major problems >> with >> >> deprecating the general sql store. However, just to clarify, Metron >> does >> >> NOT require any specific backing store. It's 100% JPA, which means >> anything >> >> that can be configured with the Spring properties we expose. I think >> the >> >> most opinionated thing we do there is ship an extremely basic table >> >> creation script for h2 and mysql as a simple example for schema. As an >> >> example, we simply use H2 in full dev, which is entirely in-memory and >> spun >> >> up automatically from configuration. The recent work by Justin Leet >> removes >> >> the need to use a SQL store at all if you choose LDAP - >> >> https://github.com/apache/metron/pull/1246. I'll let him comment >> further on >> >> this, but I think there is one small change that could be made via a >> toggle >> >> in Ambari that would even eliminate the user from seeing JDBC settings >> >> altogether during install if they choose LDAP. Again, I think I'm on >> board >> >> with deprecating the SQL backing store as I pointed this out on the >> Knox >> >> thread as well, but I just wanted to make sure everyone has an accurate >> >> picture of the current state. >> >> >> >> I had to double check on the HBase config you mentioned, but it does >> appear >> >> that we use it for the Alerts UI. I don't think I realized we were >> storing >> >> config there instead of the Zookeeper store we use for other system >> >> configuration. Ironically enough, I think that it probably makes more >> sense >> >> than the current auth info to store in a traditional sql store, however >> >> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as >> you >> >> pointed out. >> >> >> >> Whatever architectural changes we choose to add here, I think we need >> to >> >> emphasize pluggability regardless of the specific implementation. That >> is >> >> to say, I don't think we should make a hard requirement on Knox, in >> order >> >> to get LDAP, in order to deprecate an optional general SQL backing >> store. >> >> It makes sensible defaults if that's where we want to go, which is the >> way >> >> we have done things for most of the successful features I've seen in >> >> Metron. Provide all the options should a user desire them, but abstract >> >> away the complexity in the UIs. >> >> >> >> Best, >> >> Mike >> >> >> >> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < >> >> si...@simonellistonball.com> wrote: >> >> >> >>> I've been coming across a number of organisations who are blocked from >> >>> installing Metron by the MySQL auth database. >> >>> >> >>> The main problems with our MySQL default are: >> >>> >> >>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a >> >>> security platform and usually where the deployment conversation ends >> for me >> >>> * MySQL install varies from platform to platform >> >>> * An additional database to manage, backup, etc. so now I have to >> talk to a >> >>> DBA >> >>> * Harder to maintain HA for this without externalising and fighting >> against >> >>> our defaults >> >>> * There are a lot of dependencies for just storing a table of users >> >>> (Eclipse Link, JPA, the MySQL server and the need to get clients >> installed >> >>> and pushed separately because of licence requirements) >> >>> * Organisations don't want to have to manage yet another user source >> of >> >>> truth since this leads to operational complexity. >> >>> >> >>> In short, managing our own user store makes very little sense to >> operations >> >>> users. >> >>> >> >>> Some of these (licence and inconsistency for example) could be solved >> by >> >>> changing our default DB to something like Postgres, which has easier >> terms >> >>> to deal with. We could start encrypting passwords, but there would >> still be >> >>> a lot of dependencies to store users, which is a problem much better >> solved >> >>> by LDAP. >> >>> >> >>> Now that we have the option to use LDAP for user storage, I would >> suggest >> >>> that we deprecate and ultimately remove all the RDBMS and ORM >> dependencies, >> >>> which significantly reduces our dependencies and simplifies >> deployment and >> >>> long term management of Metron clusters. >> >>> >> >>> So I propose that we deprecate the RDBMS use in the next Apache >> release, >> >>> and then strip out the RDBMS stuff in the following. We would >> continue to >> >>> use LDAP for users and HBase for non-LDAPy user settings (as we >> currently >> >>> do). We should also provide a small demo LDAP for full dev. Since we >> are >> >>> looking at adding Knox into the stack, that project provides a >> convenient >> >>> mini-LDAP demo service which would do this job without the need to add >> >>> additional components. >> >>> >> >>> Thoughts? Anyone relying on MySQL for users (if so, are you aware >> that your >> >>> passwords are all plaintext? How do you currently handle the >> shortcomings >> >>> and admin overhead?) Any objections? >> >>> >> >>> Simon >> >> ------------------- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org ------------------- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
