Just to clarify this point, I am not proposing we alter any interactions with Ambari and backing stores. I want to make sure we cover all the problems and provide good solutions where we're able. Again, just to reiterate my earlier point, I agree with the move to deprecate SQL/JPA for the UI. And I agree with James about pluggability and simplifying the surface area of options we expose to users via Ambari config.
Best, Mike On Thu, Nov 15, 2018 at 12:14 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Incidentally, even without the Metron piece in the picture, what is the > answer for Ambari's database dependency? Which uses a SQL data store. Does > this actually solve the problem of "customers won't install Metron bc SQL > store?" or are there other issues we need to address? > > On Thu, Nov 15, 2018 at 9:30 AM James Sirota <jsir...@apache.org> wrote: > >> Hi Guys, >> >> My opinion on this, as is with Knox SSO, is that the code should be >> pluggable to support JDBC, but we should not continue to support the >> concrete implementation and expose it to users via a setting. This is a >> fairly minor feature and the added complexity of supporting switching >> between JDBC and LDAP is simply not worth it. We need to strike a balance >> between ease of use and capabilities/extensibility. For features that are >> worth it such as with analytics and stream processing, the extra capability >> is worth the added complexity in configuration. But for this, it is not. >> So let's keep JDBC around for a release to allow users to migrate to LDAP, >> deprecate it, and move on. >> >> Thanks, >> James >> >> 13.11.2018, 16:03, "Simon Elliston Ball" <si...@simonellistonball.com>: >> > We went over the hbase user settings thing on extensive discussions at >> the time. Storing an arbitrary blob of JSON which is only ever accessed by >> a single key (username) was concluded to be a key value problem, not a >> relational problem. Hbase was concluded to be massive overkill as a key >> value store in this usecase, unless it was already there and ready to go, >> which in the case of Metron, it is, for enrichments, threat intel and >> profiles. Hence it ended up in Hbase, as a conveniently present data store >> that matched the usage patterns. See >> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E >> and METRON-1337 for discussion. >> > >> > Simon >> > >> >> On 13 Nov 2018, at 18:50, Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >> >> >> Thanks for the write up Simon. I don't think I see any major problems >> with >> >> deprecating the general sql store. However, just to clarify, Metron >> does >> >> NOT require any specific backing store. It's 100% JPA, which means >> anything >> >> that can be configured with the Spring properties we expose. I think >> the >> >> most opinionated thing we do there is ship an extremely basic table >> >> creation script for h2 and mysql as a simple example for schema. As an >> >> example, we simply use H2 in full dev, which is entirely in-memory >> and spun >> >> up automatically from configuration. The recent work by Justin Leet >> removes >> >> the need to use a SQL store at all if you choose LDAP - >> >> https://github.com/apache/metron/pull/1246. I'll let him comment >> further on >> >> this, but I think there is one small change that could be made via a >> toggle >> >> in Ambari that would even eliminate the user from seeing JDBC settings >> >> altogether during install if they choose LDAP. Again, I think I'm on >> board >> >> with deprecating the SQL backing store as I pointed this out on the >> Knox >> >> thread as well, but I just wanted to make sure everyone has an >> accurate >> >> picture of the current state. >> >> >> >> I had to double check on the HBase config you mentioned, but it does >> appear >> >> that we use it for the Alerts UI. I don't think I realized we were >> storing >> >> config there instead of the Zookeeper store we use for other system >> >> configuration. Ironically enough, I think that it probably makes more >> sense >> >> than the current auth info to store in a traditional sql store, >> however >> >> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, >> as you >> >> pointed out. >> >> >> >> Whatever architectural changes we choose to add here, I think we need >> to >> >> emphasize pluggability regardless of the specific implementation. >> That is >> >> to say, I don't think we should make a hard requirement on Knox, in >> order >> >> to get LDAP, in order to deprecate an optional general SQL backing >> store. >> >> It makes sensible defaults if that's where we want to go, which is >> the way >> >> we have done things for most of the successful features I've seen in >> >> Metron. Provide all the options should a user desire them, but >> abstract >> >> away the complexity in the UIs. >> >> >> >> Best, >> >> Mike >> >> >> >> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < >> >> si...@simonellistonball.com> wrote: >> >> >> >>> I've been coming across a number of organisations who are blocked >> from >> >>> installing Metron by the MySQL auth database. >> >>> >> >>> The main problems with our MySQL default are: >> >>> >> >>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a >> >>> security platform and usually where the deployment conversation ends >> for me >> >>> * MySQL install varies from platform to platform >> >>> * An additional database to manage, backup, etc. so now I have to >> talk to a >> >>> DBA >> >>> * Harder to maintain HA for this without externalising and fighting >> against >> >>> our defaults >> >>> * There are a lot of dependencies for just storing a table of users >> >>> (Eclipse Link, JPA, the MySQL server and the need to get clients >> installed >> >>> and pushed separately because of licence requirements) >> >>> * Organisations don't want to have to manage yet another user source >> of >> >>> truth since this leads to operational complexity. >> >>> >> >>> In short, managing our own user store makes very little sense to >> operations >> >>> users. >> >>> >> >>> Some of these (licence and inconsistency for example) could be >> solved by >> >>> changing our default DB to something like Postgres, which has easier >> terms >> >>> to deal with. We could start encrypting passwords, but there would >> still be >> >>> a lot of dependencies to store users, which is a problem much better >> solved >> >>> by LDAP. >> >>> >> >>> Now that we have the option to use LDAP for user storage, I would >> suggest >> >>> that we deprecate and ultimately remove all the RDBMS and ORM >> dependencies, >> >>> which significantly reduces our dependencies and simplifies >> deployment and >> >>> long term management of Metron clusters. >> >>> >> >>> So I propose that we deprecate the RDBMS use in the next Apache >> release, >> >>> and then strip out the RDBMS stuff in the following. We would >> continue to >> >>> use LDAP for users and HBase for non-LDAPy user settings (as we >> currently >> >>> do). We should also provide a small demo LDAP for full dev. Since we >> are >> >>> looking at adding Knox into the stack, that project provides a >> convenient >> >>> mini-LDAP demo service which would do this job without the need to >> add >> >>> additional components. >> >>> >> >>> Thoughts? Anyone relying on MySQL for users (if so, are you aware >> that your >> >>> passwords are all plaintext? How do you currently handle the >> shortcomings >> >>> and admin overhead?) Any objections? >> >>> >> >>> Simon >> >> ------------------- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org >> >> >>
