I would like to understand the work required to move our JDBC support ( or adapt the current support to the abstraction ) to /contrib. We could default and only officially support LDAP, but have the /contrib ( or /extension_examples ) have a “this is how you would support jdbc for auth “ project.
On November 15, 2018 at 15:01:10, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: Yes, makes sense. +1 to that. On Thu, Nov 15, 2018 at 12:54 PM James Sirota <jsir...@apache.org> wrote: > To clarify my position, I don't have a problem with mySql or any other > projects relying on it. mySql in itself is not an issue. What I don't > want is for a customer to be presented with an option to chose and > configure two options for authenticating the UI, which I think is > needless. It adds complexity for not much value. Since LDAP is clearly > the better way to go that should be what we support without explicitly > giving a user an option to switch to JDBC. A user can still do so by > extending our abstractions if that is what they chose to do, but this would > not be officially supported by us. We would not be providing a config or > an mPack to do this. A user would have to do it on their own. > > James > > > > 15.11.2018, 12:15, "Michael Miklavcic" <michael.miklav...@gmail.com>: > > Incidentally, even without the Metron piece in the picture, what is the > > answer for Ambari's database dependency? Which uses a SQL data store. > Does > > this actually solve the problem of "customers won't install Metron bc SQL > > store?" or are there other issues we need to address? > > > > On Thu, Nov 15, 2018 at 9:30 AM James Sirota <jsir...@apache.org> wrote: > > > >> Hi Guys, > >> > >> My opinion on this, as is with Knox SSO, is that the code should be > >> pluggable to support JDBC, but we should not continue to support the > >> concrete implementation and expose it to users via a setting. This is a > >> fairly minor feature and the added complexity of supporting switching > >> between JDBC and LDAP is simply not worth it. We need to strike a > balance > >> between ease of use and capabilities/extensibility. For features that > are > >> worth it such as with analytics and stream processing, the extra > capability > >> is worth the added complexity in configuration. But for this, it is > not. > >> So let's keep JDBC around for a release to allow users to migrate to > LDAP, > >> deprecate it, and move on. > >> > >> Thanks, > >> James > >> > >> 13.11.2018, 16:03, "Simon Elliston Ball" <si...@simonellistonball.com > >: > >> > We went over the hbase user settings thing on extensive discussions > at > >> the time. Storing an arbitrary blob of JSON which is only ever > accessed by > >> a single key (username) was concluded to be a key value problem, not a > >> relational problem. Hbase was concluded to be massive overkill as a key > >> value store in this usecase, unless it was already there and ready to > go, > >> which in the case of Metron, it is, for enrichments, threat intel and > >> profiles. Hence it ended up in Hbase, as a conveniently present data > store > >> that matched the usage patterns. See > >> > https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E > >> and METRON-1337 for discussion. > >> > > >> > Simon > >> > > >> >> On 13 Nov 2018, at 18:50, Michael Miklavcic < > >> michael.miklav...@gmail.com> wrote: > >> >> > >> >> Thanks for the write up Simon. I don't think I see any major > problems > >> with > >> >> deprecating the general sql store. However, just to clarify, Metron > >> does > >> >> NOT require any specific backing store. It's 100% JPA, which means > >> anything > >> >> that can be configured with the Spring properties we expose. I think > >> the > >> >> most opinionated thing we do there is ship an extremely basic table > >> >> creation script for h2 and mysql as a simple example for schema. As > an > >> >> example, we simply use H2 in full dev, which is entirely in-memory > and > >> spun > >> >> up automatically from configuration. The recent work by Justin Leet > >> removes > >> >> the need to use a SQL store at all if you choose LDAP - > >> >> https://github.com/apache/metron/pull/1246. I'll let him comment > >> further on > >> >> this, but I think there is one small change that could be made via a > >> toggle > >> >> in Ambari that would even eliminate the user from seeing JDBC > settings > >> >> altogether during install if they choose LDAP. Again, I think I'm on > >> board > >> >> with deprecating the SQL backing store as I pointed this out on the > >> Knox > >> >> thread as well, but I just wanted to make sure everyone has an > accurate > >> >> picture of the current state. > >> >> > >> >> I had to double check on the HBase config you mentioned, but it does > >> appear > >> >> that we use it for the Alerts UI. I don't think I realized we were > >> storing > >> >> config there instead of the Zookeeper store we use for other system > >> >> configuration. Ironically enough, I think that it probably makes > more > >> sense > >> >> than the current auth info to store in a traditional sql store, > however > >> >> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, > as > >> you > >> >> pointed out. > >> >> > >> >> Whatever architectural changes we choose to add here, I think we > need > >> to > >> >> emphasize pluggability regardless of the specific implementation. > That > >> is > >> >> to say, I don't think we should make a hard requirement on Knox, in > >> order > >> >> to get LDAP, in order to deprecate an optional general SQL backing > >> store. > >> >> It makes sensible defaults if that's where we want to go, which is > the > >> way > >> >> we have done things for most of the successful features I've seen in > >> >> Metron. Provide all the options should a user desire them, but > abstract > >> >> away the complexity in the UIs. > >> >> > >> >> Best, > >> >> Mike > >> >> > >> >> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < > >> >> si...@simonellistonball.com> wrote: > >> >> > >> >>> I've been coming across a number of organisations who are blocked > from > >> >>> installing Metron by the MySQL auth database. > >> >>> > >> >>> The main problems with our MySQL default are: > >> >>> > >> >>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing > in a > >> >>> security platform and usually where the deployment conversation > ends > >> for me > >> >>> * MySQL install varies from platform to platform > >> >>> * An additional database to manage, backup, etc. so now I have to > >> talk to a > >> >>> DBA > >> >>> * Harder to maintain HA for this without externalising and fighting > >> against > >> >>> our defaults > >> >>> * There are a lot of dependencies for just storing a table of users > >> >>> (Eclipse Link, JPA, the MySQL server and the need to get clients > >> installed > >> >>> and pushed separately because of licence requirements) > >> >>> * Organisations don't want to have to manage yet another user > source > >> of > >> >>> truth since this leads to operational complexity. > >> >>> > >> >>> In short, managing our own user store makes very little sense to > >> operations > >> >>> users. > >> >>> > >> >>> Some of these (licence and inconsistency for example) could be > solved > >> by > >> >>> changing our default DB to something like Postgres, which has > easier > >> terms > >> >>> to deal with. We could start encrypting passwords, but there would > >> still be > >> >>> a lot of dependencies to store users, which is a problem much > better > >> solved > >> >>> by LDAP. > >> >>> > >> >>> Now that we have the option to use LDAP for user storage, I would > >> suggest > >> >>> that we deprecate and ultimately remove all the RDBMS and ORM > >> dependencies, > >> >>> which significantly reduces our dependencies and simplifies > >> deployment and > >> >>> long term management of Metron clusters. > >> >>> > >> >>> So I propose that we deprecate the RDBMS use in the next Apache > >> release, > >> >>> and then strip out the RDBMS stuff in the following. We would > >> continue to > >> >>> use LDAP for users and HBase for non-LDAPy user settings (as we > >> currently > >> >>> do). We should also provide a small demo LDAP for full dev. Since > we > >> are > >> >>> looking at adding Knox into the stack, that project provides a > >> convenient > >> >>> mini-LDAP demo service which would do this job without the need to > add > >> >>> additional components. > >> >>> > >> >>> Thoughts? Anyone relying on MySQL for users (if so, are you aware > >> that your > >> >>> passwords are all plaintext? How do you currently handle the > >> shortcomings > >> >>> and admin overhead?) Any objections? > >> >>> > >> >>> Simon > >> > >> ------------------- > >> Thank you, > >> > >> James Sirota > >> PMC- Apache Metron > >> jsirota AT apache DOT org > > ------------------- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >
