Great feature to move to LDAP integration and hopefully Ranger integration afterwards. Does it need to support LDAP and AD separately?
Cheers, Ali On Sat, Nov 17, 2018 at 3:29 AM Otto Fowler <ottobackwa...@gmail.com> wrote: > I would like to understand the work required to move our JDBC support ( or > adapt the current support to the abstraction ) to /contrib. > We could default and only officially support LDAP, but have the /contrib ( > or /extension_examples ) have a “this is how you would support jdbc for > auth “ project. > > > > On November 15, 2018 at 15:01:10, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > Yes, makes sense. +1 to that. > > On Thu, Nov 15, 2018 at 12:54 PM James Sirota <jsir...@apache.org> wrote: > > > To clarify my position, I don't have a problem with mySql or any other > > projects relying on it. mySql in itself is not an issue. What I don't > > want is for a customer to be presented with an option to chose and > > configure two options for authenticating the UI, which I think is > > needless. It adds complexity for not much value. Since LDAP is clearly > > the better way to go that should be what we support without explicitly > > giving a user an option to switch to JDBC. A user can still do so by > > extending our abstractions if that is what they chose to do, but this > would > > not be officially supported by us. We would not be providing a config or > > an mPack to do this. A user would have to do it on their own. > > > > James > > > > > > > > 15.11.2018, 12:15, "Michael Miklavcic" <michael.miklav...@gmail.com>: > > > Incidentally, even without the Metron piece in the picture, what is the > > > answer for Ambari's database dependency? Which uses a SQL data store. > > Does > > > this actually solve the problem of "customers won't install Metron bc > SQL > > > store?" or are there other issues we need to address? > > > > > > On Thu, Nov 15, 2018 at 9:30 AM James Sirota <jsir...@apache.org> > wrote: > > > > > >> Hi Guys, > > >> > > >> My opinion on this, as is with Knox SSO, is that the code should be > > >> pluggable to support JDBC, but we should not continue to support the > > >> concrete implementation and expose it to users via a setting. This is > a > > >> fairly minor feature and the added complexity of supporting switching > > >> between JDBC and LDAP is simply not worth it. We need to strike a > > balance > > >> between ease of use and capabilities/extensibility. For features that > > are > > >> worth it such as with analytics and stream processing, the extra > > capability > > >> is worth the added complexity in configuration. But for this, it is > > not. > > >> So let's keep JDBC around for a release to allow users to migrate to > > LDAP, > > >> deprecate it, and move on. > > >> > > >> Thanks, > > >> James > > >> > > >> 13.11.2018, 16:03, "Simon Elliston Ball" <si...@simonellistonball.com > > >: > > >> > We went over the hbase user settings thing on extensive discussions > > at > > >> the time. Storing an arbitrary blob of JSON which is only ever > > accessed by > > >> a single key (username) was concluded to be a key value problem, not a > > >> relational problem. Hbase was concluded to be massive overkill as a > key > > >> value store in this usecase, unless it was already there and ready to > > go, > > >> which in the case of Metron, it is, for enrichments, threat intel and > > >> profiles. Hence it ended up in Hbase, as a conveniently present data > > store > > >> that matched the usage patterns. See > > >> > > > > https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E > > >> and METRON-1337 for discussion. > > >> > > > >> > Simon > > >> > > > >> >> On 13 Nov 2018, at 18:50, Michael Miklavcic < > > >> michael.miklav...@gmail.com> wrote: > > >> >> > > >> >> Thanks for the write up Simon. I don't think I see any major > > problems > > >> with > > >> >> deprecating the general sql store. However, just to clarify, Metron > > >> does > > >> >> NOT require any specific backing store. It's 100% JPA, which means > > >> anything > > >> >> that can be configured with the Spring properties we expose. I > think > > >> the > > >> >> most opinionated thing we do there is ship an extremely basic table > > >> >> creation script for h2 and mysql as a simple example for schema. As > > an > > >> >> example, we simply use H2 in full dev, which is entirely in-memory > > and > > >> spun > > >> >> up automatically from configuration. The recent work by Justin Leet > > >> removes > > >> >> the need to use a SQL store at all if you choose LDAP - > > >> >> https://github.com/apache/metron/pull/1246. I'll let him comment > > >> further on > > >> >> this, but I think there is one small change that could be made via > a > > >> toggle > > >> >> in Ambari that would even eliminate the user from seeing JDBC > > settings > > >> >> altogether during install if they choose LDAP. Again, I think I'm > on > > >> board > > >> >> with deprecating the SQL backing store as I pointed this out on the > > >> Knox > > >> >> thread as well, but I just wanted to make sure everyone has an > > accurate > > >> >> picture of the current state. > > >> >> > > >> >> I had to double check on the HBase config you mentioned, but it > does > > >> appear > > >> >> that we use it for the Alerts UI. I don't think I realized we were > > >> storing > > >> >> config there instead of the Zookeeper store we use for other system > > >> >> configuration. Ironically enough, I think that it probably makes > > more > > >> sense > > >> >> than the current auth info to store in a traditional sql store, > > however > > >> >> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, > > as > > >> you > > >> >> pointed out. > > >> >> > > >> >> Whatever architectural changes we choose to add here, I think we > > need > > >> to > > >> >> emphasize pluggability regardless of the specific implementation. > > That > > >> is > > >> >> to say, I don't think we should make a hard requirement on Knox, in > > >> order > > >> >> to get LDAP, in order to deprecate an optional general SQL backing > > >> store. > > >> >> It makes sensible defaults if that's where we want to go, which is > > the > > >> way > > >> >> we have done things for most of the successful features I've seen > in > > >> >> Metron. Provide all the options should a user desire them, but > > abstract > > >> >> away the complexity in the UIs. > > >> >> > > >> >> Best, > > >> >> Mike > > >> >> > > >> >> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < > > >> >> si...@simonellistonball.com> wrote: > > >> >> > > >> >>> I've been coming across a number of organisations who are blocked > > from > > >> >>> installing Metron by the MySQL auth database. > > >> >>> > > >> >>> The main problems with our MySQL default are: > > >> >>> > > >> >>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing > > in a > > >> >>> security platform and usually where the deployment conversation > > ends > > >> for me > > >> >>> * MySQL install varies from platform to platform > > >> >>> * An additional database to manage, backup, etc. so now I have to > > >> talk to a > > >> >>> DBA > > >> >>> * Harder to maintain HA for this without externalising and > fighting > > >> against > > >> >>> our defaults > > >> >>> * There are a lot of dependencies for just storing a table of > users > > >> >>> (Eclipse Link, JPA, the MySQL server and the need to get clients > > >> installed > > >> >>> and pushed separately because of licence requirements) > > >> >>> * Organisations don't want to have to manage yet another user > > source > > >> of > > >> >>> truth since this leads to operational complexity. > > >> >>> > > >> >>> In short, managing our own user store makes very little sense to > > >> operations > > >> >>> users. > > >> >>> > > >> >>> Some of these (licence and inconsistency for example) could be > > solved > > >> by > > >> >>> changing our default DB to something like Postgres, which has > > easier > > >> terms > > >> >>> to deal with. We could start encrypting passwords, but there would > > >> still be > > >> >>> a lot of dependencies to store users, which is a problem much > > better > > >> solved > > >> >>> by LDAP. > > >> >>> > > >> >>> Now that we have the option to use LDAP for user storage, I would > > >> suggest > > >> >>> that we deprecate and ultimately remove all the RDBMS and ORM > > >> dependencies, > > >> >>> which significantly reduces our dependencies and simplifies > > >> deployment and > > >> >>> long term management of Metron clusters. > > >> >>> > > >> >>> So I propose that we deprecate the RDBMS use in the next Apache > > >> release, > > >> >>> and then strip out the RDBMS stuff in the following. We would > > >> continue to > > >> >>> use LDAP for users and HBase for non-LDAPy user settings (as we > > >> currently > > >> >>> do). We should also provide a small demo LDAP for full dev. Since > > we > > >> are > > >> >>> looking at adding Knox into the stack, that project provides a > > >> convenient > > >> >>> mini-LDAP demo service which would do this job without the need to > > add > > >> >>> additional components. > > >> >>> > > >> >>> Thoughts? Anyone relying on MySQL for users (if so, are you aware > > >> that your > > >> >>> passwords are all plaintext? How do you currently handle the > > >> shortcomings > > >> >>> and admin overhead?) Any objections? > > >> >>> > > >> >>> Simon > > >> > > >> ------------------- > > >> Thank you, > > >> > > >> James Sirota > > >> PMC- Apache Metron > > >> jsirota AT apache DOT org > > > > ------------------- > > Thank you, > > > > James Sirota > > PMC- Apache Metron > > jsirota AT apache DOT org > > > > > -- A.Nazemian
