Hey Scott, Solr is an install option, not a hard requirement. Users can also choose Elasticsearch currently. HBase is the only option we currently provide for streaming enrichments, ie we can depend on it being there for every install.
On Tue, Nov 13, 2018, 4:31 PM Scott C. Cote <scottcc...@gmail.com wrote: > Simon, > > Since ya’ll are going to have SOLR in your installation (is this still > true?), I could make the profile system rely upon SOLR instead of HBASE. > At Lucidworks, I did this very thing with proprietary code, but I can make > an adapter so the binaries can be stored in SOLR of arbitrary size. We > have chatted briefly about this in Slack. Am trying to stabilize the > employment situation, so that’s why I have not made any progress. > > Is there still interest here? > > SCott > Scott C. Cote > scottcc...@gmail.com > 972.900.1561 > > twitter: @scottccote > > > > > On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > > We went over the hbase user settings thing on extensive discussions at > the time. Storing an arbitrary blob of JSON which is only ever accessed by > a single key (username) was concluded to be a key value problem, not a > relational problem. Hbase was concluded to be massive overkill as a key > value store in this usecase, unless it was already there and ready to go, > which in the case of Metron, it is, for enrichments, threat intel and > profiles. Hence it ended up in Hbase, as a conveniently present data store > that matched the usage patterns. See > https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E > and METRON-1337 for discussion. > > > > Simon > > > >> On 13 Nov 2018, at 18:50, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> > >> Thanks for the write up Simon. I don't think I see any major problems > with > >> deprecating the general sql store. However, just to clarify, Metron does > >> NOT require any specific backing store. It's 100% JPA, which means > anything > >> that can be configured with the Spring properties we expose. I think the > >> most opinionated thing we do there is ship an extremely basic table > >> creation script for h2 and mysql as a simple example for schema. As an > >> example, we simply use H2 in full dev, which is entirely in-memory and > spun > >> up automatically from configuration. The recent work by Justin Leet > removes > >> the need to use a SQL store at all if you choose LDAP - > >> https://github.com/apache/metron/pull/1246. I'll let him comment > further on > >> this, but I think there is one small change that could be made via a > toggle > >> in Ambari that would even eliminate the user from seeing JDBC settings > >> altogether during install if they choose LDAP. Again, I think I'm on > board > >> with deprecating the SQL backing store as I pointed this out on the Knox > >> thread as well, but I just wanted to make sure everyone has an accurate > >> picture of the current state. > >> > >> I had to double check on the HBase config you mentioned, but it does > appear > >> that we use it for the Alerts UI. I don't think I realized we were > storing > >> config there instead of the Zookeeper store we use for other system > >> configuration. Ironically enough, I think that it probably makes more > sense > >> than the current auth info to store in a traditional sql store, however > >> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as > you > >> pointed out. > >> > >> Whatever architectural changes we choose to add here, I think we need to > >> emphasize pluggability regardless of the specific implementation. That > is > >> to say, I don't think we should make a hard requirement on Knox, in > order > >> to get LDAP, in order to deprecate an optional general SQL backing > store. > >> It makes sensible defaults if that's where we want to go, which is the > way > >> we have done things for most of the successful features I've seen in > >> Metron. Provide all the options should a user desire them, but abstract > >> away the complexity in the UIs. > >> > >> Best, > >> Mike > >> > >> > >> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < > >> si...@simonellistonball.com> wrote: > >> > >>> I've been coming across a number of organisations who are blocked from > >>> installing Metron by the MySQL auth database. > >>> > >>> The main problems with our MySQL default are: > >>> > >>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a > >>> security platform and usually where the deployment conversation ends > for me > >>> * MySQL install varies from platform to platform > >>> * An additional database to manage, backup, etc. so now I have to talk > to a > >>> DBA > >>> * Harder to maintain HA for this without externalising and fighting > against > >>> our defaults > >>> * There are a lot of dependencies for just storing a table of users > >>> (Eclipse Link, JPA, the MySQL server and the need to get clients > installed > >>> and pushed separately because of licence requirements) > >>> * Organisations don't want to have to manage yet another user source of > >>> truth since this leads to operational complexity. > >>> > >>> In short, managing our own user store makes very little sense to > operations > >>> users. > >>> > >>> Some of these (licence and inconsistency for example) could be solved > by > >>> changing our default DB to something like Postgres, which has easier > terms > >>> to deal with. We could start encrypting passwords, but there would > still be > >>> a lot of dependencies to store users, which is a problem much better > solved > >>> by LDAP. > >>> > >>> Now that we have the option to use LDAP for user storage, I would > suggest > >>> that we deprecate and ultimately remove all the RDBMS and ORM > dependencies, > >>> which significantly reduces our dependencies and simplifies deployment > and > >>> long term management of Metron clusters. > >>> > >>> So I propose that we deprecate the RDBMS use in the next Apache > release, > >>> and then strip out the RDBMS stuff in the following. We would continue > to > >>> use LDAP for users and HBase for non-LDAPy user settings (as we > currently > >>> do). We should also provide a small demo LDAP for full dev. Since we > are > >>> looking at adding Knox into the stack, that project provides a > convenient > >>> mini-LDAP demo service which would do this job without the need to add > >>> additional components. > >>> > >>> Thoughts? Anyone relying on MySQL for users (if so, are you aware that > your > >>> passwords are all plaintext? How do you currently handle the > shortcomings > >>> and admin overhead?) Any objections? > >>> > >>> Simon > >>> > >