I’m certain that I can make the “adapter” for elastic too. and the adapter compatible with HBase for those that want hbase (for other reasons). Sorry that I was not clear about that.
SCott Scott C. Cote scottcc...@gmail.com 972.900.1561 twitter: @scottccote > On Nov 13, 2018, at 5:51 PM, Michael Miklavcic <michael.miklav...@gmail.com> > wrote: > > Hey Scott, Solr is an install option, not a hard requirement. Users can > also choose Elasticsearch currently. HBase is the only option we currently > provide for streaming enrichments, ie we can depend on it being there for > every install. > > On Tue, Nov 13, 2018, 4:31 PM Scott C. Cote <scottcc...@gmail.com wrote: > >> Simon, >> >> Since ya’ll are going to have SOLR in your installation (is this still >> true?), I could make the profile system rely upon SOLR instead of HBASE. >> At Lucidworks, I did this very thing with proprietary code, but I can make >> an adapter so the binaries can be stored in SOLR of arbitrary size. We >> have chatted briefly about this in Slack. Am trying to stabilize the >> employment situation, so that’s why I have not made any progress. >> >> Is there still interest here? >> >> SCott >> Scott C. Cote >> scottcc...@gmail.com >> 972.900.1561 >> >> twitter: @scottccote >> >> >> >>> On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >>> >>> We went over the hbase user settings thing on extensive discussions at >> the time. Storing an arbitrary blob of JSON which is only ever accessed by >> a single key (username) was concluded to be a key value problem, not a >> relational problem. Hbase was concluded to be massive overkill as a key >> value store in this usecase, unless it was already there and ready to go, >> which in the case of Metron, it is, for enrichments, threat intel and >> profiles. Hence it ended up in Hbase, as a conveniently present data store >> that matched the usage patterns. See >> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E >> and METRON-1337 for discussion. >>> >>> Simon >>> >>>> On 13 Nov 2018, at 18:50, Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >>>> >>>> Thanks for the write up Simon. I don't think I see any major problems >> with >>>> deprecating the general sql store. However, just to clarify, Metron does >>>> NOT require any specific backing store. It's 100% JPA, which means >> anything >>>> that can be configured with the Spring properties we expose. I think the >>>> most opinionated thing we do there is ship an extremely basic table >>>> creation script for h2 and mysql as a simple example for schema. As an >>>> example, we simply use H2 in full dev, which is entirely in-memory and >> spun >>>> up automatically from configuration. The recent work by Justin Leet >> removes >>>> the need to use a SQL store at all if you choose LDAP - >>>> https://github.com/apache/metron/pull/1246. I'll let him comment >> further on >>>> this, but I think there is one small change that could be made via a >> toggle >>>> in Ambari that would even eliminate the user from seeing JDBC settings >>>> altogether during install if they choose LDAP. Again, I think I'm on >> board >>>> with deprecating the SQL backing store as I pointed this out on the Knox >>>> thread as well, but I just wanted to make sure everyone has an accurate >>>> picture of the current state. >>>> >>>> I had to double check on the HBase config you mentioned, but it does >> appear >>>> that we use it for the Alerts UI. I don't think I realized we were >> storing >>>> config there instead of the Zookeeper store we use for other system >>>> configuration. Ironically enough, I think that it probably makes more >> sense >>>> than the current auth info to store in a traditional sql store, however >>>> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as >> you >>>> pointed out. >>>> >>>> Whatever architectural changes we choose to add here, I think we need to >>>> emphasize pluggability regardless of the specific implementation. That >> is >>>> to say, I don't think we should make a hard requirement on Knox, in >> order >>>> to get LDAP, in order to deprecate an optional general SQL backing >> store. >>>> It makes sensible defaults if that's where we want to go, which is the >> way >>>> we have done things for most of the successful features I've seen in >>>> Metron. Provide all the options should a user desire them, but abstract >>>> away the complexity in the UIs. >>>> >>>> Best, >>>> Mike >>>> >>>> >>>> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < >>>> si...@simonellistonball.com> wrote: >>>> >>>>> I've been coming across a number of organisations who are blocked from >>>>> installing Metron by the MySQL auth database. >>>>> >>>>> The main problems with our MySQL default are: >>>>> >>>>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a >>>>> security platform and usually where the deployment conversation ends >> for me >>>>> * MySQL install varies from platform to platform >>>>> * An additional database to manage, backup, etc. so now I have to talk >> to a >>>>> DBA >>>>> * Harder to maintain HA for this without externalising and fighting >> against >>>>> our defaults >>>>> * There are a lot of dependencies for just storing a table of users >>>>> (Eclipse Link, JPA, the MySQL server and the need to get clients >> installed >>>>> and pushed separately because of licence requirements) >>>>> * Organisations don't want to have to manage yet another user source of >>>>> truth since this leads to operational complexity. >>>>> >>>>> In short, managing our own user store makes very little sense to >> operations >>>>> users. >>>>> >>>>> Some of these (licence and inconsistency for example) could be solved >> by >>>>> changing our default DB to something like Postgres, which has easier >> terms >>>>> to deal with. We could start encrypting passwords, but there would >> still be >>>>> a lot of dependencies to store users, which is a problem much better >> solved >>>>> by LDAP. >>>>> >>>>> Now that we have the option to use LDAP for user storage, I would >> suggest >>>>> that we deprecate and ultimately remove all the RDBMS and ORM >> dependencies, >>>>> which significantly reduces our dependencies and simplifies deployment >> and >>>>> long term management of Metron clusters. >>>>> >>>>> So I propose that we deprecate the RDBMS use in the next Apache >> release, >>>>> and then strip out the RDBMS stuff in the following. We would continue >> to >>>>> use LDAP for users and HBase for non-LDAPy user settings (as we >> currently >>>>> do). We should also provide a small demo LDAP for full dev. Since we >> are >>>>> looking at adding Knox into the stack, that project provides a >> convenient >>>>> mini-LDAP demo service which would do this job without the need to add >>>>> additional components. >>>>> >>>>> Thoughts? Anyone relying on MySQL for users (if so, are you aware that >> your >>>>> passwords are all plaintext? How do you currently handle the >> shortcomings >>>>> and admin overhead?) Any objections? >>>>> >>>>> Simon >>>>> >> >>
