Allow someone to remove the dependence on hbase and only use Solr. For the way you use hbase for profiles, I can make Solr fulfill the same role.
Sent from my iPhone > On Nov 13, 2018, at 5:41 PM, James Sirota <jsir...@apache.org> wrote: > > Metron already works with Solr. It's not default yet, but all the internals > are in. The profiler currently works with Hbase and I don't think we were > ever planning on switching it to Solr. Just to clarify - metron can index > telemetry into Solr. However, profiling of that telemetry is supported via > Hbase. What exactly are you planning on doing? > > Thanks, > James > > 13.11.2018, 16:31, "Scott C. Cote" <scottcc...@gmail.com>: >> Simon, >> >> Since ya’ll are going to have SOLR in your installation (is this still >> true?), I could make the profile system rely upon SOLR instead of HBASE. At >> Lucidworks, I did this very thing with proprietary code, but I can make an >> adapter so the binaries can be stored in SOLR of arbitrary size. We have >> chatted briefly about this in Slack. Am trying to stabilize the employment >> situation, so that’s why I have not made any progress. >> >> Is there still interest here? >> >> SCott >> Scott C. Cote >> scottcc...@gmail.com >> 972.900.1561 >> >> twitter: @scottccote >> >>> On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball >>> <si...@simonellistonball.com> wrote: >>> >>> We went over the hbase user settings thing on extensive discussions at the >>> time. Storing an arbitrary blob of JSON which is only ever accessed by a >>> single key (username) was concluded to be a key value problem, not a >>> relational problem. Hbase was concluded to be massive overkill as a key >>> value store in this usecase, unless it was already there and ready to go, >>> which in the case of Metron, it is, for enrichments, threat intel and >>> profiles. Hence it ended up in Hbase, as a conveniently present data store >>> that matched the usage patterns. See >>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E >>> and METRON-1337 for discussion. >>> >>> Simon >>> >>>> On 13 Nov 2018, at 18:50, Michael Miklavcic <michael.miklav...@gmail.com> >>>> wrote: >>>> >>>> Thanks for the write up Simon. I don't think I see any major problems with >>>> deprecating the general sql store. However, just to clarify, Metron does >>>> NOT require any specific backing store. It's 100% JPA, which means >>>> anything >>>> that can be configured with the Spring properties we expose. I think the >>>> most opinionated thing we do there is ship an extremely basic table >>>> creation script for h2 and mysql as a simple example for schema. As an >>>> example, we simply use H2 in full dev, which is entirely in-memory and >>>> spun >>>> up automatically from configuration. The recent work by Justin Leet >>>> removes >>>> the need to use a SQL store at all if you choose LDAP - >>>> https://github.com/apache/metron/pull/1246. I'll let him comment further >>>> on >>>> this, but I think there is one small change that could be made via a >>>> toggle >>>> in Ambari that would even eliminate the user from seeing JDBC settings >>>> altogether during install if they choose LDAP. Again, I think I'm on board >>>> with deprecating the SQL backing store as I pointed this out on the Knox >>>> thread as well, but I just wanted to make sure everyone has an accurate >>>> picture of the current state. >>>> >>>> I had to double check on the HBase config you mentioned, but it does >>>> appear >>>> that we use it for the Alerts UI. I don't think I realized we were storing >>>> config there instead of the Zookeeper store we use for other system >>>> configuration. Ironically enough, I think that it probably makes more >>>> sense >>>> than the current auth info to store in a traditional sql store, however >>>> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you >>>> pointed out. >>>> >>>> Whatever architectural changes we choose to add here, I think we need to >>>> emphasize pluggability regardless of the specific implementation. That is >>>> to say, I don't think we should make a hard requirement on Knox, in order >>>> to get LDAP, in order to deprecate an optional general SQL backing store. >>>> It makes sensible defaults if that's where we want to go, which is the way >>>> we have done things for most of the successful features I've seen in >>>> Metron. Provide all the options should a user desire them, but abstract >>>> away the complexity in the UIs. >>>> >>>> Best, >>>> Mike >>>> >>>> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < >>>> si...@simonellistonball.com> wrote: >>>> >>>>> I've been coming across a number of organisations who are blocked from >>>>> installing Metron by the MySQL auth database. >>>>> >>>>> The main problems with our MySQL default are: >>>>> >>>>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a >>>>> security platform and usually where the deployment conversation ends for >>>>> me >>>>> * MySQL install varies from platform to platform >>>>> * An additional database to manage, backup, etc. so now I have to talk >>>>> to a >>>>> DBA >>>>> * Harder to maintain HA for this without externalising and fighting >>>>> against >>>>> our defaults >>>>> * There are a lot of dependencies for just storing a table of users >>>>> (Eclipse Link, JPA, the MySQL server and the need to get clients >>>>> installed >>>>> and pushed separately because of licence requirements) >>>>> * Organisations don't want to have to manage yet another user source of >>>>> truth since this leads to operational complexity. >>>>> >>>>> In short, managing our own user store makes very little sense to >>>>> operations >>>>> users. >>>>> >>>>> Some of these (licence and inconsistency for example) could be solved by >>>>> changing our default DB to something like Postgres, which has easier >>>>> terms >>>>> to deal with. We could start encrypting passwords, but there would still >>>>> be >>>>> a lot of dependencies to store users, which is a problem much better >>>>> solved >>>>> by LDAP. >>>>> >>>>> Now that we have the option to use LDAP for user storage, I would suggest >>>>> that we deprecate and ultimately remove all the RDBMS and ORM >>>>> dependencies, >>>>> which significantly reduces our dependencies and simplifies deployment >>>>> and >>>>> long term management of Metron clusters. >>>>> >>>>> So I propose that we deprecate the RDBMS use in the next Apache release, >>>>> and then strip out the RDBMS stuff in the following. We would continue to >>>>> use LDAP for users and HBase for non-LDAPy user settings (as we currently >>>>> do). We should also provide a small demo LDAP for full dev. Since we are >>>>> looking at adding Knox into the stack, that project provides a convenient >>>>> mini-LDAP demo service which would do this job without the need to add >>>>> additional components. >>>>> >>>>> Thoughts? Anyone relying on MySQL for users (if so, are you aware that >>>>> your >>>>> passwords are all plaintext? How do you currently handle the shortcomings >>>>> and admin overhead?) Any objections? >>>>> >>>>> Simon > > ------------------- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org >
