Metron already works with Solr. It's not default yet, but all the internals are in. The profiler currently works with Hbase and I don't think we were ever planning on switching it to Solr. Just to clarify - metron can index telemetry into Solr. However, profiling of that telemetry is supported via Hbase. What exactly are you planning on doing?
Thanks, James 13.11.2018, 16:31, "Scott C. Cote" <scottcc...@gmail.com>: > Simon, > > Since ya’ll are going to have SOLR in your installation (is this still > true?), I could make the profile system rely upon SOLR instead of HBASE. At > Lucidworks, I did this very thing with proprietary code, but I can make an > adapter so the binaries can be stored in SOLR of arbitrary size. We have > chatted briefly about this in Slack. Am trying to stabilize the employment > situation, so that’s why I have not made any progress. > > Is there still interest here? > > SCott > Scott C. Cote > scottcc...@gmail.com > 972.900.1561 > > twitter: @scottccote > >> On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball >> <si...@simonellistonball.com> wrote: >> >> We went over the hbase user settings thing on extensive discussions at the >> time. Storing an arbitrary blob of JSON which is only ever accessed by a >> single key (username) was concluded to be a key value problem, not a >> relational problem. Hbase was concluded to be massive overkill as a key >> value store in this usecase, unless it was already there and ready to go, >> which in the case of Metron, it is, for enrichments, threat intel and >> profiles. Hence it ended up in Hbase, as a conveniently present data store >> that matched the usage patterns. See >> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E >> and METRON-1337 for discussion. >> >> Simon >> >>> On 13 Nov 2018, at 18:50, Michael Miklavcic <michael.miklav...@gmail.com> >>> wrote: >>> >>> Thanks for the write up Simon. I don't think I see any major problems with >>> deprecating the general sql store. However, just to clarify, Metron does >>> NOT require any specific backing store. It's 100% JPA, which means anything >>> that can be configured with the Spring properties we expose. I think the >>> most opinionated thing we do there is ship an extremely basic table >>> creation script for h2 and mysql as a simple example for schema. As an >>> example, we simply use H2 in full dev, which is entirely in-memory and spun >>> up automatically from configuration. The recent work by Justin Leet removes >>> the need to use a SQL store at all if you choose LDAP - >>> https://github.com/apache/metron/pull/1246. I'll let him comment further on >>> this, but I think there is one small change that could be made via a toggle >>> in Ambari that would even eliminate the user from seeing JDBC settings >>> altogether during install if they choose LDAP. Again, I think I'm on board >>> with deprecating the SQL backing store as I pointed this out on the Knox >>> thread as well, but I just wanted to make sure everyone has an accurate >>> picture of the current state. >>> >>> I had to double check on the HBase config you mentioned, but it does appear >>> that we use it for the Alerts UI. I don't think I realized we were storing >>> config there instead of the Zookeeper store we use for other system >>> configuration. Ironically enough, I think that it probably makes more sense >>> than the current auth info to store in a traditional sql store, however >>> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you >>> pointed out. >>> >>> Whatever architectural changes we choose to add here, I think we need to >>> emphasize pluggability regardless of the specific implementation. That is >>> to say, I don't think we should make a hard requirement on Knox, in order >>> to get LDAP, in order to deprecate an optional general SQL backing store. >>> It makes sensible defaults if that's where we want to go, which is the way >>> we have done things for most of the successful features I've seen in >>> Metron. Provide all the options should a user desire them, but abstract >>> away the complexity in the UIs. >>> >>> Best, >>> Mike >>> >>> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < >>> si...@simonellistonball.com> wrote: >>> >>>> I've been coming across a number of organisations who are blocked from >>>> installing Metron by the MySQL auth database. >>>> >>>> The main problems with our MySQL default are: >>>> >>>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a >>>> security platform and usually where the deployment conversation ends for >>>> me >>>> * MySQL install varies from platform to platform >>>> * An additional database to manage, backup, etc. so now I have to talk to >>>> a >>>> DBA >>>> * Harder to maintain HA for this without externalising and fighting >>>> against >>>> our defaults >>>> * There are a lot of dependencies for just storing a table of users >>>> (Eclipse Link, JPA, the MySQL server and the need to get clients installed >>>> and pushed separately because of licence requirements) >>>> * Organisations don't want to have to manage yet another user source of >>>> truth since this leads to operational complexity. >>>> >>>> In short, managing our own user store makes very little sense to >>>> operations >>>> users. >>>> >>>> Some of these (licence and inconsistency for example) could be solved by >>>> changing our default DB to something like Postgres, which has easier terms >>>> to deal with. We could start encrypting passwords, but there would still >>>> be >>>> a lot of dependencies to store users, which is a problem much better >>>> solved >>>> by LDAP. >>>> >>>> Now that we have the option to use LDAP for user storage, I would suggest >>>> that we deprecate and ultimately remove all the RDBMS and ORM >>>> dependencies, >>>> which significantly reduces our dependencies and simplifies deployment and >>>> long term management of Metron clusters. >>>> >>>> So I propose that we deprecate the RDBMS use in the next Apache release, >>>> and then strip out the RDBMS stuff in the following. We would continue to >>>> use LDAP for users and HBase for non-LDAPy user settings (as we currently >>>> do). We should also provide a small demo LDAP for full dev. Since we are >>>> looking at adding Knox into the stack, that project provides a convenient >>>> mini-LDAP demo service which would do this job without the need to add >>>> additional components. >>>> >>>> Thoughts? Anyone relying on MySQL for users (if so, are you aware that >>>> your >>>> passwords are all plaintext? How do you currently handle the shortcomings >>>> and admin overhead?) Any objections? >>>> >>>> Simon ------------------- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
