Allow someone to remove the dependence on hbase and only use Solr.  For the way 
you use hbase for profiles, I can make Solr fulfill the same role.

Sent from my iPhone

> On Nov 13, 2018, at 5:41 PM, James Sirota <jsir...@apache.org> wrote:
> 
> Metron already works with Solr.  It's not default yet, but all the internals 
> are in.  The profiler currently works with Hbase and I don't think we were 
> ever planning on switching it to Solr.  Just to clarify - metron can index 
> telemetry into Solr.  However, profiling of that telemetry is supported via 
> Hbase.  What exactly are you planning on doing?
> 
> Thanks,
> James 
> 
> 13.11.2018, 16:31, "Scott C. Cote" <scottcc...@gmail.com>:
>> Simon,
>> 
>> Since ya’ll are going to have SOLR in your installation (is this still 
>> true?), I could make the profile system rely upon SOLR instead of HBASE. At 
>> Lucidworks, I did this very thing with proprietary code, but I can make an 
>> adapter so the binaries can be stored in SOLR of arbitrary size. We have 
>> chatted briefly about this in Slack. Am trying to stabilize the employment 
>> situation, so that’s why I have not made any progress.
>> 
>> Is there still interest here?
>> 
>> SCott
>> Scott C. Cote
>> scottcc...@gmail.com
>> 972.900.1561
>> 
>> twitter: @scottccote
>> 
>>>  On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball 
>>> <si...@simonellistonball.com> wrote:
>>> 
>>>  We went over the hbase user settings thing on extensive discussions at the 
>>> time. Storing an arbitrary blob of JSON which is only ever accessed by a 
>>> single key (username) was concluded to be a key value problem, not a 
>>> relational problem. Hbase was concluded to be massive overkill as a key 
>>> value store in this usecase, unless it was already there and ready to go, 
>>> which in the case of Metron, it is, for enrichments, threat intel and 
>>> profiles. Hence it ended up in Hbase, as a conveniently present data store 
>>> that matched the usage patterns. See 
>>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>>>  and METRON-1337 for discussion.
>>> 
>>>  Simon
>>> 
>>>>  On 13 Nov 2018, at 18:50, Michael Miklavcic <michael.miklav...@gmail.com> 
>>>> wrote:
>>>> 
>>>>  Thanks for the write up Simon. I don't think I see any major problems with
>>>>  deprecating the general sql store. However, just to clarify, Metron does
>>>>  NOT require any specific backing store. It's 100% JPA, which means 
>>>> anything
>>>>  that can be configured with the Spring properties we expose. I think the
>>>>  most opinionated thing we do there is ship an extremely basic table
>>>>  creation script for h2 and mysql as a simple example for schema. As an
>>>>  example, we simply use H2 in full dev, which is entirely in-memory and 
>>>> spun
>>>>  up automatically from configuration. The recent work by Justin Leet 
>>>> removes
>>>>  the need to use a SQL store at all if you choose LDAP -
>>>>  https://github.com/apache/metron/pull/1246. I'll let him comment further 
>>>> on
>>>>  this, but I think there is one small change that could be made via a 
>>>> toggle
>>>>  in Ambari that would even eliminate the user from seeing JDBC settings
>>>>  altogether during install if they choose LDAP. Again, I think I'm on board
>>>>  with deprecating the SQL backing store as I pointed this out on the Knox
>>>>  thread as well, but I just wanted to make sure everyone has an accurate
>>>>  picture of the current state.
>>>> 
>>>>  I had to double check on the HBase config you mentioned, but it does 
>>>> appear
>>>>  that we use it for the Alerts UI. I don't think I realized we were storing
>>>>  config there instead of the Zookeeper store we use for other system
>>>>  configuration. Ironically enough, I think that it probably makes more 
>>>> sense
>>>>  than the current auth info to store in a traditional sql store, however
>>>>  it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
>>>>  pointed out.
>>>> 
>>>>  Whatever architectural changes we choose to add here, I think we need to
>>>>  emphasize pluggability regardless of the specific implementation. That is
>>>>  to say, I don't think we should make a hard requirement on Knox, in order
>>>>  to get LDAP, in order to deprecate an optional general SQL backing store.
>>>>  It makes sensible defaults if that's where we want to go, which is the way
>>>>  we have done things for most of the successful features I've seen in
>>>>  Metron. Provide all the options should a user desire them, but abstract
>>>>  away the complexity in the UIs.
>>>> 
>>>>  Best,
>>>>  Mike
>>>> 
>>>>  On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
>>>>  si...@simonellistonball.com> wrote:
>>>> 
>>>>>  I've been coming across a number of organisations who are blocked from
>>>>>  installing Metron by the MySQL auth database.
>>>>> 
>>>>>  The main problems with our MySQL default are:
>>>>> 
>>>>>  * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
>>>>>  security platform and usually where the deployment conversation ends for 
>>>>> me
>>>>>  * MySQL install varies from platform to platform
>>>>>  * An additional database to manage, backup, etc. so now I have to talk 
>>>>> to a
>>>>>  DBA
>>>>>  * Harder to maintain HA for this without externalising and fighting 
>>>>> against
>>>>>  our defaults
>>>>>  * There are a lot of dependencies for just storing a table of users
>>>>>  (Eclipse Link, JPA, the MySQL server and the need to get clients 
>>>>> installed
>>>>>  and pushed separately because of licence requirements)
>>>>>  * Organisations don't want to have to manage yet another user source of
>>>>>  truth since this leads to operational complexity.
>>>>> 
>>>>>  In short, managing our own user store makes very little sense to 
>>>>> operations
>>>>>  users.
>>>>> 
>>>>>  Some of these (licence and inconsistency for example) could be solved by
>>>>>  changing our default DB to something like Postgres, which has easier 
>>>>> terms
>>>>>  to deal with. We could start encrypting passwords, but there would still 
>>>>> be
>>>>>  a lot of dependencies to store users, which is a problem much better 
>>>>> solved
>>>>>  by LDAP.
>>>>> 
>>>>>  Now that we have the option to use LDAP for user storage, I would suggest
>>>>>  that we deprecate and ultimately remove all the RDBMS and ORM 
>>>>> dependencies,
>>>>>  which significantly reduces our dependencies and simplifies deployment 
>>>>> and
>>>>>  long term management of Metron clusters.
>>>>> 
>>>>>  So I propose that we deprecate the RDBMS use in the next Apache release,
>>>>>  and then strip out the RDBMS stuff in the following. We would continue to
>>>>>  use LDAP for users and HBase for non-LDAPy user settings (as we currently
>>>>>  do). We should also provide a small demo LDAP for full dev. Since we are
>>>>>  looking at adding Knox into the stack, that project provides a convenient
>>>>>  mini-LDAP demo service which would do this job without the need to add
>>>>>  additional components.
>>>>> 
>>>>>  Thoughts? Anyone relying on MySQL for users (if so, are you aware that 
>>>>> your
>>>>>  passwords are all plaintext? How do you currently handle the shortcomings
>>>>>  and admin overhead?) Any objections?
>>>>> 
>>>>>  Simon
> 
> ------------------- 
> Thank you,
> 
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
> 

Reply via email to