I’m certain that I can make the “adapter” for elastic too.   and the adapter 
compatible with HBase for those that want hbase (for other reasons).  Sorry 
that I was not clear about that.

Scott C. Cote

twitter: @scottccote

> On Nov 13, 2018, at 5:51 PM, Michael Miklavcic <michael.miklav...@gmail.com> 
> wrote:
> Hey Scott, Solr is an install option, not a hard requirement. Users can
> also choose Elasticsearch currently. HBase is the only option we currently
> provide for streaming enrichments, ie we can depend on it being there for
> every install.
> On Tue, Nov 13, 2018, 4:31 PM Scott C. Cote <scottcc...@gmail.com wrote:
>> Simon,
>> Since ya’ll are going to have SOLR in your installation (is this still
>> true?), I could make the profile system rely upon SOLR instead of HBASE.
>> At Lucidworks, I did this very thing with proprietary code, but I can make
>> an adapter so the binaries can be stored in SOLR of arbitrary size.  We
>> have chatted briefly about this in Slack.   Am trying to stabilize the
>> employment situation, so that’s why I have not made any progress.
>> Is there still interest here?
>> SCott
>> Scott C. Cote
>> scottcc...@gmail.com
>> 972.900.1561
>> twitter: @scottccote
>>> On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball <
>> si...@simonellistonball.com> wrote:
>>> We went over the hbase user settings thing on extensive discussions at
>> the time. Storing an arbitrary blob of JSON which is only ever accessed by
>> a single key (username) was concluded to be a key value problem, not a
>> relational problem. Hbase was concluded to be massive overkill as a key
>> value store in this usecase, unless it was already there and ready to go,
>> which in the case of Metron, it is, for enrichments, threat intel and
>> profiles. Hence it ended up in Hbase, as a conveniently present data store
>> that matched the usage patterns. See
>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>> and METRON-1337 for discussion.
>>> Simon
>>>> On 13 Nov 2018, at 18:50, Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>>> Thanks for the write up Simon. I don't think I see any major problems
>> with
>>>> deprecating the general sql store. However, just to clarify, Metron does
>>>> NOT require any specific backing store. It's 100% JPA, which means
>> anything
>>>> that can be configured with the Spring properties we expose. I think the
>>>> most opinionated thing we do there is ship an extremely basic table
>>>> creation script for h2 and mysql as a simple example for schema. As an
>>>> example, we simply use H2 in full dev, which is entirely in-memory and
>> spun
>>>> up automatically from configuration. The recent work by Justin Leet
>> removes
>>>> the need to use a SQL store at all if you choose LDAP -
>>>> https://github.com/apache/metron/pull/1246. I'll let him comment
>> further on
>>>> this, but I think there is one small change that could be made via a
>> toggle
>>>> in Ambari that would even eliminate the user from seeing JDBC settings
>>>> altogether during install if they choose LDAP. Again, I think I'm on
>> board
>>>> with deprecating the SQL backing store as I pointed this out on the Knox
>>>> thread as well, but I just wanted to make sure everyone has an accurate
>>>> picture of the current state.
>>>> I had to double check on the HBase config you mentioned, but it does
>> appear
>>>> that we use it for the Alerts UI. I don't think I realized we were
>> storing
>>>> config there instead of the Zookeeper store we use for other system
>>>> configuration. Ironically enough, I think that it probably makes more
>> sense
>>>> than the current auth info to store in a traditional sql store, however
>>>> it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as
>> you
>>>> pointed out.
>>>> Whatever architectural changes we choose to add here, I think we need to
>>>> emphasize pluggability regardless of the specific implementation. That
>> is
>>>> to say, I don't think we should make a hard requirement on Knox, in
>> order
>>>> to get LDAP, in order to deprecate an optional general SQL backing
>> store.
>>>> It makes sensible defaults if that's where we want to go, which is the
>> way
>>>> we have done things for most of the successful features I've seen in
>>>> Metron. Provide all the options should a user desire them, but abstract
>>>> away the complexity in the UIs.
>>>> Best,
>>>> Mike
>>>> On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
>>>> si...@simonellistonball.com> wrote:
>>>>> I've been coming across a number of organisations who are blocked from
>>>>> installing Metron by the MySQL auth database.
>>>>> The main problems with our MySQL default are:
>>>>> * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
>>>>> security platform and usually where the deployment conversation ends
>> for me
>>>>> * MySQL install varies from platform to platform
>>>>> * An additional database to manage, backup, etc. so now I have to talk
>> to a
>>>>> DBA
>>>>> * Harder to maintain HA for this without externalising and fighting
>> against
>>>>> our defaults
>>>>> * There are a lot of dependencies for just storing a table of users
>>>>> (Eclipse Link, JPA, the MySQL server and the need to get clients
>> installed
>>>>> and pushed separately because of licence requirements)
>>>>> * Organisations don't want to have to manage yet another user source of
>>>>> truth since this leads to operational complexity.
>>>>> In short, managing our own user store makes very little sense to
>> operations
>>>>> users.
>>>>> Some of these (licence and inconsistency for example) could be solved
>> by
>>>>> changing our default DB to something like Postgres, which has easier
>> terms
>>>>> to deal with. We could start encrypting passwords, but there would
>> still be
>>>>> a lot of dependencies to store users, which is a problem much better
>> solved
>>>>> by LDAP.
>>>>> Now that we have the option to use LDAP for user storage, I would
>> suggest
>>>>> that we deprecate and ultimately remove all the RDBMS and ORM
>> dependencies,
>>>>> which significantly reduces our dependencies and simplifies deployment
>> and
>>>>> long term management of Metron clusters.
>>>>> So I propose that we deprecate the RDBMS use in the next Apache
>> release,
>>>>> and then strip out the RDBMS stuff in the following. We would continue
>> to
>>>>> use LDAP for users and HBase for non-LDAPy user settings (as we
>> currently
>>>>> do). We should also provide a small demo LDAP for full dev. Since we
>> are
>>>>> looking at adding Knox into the stack, that project provides a
>> convenient
>>>>> mini-LDAP demo service which would do this job without the need to add
>>>>> additional components.
>>>>> Thoughts? Anyone relying on MySQL for users (if so, are you aware that
>> your
>>>>> passwords are all plaintext? How do you currently handle the
>> shortcomings
>>>>> and admin overhead?) Any objections?
>>>>> Simon

Reply via email to