Metron already works with Solr.  It's not default yet, but all the internals 
are in.  The profiler currently works with Hbase and I don't think we were ever 
planning on switching it to Solr.  Just to clarify - metron can index telemetry 
into Solr.  However, profiling of that telemetry is supported via Hbase.  What 
exactly are you planning on doing?

Thanks,
James 

13.11.2018, 16:31, "Scott C. Cote" <scottcc...@gmail.com>:
> Simon,
>
> Since ya’ll are going to have SOLR in your installation (is this still 
> true?), I could make the profile system rely upon SOLR instead of HBASE. At 
> Lucidworks, I did this very thing with proprietary code, but I can make an 
> adapter so the binaries can be stored in SOLR of arbitrary size. We have 
> chatted briefly about this in Slack. Am trying to stabilize the employment 
> situation, so that’s why I have not made any progress.
>
> Is there still interest here?
>
> SCott
> Scott C. Cote
> scottcc...@gmail.com
> 972.900.1561
>
> twitter: @scottccote
>
>>  On Nov 13, 2018, at 5:02 PM, Simon Elliston Ball 
>> <si...@simonellistonball.com> wrote:
>>
>>  We went over the hbase user settings thing on extensive discussions at the 
>> time. Storing an arbitrary blob of JSON which is only ever accessed by a 
>> single key (username) was concluded to be a key value problem, not a 
>> relational problem. Hbase was concluded to be massive overkill as a key 
>> value store in this usecase, unless it was already there and ready to go, 
>> which in the case of Metron, it is, for enrichments, threat intel and 
>> profiles. Hence it ended up in Hbase, as a conveniently present data store 
>> that matched the usage patterns. See 
>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>>  and METRON-1337 for discussion.
>>
>>  Simon
>>
>>>  On 13 Nov 2018, at 18:50, Michael Miklavcic <michael.miklav...@gmail.com> 
>>> wrote:
>>>
>>>  Thanks for the write up Simon. I don't think I see any major problems with
>>>  deprecating the general sql store. However, just to clarify, Metron does
>>>  NOT require any specific backing store. It's 100% JPA, which means anything
>>>  that can be configured with the Spring properties we expose. I think the
>>>  most opinionated thing we do there is ship an extremely basic table
>>>  creation script for h2 and mysql as a simple example for schema. As an
>>>  example, we simply use H2 in full dev, which is entirely in-memory and spun
>>>  up automatically from configuration. The recent work by Justin Leet removes
>>>  the need to use a SQL store at all if you choose LDAP -
>>>  https://github.com/apache/metron/pull/1246. I'll let him comment further on
>>>  this, but I think there is one small change that could be made via a toggle
>>>  in Ambari that would even eliminate the user from seeing JDBC settings
>>>  altogether during install if they choose LDAP. Again, I think I'm on board
>>>  with deprecating the SQL backing store as I pointed this out on the Knox
>>>  thread as well, but I just wanted to make sure everyone has an accurate
>>>  picture of the current state.
>>>
>>>  I had to double check on the HBase config you mentioned, but it does appear
>>>  that we use it for the Alerts UI. I don't think I realized we were storing
>>>  config there instead of the Zookeeper store we use for other system
>>>  configuration. Ironically enough, I think that it probably makes more sense
>>>  than the current auth info to store in a traditional sql store, however
>>>  it's in HBase currently so it's a non-issue wrt SQL/JPA either way, as you
>>>  pointed out.
>>>
>>>  Whatever architectural changes we choose to add here, I think we need to
>>>  emphasize pluggability regardless of the specific implementation. That is
>>>  to say, I don't think we should make a hard requirement on Knox, in order
>>>  to get LDAP, in order to deprecate an optional general SQL backing store.
>>>  It makes sensible defaults if that's where we want to go, which is the way
>>>  we have done things for most of the successful features I've seen in
>>>  Metron. Provide all the options should a user desire them, but abstract
>>>  away the complexity in the UIs.
>>>
>>>  Best,
>>>  Mike
>>>
>>>  On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
>>>  si...@simonellistonball.com> wrote:
>>>
>>>>  I've been coming across a number of organisations who are blocked from
>>>>  installing Metron by the MySQL auth database.
>>>>
>>>>  The main problems with our MySQL default are:
>>>>
>>>>  * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
>>>>  security platform and usually where the deployment conversation ends for 
>>>> me
>>>>  * MySQL install varies from platform to platform
>>>>  * An additional database to manage, backup, etc. so now I have to talk to 
>>>> a
>>>>  DBA
>>>>  * Harder to maintain HA for this without externalising and fighting 
>>>> against
>>>>  our defaults
>>>>  * There are a lot of dependencies for just storing a table of users
>>>>  (Eclipse Link, JPA, the MySQL server and the need to get clients installed
>>>>  and pushed separately because of licence requirements)
>>>>  * Organisations don't want to have to manage yet another user source of
>>>>  truth since this leads to operational complexity.
>>>>
>>>>  In short, managing our own user store makes very little sense to 
>>>> operations
>>>>  users.
>>>>
>>>>  Some of these (licence and inconsistency for example) could be solved by
>>>>  changing our default DB to something like Postgres, which has easier terms
>>>>  to deal with. We could start encrypting passwords, but there would still 
>>>> be
>>>>  a lot of dependencies to store users, which is a problem much better 
>>>> solved
>>>>  by LDAP.
>>>>
>>>>  Now that we have the option to use LDAP for user storage, I would suggest
>>>>  that we deprecate and ultimately remove all the RDBMS and ORM 
>>>> dependencies,
>>>>  which significantly reduces our dependencies and simplifies deployment and
>>>>  long term management of Metron clusters.
>>>>
>>>>  So I propose that we deprecate the RDBMS use in the next Apache release,
>>>>  and then strip out the RDBMS stuff in the following. We would continue to
>>>>  use LDAP for users and HBase for non-LDAPy user settings (as we currently
>>>>  do). We should also provide a small demo LDAP for full dev. Since we are
>>>>  looking at adding Knox into the stack, that project provides a convenient
>>>>  mini-LDAP demo service which would do this job without the need to add
>>>>  additional components.
>>>>
>>>>  Thoughts? Anyone relying on MySQL for users (if so, are you aware that 
>>>> your
>>>>  passwords are all plaintext? How do you currently handle the shortcomings
>>>>  and admin overhead?) Any objections?
>>>>
>>>>  Simon

------------------- 
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

Reply via email to