Just to clarify this point, I am not proposing we alter any interactions
with Ambari and backing stores. I want to make sure we cover all the
problems and provide good solutions where we're able. Again, just to
reiterate my earlier point, I agree with the move to deprecate SQL/JPA for
the UI. And I agree with James about pluggability and simplifying the
surface area of options we expose to users via Ambari config.

Best,
Mike

On Thu, Nov 15, 2018 at 12:14 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Incidentally, even without the Metron piece in the picture, what is the
> answer for Ambari's database dependency? Which uses a SQL data store. Does
> this actually solve the problem of "customers won't install Metron bc SQL
> store?" or are there other issues we need to address?
>
> On Thu, Nov 15, 2018 at 9:30 AM James Sirota <jsir...@apache.org> wrote:
>
>> Hi Guys,
>>
>> My opinion on this, as is with Knox SSO, is that the code should be
>> pluggable to support JDBC, but we should not continue to support the
>> concrete implementation and expose it to users via a setting.  This is a
>> fairly minor feature and the added complexity of supporting switching
>> between JDBC and LDAP is simply not worth it.  We need to strike a balance
>> between ease of use and capabilities/extensibility.  For features that are
>> worth it such as with analytics and stream processing, the extra capability
>> is worth the added complexity in configuration.  But for this, it is not.
>> So let's keep JDBC around for a release to allow users to migrate to LDAP,
>> deprecate it, and move on.
>>
>> Thanks,
>> James
>>
>> 13.11.2018, 16:03, "Simon Elliston Ball" <si...@simonellistonball.com>:
>> > We went over the hbase user settings thing on extensive discussions at
>> the time. Storing an arbitrary blob of JSON which is only ever accessed by
>> a single key (username) was concluded to be a key value problem, not a
>> relational problem. Hbase was concluded to be massive overkill as a key
>> value store in this usecase, unless it was already there and ready to go,
>> which in the case of Metron, it is, for enrichments, threat intel and
>> profiles. Hence it ended up in Hbase, as a conveniently present data store
>> that matched the usage patterns. See
>> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E
>> and METRON-1337 for discussion.
>> >
>> > Simon
>> >
>> >>  On 13 Nov 2018, at 18:50, Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>> >>
>> >>  Thanks for the write up Simon. I don't think I see any major problems
>> with
>> >>  deprecating the general sql store. However, just to clarify, Metron
>> does
>> >>  NOT require any specific backing store. It's 100% JPA, which means
>> anything
>> >>  that can be configured with the Spring properties we expose. I think
>> the
>> >>  most opinionated thing we do there is ship an extremely basic table
>> >>  creation script for h2 and mysql as a simple example for schema. As an
>> >>  example, we simply use H2 in full dev, which is entirely in-memory
>> and spun
>> >>  up automatically from configuration. The recent work by Justin Leet
>> removes
>> >>  the need to use a SQL store at all if you choose LDAP -
>> >>  https://github.com/apache/metron/pull/1246. I'll let him comment
>> further on
>> >>  this, but I think there is one small change that could be made via a
>> toggle
>> >>  in Ambari that would even eliminate the user from seeing JDBC settings
>> >>  altogether during install if they choose LDAP. Again, I think I'm on
>> board
>> >>  with deprecating the SQL backing store as I pointed this out on the
>> Knox
>> >>  thread as well, but I just wanted to make sure everyone has an
>> accurate
>> >>  picture of the current state.
>> >>
>> >>  I had to double check on the HBase config you mentioned, but it does
>> appear
>> >>  that we use it for the Alerts UI. I don't think I realized we were
>> storing
>> >>  config there instead of the Zookeeper store we use for other system
>> >>  configuration. Ironically enough, I think that it probably makes more
>> sense
>> >>  than the current auth info to store in a traditional sql store,
>> however
>> >>  it's in HBase currently so it's a non-issue wrt SQL/JPA either way,
>> as you
>> >>  pointed out.
>> >>
>> >>  Whatever architectural changes we choose to add here, I think we need
>> to
>> >>  emphasize pluggability regardless of the specific implementation.
>> That is
>> >>  to say, I don't think we should make a hard requirement on Knox, in
>> order
>> >>  to get LDAP, in order to deprecate an optional general SQL backing
>> store.
>> >>  It makes sensible defaults if that's where we want to go, which is
>> the way
>> >>  we have done things for most of the successful features I've seen in
>> >>  Metron. Provide all the options should a user desire them, but
>> abstract
>> >>  away the complexity in the UIs.
>> >>
>> >>  Best,
>> >>  Mike
>> >>
>> >>  On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball <
>> >>  si...@simonellistonball.com> wrote:
>> >>
>> >>>  I've been coming across a number of organisations who are blocked
>> from
>> >>>  installing Metron by the MySQL auth database.
>> >>>
>> >>>  The main problems with our MySQL default are:
>> >>>
>> >>>  * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a
>> >>>  security platform and usually where the deployment conversation ends
>> for me
>> >>>  * MySQL install varies from platform to platform
>> >>>  * An additional database to manage, backup, etc. so now I have to
>> talk to a
>> >>>  DBA
>> >>>  * Harder to maintain HA for this without externalising and fighting
>> against
>> >>>  our defaults
>> >>>  * There are a lot of dependencies for just storing a table of users
>> >>>  (Eclipse Link, JPA, the MySQL server and the need to get clients
>> installed
>> >>>  and pushed separately because of licence requirements)
>> >>>  * Organisations don't want to have to manage yet another user source
>> of
>> >>>  truth since this leads to operational complexity.
>> >>>
>> >>>  In short, managing our own user store makes very little sense to
>> operations
>> >>>  users.
>> >>>
>> >>>  Some of these (licence and inconsistency for example) could be
>> solved by
>> >>>  changing our default DB to something like Postgres, which has easier
>> terms
>> >>>  to deal with. We could start encrypting passwords, but there would
>> still be
>> >>>  a lot of dependencies to store users, which is a problem much better
>> solved
>> >>>  by LDAP.
>> >>>
>> >>>  Now that we have the option to use LDAP for user storage, I would
>> suggest
>> >>>  that we deprecate and ultimately remove all the RDBMS and ORM
>> dependencies,
>> >>>  which significantly reduces our dependencies and simplifies
>> deployment and
>> >>>  long term management of Metron clusters.
>> >>>
>> >>>  So I propose that we deprecate the RDBMS use in the next Apache
>> release,
>> >>>  and then strip out the RDBMS stuff in the following. We would
>> continue to
>> >>>  use LDAP for users and HBase for non-LDAPy user settings (as we
>> currently
>> >>>  do). We should also provide a small demo LDAP for full dev. Since we
>> are
>> >>>  looking at adding Knox into the stack, that project provides a
>> convenient
>> >>>  mini-LDAP demo service which would do this job without the need to
>> add
>> >>>  additional components.
>> >>>
>> >>>  Thoughts? Anyone relying on MySQL for users (if so, are you aware
>> that your
>> >>>  passwords are all plaintext? How do you currently handle the
>> shortcomings
>> >>>  and admin overhead?) Any objections?
>> >>>
>> >>>  Simon
>>
>> -------------------
>> Thank you,
>>
>> James Sirota
>> PMC- Apache Metron
>> jsirota AT apache DOT org
>>
>>
>>

Reply via email to