[DISCUSS] Deprecating MySQL

2018-11-13 Thread Simon Elliston Ball
I've been coming across a number of organisations who are blocked from installing Metron by the MySQL auth database. The main problems with our MySQL default are: * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a security platform and usually where the deployment conversation

Authorization for Configuration

2018-11-13 Thread Justin Leet
Hi all, Right now, our various configs can be modified by anyone with access to the various scripts. I'd like to start a discussion around building out some authorization to be able to add some more fine grained controls around this. Other projects have some variants on how to accomplish this.

New WIP PR, a new Full Dev option

2018-11-13 Thread Otto Fowler
I have submitted a WIP PR [#1261](https://github.com/apache/metron/pull/1261) that makes it possible to run / try the metron full dev environment with only Vagrant, VirtualBox and Docker installed, as opposed to having to have all the dev tools and ansible at the right version. I think this would

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Michael Miklavcic
Thanks for the write up Simon. I don't think I see any major problems with deprecating the general sql store. However, just to clarify, Metron does NOT require any specific backing store. It's 100% JPA, which means anything that can be configured with the Spring properties we expose. I think the

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Michael Miklavcic
Hey Scott, Solr is an install option, not a hard requirement. Users can also choose Elasticsearch currently. HBase is the only option we currently provide for streaming enrichments, ie we can depend on it being there for every install. On Tue, Nov 13, 2018, 4:31 PM Scott C. Cote Simon, > > Since

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Michael Miklavcic
Haha, yeah I had to dredge that up from February as well to remember what ultimately ended up going into HBase. Before you get your hackles up, I think you misunderstood me - I believe we're on the same page. I am saying that the SQL store would have made sense just fine in that case. Not that we

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread James Sirota
Metron already works with Solr. It's not default yet, but all the internals are in. The profiler currently works with Hbase and I don't think we were ever planning on switching it to Solr. Just to clarify - metron can index telemetry into Solr. However, profiling of that telemetry is

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Scott C. Cote
Simon, Since ya’ll are going to have SOLR in your installation (is this still true?), I could make the profile system rely upon SOLR instead of HBASE. At Lucidworks, I did this very thing with proprietary code, but I can make an adapter so the binaries can be stored in SOLR of arbitrary size.

Re: [DISCUSS] Deprecating MySQL

2018-11-13 Thread Simon Elliston Ball
We went over the hbase user settings thing on extensive discussions at the time. Storing an arbitrary blob of JSON which is only ever accessed by a single key (username) was concluded to be a key value problem, not a relational problem. Hbase was concluded to be massive overkill as a key value