Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Otto Fowler
Ok, here is something I don’t understand, but would like to. Knox comes configured with build in services for a number of other apache products and UI’s. It would seem to me, that the best integration with Knox would be to do what these other products have done. 1. Do whatever you have to do

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Otto Fowler
https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support Solr is angular for example. On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com) wrote: Ok, here is something I don’t understand, but would like to. Knox comes configured

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Simon Elliston Ball
You could say the same thing about Ambari, but that provides mpacks. Knox is also designed to be extensible through Knox service stacks since they realized they can’t support every project. The challenge is that the docs have not made it as easy as they could for the ecosystem to plug into

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

2018-11-16 Thread Otto Fowler
Maybe this is worth a confluence entry, not as a guide, but just to document what you did. On November 15, 2018 at 19:07:40, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: Ok, this is finally merged! Whew! Here's how I polished up the history at the end. I used other feature branch

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Otto Fowler
Welcome and thanks! On November 12, 2018 at 10:12:44, Sandeep Moré (moresand...@gmail.com) wrote: Hello Ryan, I am still catching up on the architecture so let me know if I am misunderstanding anything. You could have multiple serviced deployed in Knox 1. for metron (metron/api/v1) 2. for

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Ryan Merriman
Most of the research I've done around adding Metron as a Knox service is based on how other projects do it. The documentation is not easy to follow so I learned by reading other service definition files. The assumption that we are doing things drastically different is false. I completely agree

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Simon Elliston Ball
I think there is a lot to be said for defaulting to Knox on... we also that way get some 'secure by default' - at least ssl by default. The do-nothing I think you're proposing would be around the authentication, right? Knox does ship with a demo LDAP server we could have some defaults (kinda like

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-11-16 Thread Justin Leet
Given that we've had a couple major PRs (the ES client migration along with the Angular upgrade stuff, and I'm sure others), I'd be in favor of releasing both the plugin and the main repo. I'd be in favor of doing something like: metron-bro-plugin-kafka release 0.3.0 PR to update full dev metron

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

2018-11-16 Thread Michael Miklavcic
It's a good suggestion, and I've been thinking about how to best handle this. Honestly, the right answer might be to do git rebase on master from the PR branch rather than a merge. That might avoid this situation altogether. Of course, that also comes with all the obligatory warnings about

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Otto Fowler
That does sound good Simon, I think I miss understood that the default LDAP was standard with KNOX/ambari and not something we would be doing ourselves. On November 16, 2018 at 10:54:48, Simon Elliston Ball ( si...@simonellistonball.com) wrote: I think there is a lot to be said for defaulting

Re: Running MAAS in batch

2018-11-16 Thread Otto Fowler
That may be the best MAAS explanation I’ve seen Simon. On November 16, 2018 at 10:28:57, Simon Elliston Ball ( si...@simonellistonball.com) wrote: MaaS is designed to wrap model inference (scoring) an event at a time, via a REST api. As such, running it batch doesn't make a lot of sense, since

Re: [DISCUSS] Deprecating MySQL

2018-11-16 Thread Otto Fowler
I would like to understand the work required to move our JDBC support ( or adapt the current support to the abstraction ) to /contrib. We could default and only officially support LDAP, but have the /contrib ( or /extension_examples ) have a “this is how you would support jdbc for auth “ project.

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Michael Miklavcic
Ryan, what's remote debugging look like for UI testing with Knox enabled? Anything we lose from a dev testability standpoint? The discussion of defaults sounds reasonable to me, and I'd like to understand any other tradeoffs there may be for non-prod deployments like full dev. On Fri, Nov 16,

Re: Running MAAS in batch

2018-11-16 Thread deepak kumar
Thanks Simon and it makes perfect sense. On Fri, 16 Nov 2018 at 8:58 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > MaaS is designed to wrap model inference (scoring) an event at a time, via > a REST api. As such, running it batch doesn't make a lot of sense, since > each

Re: Running MAAS in batch

2018-11-16 Thread Simon Elliston Ball
MaaS is designed to wrap model inference (scoring) an event at a time, via a REST api. As such, running it batch doesn't make a lot of sense, since each message would be processed individually. Most of the models you're likely to run in MaaS however, are also likely to be easily batchable, and are

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Otto Fowler
Those are all valid points. I think it is ( was ) worth discussion at lease a little. WRT Knox and defaults: I have in the past used “do-nothing” implementations as default placeholders for functionality that needed extensive per customer configuration, or configuration outside the

Re: Running MAAS in batch

2018-11-16 Thread deepak kumar
Simon, Can you elaborate more on this: ' *wrapped up in a batch engine like Spark to takeadvantage of more efficient "mass" scoring.* ' How the mass model wrapped in spark can take advantage of mass scoring? Thanks Deepak On Fri, Nov 16, 2018 at 9:15 PM Otto Fowler wrote: > That may be the

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Ryan Merriman
I was still able to spin up the UI locally and debug in my testing. I am in complete agreement, we need to ensure the developer experience doesn't change. On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Ryan, what's remote debugging look like for UI

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-11-16 Thread Otto Fowler
Can you generate the jiras that would be included in the release? On November 16, 2018 at 10:05:50, Justin Leet (justinjl...@gmail.com) wrote: Given that we've had a couple major PRs (the ES client migration along with the Angular upgrade stuff, and I'm sure others), I'd be in favor of

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

2018-11-16 Thread Otto Fowler
Maybe if we can’t document a concrete solution, we can document or codify a procedure. Should for example the owner of the feature branch always work with the release manager when there are issues? On November 16, 2018 at 10:26:11, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: It's

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Ryan Merriman
I would also add that defaulting to Knox being on simplifies things at a technical level. On Fri, Nov 16, 2018 at 10:52 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > That's fantastic, thanks for that detail. > > Also, I'm in agreement with the recent comments from Otto and Simon.

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Michael Miklavcic
That's fantastic, thanks for that detail. Also, I'm in agreement with the recent comments from Otto and Simon. On Fri, Nov 16, 2018 at 9:49 AM Ryan Merriman wrote: > I was still able to spin up the UI locally and debug in my testing. I am > in complete agreement, we need to ensure the

Running MAAS in batch

2018-11-16 Thread deepak kumar
Hi All Right now MAAS supports running the model against real time events being streamed into metron platform. Is there any way to run the models deployed in MAAS on the batch events / data that have been indexed into hdfs ? If anyone have tried this batch model , please share some insights.

Re: Running MAAS in batch

2018-11-16 Thread Simon Elliston Ball
You model is really just a function that you wrap in a REST service in order to deploy in MaaS. In the case of something like spark, you would just wrap it in a udf instead of wrapping it in a REST service, at that point, applying it in batch is just a case of a simple dataframe query. On Fri, 16

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-16 Thread Simon Elliston Ball
It is included, yes, but is not started out of the box by default, we would also probably tweak the blueprint to change its bootstrap ldif file a bit to have more sensible user names for our defaults, but that's pretty simple load. A bit of default blueprint config is what we need there, not