Re: [DISCUSS] Knox SSO feature branch review and features
I would also add that defaulting to Knox being on simplifies things at a technical level. On Fri, Nov 16, 2018 at 10:52 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > That's fantastic, thanks for that detail. > > Also, I'm in agreement with the recent comments from Otto and Simon. > > On Fri, Nov 16, 2018 at 9:49 AM Ryan Merriman wrote: > > > I was still able to spin up the UI locally and debug in my testing. I am > > in complete agreement, we need to ensure the developer experience doesn't > > change. > > > > On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > Ryan, what's remote debugging look like for UI testing with Knox > enabled? > > > Anything we lose from a dev testability standpoint? The discussion of > > > defaults sounds reasonable to me, and I'd like to understand any other > > > tradeoffs there may be for non-prod deployments like full dev. > > > > > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman > > wrote: > > > > > > > Most of the research I've done around adding Metron as a Knox service > > is > > > > based on how other projects do it. The documentation is not easy to > > > follow > > > > so I learned by reading other service definition files. The > assumption > > > > that we are doing things drastically different is false. > > > > > > > > I completely agree with Simon. Why would we want to be dependent on > > > Knox's > > > > release cycle? How does that benefit us? It may reduce some > > operational > > > > complexity but it makes our install process more complicated because > we > > > > require a certain version of Knox (who knows when that gets > released). > > > > What do we do in the meantime? I would also like to point out that > > > Metron > > > > is inherently different than other Hadoop stack services. We are a > > > > full-blown application with multiple UIs so the way we expose > services > > > > through Knox may be a little different. > > > > > > > > I think this will be easier to discuss when we can all see what is > > > actually > > > > involved. I am working on a PR that adds Metron as a Knox service > and > > > will > > > > have that out soon. That should give everyone more context. > > > > > > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > > > > si...@simonellistonball.com> wrote: > > > > > > > > > You could say the same thing about Ambari, but that provides > mpacks. > > > Knox > > > > > is also designed to be extensible through Knox service stacks since > > > they > > > > > realized they can’t support every project. The challenge is that > the > > > docs > > > > > have not made it as easy as they could for the ecosystem to plug > into > > > > Knox, > > > > > which has led to some confusion around this being a recommended > > pattern > > > > > (which it is). > > > > > > > > > > The danger of trying to get your bits into Knox is that that ties > you > > > to > > > > > their release cycle (a problem Ambari has felt hard, hence their > > > > community > > > > > is moving away from the everything inside model towards everything > is > > > an > > > > > mpack). > > > > > > > > > > A number of implementations of Knox also use the approach Ryan is > > > > > suggesting for their own organization specific end points, so it’s > > not > > > > like > > > > > this is an uncommon, or anti-pattern, it’s more the way Knox is > > > designed > > > > to > > > > > work in the future, than the legacy of it only being able to > handle a > > > > > subset of Hadoop projects. > > > > > > > > > > Knox remains optional In our scenario, but we keep control over the > > > > > shipping of things like rewrite rules, which allows Metron to > control > > > its > > > > > release destiny should things like url patterns in the ui need to > > > change > > > > > (with a new release of angular / new module / new rest endpoint > etc) > > > > > instead of making a Metron release dependent on a Knox release. > > > > > > > > > > Imagine how we would have done with the Ambari side if we’d had to > > wait > > > > > for them to release every time we needed to change something in the > > > > > mpack... we don’t want that happening with Knox. > > > > > > > > > > Simon > > > > > > > > > > > On 16 Nov 2018, at 13:22, Otto Fowler > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > > > > > > > Solr is angular for example. > > > > > > > > > > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler ( > > > ottobackwa...@gmail.com > > > > ) > > > > > > wrote: > > > > > > > > > > > > Ok, here is something I don’t understand, but would like to. > > > > > > > > > > > > Knox comes configured with build in services for a number of > other > > > > apache > > > > > > products and UI’s. > > > > > > It would seem to me, that the best integration with Knox would be > > to > > > do > > > > > > what these other prod
Re: [DISCUSS] Knox SSO feature branch review and features
That's fantastic, thanks for that detail. Also, I'm in agreement with the recent comments from Otto and Simon. On Fri, Nov 16, 2018 at 9:49 AM Ryan Merriman wrote: > I was still able to spin up the UI locally and debug in my testing. I am > in complete agreement, we need to ensure the developer experience doesn't > change. > > On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Ryan, what's remote debugging look like for UI testing with Knox enabled? > > Anything we lose from a dev testability standpoint? The discussion of > > defaults sounds reasonable to me, and I'd like to understand any other > > tradeoffs there may be for non-prod deployments like full dev. > > > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman > wrote: > > > > > Most of the research I've done around adding Metron as a Knox service > is > > > based on how other projects do it. The documentation is not easy to > > follow > > > so I learned by reading other service definition files. The assumption > > > that we are doing things drastically different is false. > > > > > > I completely agree with Simon. Why would we want to be dependent on > > Knox's > > > release cycle? How does that benefit us? It may reduce some > operational > > > complexity but it makes our install process more complicated because we > > > require a certain version of Knox (who knows when that gets released). > > > What do we do in the meantime? I would also like to point out that > > Metron > > > is inherently different than other Hadoop stack services. We are a > > > full-blown application with multiple UIs so the way we expose services > > > through Knox may be a little different. > > > > > > I think this will be easier to discuss when we can all see what is > > actually > > > involved. I am working on a PR that adds Metron as a Knox service and > > will > > > have that out soon. That should give everyone more context. > > > > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > > > si...@simonellistonball.com> wrote: > > > > > > > You could say the same thing about Ambari, but that provides mpacks. > > Knox > > > > is also designed to be extensible through Knox service stacks since > > they > > > > realized they can’t support every project. The challenge is that the > > docs > > > > have not made it as easy as they could for the ecosystem to plug into > > > Knox, > > > > which has led to some confusion around this being a recommended > pattern > > > > (which it is). > > > > > > > > The danger of trying to get your bits into Knox is that that ties you > > to > > > > their release cycle (a problem Ambari has felt hard, hence their > > > community > > > > is moving away from the everything inside model towards everything is > > an > > > > mpack). > > > > > > > > A number of implementations of Knox also use the approach Ryan is > > > > suggesting for their own organization specific end points, so it’s > not > > > like > > > > this is an uncommon, or anti-pattern, it’s more the way Knox is > > designed > > > to > > > > work in the future, than the legacy of it only being able to handle a > > > > subset of Hadoop projects. > > > > > > > > Knox remains optional In our scenario, but we keep control over the > > > > shipping of things like rewrite rules, which allows Metron to control > > its > > > > release destiny should things like url patterns in the ui need to > > change > > > > (with a new release of angular / new module / new rest endpoint etc) > > > > instead of making a Metron release dependent on a Knox release. > > > > > > > > Imagine how we would have done with the Ambari side if we’d had to > wait > > > > for them to release every time we needed to change something in the > > > > mpack... we don’t want that happening with Knox. > > > > > > > > Simon > > > > > > > > > On 16 Nov 2018, at 13:22, Otto Fowler > > wrote: > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > > > > > Solr is angular for example. > > > > > > > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler ( > > ottobackwa...@gmail.com > > > ) > > > > > wrote: > > > > > > > > > > Ok, here is something I don’t understand, but would like to. > > > > > > > > > > Knox comes configured with build in services for a number of other > > > apache > > > > > products and UI’s. > > > > > It would seem to me, that the best integration with Knox would be > to > > do > > > > > what these other products have done. > > > > > > > > > > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > > > > 2. Create a knox service definition and provide it or try to get it > > > into > > > > > knox itself > > > > > > > > > > This would make the knox integration with metron optional and > > pluggable > > > > > wouldn’t it? > > > > > > > > > > Then knox with metron would just be the same as knox with anything > > > else. > > > > >
Re: [DISCUSS] Knox SSO feature branch review and features
I was still able to spin up the UI locally and debug in my testing. I am in complete agreement, we need to ensure the developer experience doesn't change. On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Ryan, what's remote debugging look like for UI testing with Knox enabled? > Anything we lose from a dev testability standpoint? The discussion of > defaults sounds reasonable to me, and I'd like to understand any other > tradeoffs there may be for non-prod deployments like full dev. > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman wrote: > > > Most of the research I've done around adding Metron as a Knox service is > > based on how other projects do it. The documentation is not easy to > follow > > so I learned by reading other service definition files. The assumption > > that we are doing things drastically different is false. > > > > I completely agree with Simon. Why would we want to be dependent on > Knox's > > release cycle? How does that benefit us? It may reduce some operational > > complexity but it makes our install process more complicated because we > > require a certain version of Knox (who knows when that gets released). > > What do we do in the meantime? I would also like to point out that > Metron > > is inherently different than other Hadoop stack services. We are a > > full-blown application with multiple UIs so the way we expose services > > through Knox may be a little different. > > > > I think this will be easier to discuss when we can all see what is > actually > > involved. I am working on a PR that adds Metron as a Knox service and > will > > have that out soon. That should give everyone more context. > > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > > si...@simonellistonball.com> wrote: > > > > > You could say the same thing about Ambari, but that provides mpacks. > Knox > > > is also designed to be extensible through Knox service stacks since > they > > > realized they can’t support every project. The challenge is that the > docs > > > have not made it as easy as they could for the ecosystem to plug into > > Knox, > > > which has led to some confusion around this being a recommended pattern > > > (which it is). > > > > > > The danger of trying to get your bits into Knox is that that ties you > to > > > their release cycle (a problem Ambari has felt hard, hence their > > community > > > is moving away from the everything inside model towards everything is > an > > > mpack). > > > > > > A number of implementations of Knox also use the approach Ryan is > > > suggesting for their own organization specific end points, so it’s not > > like > > > this is an uncommon, or anti-pattern, it’s more the way Knox is > designed > > to > > > work in the future, than the legacy of it only being able to handle a > > > subset of Hadoop projects. > > > > > > Knox remains optional In our scenario, but we keep control over the > > > shipping of things like rewrite rules, which allows Metron to control > its > > > release destiny should things like url patterns in the ui need to > change > > > (with a new release of angular / new module / new rest endpoint etc) > > > instead of making a Metron release dependent on a Knox release. > > > > > > Imagine how we would have done with the Ambari side if we’d had to wait > > > for them to release every time we needed to change something in the > > > mpack... we don’t want that happening with Knox. > > > > > > Simon > > > > > > > On 16 Nov 2018, at 13:22, Otto Fowler > wrote: > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > > > Solr is angular for example. > > > > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler ( > ottobackwa...@gmail.com > > ) > > > > wrote: > > > > > > > > Ok, here is something I don’t understand, but would like to. > > > > > > > > Knox comes configured with build in services for a number of other > > apache > > > > products and UI’s. > > > > It would seem to me, that the best integration with Knox would be to > do > > > > what these other products have done. > > > > > > > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > > > 2. Create a knox service definition and provide it or try to get it > > into > > > > knox itself > > > > > > > > This would make the knox integration with metron optional and > pluggable > > > > wouldn’t it? > > > > > > > > Then knox with metron would just be the same as knox with anything > > else. > > > > Please help me if I am wrong, but we seem to be going our own way > here. > > > > Why don’t we just do what these other products have done? > > > > Why don’t we try to get apache metron services accepted to the knox > > > > project? Why don’t we model our knox integration with how XYZ does > it? > > > > Have we looked at how others integrate? Having all the code and > being > > > > able to track stuff is kind of the
Re: [DISCUSS] Knox SSO feature branch review and features
Ryan, what's remote debugging look like for UI testing with Knox enabled? Anything we lose from a dev testability standpoint? The discussion of defaults sounds reasonable to me, and I'd like to understand any other tradeoffs there may be for non-prod deployments like full dev. On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman wrote: > Most of the research I've done around adding Metron as a Knox service is > based on how other projects do it. The documentation is not easy to follow > so I learned by reading other service definition files. The assumption > that we are doing things drastically different is false. > > I completely agree with Simon. Why would we want to be dependent on Knox's > release cycle? How does that benefit us? It may reduce some operational > complexity but it makes our install process more complicated because we > require a certain version of Knox (who knows when that gets released). > What do we do in the meantime? I would also like to point out that Metron > is inherently different than other Hadoop stack services. We are a > full-blown application with multiple UIs so the way we expose services > through Knox may be a little different. > > I think this will be easier to discuss when we can all see what is actually > involved. I am working on a PR that adds Metron as a Knox service and will > have that out soon. That should give everyone more context. > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > You could say the same thing about Ambari, but that provides mpacks. Knox > > is also designed to be extensible through Knox service stacks since they > > realized they can’t support every project. The challenge is that the docs > > have not made it as easy as they could for the ecosystem to plug into > Knox, > > which has led to some confusion around this being a recommended pattern > > (which it is). > > > > The danger of trying to get your bits into Knox is that that ties you to > > their release cycle (a problem Ambari has felt hard, hence their > community > > is moving away from the everything inside model towards everything is an > > mpack). > > > > A number of implementations of Knox also use the approach Ryan is > > suggesting for their own organization specific end points, so it’s not > like > > this is an uncommon, or anti-pattern, it’s more the way Knox is designed > to > > work in the future, than the legacy of it only being able to handle a > > subset of Hadoop projects. > > > > Knox remains optional In our scenario, but we keep control over the > > shipping of things like rewrite rules, which allows Metron to control its > > release destiny should things like url patterns in the ui need to change > > (with a new release of angular / new module / new rest endpoint etc) > > instead of making a Metron release dependent on a Knox release. > > > > Imagine how we would have done with the Ambari side if we’d had to wait > > for them to release every time we needed to change something in the > > mpack... we don’t want that happening with Knox. > > > > Simon > > > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > Solr is angular for example. > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com > ) > > > wrote: > > > > > > Ok, here is something I don’t understand, but would like to. > > > > > > Knox comes configured with build in services for a number of other > apache > > > products and UI’s. > > > It would seem to me, that the best integration with Knox would be to do > > > what these other products have done. > > > > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > > 2. Create a knox service definition and provide it or try to get it > into > > > knox itself > > > > > > This would make the knox integration with metron optional and pluggable > > > wouldn’t it? > > > > > > Then knox with metron would just be the same as knox with anything > else. > > > Please help me if I am wrong, but we seem to be going our own way here. > > > Why don’t we just do what these other products have done? > > > Why don’t we try to get apache metron services accepted to the knox > > > project? Why don’t we model our knox integration with how XYZ does it? > > > Have we looked at how others integrate? Having all the code and being > > > able to track stuff is kind of the point of this whole thing isn’t it? > > > > > > Maybe this is implied and I’m missing it, if so I apologize. > > > > > > I think consistency with the rest of the hadoop stack with knox helps > us. > > > > > > > > > > > > On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) > > wrote: > > > > > > 1) Sorry I misspoke. I meant to say this is not possible in the Alerts > UI > > > as far as I know. I put up a PR with a proposed solution here: > > > https://github.com/apa
Re: [DISCUSS] Deprecating MySQL
I would like to understand the work required to move our JDBC support ( or adapt the current support to the abstraction ) to /contrib. We could default and only officially support LDAP, but have the /contrib ( or /extension_examples ) have a “this is how you would support jdbc for auth “ project. On November 15, 2018 at 15:01:10, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: Yes, makes sense. +1 to that. On Thu, Nov 15, 2018 at 12:54 PM James Sirota wrote: > To clarify my position, I don't have a problem with mySql or any other > projects relying on it. mySql in itself is not an issue. What I don't > want is for a customer to be presented with an option to chose and > configure two options for authenticating the UI, which I think is > needless. It adds complexity for not much value. Since LDAP is clearly > the better way to go that should be what we support without explicitly > giving a user an option to switch to JDBC. A user can still do so by > extending our abstractions if that is what they chose to do, but this would > not be officially supported by us. We would not be providing a config or > an mPack to do this. A user would have to do it on their own. > > James > > > > 15.11.2018, 12:15, "Michael Miklavcic" : > > Incidentally, even without the Metron piece in the picture, what is the > > answer for Ambari's database dependency? Which uses a SQL data store. > Does > > this actually solve the problem of "customers won't install Metron bc SQL > > store?" or are there other issues we need to address? > > > > On Thu, Nov 15, 2018 at 9:30 AM James Sirota wrote: > > > >> Hi Guys, > >> > >> My opinion on this, as is with Knox SSO, is that the code should be > >> pluggable to support JDBC, but we should not continue to support the > >> concrete implementation and expose it to users via a setting. This is a > >> fairly minor feature and the added complexity of supporting switching > >> between JDBC and LDAP is simply not worth it. We need to strike a > balance > >> between ease of use and capabilities/extensibility. For features that > are > >> worth it such as with analytics and stream processing, the extra > capability > >> is worth the added complexity in configuration. But for this, it is > not. > >> So let's keep JDBC around for a release to allow users to migrate to > LDAP, > >> deprecate it, and move on. > >> > >> Thanks, > >> James > >> > >> 13.11.2018, 16:03, "Simon Elliston Ball" >: > >> > We went over the hbase user settings thing on extensive discussions > at > >> the time. Storing an arbitrary blob of JSON which is only ever > accessed by > >> a single key (username) was concluded to be a key value problem, not a > >> relational problem. Hbase was concluded to be massive overkill as a key > >> value store in this usecase, unless it was already there and ready to > go, > >> which in the case of Metron, it is, for enrichments, threat intel and > >> profiles. Hence it ended up in Hbase, as a conveniently present data > store > >> that matched the usage patterns. See > >> > https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E > >> and METRON-1337 for discussion. > >> > > >> > Simon > >> > > >> >> On 13 Nov 2018, at 18:50, Michael Miklavcic < > >> michael.miklav...@gmail.com> wrote: > >> >> > >> >> Thanks for the write up Simon. I don't think I see any major > problems > >> with > >> >> deprecating the general sql store. However, just to clarify, Metron > >> does > >> >> NOT require any specific backing store. It's 100% JPA, which means > >> anything > >> >> that can be configured with the Spring properties we expose. I think > >> the > >> >> most opinionated thing we do there is ship an extremely basic table > >> >> creation script for h2 and mysql as a simple example for schema. As > an > >> >> example, we simply use H2 in full dev, which is entirely in-memory > and > >> spun > >> >> up automatically from configuration. The recent work by Justin Leet > >> removes > >> >> the need to use a SQL store at all if you choose LDAP - > >> >> https://github.com/apache/metron/pull/1246. I'll let him comment > >> further on > >> >> this, but I think there is one small change that could be made via a > >> toggle > >> >> in Ambari that would even eliminate the user from seeing JDBC > settings > >> >> altogether during install if they choose LDAP. Again, I think I'm on > >> board > >> >> with deprecating the SQL backing store as I pointed this out on the > >> Knox > >> >> thread as well, but I just wanted to make sure everyone has an > accurate > >> >> picture of the current state. > >> >> > >> >> I had to double check on the HBase config you mentioned, but it does > >> appear > >> >> that we use it for the Alerts UI. I don't think I realized we were > >> storing > >> >> config there instead of the Zookeeper store we use for other system > >> >> configuration. Ironically enough, I think that it probably makes > more > >> sense
Re: [DISCUSS] Knox SSO feature branch review and features
It is included, yes, but is not started out of the box by default, we would also probably tweak the blueprint to change its bootstrap ldif file a bit to have more sensible user names for our defaults, but that's pretty simple load. A bit of default blueprint config is what we need there, not anything 'code' related per se. Simon On Fri, 16 Nov 2018 at 15:59, Otto Fowler wrote: > That does sound good Simon, I think I miss understood that the default > LDAP was standard with KNOX/ambari and not something we would be doing > ourselves. > > > On November 16, 2018 at 10:54:48, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > I think there is a lot to be said for defaulting to Knox on... we also > that > way get some 'secure by default' - at least ssl by default. The do-nothing > I think you're proposing would be around the authentication, right? Knox > does ship with a demo LDAP server we could have some defaults (kinda like > we do with the dev spring profile today) which might be a good way of > achieving a similar effect, and that's configured by default by Ambari. > Would that meet the need, or do you think we should provide a "yeah I'm > sure we don't need to authenticate that connection, let them in" identity > provider for Knox? That way we would have to have them impersonate a known > user for the REST api to work, but you would get the seemless, no auth > access. > > To be honest, I'm a fan of the first option where we give people a nice > simple, no external config and just use some sensible default users in the > demo LDAP instance that Knox owns, and then default to Knox on to give us > our nice single access point. > > Simon > > > On Fri, 16 Nov 2018 at 15:35, Otto Fowler > wrote: > > > Those are all valid points. I think it is ( was ) worth discussion at > > lease a little. > > > > WRT Knox and defaults: > > > > I have in the past used “do-nothing” implementations as default > > placeholders for functionality > > that needed extensive per customer configuration, or configuration > outside > > the responsibility of the product. > > > > Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX > > configuration with “do-nothing” providers > > for auth etc. The users would then configure the providers ( based on > the > > provider(s) we support ) at a later time. > > > > We could write the providers, as everyone has pointed out how extensible > > KNOX is ;) > > > > Would that be a valid way to simplify the issue? > > What would the fallout of that be? > > > > > > > > On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com) > > wrote: > > > > Most of the research I've done around adding Metron as a Knox service is > > based on how other projects do it. The documentation is not easy to > follow > > so I learned by reading other service definition files. The assumption > > that we are doing things drastically different is false. > > > > I completely agree with Simon. Why would we want to be dependent on > Knox's > > release cycle? How does that benefit us? It may reduce some operational > > complexity but it makes our install process more complicated because we > > require a certain version of Knox (who knows when that gets released). > > What do we do in the meantime? I would also like to point out that > Metron > > is inherently different than other Hadoop stack services. We are a > > full-blown application with multiple UIs so the way we expose services > > through Knox may be a little different. > > > > I think this will be easier to discuss when we can all see what is > actually > > involved. I am working on a PR that adds Metron as a Knox service and > will > > have that out soon. That should give everyone more context. > > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > > si...@simonellistonball.com> wrote: > > > > > You could say the same thing about Ambari, but that provides mpacks. > Knox > > > is also designed to be extensible through Knox service stacks since > they > > > realized they can’t support every project. The challenge is that the > docs > > > have not made it as easy as they could for the ecosystem to plug into > > Knox, > > > which has led to some confusion around this being a recommended > pattern > > > (which it is). > > > > > > The danger of trying to get your bits into Knox is that that ties you > to > > > their release cycle (a problem Ambari has felt hard, hence their > > community > > > is moving away from the everything inside model towards everything is > an > > > mpack). > > > > > > A number of implementations of Knox also use the approach Ryan is > > > suggesting for their own organization specific end points, so it’s not > > like > > > this is an uncommon, or anti-pattern, it’s more the way Knox is > designed > > to > > > work in the future, than the legacy of it only being able to handle a > > > subset of Hadoop projects. > > > > > > Knox remains optional In our scenario, but we keep control over the > > > shippin
Re: [DISCUSS] Knox SSO feature branch review and features
That does sound good Simon, I think I miss understood that the default LDAP was standard with KNOX/ambari and not something we would be doing ourselves. On November 16, 2018 at 10:54:48, Simon Elliston Ball ( si...@simonellistonball.com) wrote: I think there is a lot to be said for defaulting to Knox on... we also that way get some 'secure by default' - at least ssl by default. The do-nothing I think you're proposing would be around the authentication, right? Knox does ship with a demo LDAP server we could have some defaults (kinda like we do with the dev spring profile today) which might be a good way of achieving a similar effect, and that's configured by default by Ambari. Would that meet the need, or do you think we should provide a "yeah I'm sure we don't need to authenticate that connection, let them in" identity provider for Knox? That way we would have to have them impersonate a known user for the REST api to work, but you would get the seemless, no auth access. To be honest, I'm a fan of the first option where we give people a nice simple, no external config and just use some sensible default users in the demo LDAP instance that Knox owns, and then default to Knox on to give us our nice single access point. Simon On Fri, 16 Nov 2018 at 15:35, Otto Fowler wrote: > Those are all valid points. I think it is ( was ) worth discussion at > lease a little. > > WRT Knox and defaults: > > I have in the past used “do-nothing” implementations as default > placeholders for functionality > that needed extensive per customer configuration, or configuration outside > the responsibility of the product. > > Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX > configuration with “do-nothing” providers > for auth etc. The users would then configure the providers ( based on the > provider(s) we support ) at a later time. > > We could write the providers, as everyone has pointed out how extensible > KNOX is ;) > > Would that be a valid way to simplify the issue? > What would the fallout of that be? > > > > On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com) > wrote: > > Most of the research I've done around adding Metron as a Knox service is > based on how other projects do it. The documentation is not easy to follow > so I learned by reading other service definition files. The assumption > that we are doing things drastically different is false. > > I completely agree with Simon. Why would we want to be dependent on Knox's > release cycle? How does that benefit us? It may reduce some operational > complexity but it makes our install process more complicated because we > require a certain version of Knox (who knows when that gets released). > What do we do in the meantime? I would also like to point out that Metron > is inherently different than other Hadoop stack services. We are a > full-blown application with multiple UIs so the way we expose services > through Knox may be a little different. > > I think this will be easier to discuss when we can all see what is actually > involved. I am working on a PR that adds Metron as a Knox service and will > have that out soon. That should give everyone more context. > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > You could say the same thing about Ambari, but that provides mpacks. Knox > > is also designed to be extensible through Knox service stacks since they > > realized they can’t support every project. The challenge is that the docs > > have not made it as easy as they could for the ecosystem to plug into > Knox, > > which has led to some confusion around this being a recommended pattern > > (which it is). > > > > The danger of trying to get your bits into Knox is that that ties you to > > their release cycle (a problem Ambari has felt hard, hence their > community > > is moving away from the everything inside model towards everything is an > > mpack). > > > > A number of implementations of Knox also use the approach Ryan is > > suggesting for their own organization specific end points, so it’s not > like > > this is an uncommon, or anti-pattern, it’s more the way Knox is designed > to > > work in the future, than the legacy of it only being able to handle a > > subset of Hadoop projects. > > > > Knox remains optional In our scenario, but we keep control over the > > shipping of things like rewrite rules, which allows Metron to control its > > release destiny should things like url patterns in the ui need to change > > (with a new release of angular / new module / new rest endpoint etc) > > instead of making a Metron release dependent on a Knox release. > > > > Imagine how we would have done with the Ambari side if we’d had to wait > > for them to release every time we needed to change something in the > > mpack... we don’t want that happening with Knox. > > > > Simon > > > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > > > > > > https://issues.apache.org/jira
Re: Running MAAS in batch
You model is really just a function that you wrap in a REST service in order to deploy in MaaS. In the case of something like spark, you would just wrap it in a udf instead of wrapping it in a REST service, at that point, applying it in batch is just a case of a simple dataframe query. On Fri, 16 Nov 2018 at 15:51, deepak kumar wrote: > Simon, > Can you elaborate more on this: > ' > > *wrapped up in a batch engine like Spark to takeadvantage of more > efficient "mass" scoring.* > ' > How the mass model wrapped in spark can take advantage of mass scoring? > > Thanks > Deepak > > On Fri, Nov 16, 2018 at 9:15 PM Otto Fowler > wrote: > >> That may be the best MAAS explanation I’ve seen Simon. >> >> >> On November 16, 2018 at 10:28:57, Simon Elliston Ball ( >> si...@simonellistonball.com) wrote: >> >> MaaS is designed to wrap model inference (scoring) an event at a time, >> via a REST api. As such, running it batch doesn't make a lot of sense, >> since each message would be processed individually. Most of the models >> you're likely to run in MaaS however, are also likely to be easily >> batchable, and are probable better wrapped up in a batch engine like Spark >> to take advantage of more efficient "mass" scoring. >> >> Simon >> >> On Fri, 16 Nov 2018 at 15:18, deepak kumar wrote: >> >>> Hi All >>> Right now MAAS supports running the model against real time events being >>> streamed into metron platform. >>> Is there any way to run the models deployed in MAAS on the batch events >>> / data that have been indexed into hdfs ? >>> If anyone have tried this batch model , please share some insights. >>> Thanks >>> Deepak. >>> >>> >> >> -- >> -- >> simon elliston ball >> @sireb >> >> -- -- simon elliston ball @sireb
Re: [DISCUSS] Knox SSO feature branch review and features
I think there is a lot to be said for defaulting to Knox on... we also that way get some 'secure by default' - at least ssl by default. The do-nothing I think you're proposing would be around the authentication, right? Knox does ship with a demo LDAP server we could have some defaults (kinda like we do with the dev spring profile today) which might be a good way of achieving a similar effect, and that's configured by default by Ambari. Would that meet the need, or do you think we should provide a "yeah I'm sure we don't need to authenticate that connection, let them in" identity provider for Knox? That way we would have to have them impersonate a known user for the REST api to work, but you would get the seemless, no auth access. To be honest, I'm a fan of the first option where we give people a nice simple, no external config and just use some sensible default users in the demo LDAP instance that Knox owns, and then default to Knox on to give us our nice single access point. Simon On Fri, 16 Nov 2018 at 15:35, Otto Fowler wrote: > Those are all valid points. I think it is ( was ) worth discussion at > lease a little. > > WRT Knox and defaults: > > I have in the past used “do-nothing” implementations as default > placeholders for functionality > that needed extensive per customer configuration, or configuration outside > the responsibility of the product. > > Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX > configuration with “do-nothing” providers > for auth etc. The users would then configure the providers ( based on the > provider(s) we support ) at a later time. > > We could write the providers, as everyone has pointed out how extensible > KNOX is ;) > > Would that be a valid way to simplify the issue? > What would the fallout of that be? > > > > On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com) > wrote: > > Most of the research I've done around adding Metron as a Knox service is > based on how other projects do it. The documentation is not easy to follow > so I learned by reading other service definition files. The assumption > that we are doing things drastically different is false. > > I completely agree with Simon. Why would we want to be dependent on Knox's > release cycle? How does that benefit us? It may reduce some operational > complexity but it makes our install process more complicated because we > require a certain version of Knox (who knows when that gets released). > What do we do in the meantime? I would also like to point out that Metron > is inherently different than other Hadoop stack services. We are a > full-blown application with multiple UIs so the way we expose services > through Knox may be a little different. > > I think this will be easier to discuss when we can all see what is actually > involved. I am working on a PR that adds Metron as a Knox service and will > have that out soon. That should give everyone more context. > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > You could say the same thing about Ambari, but that provides mpacks. Knox > > is also designed to be extensible through Knox service stacks since they > > realized they can’t support every project. The challenge is that the docs > > have not made it as easy as they could for the ecosystem to plug into > Knox, > > which has led to some confusion around this being a recommended pattern > > (which it is). > > > > The danger of trying to get your bits into Knox is that that ties you to > > their release cycle (a problem Ambari has felt hard, hence their > community > > is moving away from the everything inside model towards everything is an > > mpack). > > > > A number of implementations of Knox also use the approach Ryan is > > suggesting for their own organization specific end points, so it’s not > like > > this is an uncommon, or anti-pattern, it’s more the way Knox is designed > to > > work in the future, than the legacy of it only being able to handle a > > subset of Hadoop projects. > > > > Knox remains optional In our scenario, but we keep control over the > > shipping of things like rewrite rules, which allows Metron to control its > > release destiny should things like url patterns in the ui need to change > > (with a new release of angular / new module / new rest endpoint etc) > > instead of making a Metron release dependent on a Knox release. > > > > Imagine how we would have done with the Ambari side if we’d had to wait > > for them to release every time we needed to change something in the > > mpack... we don’t want that happening with Knox. > > > > Simon > > > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > Solr is angular for example. > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com > ) > > > wrote: > > > > > > Ok, here is someth
Re: Running MAAS in batch
Simon, Can you elaborate more on this: ' *wrapped up in a batch engine like Spark to takeadvantage of more efficient "mass" scoring.* ' How the mass model wrapped in spark can take advantage of mass scoring? Thanks Deepak On Fri, Nov 16, 2018 at 9:15 PM Otto Fowler wrote: > That may be the best MAAS explanation I’ve seen Simon. > > > On November 16, 2018 at 10:28:57, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > MaaS is designed to wrap model inference (scoring) an event at a time, via > a REST api. As such, running it batch doesn't make a lot of sense, since > each message would be processed individually. Most of the models you're > likely to run in MaaS however, are also likely to be easily batchable, and > are probable better wrapped up in a batch engine like Spark to take > advantage of more efficient "mass" scoring. > > Simon > > On Fri, 16 Nov 2018 at 15:18, deepak kumar wrote: > >> Hi All >> Right now MAAS supports running the model against real time events being >> streamed into metron platform. >> Is there any way to run the models deployed in MAAS on the batch events / >> data that have been indexed into hdfs ? >> If anyone have tried this batch model , please share some insights. >> Thanks >> Deepak. >> >> > > -- > -- > simon elliston ball > @sireb > >
Re: Running MAAS in batch
That may be the best MAAS explanation I’ve seen Simon. On November 16, 2018 at 10:28:57, Simon Elliston Ball ( si...@simonellistonball.com) wrote: MaaS is designed to wrap model inference (scoring) an event at a time, via a REST api. As such, running it batch doesn't make a lot of sense, since each message would be processed individually. Most of the models you're likely to run in MaaS however, are also likely to be easily batchable, and are probable better wrapped up in a batch engine like Spark to take advantage of more efficient "mass" scoring. Simon On Fri, 16 Nov 2018 at 15:18, deepak kumar wrote: > Hi All > Right now MAAS supports running the model against real time events being > streamed into metron platform. > Is there any way to run the models deployed in MAAS on the batch events / > data that have been indexed into hdfs ? > If anyone have tried this batch model , please share some insights. > Thanks > Deepak. > > -- -- simon elliston ball @sireb
Re: [DISCUSS] Attribution and merging the Elasticsearch client migration
Maybe if we can’t document a concrete solution, we can document or codify a procedure. Should for example the owner of the feature branch always work with the release manager when there are issues? On November 16, 2018 at 10:26:11, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: It's a good suggestion, and I've been thinking about how to best handle this. Honestly, the right answer might be to do git rebase on master from the PR branch rather than a merge. That might avoid this situation altogether. Of course, that also comes with all the obligatory warnings about rebasing publicly shared branches, and that anyone how has their own copy will get conflicts. But I think that risk is probably minimal. I'll put something together that covers both options along with the problems and solutions for each. Hopefully that will make future collaboration that fits this pattern easier for people. On Fri, Nov 16, 2018, 5:42 AM Otto Fowler > Maybe this is worth a confluence entry, not as a guide, but just to > document what you did. > > On November 15, 2018 at 19:07:40, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > Ok, this is finally merged! Whew! > > Here's how I polished up the history at the end. I used other feature > branch merges as a guideline around commit messaging. > > * fcd644ca 2018-11-15 | METRON-1834: Migrate Elasticsearch from > TransportClient to new Java REST API (mmiklavc via mmiklavc) (HEAD -> > master, origin/master, origin/HEAD, master-merge) [mmiklavc] > |\ > | * 8bf3b6ec 2018-11-15 | METRON-1834: Migrate Elasticsearch from > TransportClient to new Java REST API (mmiklavc via mmiklavc) closes > apache/metron#1242 (stella-es-base2) [mmiklavc] > | * e7e19fbb 2018-10-08 | METRON-1834: Migrate Elasticsearch from > TransportClient to new Java REST API (cstella via mmiklavc) [cstella] > * | 0c4c622b 2018-11-14 | METRON-1749 Update Angular to latest release in > Management UI (sardell via nickwallen) closes apache/metron#1217 [sardell] > > On Thu, Nov 15, 2018 at 4:29 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Absolutely, that's part of what I did to validate. This output below also >> exactly matches the diff I get when I run it from the raw PR branch. >> >> git diff master --stat >> Upgrading.md >> | 7 +++ >> dependencies_with_url.csv >> | 2 + >> metron-deployment/Kerberos-manual-setup.md >> | 154 >> ++--- >> >> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml >> | 9 >> >> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py >> | 2 - >> >> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py >> | 3 +- >> >> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json >> | 10 >> >> metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/MetaAlertServiceImpl.java >> | 2 +- >> metron-platform/elasticsearch-shaded/pom.xml >> | 47 >> +++- >> >> metron-platform/elasticsearch-shaded/src/main/resources/META-INF/log4j-provider.properties >> | 18 --- >> metron-platform/metron-common/README.md >> | 48 >> + >> metron-platform/metron-common/src/main/config/zookeeper/global.json >> | 1 - >> >> metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigOption.java >> | 7 +++ >> metron-platform/metron-elasticsearch/README.md >> | 45 >> +++- >> metron-platform/metron-elasticsearch/pom.xml >> | 32 >> +-- >> >> metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java >> | 245 >> >> >> metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClientFactory.java >>| 189 >> + >> >> metron-platform/metron-elasticsearch/src/ma
Re: [DISCUSS] Knox SSO feature branch review and features
Those are all valid points. I think it is ( was ) worth discussion at lease a little. WRT Knox and defaults: I have in the past used “do-nothing” implementations as default placeholders for functionality that needed extensive per customer configuration, or configuration outside the responsibility of the product. Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX configuration with “do-nothing” providers for auth etc. The users would then configure the providers ( based on the provider(s) we support ) at a later time. We could write the providers, as everyone has pointed out how extensible KNOX is ;) Would that be a valid way to simplify the issue? What would the fallout of that be? On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com) wrote: Most of the research I've done around adding Metron as a Knox service is based on how other projects do it. The documentation is not easy to follow so I learned by reading other service definition files. The assumption that we are doing things drastically different is false. I completely agree with Simon. Why would we want to be dependent on Knox's release cycle? How does that benefit us? It may reduce some operational complexity but it makes our install process more complicated because we require a certain version of Knox (who knows when that gets released). What do we do in the meantime? I would also like to point out that Metron is inherently different than other Hadoop stack services. We are a full-blown application with multiple UIs so the way we expose services through Knox may be a little different. I think this will be easier to discuss when we can all see what is actually involved. I am working on a PR that adds Metron as a Knox service and will have that out soon. That should give everyone more context. On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > You could say the same thing about Ambari, but that provides mpacks. Knox > is also designed to be extensible through Knox service stacks since they > realized they can’t support every project. The challenge is that the docs > have not made it as easy as they could for the ecosystem to plug into Knox, > which has led to some confusion around this being a recommended pattern > (which it is). > > The danger of trying to get your bits into Knox is that that ties you to > their release cycle (a problem Ambari has felt hard, hence their community > is moving away from the everything inside model towards everything is an > mpack). > > A number of implementations of Knox also use the approach Ryan is > suggesting for their own organization specific end points, so it’s not like > this is an uncommon, or anti-pattern, it’s more the way Knox is designed to > work in the future, than the legacy of it only being able to handle a > subset of Hadoop projects. > > Knox remains optional In our scenario, but we keep control over the > shipping of things like rewrite rules, which allows Metron to control its > release destiny should things like url patterns in the ui need to change > (with a new release of angular / new module / new rest endpoint etc) > instead of making a Metron release dependent on a Knox release. > > Imagine how we would have done with the Ambari side if we’d had to wait > for them to release every time we needed to change something in the > mpack... we don’t want that happening with Knox. > > Simon > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > Solr is angular for example. > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com) > > wrote: > > > > Ok, here is something I don’t understand, but would like to. > > > > Knox comes configured with build in services for a number of other apache > > products and UI’s. > > It would seem to me, that the best integration with Knox would be to do > > what these other products have done. > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > 2. Create a knox service definition and provide it or try to get it into > > knox itself > > > > This would make the knox integration with metron optional and pluggable > > wouldn’t it? > > > > Then knox with metron would just be the same as knox with anything else. > > Please help me if I am wrong, but we seem to be going our own way here. > > Why don’t we just do what these other products have done? > > Why don’t we try to get apache metron services accepted to the knox > > project? Why don’t we model our knox integration with how XYZ does it? > > Have we looked at how others integrate? Having all the code and being > > able to track stuff is kind of the point of this whole thing isn’t it? > > > > Maybe this is implied and I’m missing it, if so I apologize. > > > > I think consistency with the rest of the hadoop stack with knox helps us. > > > > > > > > On November 15, 20
Re: Running MAAS in batch
Thanks Simon and it makes perfect sense. On Fri, 16 Nov 2018 at 8:58 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > MaaS is designed to wrap model inference (scoring) an event at a time, via > a REST api. As such, running it batch doesn't make a lot of sense, since > each message would be processed individually. Most of the models you're > likely to run in MaaS however, are also likely to be easily batchable, and > are probable better wrapped up in a batch engine like Spark to take > advantage of more efficient "mass" scoring. > > Simon > > On Fri, 16 Nov 2018 at 15:18, deepak kumar wrote: > > > Hi All > > Right now MAAS supports running the model against real time events being > > streamed into metron platform. > > Is there any way to run the models deployed in MAAS on the batch events / > > data that have been indexed into hdfs ? > > If anyone have tried this batch model , please share some insights. > > Thanks > > Deepak. > > > > > > -- > -- > simon elliston ball > @sireb >
Re: Running MAAS in batch
MaaS is designed to wrap model inference (scoring) an event at a time, via a REST api. As such, running it batch doesn't make a lot of sense, since each message would be processed individually. Most of the models you're likely to run in MaaS however, are also likely to be easily batchable, and are probable better wrapped up in a batch engine like Spark to take advantage of more efficient "mass" scoring. Simon On Fri, 16 Nov 2018 at 15:18, deepak kumar wrote: > Hi All > Right now MAAS supports running the model against real time events being > streamed into metron platform. > Is there any way to run the models deployed in MAAS on the batch events / > data that have been indexed into hdfs ? > If anyone have tried this batch model , please share some insights. > Thanks > Deepak. > > -- -- simon elliston ball @sireb
Re: [DISCUSS] Attribution and merging the Elasticsearch client migration
It's a good suggestion, and I've been thinking about how to best handle this. Honestly, the right answer might be to do git rebase on master from the PR branch rather than a merge. That might avoid this situation altogether. Of course, that also comes with all the obligatory warnings about rebasing publicly shared branches, and that anyone how has their own copy will get conflicts. But I think that risk is probably minimal. I'll put something together that covers both options along with the problems and solutions for each. Hopefully that will make future collaboration that fits this pattern easier for people. On Fri, Nov 16, 2018, 5:42 AM Otto Fowler > Maybe this is worth a confluence entry, not as a guide, but just to > document what you did. > > On November 15, 2018 at 19:07:40, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > Ok, this is finally merged! Whew! > > Here's how I polished up the history at the end. I used other feature > branch merges as a guideline around commit messaging. > > * fcd644ca 2018-11-15 | METRON-1834: Migrate Elasticsearch from > TransportClient to new Java REST API (mmiklavc via mmiklavc) (HEAD -> > master, origin/master, origin/HEAD, master-merge) [mmiklavc] > |\ > | * 8bf3b6ec 2018-11-15 | METRON-1834: Migrate Elasticsearch from > TransportClient to new Java REST API (mmiklavc via mmiklavc) closes > apache/metron#1242 (stella-es-base2) [mmiklavc] > | * e7e19fbb 2018-10-08 | METRON-1834: Migrate Elasticsearch from > TransportClient to new Java REST API (cstella via mmiklavc) [cstella] > * | 0c4c622b 2018-11-14 | METRON-1749 Update Angular to latest release in > Management UI (sardell via nickwallen) closes apache/metron#1217 [sardell] > > On Thu, Nov 15, 2018 at 4:29 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Absolutely, that's part of what I did to validate. This output below also >> exactly matches the diff I get when I run it from the raw PR branch. >> >> git diff master --stat >> Upgrading.md >> | 7 +++ >> dependencies_with_url.csv >> | 2 + >> metron-deployment/Kerberos-manual-setup.md >> | 154 >> ++--- >> >> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml >> | 9 >> >> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py >> | 2 - >> >> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py >> | 3 +- >> >> metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json >> | 10 >> >> metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/MetaAlertServiceImpl.java >> | 2 +- >> metron-platform/elasticsearch-shaded/pom.xml >> | 47 >> +++- >> >> metron-platform/elasticsearch-shaded/src/main/resources/META-INF/log4j-provider.properties >> | 18 --- >> metron-platform/metron-common/README.md >> | 48 >> + >> metron-platform/metron-common/src/main/config/zookeeper/global.json >> | 1 - >> >> metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigOption.java >> | 7 +++ >> metron-platform/metron-elasticsearch/README.md >> | 45 >> +++- >> metron-platform/metron-elasticsearch/pom.xml >> | 32 >> +-- >> >> metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java >> | 245 >> >> >> metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClientFactory.java >>| 189 >> + >> >> metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/config/ElasticsearchClientConfig.java >> | 187 >> >> >> metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/config/ElasticsearchClientOptio
Running MAAS in batch
Hi All Right now MAAS supports running the model against real time events being streamed into metron platform. Is there any way to run the models deployed in MAAS on the batch events / data that have been indexed into hdfs ? If anyone have tried this batch model , please share some insights. Thanks Deepak.
Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?
Can you generate the jiras that would be included in the release? On November 16, 2018 at 10:05:50, Justin Leet (justinjl...@gmail.com) wrote: Given that we've had a couple major PRs (the ES client migration along with the Angular upgrade stuff, and I'm sure others), I'd be in favor of releasing both the plugin and the main repo. I'd be in favor of doing something like: metron-bro-plugin-kafka release 0.3.0 PR to update full dev metron release 0.7.0 (given that things changed a good amount) I would also expect this process to most likely be kicked off post Thanksgiving break (Thursday 22nd and Friday 23rd for anybody not in the US). I can kick out another summarization thread if we want, but basically: * Nov 26 - Start the Bro plugin release process * Once that finishes, PR to update full dev with new plugin version * Once PR is in, start metron release process (hopefully) sometime the week of the 3rd? Are there any objections to staggering the releases like that? They could also be done together, but it means that we have to update full dev to match the plugin version post release. On Wed, Nov 14, 2018 at 10:29 AM zeo...@gmail.com wrote: > In my opinion metron-bro-plugin-kafka is ready for a release. Anything > else people would want to see? Once it gets released, I would like to > update full dev to use the newest version prior to any future metron > release (0.6.1 or whatever we choose). > > Jon > > On Wed, Nov 7, 2018 at 8:07 PM zeo...@gmail.com wrote: > > > So, about this release, anybody have time to review > > apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13? > > > > Jon > > > > On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > >> And I do think we will be ready to roll another Metron release in the > near > >> future as well. > >> > >> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet > >> wrote: > >> > >> > I tend to agree with Mike. I think we could do a release of the main > >> > project and get benefit, but I think skipping this cycle (especially > >> since > >> > we had a release last month) let's us get things like the ES client > >> > migration in and settled and just generally make recommending a newer > >> > version easier. > >> > > >> > The main reason for doing a release for the Bro plugin is to address > >> > shortcomings that we discovered and are addressing. > >> > > >> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic < > >> > michael.miklav...@gmail.com> wrote: > >> > > >> > > Migrating the ES client, refactoring parser bolt. There are some > >> > interface > >> > > changes in flight right now that I think would be beneficial to see > in > >> > the > >> > > next release. > >> > > > >> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen > >> wrote: > >> > > > >> > > > I am in favor of a release for both. > >> > > > > >> > > > There are a lot of really useful bug fixes, management of pcap > >> through > >> > > > Ambari, more flexibility for configuring JAAS in Ambari, increased > >> > > > Elasticsearch performance, the Syslog parser, and the Batch > >> Profiler, > >> > > among > >> > > > others. I would be happy with calling it a 0.6.1 point release. > >> > > > > >> > > > Mike - What is outstanding that you would like to see in the > >> release? > >> > > > > >> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic < > >> > > > michael.miklav...@gmail.com> wrote: > >> > > > > >> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin > release. > >> It > >> > > > seems > >> > > > > like it's ready to go, and I think there are a few more things > I'd > >> > like > >> > > > to > >> > > > > see get into our next Metron release so I'm good with holding > off > >> > > there. > >> > > > > > >> > > > > Mike > >> > > > > > >> > > > > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet < > >> justinjl...@gmail.com> > >> > > > > wrote: > >> > > > > > >> > > > > > Hi all, > >> > > > > > > >> > > > > > As you might recall from a prior discussion about release > >> cadence, > >> > we > >> > > > > were > >> > > > > > interested in initiating release threads near our board > reports > >> to > >> > > see > >> > > > if > >> > > > > > we wanted to do releases or not. Additionally, the work is > done > >> to > >> > do > >> > > > two > >> > > > > > separate releases, so our options are releasing both, a single > >> one, > >> > > or > >> > > > > > neither. > >> > > > > > > >> > > > > > Having said that, a metron-bro-kafka-plugin 0.3.0 release came > >> up > >> > on > >> > > > this > >> > > > > > thread > >> > > > > > < > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E > >> > > > > > >. > >> > > > > > In particular, the prospect of a release came up in the > context > >> of > >> > > > > having a > >> > > > > > version with better (and working) testing. > >> > > > > > > >> > > > > > Version Number >
Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?
Given that we've had a couple major PRs (the ES client migration along with the Angular upgrade stuff, and I'm sure others), I'd be in favor of releasing both the plugin and the main repo. I'd be in favor of doing something like: metron-bro-plugin-kafka release 0.3.0 PR to update full dev metron release 0.7.0 (given that things changed a good amount) I would also expect this process to most likely be kicked off post Thanksgiving break (Thursday 22nd and Friday 23rd for anybody not in the US). I can kick out another summarization thread if we want, but basically: * Nov 26 - Start the Bro plugin release process * Once that finishes, PR to update full dev with new plugin version * Once PR is in, start metron release process (hopefully) sometime the week of the 3rd? Are there any objections to staggering the releases like that? They could also be done together, but it means that we have to update full dev to match the plugin version post release. On Wed, Nov 14, 2018 at 10:29 AM zeo...@gmail.com wrote: > In my opinion metron-bro-plugin-kafka is ready for a release. Anything > else people would want to see? Once it gets released, I would like to > update full dev to use the newest version prior to any future metron > release (0.6.1 or whatever we choose). > > Jon > > On Wed, Nov 7, 2018 at 8:07 PM zeo...@gmail.com wrote: > > > So, about this release, anybody have time to review > > apache/metron-bro-plugin-kafka#2 and apache/metron-bro-plugin-kafka#13? > > > > Jon > > > > On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > >> And I do think we will be ready to roll another Metron release in the > near > >> future as well. > >> > >> On Wed, Oct 17, 2018 at 8:26 AM Justin Leet > >> wrote: > >> > >> > I tend to agree with Mike. I think we could do a release of the main > >> > project and get benefit, but I think skipping this cycle (especially > >> since > >> > we had a release last month) let's us get things like the ES client > >> > migration in and settled and just generally make recommending a newer > >> > version easier. > >> > > >> > The main reason for doing a release for the Bro plugin is to address > >> > shortcomings that we discovered and are addressing. > >> > > >> > On Tue, Oct 16, 2018 at 4:42 PM Michael Miklavcic < > >> > michael.miklav...@gmail.com> wrote: > >> > > >> > > Migrating the ES client, refactoring parser bolt. There are some > >> > interface > >> > > changes in flight right now that I think would be beneficial to see > in > >> > the > >> > > next release. > >> > > > >> > > On Tue, Oct 16, 2018 at 2:01 PM Nick Allen > >> wrote: > >> > > > >> > > > I am in favor of a release for both. > >> > > > > >> > > > There are a lot of really useful bug fixes, management of pcap > >> through > >> > > > Ambari, more flexibility for configuring JAAS in Ambari, increased > >> > > > Elasticsearch performance, the Syslog parser, and the Batch > >> Profiler, > >> > > among > >> > > > others. I would be happy with calling it a 0.6.1 point release. > >> > > > > >> > > > Mike - What is outstanding that you would like to see in the > >> release? > >> > > > > >> > > > On Tue, Oct 16, 2018, 12:21 PM Michael Miklavcic < > >> > > > michael.miklav...@gmail.com> wrote: > >> > > > > >> > > > > I'd be +1 on going with just the metron-bro-kafka-plugin > release. > >> It > >> > > > seems > >> > > > > like it's ready to go, and I think there are a few more things > I'd > >> > like > >> > > > to > >> > > > > see get into our next Metron release so I'm good with holding > off > >> > > there. > >> > > > > > >> > > > > Mike > >> > > > > > >> > > > > On Tue, Oct 16, 2018 at 10:26 AM Justin Leet < > >> justinjl...@gmail.com> > >> > > > > wrote: > >> > > > > > >> > > > > > Hi all, > >> > > > > > > >> > > > > > As you might recall from a prior discussion about release > >> cadence, > >> > we > >> > > > > were > >> > > > > > interested in initiating release threads near our board > reports > >> to > >> > > see > >> > > > if > >> > > > > > we wanted to do releases or not. Additionally, the work is > done > >> to > >> > do > >> > > > two > >> > > > > > separate releases, so our options are releasing both, a single > >> one, > >> > > or > >> > > > > > neither. > >> > > > > > > >> > > > > > Having said that, a metron-bro-kafka-plugin 0.3.0 release came > >> up > >> > on > >> > > > this > >> > > > > > thread > >> > > > > > < > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://lists.apache.org/thread.html/3c18c3aba6b436b11032831e7db541d50eb7cb1e3ae54b7423057c88@%3Cdev.metron.apache.org%3E > >> > > > > > >. > >> > > > > > In particular, the prospect of a release came up in the > context > >> of > >> > > > > having a > >> > > > > > version with better (and working) testing. > >> > > > > > > >> > > > > > Version Number > >> > > > > > If we choose to do a core Metron release, I propose 0.6.1. For > >> > > > > > metron-bro-kafka-plugin, I propose 0.3.0. Keep in m
Re: [DISCUSS] Knox SSO feature branch review and features
Most of the research I've done around adding Metron as a Knox service is based on how other projects do it. The documentation is not easy to follow so I learned by reading other service definition files. The assumption that we are doing things drastically different is false. I completely agree with Simon. Why would we want to be dependent on Knox's release cycle? How does that benefit us? It may reduce some operational complexity but it makes our install process more complicated because we require a certain version of Knox (who knows when that gets released). What do we do in the meantime? I would also like to point out that Metron is inherently different than other Hadoop stack services. We are a full-blown application with multiple UIs so the way we expose services through Knox may be a little different. I think this will be easier to discuss when we can all see what is actually involved. I am working on a PR that adds Metron as a Knox service and will have that out soon. That should give everyone more context. On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > You could say the same thing about Ambari, but that provides mpacks. Knox > is also designed to be extensible through Knox service stacks since they > realized they can’t support every project. The challenge is that the docs > have not made it as easy as they could for the ecosystem to plug into Knox, > which has led to some confusion around this being a recommended pattern > (which it is). > > The danger of trying to get your bits into Knox is that that ties you to > their release cycle (a problem Ambari has felt hard, hence their community > is moving away from the everything inside model towards everything is an > mpack). > > A number of implementations of Knox also use the approach Ryan is > suggesting for their own organization specific end points, so it’s not like > this is an uncommon, or anti-pattern, it’s more the way Knox is designed to > work in the future, than the legacy of it only being able to handle a > subset of Hadoop projects. > > Knox remains optional In our scenario, but we keep control over the > shipping of things like rewrite rules, which allows Metron to control its > release destiny should things like url patterns in the ui need to change > (with a new release of angular / new module / new rest endpoint etc) > instead of making a Metron release dependent on a Knox release. > > Imagine how we would have done with the Ambari side if we’d had to wait > for them to release every time we needed to change something in the > mpack... we don’t want that happening with Knox. > > Simon > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > Solr is angular for example. > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com) > > wrote: > > > > Ok, here is something I don’t understand, but would like to. > > > > Knox comes configured with build in services for a number of other apache > > products and UI’s. > > It would seem to me, that the best integration with Knox would be to do > > what these other products have done. > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > 2. Create a knox service definition and provide it or try to get it into > > knox itself > > > > This would make the knox integration with metron optional and pluggable > > wouldn’t it? > > > > Then knox with metron would just be the same as knox with anything else. > > Please help me if I am wrong, but we seem to be going our own way here. > > Why don’t we just do what these other products have done? > > Why don’t we try to get apache metron services accepted to the knox > > project? Why don’t we model our knox integration with how XYZ does it? > > Have we looked at how others integrate? Having all the code and being > > able to track stuff is kind of the point of this whole thing isn’t it? > > > > Maybe this is implied and I’m missing it, if so I apologize. > > > > I think consistency with the rest of the hadoop stack with knox helps us. > > > > > > > > On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) > wrote: > > > > 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI > > as far as I know. I put up a PR with a proposed solution here: > > https://github.com/apache/metron/pull/1266. > > 2) Yes Knox is a service you can install with Ambari, similar to Ranger > or > > Spark. There are some things that are specifically configured in Knox and > > there are some things specific to Metron. I will put up a PR with the > > changes needed so you can see exactly what is involved. > > 3) I don't understand what you mean here. Is this a question? > > 4) I think it's a little early to predict the Ambari changes required. > > This will depend on how tasks 1-3 go. I imagine it's similar to other > > mpack work: e
Re: [DISCUSS] Knox SSO feature branch review and features
You could say the same thing about Ambari, but that provides mpacks. Knox is also designed to be extensible through Knox service stacks since they realized they can’t support every project. The challenge is that the docs have not made it as easy as they could for the ecosystem to plug into Knox, which has led to some confusion around this being a recommended pattern (which it is). The danger of trying to get your bits into Knox is that that ties you to their release cycle (a problem Ambari has felt hard, hence their community is moving away from the everything inside model towards everything is an mpack). A number of implementations of Knox also use the approach Ryan is suggesting for their own organization specific end points, so it’s not like this is an uncommon, or anti-pattern, it’s more the way Knox is designed to work in the future, than the legacy of it only being able to handle a subset of Hadoop projects. Knox remains optional In our scenario, but we keep control over the shipping of things like rewrite rules, which allows Metron to control its release destiny should things like url patterns in the ui need to change (with a new release of angular / new module / new rest endpoint etc) instead of making a Metron release dependent on a Knox release. Imagine how we would have done with the Ambari side if we’d had to wait for them to release every time we needed to change something in the mpack... we don’t want that happening with Knox. Simon > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > Solr is angular for example. > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > Ok, here is something I don’t understand, but would like to. > > Knox comes configured with build in services for a number of other apache > products and UI’s. > It would seem to me, that the best integration with Knox would be to do > what these other products have done. > > > 1. Do whatever you have to do to make your own stuff compatible. > 2. Create a knox service definition and provide it or try to get it into > knox itself > > This would make the knox integration with metron optional and pluggable > wouldn’t it? > > Then knox with metron would just be the same as knox with anything else. > Please help me if I am wrong, but we seem to be going our own way here. > Why don’t we just do what these other products have done? > Why don’t we try to get apache metron services accepted to the knox > project? Why don’t we model our knox integration with how XYZ does it? > Have we looked at how others integrate? Having all the code and being > able to track stuff is kind of the point of this whole thing isn’t it? > > Maybe this is implied and I’m missing it, if so I apologize. > > I think consistency with the rest of the hadoop stack with knox helps us. > > > > On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) wrote: > > 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI > as far as I know. I put up a PR with a proposed solution here: > https://github.com/apache/metron/pull/1266. > 2) Yes Knox is a service you can install with Ambari, similar to Ranger or > Spark. There are some things that are specifically configured in Knox and > there are some things specific to Metron. I will put up a PR with the > changes needed so you can see exactly what is involved. > 3) I don't understand what you mean here. Is this a question? > 4) I think it's a little early to predict the Ambari changes required. > This will depend on how tasks 1-3 go. I imagine it's similar to other > mpack work: expose some parameters in ambari and bind those to config > files. My understanding from this thread so far is that we should focus on > a manual, documented approach to start. > > On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Thanks Ryan, >> >> 1) Can you clarify "not a good way to do this?" Are you saying we don't >> have a way to set this and need to add the config option, or that a >> solution is not obvious and it's unclear what to do? It seems to me you're >> saying the former, but I'd like to be sure. >> 2) Is Knox not a service made available by Ambari similar to Ranger or >> Spark? I'm assuming that similar to Kerberos, there are some things that >> are specifically configured in Knox and others that are app-specific. Some >> explanation of what this looks like would be helpful. >> 3) Sounds like this follows pretty naturally from #1 >> 4) Relates to #2. I think we need some guidance on what a manual vs >> MPack/automated install would look like. >> >> Cheers, >> Mike >> >> >>> On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman wrote: >>> >>> Wanted to give an update on the context path issue. I investigated >>> rewriting url links in the outgoing static assets with Knox and it was
Re: [DISCUSS] Knox SSO feature branch review and features
https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support Solr is angular for example. On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com) wrote: Ok, here is something I don’t understand, but would like to. Knox comes configured with build in services for a number of other apache products and UI’s. It would seem to me, that the best integration with Knox would be to do what these other products have done. 1. Do whatever you have to do to make your own stuff compatible. 2. Create a knox service definition and provide it or try to get it into knox itself This would make the knox integration with metron optional and pluggable wouldn’t it? Then knox with metron would just be the same as knox with anything else. Please help me if I am wrong, but we seem to be going our own way here. Why don’t we just do what these other products have done? Why don’t we try to get apache metron services accepted to the knox project? Why don’t we model our knox integration with how XYZ does it? Have we looked at how others integrate? Having all the code and being able to track stuff is kind of the point of this whole thing isn’t it? Maybe this is implied and I’m missing it, if so I apologize. I think consistency with the rest of the hadoop stack with knox helps us. On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) wrote: 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI as far as I know. I put up a PR with a proposed solution here: https://github.com/apache/metron/pull/1266. 2) Yes Knox is a service you can install with Ambari, similar to Ranger or Spark. There are some things that are specifically configured in Knox and there are some things specific to Metron. I will put up a PR with the changes needed so you can see exactly what is involved. 3) I don't understand what you mean here. Is this a question? 4) I think it's a little early to predict the Ambari changes required. This will depend on how tasks 1-3 go. I imagine it's similar to other mpack work: expose some parameters in ambari and bind those to config files. My understanding from this thread so far is that we should focus on a manual, documented approach to start. On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Thanks Ryan, > > 1) Can you clarify "not a good way to do this?" Are you saying we don't > have a way to set this and need to add the config option, or that a > solution is not obvious and it's unclear what to do? It seems to me you're > saying the former, but I'd like to be sure. > 2) Is Knox not a service made available by Ambari similar to Ranger or > Spark? I'm assuming that similar to Kerberos, there are some things that > are specifically configured in Knox and others that are app-specific. Some > explanation of what this looks like would be helpful. > 3) Sounds like this follows pretty naturally from #1 > 4) Relates to #2. I think we need some guidance on what a manual vs > MPack/automated install would look like. > > Cheers, > Mike > > > On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman wrote: > > > Wanted to give an update on the context path issue. I investigated > > rewriting url links in the outgoing static assets with Knox and it was > not > > trivial. Fortunately I found a simple solution that works with or > without > > Knox. I changed the base tag in index.html from to > href="./">, or in other words made the base href relative. > > > > I believe I am at the point where I can task this out and provide a high > > level overview of the changes needed. I think that each task will be a > > manageable size and can stand alone so I don't think we need a feature > > branch. > > > > The first task involves a general change to the UI code. We need a way > to > > set the path to the REST service with a configuration setting because it > is > > different with and without Knox. Currently there is not a good way to do > > this in the UI. We can use the environment files but that is a build > time > > setting and is not flexible. I can see this capability being useful for > > other use cases in the future. I think we could even split this up into > 2 > > separate tasks, one for the alerts UI and one for the management UI. > > > > The second task involves adding Knox to our stack either by default as a > > dependency in the mpack or with a documented approach. We would add our > > REST service, Alerts UI, and Management UI as services in Knox. > Everything > > would continue to function as it currently does but with all > communication > > going through Knox. LDAP authentication would be required when using > Knox > > and Knox will authenticate with the REST service by passing along an > > Authorization header. Enabling Knox would be a manual process that > > involves deploying assets (Knox descriptor files) and changing > > configuration. There would be no change to how the UI functions by > defa
Re: [DISCUSS] Knox SSO feature branch review and features
Ok, here is something I don’t understand, but would like to. Knox comes configured with build in services for a number of other apache products and UI’s. It would seem to me, that the best integration with Knox would be to do what these other products have done. 1. Do whatever you have to do to make your own stuff compatible. 2. Create a knox service definition and provide it or try to get it into knox itself This would make the knox integration with metron optional and pluggable wouldn’t it? Then knox with metron would just be the same as knox with anything else. Please help me if I am wrong, but we seem to be going our own way here. Why don’t we just do what these other products have done? Why don’t we try to get apache metron services accepted to the knox project? Why don’t we model our knox integration with how XYZ does it? Have we looked at how others integrate? Having all the code and being able to track stuff is kind of the point of this whole thing isn’t it? Maybe this is implied and I’m missing it, if so I apologize. I think consistency with the rest of the hadoop stack with knox helps us. On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) wrote: 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI as far as I know. I put up a PR with a proposed solution here: https://github.com/apache/metron/pull/1266. 2) Yes Knox is a service you can install with Ambari, similar to Ranger or Spark. There are some things that are specifically configured in Knox and there are some things specific to Metron. I will put up a PR with the changes needed so you can see exactly what is involved. 3) I don't understand what you mean here. Is this a question? 4) I think it's a little early to predict the Ambari changes required. This will depend on how tasks 1-3 go. I imagine it's similar to other mpack work: expose some parameters in ambari and bind those to config files. My understanding from this thread so far is that we should focus on a manual, documented approach to start. On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Thanks Ryan, > > 1) Can you clarify "not a good way to do this?" Are you saying we don't > have a way to set this and need to add the config option, or that a > solution is not obvious and it's unclear what to do? It seems to me you're > saying the former, but I'd like to be sure. > 2) Is Knox not a service made available by Ambari similar to Ranger or > Spark? I'm assuming that similar to Kerberos, there are some things that > are specifically configured in Knox and others that are app-specific. Some > explanation of what this looks like would be helpful. > 3) Sounds like this follows pretty naturally from #1 > 4) Relates to #2. I think we need some guidance on what a manual vs > MPack/automated install would look like. > > Cheers, > Mike > > > On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman wrote: > > > Wanted to give an update on the context path issue. I investigated > > rewriting url links in the outgoing static assets with Knox and it was > not > > trivial. Fortunately I found a simple solution that works with or > without > > Knox. I changed the base tag in index.html from to > href="./">, or in other words made the base href relative. > > > > I believe I am at the point where I can task this out and provide a high > > level overview of the changes needed. I think that each task will be a > > manageable size and can stand alone so I don't think we need a feature > > branch. > > > > The first task involves a general change to the UI code. We need a way > to > > set the path to the REST service with a configuration setting because it > is > > different with and without Knox. Currently there is not a good way to do > > this in the UI. We can use the environment files but that is a build > time > > setting and is not flexible. I can see this capability being useful for > > other use cases in the future. I think we could even split this up into > 2 > > separate tasks, one for the alerts UI and one for the management UI. > > > > The second task involves adding Knox to our stack either by default as a > > dependency in the mpack or with a documented approach. We would add our > > REST service, Alerts UI, and Management UI as services in Knox. > Everything > > would continue to function as it currently does but with all > communication > > going through Knox. LDAP authentication would be required when using > Knox > > and Knox will authenticate with the REST service by passing along an > > Authorization header. Enabling Knox would be a manual process that > > involves deploying assets (Knox descriptor files) and changing > > configuration. There would be no change to how the UI functions by > default > > (without Knox) and either LDAP or JDBC authentication could still be > used.. > > > > The third task involves enabling SSO with Knox. We would update the REST > > service so that it can authenticate with
Re: [DISCUSS] Knox SSO feature branch review and features
Welcome and thanks! On November 12, 2018 at 10:12:44, Sandeep Moré (moresand...@gmail.com) wrote: Hello Ryan, I am still catching up on the architecture so let me know if I am misunderstanding anything. You could have multiple serviced deployed in Knox 1. for metron (metron/api/v1) 2. for alerts-ui (metron-alerts-ui/alerts-list) and have them run in one Knox instance and you could have one service reference from other (not recommended but possible). You can tailor rewrite rules to update the context path for the assets as well, pointed out by Simon. Knox Wiki [1] has some blogs and tutorials that you can look at. This [2] is a good tutorial on how rewriting static assets, there is also a blog [3] on basics of rewrite rules that should be a good reference. I would also be glad to look at the service definitions you have and answer any questions. [1] https://cwiki.apache.org/confluence/display/KNOX/Index [2] https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox [3] https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox Best, Sandeep P.S. I am a Knox committer and new to Metron. On Mon, Nov 12, 2018 at 9:59 AM Ryan Merriman wrote: > I'm just coming up to speed on Knox so maybe rewriting assets links are > trivial. If anyone has a good example of how to do that or can point to > some documentation, please share. > > On Mon, Nov 12, 2018 at 8:54 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > Doing the Knox proxy work first certainly does make a lot of sense vs the > > SSO first approach, so I'm in favour of this. It bypasses all the > anti-CORS > > proxying stuff the other solution needed by being on the same URL space. > > > > Is there are reason we're not re-writing the asset link URLs in Knox? We > > should have a reverse content rewrite rule to avoid that problem and make > > it entirely transparent whether there is Knox or not. We shouldn't be > > changing anything about the UI services themselves. If the rewrite > service > > is complete, there is no change to base ref in the UI code, Knox would > > effectively apply it by content filtering. Note also that the gateway URL > > is configurable and likely to vary from Knox to Knox, so baking it into > the > > ng build will break non-full-dev builds. (e.g. gateway/default could well > > be gateway/xyz). > > > > I would also like to discuss removing the JDBC auth, because it's a set > of > > plaintext passwords in a mysql DB... it introduces a problematic > dependency > > (mysql) a ton of java dependencies we could cut out (JPA, eclipselink) > and > > opens up a massive security hole. I personally know of several > > organisations who are blocked from using Metron by the presence of the > JDBC > > authentication method in its current form. > > > > Simon > > > > On Mon, 12 Nov 2018 at 14:36, Ryan Merriman wrote: > > > > > Let me clarify on exposing both legacy and Knox URLs at the same time. > > The > > > base urls will look something like this: > > > > > > Legacy REST - http://node1:8082/api/v1 > > > Legacy Alerts UI - http://node1:4201:/alerts-list > > > > > > Knox REST - https://node1:8443/gateway/default/metron/api/v1 > > > Knox Alerts UI - > > > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > > > > > If Knox were turned on and the alerts UI deployed as is, it would not > > > work. This is because static assets are referenced with > > > http://node1:4201/assets/some-asset.js which does not include the > > correct > > > context path to the alerts UI in knox. To make it work, you have to > set > > > the base ref to "/gateway/default/metron-alerts-ui" so that static > assets > > > are referenced at > > > > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js > > . > > > When you do that, the legacy alerts UI will no longer work. I guess > the > > > point I'm trying to make is that we would have to switch between them > or > > > have 2 separate application running. I imagine most users only need > one > > or > > > the other running so probably not an issue. > > > > > > Jon, the primary upgrade consideration I see is with authentication. > To > > be > > > able to use Knox, you would have to upgrade to LDAP-based > authentication > > if > > > you were still using JDBC-based authentication in REST. The urls would > > > also change obviously. > > > > > > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com > > wrote: > > > > > > > Phew, that was quite the thread to catch up on. > > > > > > > > I agree that this should be optional/pluggable to start, and I'm > > > interested > > > > to hear the issues as they relate to upgrading an existing cluster > > (given > > > > the suggested approach) and exposing both legacy and knox URLs at the > > > same > > > > time. > > > > > > > > Jon > > > > > > > > On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < > > > > michael.miklav...@gmail.com> > > > > wrote: > > > > > > > > > A couple more things, and I think this goes
Re: [DISCUSS] Attribution and merging the Elasticsearch client migration
Maybe this is worth a confluence entry, not as a guide, but just to document what you did. On November 15, 2018 at 19:07:40, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: Ok, this is finally merged! Whew! Here's how I polished up the history at the end. I used other feature branch merges as a guideline around commit messaging. * fcd644ca 2018-11-15 | METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) (HEAD -> master, origin/master, origin/HEAD, master-merge) [mmiklavc] |\ | * 8bf3b6ec 2018-11-15 | METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242 (stella-es-base2) [mmiklavc] | * e7e19fbb 2018-10-08 | METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (cstella via mmiklavc) [cstella] * | 0c4c622b 2018-11-14 | METRON-1749 Update Angular to latest release in Management UI (sardell via nickwallen) closes apache/metron#1217 [sardell] On Thu, Nov 15, 2018 at 4:29 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Absolutely, that's part of what I did to validate. This output below also > exactly matches the diff I get when I run it from the raw PR branch. > > git diff master --stat > Upgrading.md > | 7 +++ > dependencies_with_url.csv > | 2 + > metron-deployment/Kerberos-manual-setup.md > | 154 > ++--- > > metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml > | 9 > > metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py > | 2 - > > metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py > | 3 +- > > metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json > | 10 > > metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/MetaAlertServiceImpl.java > | 2 +- > metron-platform/elasticsearch-shaded/pom.xml > | 47 > +++- > > metron-platform/elasticsearch-shaded/src/main/resources/META-INF/log4j-provider.properties > | 18 --- > metron-platform/metron-common/README.md > | 48 > + > metron-platform/metron-common/src/main/config/zookeeper/global.json > | 1 - > > metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/ConfigOption.java > | 7 +++ > metron-platform/metron-elasticsearch/README.md > | 45 > +++- > metron-platform/metron-elasticsearch/pom.xml > | 32 > +-- > > metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java > | 245 > > > metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClientFactory.java >| 189 > + > > metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/config/ElasticsearchClientConfig.java > | 187 > > > metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/config/ElasticsearchClientOptions.java >| 60 + > > metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchColumnMetadataDao.java > | 101 ++- > > metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java > | 21 > > metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDao.java >| 2 +- > > metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertSearchDao.java > | 6 ++- > > metron-platform/metron-elasticsearch/src/main/java/org/apache/
