Re: [DISCUSS] Knox SSO feature branch review and features
I just put up a PR that adds Metron as a Knox service here: https://github.com/apache/metron/pull/1275. This should give everyone a good idea of what is involved. I added a section on outstanding items that highlights some of the things we have been discussion here. On Fri, Nov 16, 2018 at 10:54 AM Ryan Merriman wrote: > I would also add that defaulting to Knox being on simplifies things at a > technical level. > > On Fri, Nov 16, 2018 at 10:52 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> That's fantastic, thanks for that detail. >> >> Also, I'm in agreement with the recent comments from Otto and Simon. >> >> On Fri, Nov 16, 2018 at 9:49 AM Ryan Merriman >> wrote: >> >> > I was still able to spin up the UI locally and debug in my testing. I >> am >> > in complete agreement, we need to ensure the developer experience >> doesn't >> > change. >> > >> > On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic < >> > michael.miklav...@gmail.com> wrote: >> > >> > > Ryan, what's remote debugging look like for UI testing with Knox >> enabled? >> > > Anything we lose from a dev testability standpoint? The discussion of >> > > defaults sounds reasonable to me, and I'd like to understand any other >> > > tradeoffs there may be for non-prod deployments like full dev. >> > > >> > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman >> > wrote: >> > > >> > > > Most of the research I've done around adding Metron as a Knox >> service >> > is >> > > > based on how other projects do it. The documentation is not easy to >> > > follow >> > > > so I learned by reading other service definition files. The >> assumption >> > > > that we are doing things drastically different is false. >> > > > >> > > > I completely agree with Simon. Why would we want to be dependent on >> > > Knox's >> > > > release cycle? How does that benefit us? It may reduce some >> > operational >> > > > complexity but it makes our install process more complicated >> because we >> > > > require a certain version of Knox (who knows when that gets >> released). >> > > > What do we do in the meantime? I would also like to point out that >> > > Metron >> > > > is inherently different than other Hadoop stack services. We are a >> > > > full-blown application with multiple UIs so the way we expose >> services >> > > > through Knox may be a little different. >> > > > >> > > > I think this will be easier to discuss when we can all see what is >> > > actually >> > > > involved. I am working on a PR that adds Metron as a Knox service >> and >> > > will >> > > > have that out soon. That should give everyone more context. >> > > > >> > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < >> > > > si...@simonellistonball.com> wrote: >> > > > >> > > > > You could say the same thing about Ambari, but that provides >> mpacks. >> > > Knox >> > > > > is also designed to be extensible through Knox service stacks >> since >> > > they >> > > > > realized they can’t support every project. The challenge is that >> the >> > > docs >> > > > > have not made it as easy as they could for the ecosystem to plug >> into >> > > > Knox, >> > > > > which has led to some confusion around this being a recommended >> > pattern >> > > > > (which it is). >> > > > > >> > > > > The danger of trying to get your bits into Knox is that that ties >> you >> > > to >> > > > > their release cycle (a problem Ambari has felt hard, hence their >> > > > community >> > > > > is moving away from the everything inside model towards >> everything is >> > > an >> > > > > mpack). >> > > > > >> > > > > A number of implementations of Knox also use the approach Ryan is >> > > > > suggesting for their own organization specific end points, so it’s >> > not >> > > > like >> > > > > this is an uncommon, or anti-pattern, it’s more the way Knox is >> > > designed >> > > > to >> > > > > work in the future, than the legacy of it only being able to >> handle a >> > > > > subset of Hadoop projects. >> > > > > >> > > > > Knox remains optional In our scenario, but we keep control over >> the >> > > > > shipping of things like rewrite rules, which allows Metron to >> control >> > > its >> > > > > release destiny should things like url patterns in the ui need to >> > > change >> > > > > (with a new release of angular / new module / new rest endpoint >> etc) >> > > > > instead of making a Metron release dependent on a Knox release. >> > > > > >> > > > > Imagine how we would have done with the Ambari side if we’d had to >> > wait >> > > > > for them to release every time we needed to change something in >> the >> > > > > mpack... we don’t want that happening with Knox. >> > > > > >> > > > > Simon >> > > > > >> > > > > > On 16 Nov 2018, at 13:22, Otto Fowler >> > > wrote: >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support >> > > > > > >> > > > > > Solr is angular for example. >> > > > > > >> >
Re: [DISCUSS] Knox SSO feature branch review and features
I would also add that defaulting to Knox being on simplifies things at a technical level. On Fri, Nov 16, 2018 at 10:52 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > That's fantastic, thanks for that detail. > > Also, I'm in agreement with the recent comments from Otto and Simon. > > On Fri, Nov 16, 2018 at 9:49 AM Ryan Merriman wrote: > > > I was still able to spin up the UI locally and debug in my testing. I am > > in complete agreement, we need to ensure the developer experience doesn't > > change. > > > > On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > Ryan, what's remote debugging look like for UI testing with Knox > enabled? > > > Anything we lose from a dev testability standpoint? The discussion of > > > defaults sounds reasonable to me, and I'd like to understand any other > > > tradeoffs there may be for non-prod deployments like full dev. > > > > > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman > > wrote: > > > > > > > Most of the research I've done around adding Metron as a Knox service > > is > > > > based on how other projects do it. The documentation is not easy to > > > follow > > > > so I learned by reading other service definition files. The > assumption > > > > that we are doing things drastically different is false. > > > > > > > > I completely agree with Simon. Why would we want to be dependent on > > > Knox's > > > > release cycle? How does that benefit us? It may reduce some > > operational > > > > complexity but it makes our install process more complicated because > we > > > > require a certain version of Knox (who knows when that gets > released). > > > > What do we do in the meantime? I would also like to point out that > > > Metron > > > > is inherently different than other Hadoop stack services. We are a > > > > full-blown application with multiple UIs so the way we expose > services > > > > through Knox may be a little different. > > > > > > > > I think this will be easier to discuss when we can all see what is > > > actually > > > > involved. I am working on a PR that adds Metron as a Knox service > and > > > will > > > > have that out soon. That should give everyone more context. > > > > > > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > > > > si...@simonellistonball.com> wrote: > > > > > > > > > You could say the same thing about Ambari, but that provides > mpacks. > > > Knox > > > > > is also designed to be extensible through Knox service stacks since > > > they > > > > > realized they can’t support every project. The challenge is that > the > > > docs > > > > > have not made it as easy as they could for the ecosystem to plug > into > > > > Knox, > > > > > which has led to some confusion around this being a recommended > > pattern > > > > > (which it is). > > > > > > > > > > The danger of trying to get your bits into Knox is that that ties > you > > > to > > > > > their release cycle (a problem Ambari has felt hard, hence their > > > > community > > > > > is moving away from the everything inside model towards everything > is > > > an > > > > > mpack). > > > > > > > > > > A number of implementations of Knox also use the approach Ryan is > > > > > suggesting for their own organization specific end points, so it’s > > not > > > > like > > > > > this is an uncommon, or anti-pattern, it’s more the way Knox is > > > designed > > > > to > > > > > work in the future, than the legacy of it only being able to > handle a > > > > > subset of Hadoop projects. > > > > > > > > > > Knox remains optional In our scenario, but we keep control over the > > > > > shipping of things like rewrite rules, which allows Metron to > control > > > its > > > > > release destiny should things like url patterns in the ui need to > > > change > > > > > (with a new release of angular / new module / new rest endpoint > etc) > > > > > instead of making a Metron release dependent on a Knox release. > > > > > > > > > > Imagine how we would have done with the Ambari side if we’d had to > > wait > > > > > for them to release every time we needed to change something in the > > > > > mpack... we don’t want that happening with Knox. > > > > > > > > > > Simon > > > > > > > > > > > On 16 Nov 2018, at 13:22, Otto Fowler > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > > > > > > > Solr is angular for example. > > > > > > > > > > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler ( > > > ottobackwa...@gmail.com > > > > ) > > > > > > wrote: > > > > > > > > > > > > Ok, here is something I don’t understand, but would like to. > > > > > > > > > > > > Knox comes configured with build in services for a number of > other > > > > apache > > > > > > products and UI’s. > > > > > > It would seem to me, that the best integration with Knox would be > > to > > > do > > > > > > what these other
Re: [DISCUSS] Knox SSO feature branch review and features
That's fantastic, thanks for that detail. Also, I'm in agreement with the recent comments from Otto and Simon. On Fri, Nov 16, 2018 at 9:49 AM Ryan Merriman wrote: > I was still able to spin up the UI locally and debug in my testing. I am > in complete agreement, we need to ensure the developer experience doesn't > change. > > On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Ryan, what's remote debugging look like for UI testing with Knox enabled? > > Anything we lose from a dev testability standpoint? The discussion of > > defaults sounds reasonable to me, and I'd like to understand any other > > tradeoffs there may be for non-prod deployments like full dev. > > > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman > wrote: > > > > > Most of the research I've done around adding Metron as a Knox service > is > > > based on how other projects do it. The documentation is not easy to > > follow > > > so I learned by reading other service definition files. The assumption > > > that we are doing things drastically different is false. > > > > > > I completely agree with Simon. Why would we want to be dependent on > > Knox's > > > release cycle? How does that benefit us? It may reduce some > operational > > > complexity but it makes our install process more complicated because we > > > require a certain version of Knox (who knows when that gets released). > > > What do we do in the meantime? I would also like to point out that > > Metron > > > is inherently different than other Hadoop stack services. We are a > > > full-blown application with multiple UIs so the way we expose services > > > through Knox may be a little different. > > > > > > I think this will be easier to discuss when we can all see what is > > actually > > > involved. I am working on a PR that adds Metron as a Knox service and > > will > > > have that out soon. That should give everyone more context. > > > > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > > > si...@simonellistonball.com> wrote: > > > > > > > You could say the same thing about Ambari, but that provides mpacks. > > Knox > > > > is also designed to be extensible through Knox service stacks since > > they > > > > realized they can’t support every project. The challenge is that the > > docs > > > > have not made it as easy as they could for the ecosystem to plug into > > > Knox, > > > > which has led to some confusion around this being a recommended > pattern > > > > (which it is). > > > > > > > > The danger of trying to get your bits into Knox is that that ties you > > to > > > > their release cycle (a problem Ambari has felt hard, hence their > > > community > > > > is moving away from the everything inside model towards everything is > > an > > > > mpack). > > > > > > > > A number of implementations of Knox also use the approach Ryan is > > > > suggesting for their own organization specific end points, so it’s > not > > > like > > > > this is an uncommon, or anti-pattern, it’s more the way Knox is > > designed > > > to > > > > work in the future, than the legacy of it only being able to handle a > > > > subset of Hadoop projects. > > > > > > > > Knox remains optional In our scenario, but we keep control over the > > > > shipping of things like rewrite rules, which allows Metron to control > > its > > > > release destiny should things like url patterns in the ui need to > > change > > > > (with a new release of angular / new module / new rest endpoint etc) > > > > instead of making a Metron release dependent on a Knox release. > > > > > > > > Imagine how we would have done with the Ambari side if we’d had to > wait > > > > for them to release every time we needed to change something in the > > > > mpack... we don’t want that happening with Knox. > > > > > > > > Simon > > > > > > > > > On 16 Nov 2018, at 13:22, Otto Fowler > > wrote: > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > > > > > Solr is angular for example. > > > > > > > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler ( > > ottobackwa...@gmail.com > > > ) > > > > > wrote: > > > > > > > > > > Ok, here is something I don’t understand, but would like to. > > > > > > > > > > Knox comes configured with build in services for a number of other > > > apache > > > > > products and UI’s. > > > > > It would seem to me, that the best integration with Knox would be > to > > do > > > > > what these other products have done. > > > > > > > > > > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > > > > 2. Create a knox service definition and provide it or try to get it > > > into > > > > > knox itself > > > > > > > > > > This would make the knox integration with metron optional and > > pluggable > > > > > wouldn’t it? > > > > > > > > > > Then knox with metron would just be the same as knox with anything > > > else. > > > > >
Re: [DISCUSS] Knox SSO feature branch review and features
I was still able to spin up the UI locally and debug in my testing. I am in complete agreement, we need to ensure the developer experience doesn't change. On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Ryan, what's remote debugging look like for UI testing with Knox enabled? > Anything we lose from a dev testability standpoint? The discussion of > defaults sounds reasonable to me, and I'd like to understand any other > tradeoffs there may be for non-prod deployments like full dev. > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman wrote: > > > Most of the research I've done around adding Metron as a Knox service is > > based on how other projects do it. The documentation is not easy to > follow > > so I learned by reading other service definition files. The assumption > > that we are doing things drastically different is false. > > > > I completely agree with Simon. Why would we want to be dependent on > Knox's > > release cycle? How does that benefit us? It may reduce some operational > > complexity but it makes our install process more complicated because we > > require a certain version of Knox (who knows when that gets released). > > What do we do in the meantime? I would also like to point out that > Metron > > is inherently different than other Hadoop stack services. We are a > > full-blown application with multiple UIs so the way we expose services > > through Knox may be a little different. > > > > I think this will be easier to discuss when we can all see what is > actually > > involved. I am working on a PR that adds Metron as a Knox service and > will > > have that out soon. That should give everyone more context. > > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > > si...@simonellistonball.com> wrote: > > > > > You could say the same thing about Ambari, but that provides mpacks. > Knox > > > is also designed to be extensible through Knox service stacks since > they > > > realized they can’t support every project. The challenge is that the > docs > > > have not made it as easy as they could for the ecosystem to plug into > > Knox, > > > which has led to some confusion around this being a recommended pattern > > > (which it is). > > > > > > The danger of trying to get your bits into Knox is that that ties you > to > > > their release cycle (a problem Ambari has felt hard, hence their > > community > > > is moving away from the everything inside model towards everything is > an > > > mpack). > > > > > > A number of implementations of Knox also use the approach Ryan is > > > suggesting for their own organization specific end points, so it’s not > > like > > > this is an uncommon, or anti-pattern, it’s more the way Knox is > designed > > to > > > work in the future, than the legacy of it only being able to handle a > > > subset of Hadoop projects. > > > > > > Knox remains optional In our scenario, but we keep control over the > > > shipping of things like rewrite rules, which allows Metron to control > its > > > release destiny should things like url patterns in the ui need to > change > > > (with a new release of angular / new module / new rest endpoint etc) > > > instead of making a Metron release dependent on a Knox release. > > > > > > Imagine how we would have done with the Ambari side if we’d had to wait > > > for them to release every time we needed to change something in the > > > mpack... we don’t want that happening with Knox. > > > > > > Simon > > > > > > > On 16 Nov 2018, at 13:22, Otto Fowler > wrote: > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > > > Solr is angular for example. > > > > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler ( > ottobackwa...@gmail.com > > ) > > > > wrote: > > > > > > > > Ok, here is something I don’t understand, but would like to. > > > > > > > > Knox comes configured with build in services for a number of other > > apache > > > > products and UI’s. > > > > It would seem to me, that the best integration with Knox would be to > do > > > > what these other products have done. > > > > > > > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > > > 2. Create a knox service definition and provide it or try to get it > > into > > > > knox itself > > > > > > > > This would make the knox integration with metron optional and > pluggable > > > > wouldn’t it? > > > > > > > > Then knox with metron would just be the same as knox with anything > > else. > > > > Please help me if I am wrong, but we seem to be going our own way > here. > > > > Why don’t we just do what these other products have done? > > > > Why don’t we try to get apache metron services accepted to the knox > > > > project? Why don’t we model our knox integration with how XYZ does > it? > > > > Have we looked at how others integrate? Having all the code and > being > > > > able to track stuff is kind of
Re: [DISCUSS] Knox SSO feature branch review and features
Ryan, what's remote debugging look like for UI testing with Knox enabled? Anything we lose from a dev testability standpoint? The discussion of defaults sounds reasonable to me, and I'd like to understand any other tradeoffs there may be for non-prod deployments like full dev. On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman wrote: > Most of the research I've done around adding Metron as a Knox service is > based on how other projects do it. The documentation is not easy to follow > so I learned by reading other service definition files. The assumption > that we are doing things drastically different is false. > > I completely agree with Simon. Why would we want to be dependent on Knox's > release cycle? How does that benefit us? It may reduce some operational > complexity but it makes our install process more complicated because we > require a certain version of Knox (who knows when that gets released). > What do we do in the meantime? I would also like to point out that Metron > is inherently different than other Hadoop stack services. We are a > full-blown application with multiple UIs so the way we expose services > through Knox may be a little different. > > I think this will be easier to discuss when we can all see what is actually > involved. I am working on a PR that adds Metron as a Knox service and will > have that out soon. That should give everyone more context. > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > You could say the same thing about Ambari, but that provides mpacks. Knox > > is also designed to be extensible through Knox service stacks since they > > realized they can’t support every project. The challenge is that the docs > > have not made it as easy as they could for the ecosystem to plug into > Knox, > > which has led to some confusion around this being a recommended pattern > > (which it is). > > > > The danger of trying to get your bits into Knox is that that ties you to > > their release cycle (a problem Ambari has felt hard, hence their > community > > is moving away from the everything inside model towards everything is an > > mpack). > > > > A number of implementations of Knox also use the approach Ryan is > > suggesting for their own organization specific end points, so it’s not > like > > this is an uncommon, or anti-pattern, it’s more the way Knox is designed > to > > work in the future, than the legacy of it only being able to handle a > > subset of Hadoop projects. > > > > Knox remains optional In our scenario, but we keep control over the > > shipping of things like rewrite rules, which allows Metron to control its > > release destiny should things like url patterns in the ui need to change > > (with a new release of angular / new module / new rest endpoint etc) > > instead of making a Metron release dependent on a Knox release. > > > > Imagine how we would have done with the Ambari side if we’d had to wait > > for them to release every time we needed to change something in the > > mpack... we don’t want that happening with Knox. > > > > Simon > > > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > Solr is angular for example. > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com > ) > > > wrote: > > > > > > Ok, here is something I don’t understand, but would like to. > > > > > > Knox comes configured with build in services for a number of other > apache > > > products and UI’s. > > > It would seem to me, that the best integration with Knox would be to do > > > what these other products have done. > > > > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > > 2. Create a knox service definition and provide it or try to get it > into > > > knox itself > > > > > > This would make the knox integration with metron optional and pluggable > > > wouldn’t it? > > > > > > Then knox with metron would just be the same as knox with anything > else. > > > Please help me if I am wrong, but we seem to be going our own way here. > > > Why don’t we just do what these other products have done? > > > Why don’t we try to get apache metron services accepted to the knox > > > project? Why don’t we model our knox integration with how XYZ does it? > > > Have we looked at how others integrate? Having all the code and being > > > able to track stuff is kind of the point of this whole thing isn’t it? > > > > > > Maybe this is implied and I’m missing it, if so I apologize. > > > > > > I think consistency with the rest of the hadoop stack with knox helps > us. > > > > > > > > > > > > On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) > > wrote: > > > > > > 1) Sorry I misspoke. I meant to say this is not possible in the Alerts > UI > > > as far as I know. I put up a PR with a proposed solution here: > > >
Re: [DISCUSS] Knox SSO feature branch review and features
It is included, yes, but is not started out of the box by default, we would also probably tweak the blueprint to change its bootstrap ldif file a bit to have more sensible user names for our defaults, but that's pretty simple load. A bit of default blueprint config is what we need there, not anything 'code' related per se. Simon On Fri, 16 Nov 2018 at 15:59, Otto Fowler wrote: > That does sound good Simon, I think I miss understood that the default > LDAP was standard with KNOX/ambari and not something we would be doing > ourselves. > > > On November 16, 2018 at 10:54:48, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > I think there is a lot to be said for defaulting to Knox on... we also > that > way get some 'secure by default' - at least ssl by default. The do-nothing > I think you're proposing would be around the authentication, right? Knox > does ship with a demo LDAP server we could have some defaults (kinda like > we do with the dev spring profile today) which might be a good way of > achieving a similar effect, and that's configured by default by Ambari. > Would that meet the need, or do you think we should provide a "yeah I'm > sure we don't need to authenticate that connection, let them in" identity > provider for Knox? That way we would have to have them impersonate a known > user for the REST api to work, but you would get the seemless, no auth > access. > > To be honest, I'm a fan of the first option where we give people a nice > simple, no external config and just use some sensible default users in the > demo LDAP instance that Knox owns, and then default to Knox on to give us > our nice single access point. > > Simon > > > On Fri, 16 Nov 2018 at 15:35, Otto Fowler > wrote: > > > Those are all valid points. I think it is ( was ) worth discussion at > > lease a little. > > > > WRT Knox and defaults: > > > > I have in the past used “do-nothing” implementations as default > > placeholders for functionality > > that needed extensive per customer configuration, or configuration > outside > > the responsibility of the product. > > > > Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX > > configuration with “do-nothing” providers > > for auth etc. The users would then configure the providers ( based on > the > > provider(s) we support ) at a later time. > > > > We could write the providers, as everyone has pointed out how extensible > > KNOX is ;) > > > > Would that be a valid way to simplify the issue? > > What would the fallout of that be? > > > > > > > > On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com) > > wrote: > > > > Most of the research I've done around adding Metron as a Knox service is > > based on how other projects do it. The documentation is not easy to > follow > > so I learned by reading other service definition files. The assumption > > that we are doing things drastically different is false. > > > > I completely agree with Simon. Why would we want to be dependent on > Knox's > > release cycle? How does that benefit us? It may reduce some operational > > complexity but it makes our install process more complicated because we > > require a certain version of Knox (who knows when that gets released). > > What do we do in the meantime? I would also like to point out that > Metron > > is inherently different than other Hadoop stack services. We are a > > full-blown application with multiple UIs so the way we expose services > > through Knox may be a little different. > > > > I think this will be easier to discuss when we can all see what is > actually > > involved. I am working on a PR that adds Metron as a Knox service and > will > > have that out soon. That should give everyone more context. > > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > > si...@simonellistonball.com> wrote: > > > > > You could say the same thing about Ambari, but that provides mpacks. > Knox > > > is also designed to be extensible through Knox service stacks since > they > > > realized they can’t support every project. The challenge is that the > docs > > > have not made it as easy as they could for the ecosystem to plug into > > Knox, > > > which has led to some confusion around this being a recommended > pattern > > > (which it is). > > > > > > The danger of trying to get your bits into Knox is that that ties you > to > > > their release cycle (a problem Ambari has felt hard, hence their > > community > > > is moving away from the everything inside model towards everything is > an > > > mpack). > > > > > > A number of implementations of Knox also use the approach Ryan is > > > suggesting for their own organization specific end points, so it’s not > > like > > > this is an uncommon, or anti-pattern, it’s more the way Knox is > designed > > to > > > work in the future, than the legacy of it only being able to handle a > > > subset of Hadoop projects. > > > > > > Knox remains optional In our scenario, but we keep control over the > > >
Re: [DISCUSS] Knox SSO feature branch review and features
That does sound good Simon, I think I miss understood that the default LDAP was standard with KNOX/ambari and not something we would be doing ourselves. On November 16, 2018 at 10:54:48, Simon Elliston Ball ( si...@simonellistonball.com) wrote: I think there is a lot to be said for defaulting to Knox on... we also that way get some 'secure by default' - at least ssl by default. The do-nothing I think you're proposing would be around the authentication, right? Knox does ship with a demo LDAP server we could have some defaults (kinda like we do with the dev spring profile today) which might be a good way of achieving a similar effect, and that's configured by default by Ambari. Would that meet the need, or do you think we should provide a "yeah I'm sure we don't need to authenticate that connection, let them in" identity provider for Knox? That way we would have to have them impersonate a known user for the REST api to work, but you would get the seemless, no auth access. To be honest, I'm a fan of the first option where we give people a nice simple, no external config and just use some sensible default users in the demo LDAP instance that Knox owns, and then default to Knox on to give us our nice single access point. Simon On Fri, 16 Nov 2018 at 15:35, Otto Fowler wrote: > Those are all valid points. I think it is ( was ) worth discussion at > lease a little. > > WRT Knox and defaults: > > I have in the past used “do-nothing” implementations as default > placeholders for functionality > that needed extensive per customer configuration, or configuration outside > the responsibility of the product. > > Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX > configuration with “do-nothing” providers > for auth etc. The users would then configure the providers ( based on the > provider(s) we support ) at a later time. > > We could write the providers, as everyone has pointed out how extensible > KNOX is ;) > > Would that be a valid way to simplify the issue? > What would the fallout of that be? > > > > On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com) > wrote: > > Most of the research I've done around adding Metron as a Knox service is > based on how other projects do it. The documentation is not easy to follow > so I learned by reading other service definition files. The assumption > that we are doing things drastically different is false. > > I completely agree with Simon. Why would we want to be dependent on Knox's > release cycle? How does that benefit us? It may reduce some operational > complexity but it makes our install process more complicated because we > require a certain version of Knox (who knows when that gets released). > What do we do in the meantime? I would also like to point out that Metron > is inherently different than other Hadoop stack services. We are a > full-blown application with multiple UIs so the way we expose services > through Knox may be a little different. > > I think this will be easier to discuss when we can all see what is actually > involved. I am working on a PR that adds Metron as a Knox service and will > have that out soon. That should give everyone more context. > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > You could say the same thing about Ambari, but that provides mpacks. Knox > > is also designed to be extensible through Knox service stacks since they > > realized they can’t support every project. The challenge is that the docs > > have not made it as easy as they could for the ecosystem to plug into > Knox, > > which has led to some confusion around this being a recommended pattern > > (which it is). > > > > The danger of trying to get your bits into Knox is that that ties you to > > their release cycle (a problem Ambari has felt hard, hence their > community > > is moving away from the everything inside model towards everything is an > > mpack). > > > > A number of implementations of Knox also use the approach Ryan is > > suggesting for their own organization specific end points, so it’s not > like > > this is an uncommon, or anti-pattern, it’s more the way Knox is designed > to > > work in the future, than the legacy of it only being able to handle a > > subset of Hadoop projects. > > > > Knox remains optional In our scenario, but we keep control over the > > shipping of things like rewrite rules, which allows Metron to control its > > release destiny should things like url patterns in the ui need to change > > (with a new release of angular / new module / new rest endpoint etc) > > instead of making a Metron release dependent on a Knox release. > > > > Imagine how we would have done with the Ambari side if we’d had to wait > > for them to release every time we needed to change something in the > > mpack... we don’t want that happening with Knox. > > > > Simon > > > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > > > > > >
Re: [DISCUSS] Knox SSO feature branch review and features
I think there is a lot to be said for defaulting to Knox on... we also that way get some 'secure by default' - at least ssl by default. The do-nothing I think you're proposing would be around the authentication, right? Knox does ship with a demo LDAP server we could have some defaults (kinda like we do with the dev spring profile today) which might be a good way of achieving a similar effect, and that's configured by default by Ambari. Would that meet the need, or do you think we should provide a "yeah I'm sure we don't need to authenticate that connection, let them in" identity provider for Knox? That way we would have to have them impersonate a known user for the REST api to work, but you would get the seemless, no auth access. To be honest, I'm a fan of the first option where we give people a nice simple, no external config and just use some sensible default users in the demo LDAP instance that Knox owns, and then default to Knox on to give us our nice single access point. Simon On Fri, 16 Nov 2018 at 15:35, Otto Fowler wrote: > Those are all valid points. I think it is ( was ) worth discussion at > lease a little. > > WRT Knox and defaults: > > I have in the past used “do-nothing” implementations as default > placeholders for functionality > that needed extensive per customer configuration, or configuration outside > the responsibility of the product. > > Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX > configuration with “do-nothing” providers > for auth etc. The users would then configure the providers ( based on the > provider(s) we support ) at a later time. > > We could write the providers, as everyone has pointed out how extensible > KNOX is ;) > > Would that be a valid way to simplify the issue? > What would the fallout of that be? > > > > On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com) > wrote: > > Most of the research I've done around adding Metron as a Knox service is > based on how other projects do it. The documentation is not easy to follow > so I learned by reading other service definition files. The assumption > that we are doing things drastically different is false. > > I completely agree with Simon. Why would we want to be dependent on Knox's > release cycle? How does that benefit us? It may reduce some operational > complexity but it makes our install process more complicated because we > require a certain version of Knox (who knows when that gets released). > What do we do in the meantime? I would also like to point out that Metron > is inherently different than other Hadoop stack services. We are a > full-blown application with multiple UIs so the way we expose services > through Knox may be a little different. > > I think this will be easier to discuss when we can all see what is actually > involved. I am working on a PR that adds Metron as a Knox service and will > have that out soon. That should give everyone more context. > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > You could say the same thing about Ambari, but that provides mpacks. Knox > > is also designed to be extensible through Knox service stacks since they > > realized they can’t support every project. The challenge is that the docs > > have not made it as easy as they could for the ecosystem to plug into > Knox, > > which has led to some confusion around this being a recommended pattern > > (which it is). > > > > The danger of trying to get your bits into Knox is that that ties you to > > their release cycle (a problem Ambari has felt hard, hence their > community > > is moving away from the everything inside model towards everything is an > > mpack). > > > > A number of implementations of Knox also use the approach Ryan is > > suggesting for their own organization specific end points, so it’s not > like > > this is an uncommon, or anti-pattern, it’s more the way Knox is designed > to > > work in the future, than the legacy of it only being able to handle a > > subset of Hadoop projects. > > > > Knox remains optional In our scenario, but we keep control over the > > shipping of things like rewrite rules, which allows Metron to control its > > release destiny should things like url patterns in the ui need to change > > (with a new release of angular / new module / new rest endpoint etc) > > instead of making a Metron release dependent on a Knox release. > > > > Imagine how we would have done with the Ambari side if we’d had to wait > > for them to release every time we needed to change something in the > > mpack... we don’t want that happening with Knox. > > > > Simon > > > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > Solr is angular for example. > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com > ) > > > wrote: > > > > > > Ok, here is
Re: [DISCUSS] Knox SSO feature branch review and features
Those are all valid points. I think it is ( was ) worth discussion at lease a little. WRT Knox and defaults: I have in the past used “do-nothing” implementations as default placeholders for functionality that needed extensive per customer configuration, or configuration outside the responsibility of the product. Would it be simpler if we ALWAYS used Knox, but defaulted to a KNOX configuration with “do-nothing” providers for auth etc. The users would then configure the providers ( based on the provider(s) we support ) at a later time. We could write the providers, as everyone has pointed out how extensible KNOX is ;) Would that be a valid way to simplify the issue? What would the fallout of that be? On November 16, 2018 at 09:20:53, Ryan Merriman (merrim...@gmail.com) wrote: Most of the research I've done around adding Metron as a Knox service is based on how other projects do it. The documentation is not easy to follow so I learned by reading other service definition files. The assumption that we are doing things drastically different is false. I completely agree with Simon. Why would we want to be dependent on Knox's release cycle? How does that benefit us? It may reduce some operational complexity but it makes our install process more complicated because we require a certain version of Knox (who knows when that gets released). What do we do in the meantime? I would also like to point out that Metron is inherently different than other Hadoop stack services. We are a full-blown application with multiple UIs so the way we expose services through Knox may be a little different. I think this will be easier to discuss when we can all see what is actually involved. I am working on a PR that adds Metron as a Knox service and will have that out soon. That should give everyone more context. On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > You could say the same thing about Ambari, but that provides mpacks. Knox > is also designed to be extensible through Knox service stacks since they > realized they can’t support every project. The challenge is that the docs > have not made it as easy as they could for the ecosystem to plug into Knox, > which has led to some confusion around this being a recommended pattern > (which it is). > > The danger of trying to get your bits into Knox is that that ties you to > their release cycle (a problem Ambari has felt hard, hence their community > is moving away from the everything inside model towards everything is an > mpack). > > A number of implementations of Knox also use the approach Ryan is > suggesting for their own organization specific end points, so it’s not like > this is an uncommon, or anti-pattern, it’s more the way Knox is designed to > work in the future, than the legacy of it only being able to handle a > subset of Hadoop projects. > > Knox remains optional In our scenario, but we keep control over the > shipping of things like rewrite rules, which allows Metron to control its > release destiny should things like url patterns in the ui need to change > (with a new release of angular / new module / new rest endpoint etc) > instead of making a Metron release dependent on a Knox release. > > Imagine how we would have done with the Ambari side if we’d had to wait > for them to release every time we needed to change something in the > mpack... we don’t want that happening with Knox. > > Simon > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > Solr is angular for example. > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com) > > wrote: > > > > Ok, here is something I don’t understand, but would like to. > > > > Knox comes configured with build in services for a number of other apache > > products and UI’s. > > It would seem to me, that the best integration with Knox would be to do > > what these other products have done. > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > 2. Create a knox service definition and provide it or try to get it into > > knox itself > > > > This would make the knox integration with metron optional and pluggable > > wouldn’t it? > > > > Then knox with metron would just be the same as knox with anything else. > > Please help me if I am wrong, but we seem to be going our own way here. > > Why don’t we just do what these other products have done? > > Why don’t we try to get apache metron services accepted to the knox > > project? Why don’t we model our knox integration with how XYZ does it? > > Have we looked at how others integrate? Having all the code and being > > able to track stuff is kind of the point of this whole thing isn’t it? > > > > Maybe this is implied and I’m missing it, if so I apologize. > > > > I think consistency with the rest of the hadoop stack with knox helps us. > > > > > > > > On November 15,
Re: [DISCUSS] Knox SSO feature branch review and features
Most of the research I've done around adding Metron as a Knox service is based on how other projects do it. The documentation is not easy to follow so I learned by reading other service definition files. The assumption that we are doing things drastically different is false. I completely agree with Simon. Why would we want to be dependent on Knox's release cycle? How does that benefit us? It may reduce some operational complexity but it makes our install process more complicated because we require a certain version of Knox (who knows when that gets released). What do we do in the meantime? I would also like to point out that Metron is inherently different than other Hadoop stack services. We are a full-blown application with multiple UIs so the way we expose services through Knox may be a little different. I think this will be easier to discuss when we can all see what is actually involved. I am working on a PR that adds Metron as a Knox service and will have that out soon. That should give everyone more context. On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > You could say the same thing about Ambari, but that provides mpacks. Knox > is also designed to be extensible through Knox service stacks since they > realized they can’t support every project. The challenge is that the docs > have not made it as easy as they could for the ecosystem to plug into Knox, > which has led to some confusion around this being a recommended pattern > (which it is). > > The danger of trying to get your bits into Knox is that that ties you to > their release cycle (a problem Ambari has felt hard, hence their community > is moving away from the everything inside model towards everything is an > mpack). > > A number of implementations of Knox also use the approach Ryan is > suggesting for their own organization specific end points, so it’s not like > this is an uncommon, or anti-pattern, it’s more the way Knox is designed to > work in the future, than the legacy of it only being able to handle a > subset of Hadoop projects. > > Knox remains optional In our scenario, but we keep control over the > shipping of things like rewrite rules, which allows Metron to control its > release destiny should things like url patterns in the ui need to change > (with a new release of angular / new module / new rest endpoint etc) > instead of making a Metron release dependent on a Knox release. > > Imagine how we would have done with the Ambari side if we’d had to wait > for them to release every time we needed to change something in the > mpack... we don’t want that happening with Knox. > > Simon > > > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > Solr is angular for example. > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com) > > wrote: > > > > Ok, here is something I don’t understand, but would like to. > > > > Knox comes configured with build in services for a number of other apache > > products and UI’s. > > It would seem to me, that the best integration with Knox would be to do > > what these other products have done. > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > 2. Create a knox service definition and provide it or try to get it into > > knox itself > > > > This would make the knox integration with metron optional and pluggable > > wouldn’t it? > > > > Then knox with metron would just be the same as knox with anything else. > > Please help me if I am wrong, but we seem to be going our own way here. > > Why don’t we just do what these other products have done? > > Why don’t we try to get apache metron services accepted to the knox > > project? Why don’t we model our knox integration with how XYZ does it? > > Have we looked at how others integrate? Having all the code and being > > able to track stuff is kind of the point of this whole thing isn’t it? > > > > Maybe this is implied and I’m missing it, if so I apologize. > > > > I think consistency with the rest of the hadoop stack with knox helps us. > > > > > > > > On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) > wrote: > > > > 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI > > as far as I know. I put up a PR with a proposed solution here: > > https://github.com/apache/metron/pull/1266. > > 2) Yes Knox is a service you can install with Ambari, similar to Ranger > or > > Spark. There are some things that are specifically configured in Knox and > > there are some things specific to Metron. I will put up a PR with the > > changes needed so you can see exactly what is involved. > > 3) I don't understand what you mean here. Is this a question? > > 4) I think it's a little early to predict the Ambari changes required. > > This will depend on how tasks 1-3 go. I imagine it's similar to other > > mpack work:
Re: [DISCUSS] Knox SSO feature branch review and features
You could say the same thing about Ambari, but that provides mpacks. Knox is also designed to be extensible through Knox service stacks since they realized they can’t support every project. The challenge is that the docs have not made it as easy as they could for the ecosystem to plug into Knox, which has led to some confusion around this being a recommended pattern (which it is). The danger of trying to get your bits into Knox is that that ties you to their release cycle (a problem Ambari has felt hard, hence their community is moving away from the everything inside model towards everything is an mpack). A number of implementations of Knox also use the approach Ryan is suggesting for their own organization specific end points, so it’s not like this is an uncommon, or anti-pattern, it’s more the way Knox is designed to work in the future, than the legacy of it only being able to handle a subset of Hadoop projects. Knox remains optional In our scenario, but we keep control over the shipping of things like rewrite rules, which allows Metron to control its release destiny should things like url patterns in the ui need to change (with a new release of angular / new module / new rest endpoint etc) instead of making a Metron release dependent on a Knox release. Imagine how we would have done with the Ambari side if we’d had to wait for them to release every time we needed to change something in the mpack... we don’t want that happening with Knox. Simon > On 16 Nov 2018, at 13:22, Otto Fowler wrote: > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > Solr is angular for example. > > > On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > Ok, here is something I don’t understand, but would like to. > > Knox comes configured with build in services for a number of other apache > products and UI’s. > It would seem to me, that the best integration with Knox would be to do > what these other products have done. > > > 1. Do whatever you have to do to make your own stuff compatible. > 2. Create a knox service definition and provide it or try to get it into > knox itself > > This would make the knox integration with metron optional and pluggable > wouldn’t it? > > Then knox with metron would just be the same as knox with anything else. > Please help me if I am wrong, but we seem to be going our own way here. > Why don’t we just do what these other products have done? > Why don’t we try to get apache metron services accepted to the knox > project? Why don’t we model our knox integration with how XYZ does it? > Have we looked at how others integrate? Having all the code and being > able to track stuff is kind of the point of this whole thing isn’t it? > > Maybe this is implied and I’m missing it, if so I apologize. > > I think consistency with the rest of the hadoop stack with knox helps us. > > > > On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) wrote: > > 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI > as far as I know. I put up a PR with a proposed solution here: > https://github.com/apache/metron/pull/1266. > 2) Yes Knox is a service you can install with Ambari, similar to Ranger or > Spark. There are some things that are specifically configured in Knox and > there are some things specific to Metron. I will put up a PR with the > changes needed so you can see exactly what is involved. > 3) I don't understand what you mean here. Is this a question? > 4) I think it's a little early to predict the Ambari changes required. > This will depend on how tasks 1-3 go. I imagine it's similar to other > mpack work: expose some parameters in ambari and bind those to config > files. My understanding from this thread so far is that we should focus on > a manual, documented approach to start. > > On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Thanks Ryan, >> >> 1) Can you clarify "not a good way to do this?" Are you saying we don't >> have a way to set this and need to add the config option, or that a >> solution is not obvious and it's unclear what to do? It seems to me you're >> saying the former, but I'd like to be sure. >> 2) Is Knox not a service made available by Ambari similar to Ranger or >> Spark? I'm assuming that similar to Kerberos, there are some things that >> are specifically configured in Knox and others that are app-specific. Some >> explanation of what this looks like would be helpful. >> 3) Sounds like this follows pretty naturally from #1 >> 4) Relates to #2. I think we need some guidance on what a manual vs >> MPack/automated install would look like. >> >> Cheers, >> Mike >> >> >>> On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman wrote: >>> >>> Wanted to give an update on the context path issue. I investigated >>> rewriting url links in the outgoing static assets with Knox and it
Re: [DISCUSS] Knox SSO feature branch review and features
https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support Solr is angular for example. On November 16, 2018 at 08:12:55, Otto Fowler (ottobackwa...@gmail.com) wrote: Ok, here is something I don’t understand, but would like to. Knox comes configured with build in services for a number of other apache products and UI’s. It would seem to me, that the best integration with Knox would be to do what these other products have done. 1. Do whatever you have to do to make your own stuff compatible. 2. Create a knox service definition and provide it or try to get it into knox itself This would make the knox integration with metron optional and pluggable wouldn’t it? Then knox with metron would just be the same as knox with anything else. Please help me if I am wrong, but we seem to be going our own way here. Why don’t we just do what these other products have done? Why don’t we try to get apache metron services accepted to the knox project? Why don’t we model our knox integration with how XYZ does it? Have we looked at how others integrate? Having all the code and being able to track stuff is kind of the point of this whole thing isn’t it? Maybe this is implied and I’m missing it, if so I apologize. I think consistency with the rest of the hadoop stack with knox helps us. On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) wrote: 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI as far as I know. I put up a PR with a proposed solution here: https://github.com/apache/metron/pull/1266. 2) Yes Knox is a service you can install with Ambari, similar to Ranger or Spark. There are some things that are specifically configured in Knox and there are some things specific to Metron. I will put up a PR with the changes needed so you can see exactly what is involved. 3) I don't understand what you mean here. Is this a question? 4) I think it's a little early to predict the Ambari changes required. This will depend on how tasks 1-3 go. I imagine it's similar to other mpack work: expose some parameters in ambari and bind those to config files. My understanding from this thread so far is that we should focus on a manual, documented approach to start. On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Thanks Ryan, > > 1) Can you clarify "not a good way to do this?" Are you saying we don't > have a way to set this and need to add the config option, or that a > solution is not obvious and it's unclear what to do? It seems to me you're > saying the former, but I'd like to be sure. > 2) Is Knox not a service made available by Ambari similar to Ranger or > Spark? I'm assuming that similar to Kerberos, there are some things that > are specifically configured in Knox and others that are app-specific. Some > explanation of what this looks like would be helpful. > 3) Sounds like this follows pretty naturally from #1 > 4) Relates to #2. I think we need some guidance on what a manual vs > MPack/automated install would look like. > > Cheers, > Mike > > > On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman wrote: > > > Wanted to give an update on the context path issue. I investigated > > rewriting url links in the outgoing static assets with Knox and it was > not > > trivial. Fortunately I found a simple solution that works with or > without > > Knox. I changed the base tag in index.html from to > href="./">, or in other words made the base href relative. > > > > I believe I am at the point where I can task this out and provide a high > > level overview of the changes needed. I think that each task will be a > > manageable size and can stand alone so I don't think we need a feature > > branch. > > > > The first task involves a general change to the UI code. We need a way > to > > set the path to the REST service with a configuration setting because it > is > > different with and without Knox. Currently there is not a good way to do > > this in the UI. We can use the environment files but that is a build > time > > setting and is not flexible. I can see this capability being useful for > > other use cases in the future. I think we could even split this up into > 2 > > separate tasks, one for the alerts UI and one for the management UI. > > > > The second task involves adding Knox to our stack either by default as a > > dependency in the mpack or with a documented approach. We would add our > > REST service, Alerts UI, and Management UI as services in Knox. > Everything > > would continue to function as it currently does but with all > communication > > going through Knox. LDAP authentication would be required when using > Knox > > and Knox will authenticate with the REST service by passing along an > > Authorization header. Enabling Knox would be a manual process that > > involves deploying assets (Knox descriptor files) and changing > > configuration. There would be no change to how the UI functions by >
Re: [DISCUSS] Knox SSO feature branch review and features
Ok, here is something I don’t understand, but would like to. Knox comes configured with build in services for a number of other apache products and UI’s. It would seem to me, that the best integration with Knox would be to do what these other products have done. 1. Do whatever you have to do to make your own stuff compatible. 2. Create a knox service definition and provide it or try to get it into knox itself This would make the knox integration with metron optional and pluggable wouldn’t it? Then knox with metron would just be the same as knox with anything else. Please help me if I am wrong, but we seem to be going our own way here. Why don’t we just do what these other products have done? Why don’t we try to get apache metron services accepted to the knox project? Why don’t we model our knox integration with how XYZ does it? Have we looked at how others integrate? Having all the code and being able to track stuff is kind of the point of this whole thing isn’t it? Maybe this is implied and I’m missing it, if so I apologize. I think consistency with the rest of the hadoop stack with knox helps us. On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) wrote: 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI as far as I know. I put up a PR with a proposed solution here: https://github.com/apache/metron/pull/1266. 2) Yes Knox is a service you can install with Ambari, similar to Ranger or Spark. There are some things that are specifically configured in Knox and there are some things specific to Metron. I will put up a PR with the changes needed so you can see exactly what is involved. 3) I don't understand what you mean here. Is this a question? 4) I think it's a little early to predict the Ambari changes required. This will depend on how tasks 1-3 go. I imagine it's similar to other mpack work: expose some parameters in ambari and bind those to config files. My understanding from this thread so far is that we should focus on a manual, documented approach to start. On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Thanks Ryan, > > 1) Can you clarify "not a good way to do this?" Are you saying we don't > have a way to set this and need to add the config option, or that a > solution is not obvious and it's unclear what to do? It seems to me you're > saying the former, but I'd like to be sure. > 2) Is Knox not a service made available by Ambari similar to Ranger or > Spark? I'm assuming that similar to Kerberos, there are some things that > are specifically configured in Knox and others that are app-specific. Some > explanation of what this looks like would be helpful. > 3) Sounds like this follows pretty naturally from #1 > 4) Relates to #2. I think we need some guidance on what a manual vs > MPack/automated install would look like. > > Cheers, > Mike > > > On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman wrote: > > > Wanted to give an update on the context path issue. I investigated > > rewriting url links in the outgoing static assets with Knox and it was > not > > trivial. Fortunately I found a simple solution that works with or > without > > Knox. I changed the base tag in index.html from to > href="./">, or in other words made the base href relative. > > > > I believe I am at the point where I can task this out and provide a high > > level overview of the changes needed. I think that each task will be a > > manageable size and can stand alone so I don't think we need a feature > > branch. > > > > The first task involves a general change to the UI code. We need a way > to > > set the path to the REST service with a configuration setting because it > is > > different with and without Knox. Currently there is not a good way to do > > this in the UI. We can use the environment files but that is a build > time > > setting and is not flexible. I can see this capability being useful for > > other use cases in the future. I think we could even split this up into > 2 > > separate tasks, one for the alerts UI and one for the management UI. > > > > The second task involves adding Knox to our stack either by default as a > > dependency in the mpack or with a documented approach. We would add our > > REST service, Alerts UI, and Management UI as services in Knox. > Everything > > would continue to function as it currently does but with all > communication > > going through Knox. LDAP authentication would be required when using > Knox > > and Knox will authenticate with the REST service by passing along an > > Authorization header. Enabling Knox would be a manual process that > > involves deploying assets (Knox descriptor files) and changing > > configuration. There would be no change to how the UI functions by > default > > (without Knox) and either LDAP or JDBC authentication could still be > used.. > > > > The third task involves enabling SSO with Knox. We would update the REST > > service so that it can authenticate
Re: [DISCUSS] Knox SSO feature branch review and features
Welcome and thanks! On November 12, 2018 at 10:12:44, Sandeep Moré (moresand...@gmail.com) wrote: Hello Ryan, I am still catching up on the architecture so let me know if I am misunderstanding anything. You could have multiple serviced deployed in Knox 1. for metron (metron/api/v1) 2. for alerts-ui (metron-alerts-ui/alerts-list) and have them run in one Knox instance and you could have one service reference from other (not recommended but possible). You can tailor rewrite rules to update the context path for the assets as well, pointed out by Simon. Knox Wiki [1] has some blogs and tutorials that you can look at. This [2] is a good tutorial on how rewriting static assets, there is also a blog [3] on basics of rewrite rules that should be a good reference. I would also be glad to look at the service definitions you have and answer any questions. [1] https://cwiki.apache.org/confluence/display/KNOX/Index [2] https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox [3] https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox Best, Sandeep P.S. I am a Knox committer and new to Metron. On Mon, Nov 12, 2018 at 9:59 AM Ryan Merriman wrote: > I'm just coming up to speed on Knox so maybe rewriting assets links are > trivial. If anyone has a good example of how to do that or can point to > some documentation, please share. > > On Mon, Nov 12, 2018 at 8:54 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > Doing the Knox proxy work first certainly does make a lot of sense vs the > > SSO first approach, so I'm in favour of this. It bypasses all the > anti-CORS > > proxying stuff the other solution needed by being on the same URL space. > > > > Is there are reason we're not re-writing the asset link URLs in Knox? We > > should have a reverse content rewrite rule to avoid that problem and make > > it entirely transparent whether there is Knox or not. We shouldn't be > > changing anything about the UI services themselves. If the rewrite > service > > is complete, there is no change to base ref in the UI code, Knox would > > effectively apply it by content filtering. Note also that the gateway URL > > is configurable and likely to vary from Knox to Knox, so baking it into > the > > ng build will break non-full-dev builds. (e.g. gateway/default could well > > be gateway/xyz). > > > > I would also like to discuss removing the JDBC auth, because it's a set > of > > plaintext passwords in a mysql DB... it introduces a problematic > dependency > > (mysql) a ton of java dependencies we could cut out (JPA, eclipselink) > and > > opens up a massive security hole. I personally know of several > > organisations who are blocked from using Metron by the presence of the > JDBC > > authentication method in its current form. > > > > Simon > > > > On Mon, 12 Nov 2018 at 14:36, Ryan Merriman wrote: > > > > > Let me clarify on exposing both legacy and Knox URLs at the same time. > > The > > > base urls will look something like this: > > > > > > Legacy REST - http://node1:8082/api/v1 > > > Legacy Alerts UI - http://node1:4201:/alerts-list > > > > > > Knox REST - https://node1:8443/gateway/default/metron/api/v1 > > > Knox Alerts UI - > > > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > > > > > If Knox were turned on and the alerts UI deployed as is, it would not > > > work. This is because static assets are referenced with > > > http://node1:4201/assets/some-asset.js which does not include the > > correct > > > context path to the alerts UI in knox. To make it work, you have to > set > > > the base ref to "/gateway/default/metron-alerts-ui" so that static > assets > > > are referenced at > > > > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js > > . > > > When you do that, the legacy alerts UI will no longer work. I guess > the > > > point I'm trying to make is that we would have to switch between them > or > > > have 2 separate application running. I imagine most users only need > one > > or > > > the other running so probably not an issue. > > > > > > Jon, the primary upgrade consideration I see is with authentication. > To > > be > > > able to use Knox, you would have to upgrade to LDAP-based > authentication > > if > > > you were still using JDBC-based authentication in REST. The urls would > > > also change obviously. > > > > > > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com > > wrote: > > > > > > > Phew, that was quite the thread to catch up on. > > > > > > > > I agree that this should be optional/pluggable to start, and I'm > > > interested > > > > to hear the issues as they relate to upgrading an existing cluster > > (given > > > > the suggested approach) and exposing both legacy and knox URLs at the > > > same > > > > time. > > > > > > > > Jon > > > > > > > > On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < > > > > michael.miklav...@gmail.com> > > > > wrote: > > > > > > > > > A couple more things, and I think this goes
Re: [DISCUSS] Knox SSO feature branch review and features
1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI as far as I know. I put up a PR with a proposed solution here: https://github.com/apache/metron/pull/1266. 2) Yes Knox is a service you can install with Ambari, similar to Ranger or Spark. There are some things that are specifically configured in Knox and there are some things specific to Metron. I will put up a PR with the changes needed so you can see exactly what is involved. 3) I don't understand what you mean here. Is this a question? 4) I think it's a little early to predict the Ambari changes required. This will depend on how tasks 1-3 go. I imagine it's similar to other mpack work: expose some parameters in ambari and bind those to config files. My understanding from this thread so far is that we should focus on a manual, documented approach to start. On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Thanks Ryan, > > 1) Can you clarify "not a good way to do this?" Are you saying we don't > have a way to set this and need to add the config option, or that a > solution is not obvious and it's unclear what to do? It seems to me you're > saying the former, but I'd like to be sure. > 2) Is Knox not a service made available by Ambari similar to Ranger or > Spark? I'm assuming that similar to Kerberos, there are some things that > are specifically configured in Knox and others that are app-specific. Some > explanation of what this looks like would be helpful. > 3) Sounds like this follows pretty naturally from #1 > 4) Relates to #2. I think we need some guidance on what a manual vs > MPack/automated install would look like. > > Cheers, > Mike > > > On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman wrote: > > > Wanted to give an update on the context path issue. I investigated > > rewriting url links in the outgoing static assets with Knox and it was > not > > trivial. Fortunately I found a simple solution that works with or > without > > Knox. I changed the base tag in index.html from to > href="./">, or in other words made the base href relative. > > > > I believe I am at the point where I can task this out and provide a high > > level overview of the changes needed. I think that each task will be a > > manageable size and can stand alone so I don't think we need a feature > > branch. > > > > The first task involves a general change to the UI code. We need a way > to > > set the path to the REST service with a configuration setting because it > is > > different with and without Knox. Currently there is not a good way to do > > this in the UI. We can use the environment files but that is a build > time > > setting and is not flexible. I can see this capability being useful for > > other use cases in the future. I think we could even split this up into > 2 > > separate tasks, one for the alerts UI and one for the management UI. > > > > The second task involves adding Knox to our stack either by default as a > > dependency in the mpack or with a documented approach. We would add our > > REST service, Alerts UI, and Management UI as services in Knox. > Everything > > would continue to function as it currently does but with all > communication > > going through Knox. LDAP authentication would be required when using > Knox > > and Knox will authenticate with the REST service by passing along an > > Authorization header. Enabling Knox would be a manual process that > > involves deploying assets (Knox descriptor files) and changing > > configuration. There would be no change to how the UI functions by > default > > (without Knox) and either LDAP or JDBC authentication could still be > used.. > > > > The third task involves enabling SSO with Knox. We would update the REST > > service so that it can authenticate with a Knox SSO token. We would > > provide documentation on how to update the Knox settings in Ambari to > > enable SSO. The Alerts/Management UI would need to expose configuration > > properties for the REST url and login url since these would be different > > when Knox is enabled. We would also need to provide documentation on how > > to make these UI configuration changes, based on the work done in task 1. > > > > An optional forth task would be exposing configuration settings and > > enabling Knox with Ambari. We would eliminate the manual steps necessary > > for enabling Knox and instead automate those steps with an Ambari input > > control, similar to how LDAP is enabled. > > > > Thoughts on this plan? > > > > On Thu, Nov 15, 2018 at 10:22 AM James Sirota > wrote: > > > > > In my view Knox SSO is such a minor feature when it comes to Metron's > > > capabilities than it's not worth supporting multiple scenarios where it > > > works with Knox or without Knox. Where we should be configurable (and > > are > > > configurable) is on the analytics and stream processing. But this? As > > > long as the UI authenticates securely I don't think anyone is going to > >
Re: [DISCUSS] Knox SSO feature branch review and features
Thanks Ryan, 1) Can you clarify "not a good way to do this?" Are you saying we don't have a way to set this and need to add the config option, or that a solution is not obvious and it's unclear what to do? It seems to me you're saying the former, but I'd like to be sure. 2) Is Knox not a service made available by Ambari similar to Ranger or Spark? I'm assuming that similar to Kerberos, there are some things that are specifically configured in Knox and others that are app-specific. Some explanation of what this looks like would be helpful. 3) Sounds like this follows pretty naturally from #1 4) Relates to #2. I think we need some guidance on what a manual vs MPack/automated install would look like. Cheers, Mike On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman wrote: > Wanted to give an update on the context path issue. I investigated > rewriting url links in the outgoing static assets with Knox and it was not > trivial. Fortunately I found a simple solution that works with or without > Knox. I changed the base tag in index.html from to href="./">, or in other words made the base href relative. > > I believe I am at the point where I can task this out and provide a high > level overview of the changes needed. I think that each task will be a > manageable size and can stand alone so I don't think we need a feature > branch. > > The first task involves a general change to the UI code. We need a way to > set the path to the REST service with a configuration setting because it is > different with and without Knox. Currently there is not a good way to do > this in the UI. We can use the environment files but that is a build time > setting and is not flexible. I can see this capability being useful for > other use cases in the future. I think we could even split this up into 2 > separate tasks, one for the alerts UI and one for the management UI. > > The second task involves adding Knox to our stack either by default as a > dependency in the mpack or with a documented approach. We would add our > REST service, Alerts UI, and Management UI as services in Knox. Everything > would continue to function as it currently does but with all communication > going through Knox. LDAP authentication would be required when using Knox > and Knox will authenticate with the REST service by passing along an > Authorization header. Enabling Knox would be a manual process that > involves deploying assets (Knox descriptor files) and changing > configuration. There would be no change to how the UI functions by default > (without Knox) and either LDAP or JDBC authentication could still be used.. > > The third task involves enabling SSO with Knox. We would update the REST > service so that it can authenticate with a Knox SSO token. We would > provide documentation on how to update the Knox settings in Ambari to > enable SSO. The Alerts/Management UI would need to expose configuration > properties for the REST url and login url since these would be different > when Knox is enabled. We would also need to provide documentation on how > to make these UI configuration changes, based on the work done in task 1. > > An optional forth task would be exposing configuration settings and > enabling Knox with Ambari. We would eliminate the manual steps necessary > for enabling Knox and instead automate those steps with an Ambari input > control, similar to how LDAP is enabled. > > Thoughts on this plan? > > On Thu, Nov 15, 2018 at 10:22 AM James Sirota wrote: > > > In my view Knox SSO is such a minor feature when it comes to Metron's > > capabilities than it's not worth supporting multiple scenarios where it > > works with Knox or without Knox. Where we should be configurable (and > are > > configurable) is on the analytics and stream processing. But this? As > > long as the UI authenticates securely I don't think anyone is going to > care > > what proxy it's using. The code itself should be written in a way that > > it's pluggable so if we ever wanted to use another proxy or disable it > all > > together we could. But this should not be a configuration we pass on to > > the user. The added complexity is simply not worth it here. We have to > > start being opinionated about making sensible choices on behalf of the > > user. A sensible choice here is to run with Knox and LDAP. The JDBC > > component should exist for another release to allow the community to > > migrate over to LDAP and then be deprecated. The code should still be > > pluggable and if anyone wanted to extend it to work with JDBC they could, > > or if people wanted to plug in another proxy they could, but this is not > > something we would officially support. > > > > Thanks, > > James > > > > 12.11.2018, 07:36, "Ryan Merriman" : > > > Let me clarify on exposing both legacy and Knox URLs at the same time. > > The > > > base urls will look something like this: > > > > > > Legacy REST - http://node1:8082/api/v1 > > > Legacy Alerts UI -
Re: [DISCUSS] Knox SSO feature branch review and features
Wanted to give an update on the context path issue. I investigated rewriting url links in the outgoing static assets with Knox and it was not trivial. Fortunately I found a simple solution that works with or without Knox. I changed the base tag in index.html from to , or in other words made the base href relative. I believe I am at the point where I can task this out and provide a high level overview of the changes needed. I think that each task will be a manageable size and can stand alone so I don't think we need a feature branch. The first task involves a general change to the UI code. We need a way to set the path to the REST service with a configuration setting because it is different with and without Knox. Currently there is not a good way to do this in the UI. We can use the environment files but that is a build time setting and is not flexible. I can see this capability being useful for other use cases in the future. I think we could even split this up into 2 separate tasks, one for the alerts UI and one for the management UI. The second task involves adding Knox to our stack either by default as a dependency in the mpack or with a documented approach. We would add our REST service, Alerts UI, and Management UI as services in Knox. Everything would continue to function as it currently does but with all communication going through Knox. LDAP authentication would be required when using Knox and Knox will authenticate with the REST service by passing along an Authorization header. Enabling Knox would be a manual process that involves deploying assets (Knox descriptor files) and changing configuration. There would be no change to how the UI functions by default (without Knox) and either LDAP or JDBC authentication could still be used.. The third task involves enabling SSO with Knox. We would update the REST service so that it can authenticate with a Knox SSO token. We would provide documentation on how to update the Knox settings in Ambari to enable SSO. The Alerts/Management UI would need to expose configuration properties for the REST url and login url since these would be different when Knox is enabled. We would also need to provide documentation on how to make these UI configuration changes, based on the work done in task 1. An optional forth task would be exposing configuration settings and enabling Knox with Ambari. We would eliminate the manual steps necessary for enabling Knox and instead automate those steps with an Ambari input control, similar to how LDAP is enabled. Thoughts on this plan? On Thu, Nov 15, 2018 at 10:22 AM James Sirota wrote: > In my view Knox SSO is such a minor feature when it comes to Metron's > capabilities than it's not worth supporting multiple scenarios where it > works with Knox or without Knox. Where we should be configurable (and are > configurable) is on the analytics and stream processing. But this? As > long as the UI authenticates securely I don't think anyone is going to care > what proxy it's using. The code itself should be written in a way that > it's pluggable so if we ever wanted to use another proxy or disable it all > together we could. But this should not be a configuration we pass on to > the user. The added complexity is simply not worth it here. We have to > start being opinionated about making sensible choices on behalf of the > user. A sensible choice here is to run with Knox and LDAP. The JDBC > component should exist for another release to allow the community to > migrate over to LDAP and then be deprecated. The code should still be > pluggable and if anyone wanted to extend it to work with JDBC they could, > or if people wanted to plug in another proxy they could, but this is not > something we would officially support. > > Thanks, > James > > 12.11.2018, 07:36, "Ryan Merriman" : > > Let me clarify on exposing both legacy and Knox URLs at the same time. > The > > base urls will look something like this: > > > > Legacy REST - http://node1:8082/api/v1 > > Legacy Alerts UI - http://node1:4201:/alerts-list > > > > Knox REST - https://node1:8443/gateway/default/metron/api/v1 > > Knox Alerts UI - > > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > > > If Knox were turned on and the alerts UI deployed as is, it would not > > work. This is because static assets are referenced with > > http://node1:4201/assets/some-asset.js which does not include the > correct > > context path to the alerts UI in knox. To make it work, you have to set > > the base ref to "/gateway/default/metron-alerts-ui" so that static assets > > are referenced at > > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js > . > > When you do that, the legacy alerts UI will no longer work. I guess the > > point I'm trying to make is that we would have to switch between them or > > have 2 separate application running. I imagine most users only need one > or > > the other running so probably not an issue. > > >
Re: [DISCUSS] Knox SSO feature branch review and features
In my view Knox SSO is such a minor feature when it comes to Metron's capabilities than it's not worth supporting multiple scenarios where it works with Knox or without Knox. Where we should be configurable (and are configurable) is on the analytics and stream processing. But this? As long as the UI authenticates securely I don't think anyone is going to care what proxy it's using. The code itself should be written in a way that it's pluggable so if we ever wanted to use another proxy or disable it all together we could. But this should not be a configuration we pass on to the user. The added complexity is simply not worth it here. We have to start being opinionated about making sensible choices on behalf of the user. A sensible choice here is to run with Knox and LDAP. The JDBC component should exist for another release to allow the community to migrate over to LDAP and then be deprecated. The code should still be pluggable and if anyone wanted to extend it to work with JDBC they could, or if people wanted to plug in another proxy they could, but this is not something we would officially support. Thanks, James 12.11.2018, 07:36, "Ryan Merriman" : > Let me clarify on exposing both legacy and Knox URLs at the same time. The > base urls will look something like this: > > Legacy REST - http://node1:8082/api/v1 > Legacy Alerts UI - http://node1:4201:/alerts-list > > Knox REST - https://node1:8443/gateway/default/metron/api/v1 > Knox Alerts UI - > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > If Knox were turned on and the alerts UI deployed as is, it would not > work. This is because static assets are referenced with > http://node1:4201/assets/some-asset.js which does not include the correct > context path to the alerts UI in knox. To make it work, you have to set > the base ref to "/gateway/default/metron-alerts-ui" so that static assets > are referenced at > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js. > When you do that, the legacy alerts UI will no longer work. I guess the > point I'm trying to make is that we would have to switch between them or > have 2 separate application running. I imagine most users only need one or > the other running so probably not an issue. > > Jon, the primary upgrade consideration I see is with authentication. To be > able to use Knox, you would have to upgrade to LDAP-based authentication if > you were still using JDBC-based authentication in REST. The urls would > also change obviously. > > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com wrote: > >> Phew, that was quite the thread to catch up on. >> >> I agree that this should be optional/pluggable to start, and I'm interested >> to hear the issues as they relate to upgrading an existing cluster (given >> the suggested approach) and exposing both legacy and knox URLs at the same >> time. >> >> Jon >> >> On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < >> michael.miklav...@gmail.com> >> wrote: >> >> > A couple more things, and I think this goes without saying - whatever we >> do >> > with Knox should NOT >> > >> > 1. Require unit and integration tests to use Knox >> > 2. Break fulldev >> > >> > Also, I don't know that I saw you mention this, but I'm unsure how we >> > should leverage Knox as a core piece of the platform. i.e. should we make >> > this required or optional? I'm open to hearing opinions on this, but I'm >> > inclined to keep this a pluggable option. >> > >> > Mike >> > >> > >> > On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic < >> > michael.miklav...@gmail.com> wrote: >> > >> > > Thanks for the update Ryan. Per my earlier comments, I thought it might >> > be >> > > the case that we could dramatically simplify this by leveraging Knox's >> > > proxy capabilities, and per your research that appears to be the case. >> > This >> > > is a dramatic simplification and improvement of this feature imo, +1. >> I'm >> > > also +1 on a couple distinct steps that you've laid out: fix the UI >> > issues >> > > in master, then add Knox for SSO. That should help mitigate issues with >> > > merge conflicts with ongoing development. >> > > >> > > > I think it will be a challenge exposing the UIs through both the Knox >> > > url and legacy urls at the same time. >> > > I'm not sure I understand the issue here. Are you referring to this >> > > comment? "Added a ng build option to build the UI with base href set to >> > > Knox base path." Isn't it just a matter of URL rewriting/forwarding? I >> > > thought we'd be exposing the URL's directly in one context, and through >> > > Knox in the other. Either way, it seems like we should be able to >> > provide a >> > > dynamic base path through configuration in our web applications. I'd >> > expect >> > > to modify that property based on whether Knox is configured or not. >> > > >> > > > I'm also not clear on how one would use Knox with REST set to legacy
Re: [DISCUSS] Knox SSO feature branch review and features
What you're looking for is an OUT rewrite rule, and a filter rule on content-type. It's not spectacularly well documented, but https://knox.apache.org/books/knox-1-0-0/dev-guide.html#Rewrite+Provider and specifically https://knox.apache.org/books/knox-1-0-0/dev-guide.html#Rewrite+Steps is a starting point. There are some reasonable examples in Knox itself for the webhdfs service, which uses this mechanism: https://github.com/apache/knox/blob/master/gateway-service-webhdfs/src/main/resources/org/apache/knox/gateway/hdfs/WebHdfsDeploymentContributor/rewrite.xml Hope that helps. It's not well doc-ed sadly, and not massively flexible, but should work. I suspect from my previous experiments with this you may also need to build this file as part of the UI builds, so it is aware of the bundle names generated, because the Knox matching rules don't have proper back reference capabilities. I did a POC of this sometime back in March, that I might be able to dig out if it would help. Simon On Mon, 12 Nov 2018 at 14:59, Ryan Merriman wrote: > I'm just coming up to speed on Knox so maybe rewriting assets links are > trivial. If anyone has a good example of how to do that or can point to > some documentation, please share. > > On Mon, Nov 12, 2018 at 8:54 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > Doing the Knox proxy work first certainly does make a lot of sense vs the > > SSO first approach, so I'm in favour of this. It bypasses all the > anti-CORS > > proxying stuff the other solution needed by being on the same URL space. > > > > Is there are reason we're not re-writing the asset link URLs in Knox? We > > should have a reverse content rewrite rule to avoid that problem and make > > it entirely transparent whether there is Knox or not. We shouldn't be > > changing anything about the UI services themselves. If the rewrite > service > > is complete, there is no change to base ref in the UI code, Knox would > > effectively apply it by content filtering. Note also that the gateway URL > > is configurable and likely to vary from Knox to Knox, so baking it into > the > > ng build will break non-full-dev builds. (e.g. gateway/default could well > > be gateway/xyz). > > > > I would also like to discuss removing the JDBC auth, because it's a set > of > > plaintext passwords in a mysql DB... it introduces a problematic > dependency > > (mysql) a ton of java dependencies we could cut out (JPA, eclipselink) > and > > opens up a massive security hole. I personally know of several > > organisations who are blocked from using Metron by the presence of the > JDBC > > authentication method in its current form. > > > > Simon > > > > On Mon, 12 Nov 2018 at 14:36, Ryan Merriman wrote: > > > > > Let me clarify on exposing both legacy and Knox URLs at the same time. > > The > > > base urls will look something like this: > > > > > > Legacy REST - http://node1:8082/api/v1 > > > Legacy Alerts UI - http://node1:4201:/alerts-list > > > > > > Knox REST - https://node1:8443/gateway/default/metron/api/v1 > > > Knox Alerts UI - > > > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > > > > > If Knox were turned on and the alerts UI deployed as is, it would not > > > work. This is because static assets are referenced with > > > http://node1:4201/assets/some-asset.js which does not include the > > correct > > > context path to the alerts UI in knox. To make it work, you have to > set > > > the base ref to "/gateway/default/metron-alerts-ui" so that static > assets > > > are referenced at > > > > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js > > . > > > When you do that, the legacy alerts UI will no longer work. I guess > the > > > point I'm trying to make is that we would have to switch between them > or > > > have 2 separate application running. I imagine most users only need > one > > or > > > the other running so probably not an issue. > > > > > > Jon, the primary upgrade consideration I see is with authentication. > To > > be > > > able to use Knox, you would have to upgrade to LDAP-based > authentication > > if > > > you were still using JDBC-based authentication in REST. The urls would > > > also change obviously. > > > > > > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com > > wrote: > > > > > > > Phew, that was quite the thread to catch up on. > > > > > > > > I agree that this should be optional/pluggable to start, and I'm > > > interested > > > > to hear the issues as they relate to upgrading an existing cluster > > (given > > > > the suggested approach) and exposing both legacy and knox URLs at the > > > same > > > > time. > > > > > > > > Jon > > > > > > > > On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < > > > > michael.miklav...@gmail.com> > > > > wrote: > > > > > > > > > A couple more things, and I think this goes without saying - > whatever > > > we > > > > do > > > > > with Knox should NOT > > > > > > > > > >1. Require unit and integration tests
Re: [DISCUSS] Knox SSO feature branch review and features
Hello Ryan, I am still catching up on the architecture so let me know if I am misunderstanding anything. You could have multiple serviced deployed in Knox 1. for metron (metron/api/v1) 2. for alerts-ui (metron-alerts-ui/alerts-list) and have them run in one Knox instance and you could have one service reference from other (not recommended but possible). You can tailor rewrite rules to update the context path for the assets as well, pointed out by Simon. Knox Wiki [1] has some blogs and tutorials that you can look at. This [2] is a good tutorial on how rewriting static assets, there is also a blog [3] on basics of rewrite rules that should be a good reference. I would also be glad to look at the service definitions you have and answer any questions. [1] https://cwiki.apache.org/confluence/display/KNOX/Index [2] https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox [3] https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox Best, Sandeep P.S. I am a Knox committer and new to Metron. On Mon, Nov 12, 2018 at 9:59 AM Ryan Merriman wrote: > I'm just coming up to speed on Knox so maybe rewriting assets links are > trivial. If anyone has a good example of how to do that or can point to > some documentation, please share. > > On Mon, Nov 12, 2018 at 8:54 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > Doing the Knox proxy work first certainly does make a lot of sense vs the > > SSO first approach, so I'm in favour of this. It bypasses all the > anti-CORS > > proxying stuff the other solution needed by being on the same URL space. > > > > Is there are reason we're not re-writing the asset link URLs in Knox? We > > should have a reverse content rewrite rule to avoid that problem and make > > it entirely transparent whether there is Knox or not. We shouldn't be > > changing anything about the UI services themselves. If the rewrite > service > > is complete, there is no change to base ref in the UI code, Knox would > > effectively apply it by content filtering. Note also that the gateway URL > > is configurable and likely to vary from Knox to Knox, so baking it into > the > > ng build will break non-full-dev builds. (e.g. gateway/default could well > > be gateway/xyz). > > > > I would also like to discuss removing the JDBC auth, because it's a set > of > > plaintext passwords in a mysql DB... it introduces a problematic > dependency > > (mysql) a ton of java dependencies we could cut out (JPA, eclipselink) > and > > opens up a massive security hole. I personally know of several > > organisations who are blocked from using Metron by the presence of the > JDBC > > authentication method in its current form. > > > > Simon > > > > On Mon, 12 Nov 2018 at 14:36, Ryan Merriman wrote: > > > > > Let me clarify on exposing both legacy and Knox URLs at the same time. > > The > > > base urls will look something like this: > > > > > > Legacy REST - http://node1:8082/api/v1 > > > Legacy Alerts UI - http://node1:4201:/alerts-list > > > > > > Knox REST - https://node1:8443/gateway/default/metron/api/v1 > > > Knox Alerts UI - > > > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > > > > > If Knox were turned on and the alerts UI deployed as is, it would not > > > work. This is because static assets are referenced with > > > http://node1:4201/assets/some-asset.js which does not include the > > correct > > > context path to the alerts UI in knox. To make it work, you have to > set > > > the base ref to "/gateway/default/metron-alerts-ui" so that static > assets > > > are referenced at > > > > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js > > . > > > When you do that, the legacy alerts UI will no longer work. I guess > the > > > point I'm trying to make is that we would have to switch between them > or > > > have 2 separate application running. I imagine most users only need > one > > or > > > the other running so probably not an issue. > > > > > > Jon, the primary upgrade consideration I see is with authentication. > To > > be > > > able to use Knox, you would have to upgrade to LDAP-based > authentication > > if > > > you were still using JDBC-based authentication in REST. The urls would > > > also change obviously. > > > > > > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com > > wrote: > > > > > > > Phew, that was quite the thread to catch up on. > > > > > > > > I agree that this should be optional/pluggable to start, and I'm > > > interested > > > > to hear the issues as they relate to upgrading an existing cluster > > (given > > > > the suggested approach) and exposing both legacy and knox URLs at the > > > same > > > > time. > > > > > > > > Jon > > > > > > > > On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < > > > > michael.miklav...@gmail.com> > > > > wrote: > > > > > > > > > A couple more things, and I think this goes without saying - > whatever > > > we > > > > do > > > > > with Knox should NOT > > > > >
Re: [DISCUSS] Knox SSO feature branch review and features
I'm just coming up to speed on Knox so maybe rewriting assets links are trivial. If anyone has a good example of how to do that or can point to some documentation, please share. On Mon, Nov 12, 2018 at 8:54 AM Simon Elliston Ball < si...@simonellistonball.com> wrote: > Doing the Knox proxy work first certainly does make a lot of sense vs the > SSO first approach, so I'm in favour of this. It bypasses all the anti-CORS > proxying stuff the other solution needed by being on the same URL space. > > Is there are reason we're not re-writing the asset link URLs in Knox? We > should have a reverse content rewrite rule to avoid that problem and make > it entirely transparent whether there is Knox or not. We shouldn't be > changing anything about the UI services themselves. If the rewrite service > is complete, there is no change to base ref in the UI code, Knox would > effectively apply it by content filtering. Note also that the gateway URL > is configurable and likely to vary from Knox to Knox, so baking it into the > ng build will break non-full-dev builds. (e.g. gateway/default could well > be gateway/xyz). > > I would also like to discuss removing the JDBC auth, because it's a set of > plaintext passwords in a mysql DB... it introduces a problematic dependency > (mysql) a ton of java dependencies we could cut out (JPA, eclipselink) and > opens up a massive security hole. I personally know of several > organisations who are blocked from using Metron by the presence of the JDBC > authentication method in its current form. > > Simon > > On Mon, 12 Nov 2018 at 14:36, Ryan Merriman wrote: > > > Let me clarify on exposing both legacy and Knox URLs at the same time. > The > > base urls will look something like this: > > > > Legacy REST - http://node1:8082/api/v1 > > Legacy Alerts UI - http://node1:4201:/alerts-list > > > > Knox REST - https://node1:8443/gateway/default/metron/api/v1 > > Knox Alerts UI - > > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > > > If Knox were turned on and the alerts UI deployed as is, it would not > > work. This is because static assets are referenced with > > http://node1:4201/assets/some-asset.js which does not include the > correct > > context path to the alerts UI in knox. To make it work, you have to set > > the base ref to "/gateway/default/metron-alerts-ui" so that static assets > > are referenced at > > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js > . > > When you do that, the legacy alerts UI will no longer work. I guess the > > point I'm trying to make is that we would have to switch between them or > > have 2 separate application running. I imagine most users only need one > or > > the other running so probably not an issue. > > > > Jon, the primary upgrade consideration I see is with authentication. To > be > > able to use Knox, you would have to upgrade to LDAP-based authentication > if > > you were still using JDBC-based authentication in REST. The urls would > > also change obviously. > > > > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com > wrote: > > > > > Phew, that was quite the thread to catch up on. > > > > > > I agree that this should be optional/pluggable to start, and I'm > > interested > > > to hear the issues as they relate to upgrading an existing cluster > (given > > > the suggested approach) and exposing both legacy and knox URLs at the > > same > > > time. > > > > > > Jon > > > > > > On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < > > > michael.miklav...@gmail.com> > > > wrote: > > > > > > > A couple more things, and I think this goes without saying - whatever > > we > > > do > > > > with Knox should NOT > > > > > > > >1. Require unit and integration tests to use Knox > > > >2. Break fulldev > > > > > > > > Also, I don't know that I saw you mention this, but I'm unsure how we > > > > should leverage Knox as a core piece of the platform. i.e. should we > > make > > > > this required or optional? I'm open to hearing opinions on this, but > > I'm > > > > inclined to keep this a pluggable option. > > > > > > > > Mike > > > > > > > > > > > > On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic < > > > > michael.miklav...@gmail.com> wrote: > > > > > > > > > Thanks for the update Ryan. Per my earlier comments, I thought it > > might > > > > be > > > > > the case that we could dramatically simplify this by leveraging > > Knox's > > > > > proxy capabilities, and per your research that appears to be the > > case. > > > > This > > > > > is a dramatic simplification and improvement of this feature imo, > +1. > > > I'm > > > > > also +1 on a couple distinct steps that you've laid out: fix the UI > > > > issues > > > > > in master, then add Knox for SSO. That should help mitigate issues > > with > > > > > merge conflicts with ongoing development. > > > > > > > > > > > I think it will be a challenge exposing the UIs through both the > > Knox > > > > > url and legacy urls at the same time. > > > > > I'm not
Re: [DISCUSS] Knox SSO feature branch review and features
Phew, that was quite the thread to catch up on. I agree that this should be optional/pluggable to start, and I'm interested to hear the issues as they relate to upgrading an existing cluster (given the suggested approach) and exposing both legacy and knox URLs at the same time. Jon On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic wrote: > A couple more things, and I think this goes without saying - whatever we do > with Knox should NOT > >1. Require unit and integration tests to use Knox >2. Break fulldev > > Also, I don't know that I saw you mention this, but I'm unsure how we > should leverage Knox as a core piece of the platform. i.e. should we make > this required or optional? I'm open to hearing opinions on this, but I'm > inclined to keep this a pluggable option. > > Mike > > > On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Thanks for the update Ryan. Per my earlier comments, I thought it might > be > > the case that we could dramatically simplify this by leveraging Knox's > > proxy capabilities, and per your research that appears to be the case. > This > > is a dramatic simplification and improvement of this feature imo, +1. I'm > > also +1 on a couple distinct steps that you've laid out: fix the UI > issues > > in master, then add Knox for SSO. That should help mitigate issues with > > merge conflicts with ongoing development. > > > > > I think it will be a challenge exposing the UIs through both the Knox > > url and legacy urls at the same time. > > I'm not sure I understand the issue here. Are you referring to this > > comment? "Added a ng build option to build the UI with base href set to > > Knox base path." Isn't it just a matter of URL rewriting/forwarding? I > > thought we'd be exposing the URL's directly in one context, and through > > Knox in the other. Either way, it seems like we should be able to > provide a > > dynamic base path through configuration in our web applications. I'd > expect > > to modify that property based on whether Knox is configured or not. > > > > > I'm also not clear on how one would use Knox with REST set to legacy > > JDBC-based authentication. As far as I know Knox does not support JDBC so > > there would be a mismatch between Knox and REST. > > I'm OK with not having Knox work with JDBC. That's a feature of Knox and > > probably not something we care much about. > > > > >We could initially make Knox an optional feature that requires setup > with > > the help of some documentation (like Kerberos) while keeping the system > the > > way it is now by default. > > Sounds good to me. > > > > > I imagine we'll deprecate JDBC-based authentication at some point so > > that may be a good time to switch. > > I would like to announce deprecation in our next release and move to > > remove it in a following release. > > > > Thanks for taking this on and great job laying things out. > > > > Thanks, > > Mike > > > > On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman > wrote: > > > >> I have spent some time recently reviewing this discussion and the > feature > >> branch that Simon put out. I think this is an important feature and > want > >> to move it forward. I started another discussion on adding Knox to our > >> stack but this discussion has a lot of good context so I will continue > it > >> here. > >> > >> I think the main point of contention was that this feature branch > included > >> several different architectural changes and it was unclear if they were > >> needed and if so, could be done separately. Fortunately LDAP > >> authentication has been accepted into master so we can cross it off the > >> list. From my understanding of the points people have made, that leaves > >> Knox related SSO changes and migrating expressjs to a different, > JVM-based > >> web server that includes proxying capabilities (Zuul). > >> > >> I think everyone agrees that if we can limit the scope to just Knox > >> related > >> SSO changes that would be ideal. I believe I have found a way to do > that > >> while working on a small POC this week. The key to this (Simon alluded > to > >> it earlier) is to put both REST and our UIs behind Knox. I initially > was > >> focused on just adding REST as a service in Knox and decided to > experiment > >> with also adding our UIs. After I did this it became clear that this > >> simplifies things considerably: > >> > >>- The REST app and the UIs are now served from the same host so CORS > >>concerns go away. > >>- We no longer need to worry about proxying REST requests from the > UIs > >>with express or Zuul because Knox handles that for us. This will > make > >> our > >>express configuration even simpler. In fact, all we need is a simple > >> way > >>to serve static UI assets. > >>- We no longer need to check for SSO tokens and redirect in the UI > >>web/app servers (or the REST app for that matter) because Knox > handles > >> that > >>for us. > >>-
Re: [DISCUSS] Knox SSO feature branch review and features
A couple more things, and I think this goes without saying - whatever we do with Knox should NOT 1. Require unit and integration tests to use Knox 2. Break fulldev Also, I don't know that I saw you mention this, but I'm unsure how we should leverage Knox as a core piece of the platform. i.e. should we make this required or optional? I'm open to hearing opinions on this, but I'm inclined to keep this a pluggable option. Mike On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Thanks for the update Ryan. Per my earlier comments, I thought it might be > the case that we could dramatically simplify this by leveraging Knox's > proxy capabilities, and per your research that appears to be the case. This > is a dramatic simplification and improvement of this feature imo, +1. I'm > also +1 on a couple distinct steps that you've laid out: fix the UI issues > in master, then add Knox for SSO. That should help mitigate issues with > merge conflicts with ongoing development. > > > I think it will be a challenge exposing the UIs through both the Knox > url and legacy urls at the same time. > I'm not sure I understand the issue here. Are you referring to this > comment? "Added a ng build option to build the UI with base href set to > Knox base path." Isn't it just a matter of URL rewriting/forwarding? I > thought we'd be exposing the URL's directly in one context, and through > Knox in the other. Either way, it seems like we should be able to provide a > dynamic base path through configuration in our web applications. I'd expect > to modify that property based on whether Knox is configured or not. > > > I'm also not clear on how one would use Knox with REST set to legacy > JDBC-based authentication. As far as I know Knox does not support JDBC so > there would be a mismatch between Knox and REST. > I'm OK with not having Knox work with JDBC. That's a feature of Knox and > probably not something we care much about. > > >We could initially make Knox an optional feature that requires setup with > the help of some documentation (like Kerberos) while keeping the system the > way it is now by default. > Sounds good to me. > > > I imagine we'll deprecate JDBC-based authentication at some point so > that may be a good time to switch. > I would like to announce deprecation in our next release and move to > remove it in a following release. > > Thanks for taking this on and great job laying things out. > > Thanks, > Mike > > On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman wrote: > >> I have spent some time recently reviewing this discussion and the feature >> branch that Simon put out. I think this is an important feature and want >> to move it forward. I started another discussion on adding Knox to our >> stack but this discussion has a lot of good context so I will continue it >> here. >> >> I think the main point of contention was that this feature branch included >> several different architectural changes and it was unclear if they were >> needed and if so, could be done separately. Fortunately LDAP >> authentication has been accepted into master so we can cross it off the >> list. From my understanding of the points people have made, that leaves >> Knox related SSO changes and migrating expressjs to a different, JVM-based >> web server that includes proxying capabilities (Zuul). >> >> I think everyone agrees that if we can limit the scope to just Knox >> related >> SSO changes that would be ideal. I believe I have found a way to do that >> while working on a small POC this week. The key to this (Simon alluded to >> it earlier) is to put both REST and our UIs behind Knox. I initially was >> focused on just adding REST as a service in Knox and decided to experiment >> with also adding our UIs. After I did this it became clear that this >> simplifies things considerably: >> >>- The REST app and the UIs are now served from the same host so CORS >>concerns go away. >>- We no longer need to worry about proxying REST requests from the UIs >>with express or Zuul because Knox handles that for us. This will make >> our >>express configuration even simpler. In fact, all we need is a simple >> way >>to serve static UI assets. >>- We no longer need to check for SSO tokens and redirect in the UI >>web/app servers (or the REST app for that matter) because Knox handles >> that >>for us. >>- The UIs can now easily access any Knox service (not just our REST >> app) >>without any extra proxy configuration. >>- SSO token authentication is only necessary in REST so there is no >> need >>to create shared Spring modules or split functionality out. >> >> The most significant change I had to make (borrowed from Simon's feature >> branch) was the SSO token authentication mentioned above. The primary >> short term benefit with this approach is that outside of some general >> deficiencies unrelated to this our UI architecture doesn't need to >>
Re: [DISCUSS] Knox SSO feature branch review and features
Thanks for the update Ryan. Per my earlier comments, I thought it might be the case that we could dramatically simplify this by leveraging Knox's proxy capabilities, and per your research that appears to be the case. This is a dramatic simplification and improvement of this feature imo, +1. I'm also +1 on a couple distinct steps that you've laid out: fix the UI issues in master, then add Knox for SSO. That should help mitigate issues with merge conflicts with ongoing development. > I think it will be a challenge exposing the UIs through both the Knox url and legacy urls at the same time. I'm not sure I understand the issue here. Are you referring to this comment? "Added a ng build option to build the UI with base href set to Knox base path." Isn't it just a matter of URL rewriting/forwarding? I thought we'd be exposing the URL's directly in one context, and through Knox in the other. Either way, it seems like we should be able to provide a dynamic base path through configuration in our web applications. I'd expect to modify that property based on whether Knox is configured or not. > I'm also not clear on how one would use Knox with REST set to legacy JDBC-based authentication. As far as I know Knox does not support JDBC so there would be a mismatch between Knox and REST. I'm OK with not having Knox work with JDBC. That's a feature of Knox and probably not something we care much about. >We could initially make Knox an optional feature that requires setup with the help of some documentation (like Kerberos) while keeping the system the way it is now by default. Sounds good to me. > I imagine we'll deprecate JDBC-based authentication at some point so that may be a good time to switch. I would like to announce deprecation in our next release and move to remove it in a following release. Thanks for taking this on and great job laying things out. Thanks, Mike On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman wrote: > I have spent some time recently reviewing this discussion and the feature > branch that Simon put out. I think this is an important feature and want > to move it forward. I started another discussion on adding Knox to our > stack but this discussion has a lot of good context so I will continue it > here. > > I think the main point of contention was that this feature branch included > several different architectural changes and it was unclear if they were > needed and if so, could be done separately. Fortunately LDAP > authentication has been accepted into master so we can cross it off the > list. From my understanding of the points people have made, that leaves > Knox related SSO changes and migrating expressjs to a different, JVM-based > web server that includes proxying capabilities (Zuul). > > I think everyone agrees that if we can limit the scope to just Knox related > SSO changes that would be ideal. I believe I have found a way to do that > while working on a small POC this week. The key to this (Simon alluded to > it earlier) is to put both REST and our UIs behind Knox. I initially was > focused on just adding REST as a service in Knox and decided to experiment > with also adding our UIs. After I did this it became clear that this > simplifies things considerably: > >- The REST app and the UIs are now served from the same host so CORS >concerns go away. >- We no longer need to worry about proxying REST requests from the UIs >with express or Zuul because Knox handles that for us. This will make > our >express configuration even simpler. In fact, all we need is a simple > way >to serve static UI assets. >- We no longer need to check for SSO tokens and redirect in the UI >web/app servers (or the REST app for that matter) because Knox handles > that >for us. >- The UIs can now easily access any Knox service (not just our REST app) >without any extra proxy configuration. >- SSO token authentication is only necessary in REST so there is no need >to create shared Spring modules or split functionality out. > > The most significant change I had to make (borrowed from Simon's feature > branch) was the SSO token authentication mentioned above. The primary > short term benefit with this approach is that outside of some general > deficiencies unrelated to this our UI architecture doesn't need to > fundamentally change. I could summarize the changes as: > >- Knox install and configuration (setting up REST and the alerts UI as >Knox services) >- Added Knox SSO token authentication to REST >- Updated REST urls in the UI code (should be configurable) >- Fixed a few UI bugs where relative paths were not being used >- Added a ng build option to build the UI with base href set to Knox >base path ( > > https://github.com/angular/angular-cli/wiki/build#base-tag-handling-in-indexhtml >) > > Most the UI changes are preexisting, minor issues that could be fixed > directly in master. We would need to think of an
Re: [DISCUSS] Knox SSO feature branch review and features
I have spent some time recently reviewing this discussion and the feature branch that Simon put out. I think this is an important feature and want to move it forward. I started another discussion on adding Knox to our stack but this discussion has a lot of good context so I will continue it here. I think the main point of contention was that this feature branch included several different architectural changes and it was unclear if they were needed and if so, could be done separately. Fortunately LDAP authentication has been accepted into master so we can cross it off the list. From my understanding of the points people have made, that leaves Knox related SSO changes and migrating expressjs to a different, JVM-based web server that includes proxying capabilities (Zuul). I think everyone agrees that if we can limit the scope to just Knox related SSO changes that would be ideal. I believe I have found a way to do that while working on a small POC this week. The key to this (Simon alluded to it earlier) is to put both REST and our UIs behind Knox. I initially was focused on just adding REST as a service in Knox and decided to experiment with also adding our UIs. After I did this it became clear that this simplifies things considerably: - The REST app and the UIs are now served from the same host so CORS concerns go away. - We no longer need to worry about proxying REST requests from the UIs with express or Zuul because Knox handles that for us. This will make our express configuration even simpler. In fact, all we need is a simple way to serve static UI assets. - We no longer need to check for SSO tokens and redirect in the UI web/app servers (or the REST app for that matter) because Knox handles that for us. - The UIs can now easily access any Knox service (not just our REST app) without any extra proxy configuration. - SSO token authentication is only necessary in REST so there is no need to create shared Spring modules or split functionality out. The most significant change I had to make (borrowed from Simon's feature branch) was the SSO token authentication mentioned above. The primary short term benefit with this approach is that outside of some general deficiencies unrelated to this our UI architecture doesn't need to fundamentally change. I could summarize the changes as: - Knox install and configuration (setting up REST and the alerts UI as Knox services) - Added Knox SSO token authentication to REST - Updated REST urls in the UI code (should be configurable) - Fixed a few UI bugs where relative paths were not being used - Added a ng build option to build the UI with base href set to Knox base path ( https://github.com/angular/angular-cli/wiki/build#base-tag-handling-in-indexhtml ) Most the UI changes are preexisting, minor issues that could be fixed directly in master. We would need to think of an approach for the base href build requirement but I'm sure it's not that bad. However there will be some backwards compatibility issues we would need to think through. I think it will be a challenge exposing the UIs through both the Knox url and legacy urls at the same time. I'm also not clear on how one would use Knox with REST set to legacy JDBC-based authentication. As far as I know Knox does not support JDBC so there would be a mismatch between Knox and REST. Knox does have the ability to pass along basic authentication headers so LDAP in REST would work. We could initially make Knox an optional feature that requires setup with the help of some documentation (like Kerberos) while keeping the system the way it is now by default. I imagine we'll deprecate JDBC-based authentication at some point so that may be a good time to switch. What do people think about this approach? Concerns? Are there any huge holes in this I'm not thinking about? I want to highlight that the work Simon did in his feature branch was crucial to better understanding this. I am pretty sure we'll end up reusing a lot code from that branch. On Thu, Sep 27, 2018 at 6:30 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Apparently, I hit send on my last email before finishing my synopsis (per > @Otto's Q in Slack). To summarize, based on my current understanding I > believe that each of the feature branch changes I've outline above are > units of work that are related, yet should be executed on independently. > Knox SSO in its own feature branch. Migrating technologies like NodeJs or > migrating the auth DB to LDAP seem like they belong in their own separate > PR's or feature branches. > > Thanks, > Mike > > On Thu, Sep 27, 2018 at 4:08 PM Casey Stella wrote: > > > I'm coming in late to the game here, but for my mind a feature branch > > should involve the minimum architectural change to accomplish a given > > feature. > > The feature in question is SSO integration. It seems to me that the > > operative question is can we do the
Re: [DISCUSS] Knox SSO feature branch review and features
Apparently, I hit send on my last email before finishing my synopsis (per @Otto's Q in Slack). To summarize, based on my current understanding I believe that each of the feature branch changes I've outline above are units of work that are related, yet should be executed on independently. Knox SSO in its own feature branch. Migrating technologies like NodeJs or migrating the auth DB to LDAP seem like they belong in their own separate PR's or feature branches. Thanks, Mike On Thu, Sep 27, 2018 at 4:08 PM Casey Stella wrote: > I'm coming in late to the game here, but for my mind a feature branch > should involve the minimum architectural change to accomplish a given > feature. > The feature in question is SSO integration. It seems to me that the > operative question is can we do the feature without making the OTHER > architectural change > (e.g. migrating from expressjs to spring boot + zuul). I would argue that > if we WANT to do that, then it should be a separate feature branch. > > Thus, I leave with a question: is there a way to accomplish this feature > without ripping out expressjs? > >- If so and it is feasible, I would argue that we should decouple this >into a separate feature branch. >- If so and it is infeasible, I'd like to hear an argument as to the >infeasibility and let's decide given that >- If it is not possible, then I'd argue that we should keep them coupled >and move this through as-is. > > On a side-note, it feels a bit weird that we're narrowing to a bundled > proxy, rather than having that be a pluggable thing. I'm not super > knowledgeable in this space, so I apologize > in advance if this is naive, but isn't this a pluggable, external component > (e.g. nginx)? > > On Thu, Sep 27, 2018 at 5:05 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > I've spent some more time reading through Simon's response and the added > > sequence diagram. This is definitely helpful - thank you Simon. > > > > I need to redact my initial list: > > > >1. Node migrated to Spring Boot, expressjs migrated to a > >non-JS/non-NodeJs proxying mechanism (ie Zuul in this case) > >2. JDBC removed completely in favor of LDAP > >3. Knox/SSO > > > > I'm a bit conflicted on the best way to move forward and would like some > > thoughts from other community members on this. I think an argument can be > > made that 1 and 2 are independent of 3, and should/could really be > > independent PR's against master. > > > > The need for a replacement for expressjs (Zuul in this case) is an > artifact > > that our request/response cycle for REST calls is a simple matter of > > forwarding with some additional headers for authentication. There's a > > JSESSIONID managed by the client browser in our current architecture, for > > example. You login to the alerts or the management UI which forwards a > > request to REST, which looks up credentials in a backend database, and > > passes the results back up the chain. All browser requests go directly to > > the specific UI you're working with - this is the CORS problem. You > can't, > > without some effort with headers for adding other domains to the safe > list > > or disabling the security check for CORS, make remote calls directly to > > REST. That's why we proxy. Switching over to Spring Boot leaves a gap > with > > expressjs having handled the proxying and filtering, since it's only > > available to a NodeJs application (it's server-side javascript vs the > > client side javascript deployed via our Angular applications). Enter > Zuul, > > which now effectively handles that. At runtime, Zuul is a part of the > > Spring app that serves up our UI's. It handles the requests via > filtering, > > forwards them to REST, manages the response back to the client. Very > > similar to what expressjs was doing, per my current understanding. The > > sequence diagrams Simon added are useful, and I think some of what was > less > > clear was what we currently vs what the new changes are doing to the > > architecture. This is no fault of Simon's - there simply wasn't any > > architecture diagrams/documents around this before. Here's my impression > of > > the very very basic current state - someone more familiar with this > > architecture please advise if I'm incorrect about anything (probably > Ryan). > > > > https://imgur.com/f8GtSmh > > > > Zuul would be replacing the bit about expressjs in the diagram, and > instead > > of node we have spring boot. This covers 1. 2 and 3 are other issues. I'd > > like to see similar exposition of those server processes with knox > > involved. I imagine in that case we bump up from 3 to 4 server instances > > for the additional knox endpoint. > > > > Mike > > > > > > > > > > > > On Wed, Sep 19, 2018 at 11:28 AM James Sirota > wrote: > > > > > Thank you, Simon. The diagrams help a lot > > > > > > 19.09.2018, 21:27, "Simon Elliston Ball" >: > > > > To clarify some of this I've put some documentation
Re: [DISCUSS] Knox SSO feature branch review and features
I'm coming in late to the game here, but for my mind a feature branch should involve the minimum architectural change to accomplish a given feature. The feature in question is SSO integration. It seems to me that the operative question is can we do the feature without making the OTHER architectural change (e.g. migrating from expressjs to spring boot + zuul). I would argue that if we WANT to do that, then it should be a separate feature branch. Thus, I leave with a question: is there a way to accomplish this feature without ripping out expressjs? - If so and it is feasible, I would argue that we should decouple this into a separate feature branch. - If so and it is infeasible, I'd like to hear an argument as to the infeasibility and let's decide given that - If it is not possible, then I'd argue that we should keep them coupled and move this through as-is. On a side-note, it feels a bit weird that we're narrowing to a bundled proxy, rather than having that be a pluggable thing. I'm not super knowledgeable in this space, so I apologize in advance if this is naive, but isn't this a pluggable, external component (e.g. nginx)? On Thu, Sep 27, 2018 at 5:05 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I've spent some more time reading through Simon's response and the added > sequence diagram. This is definitely helpful - thank you Simon. > > I need to redact my initial list: > >1. Node migrated to Spring Boot, expressjs migrated to a >non-JS/non-NodeJs proxying mechanism (ie Zuul in this case) >2. JDBC removed completely in favor of LDAP >3. Knox/SSO > > I'm a bit conflicted on the best way to move forward and would like some > thoughts from other community members on this. I think an argument can be > made that 1 and 2 are independent of 3, and should/could really be > independent PR's against master. > > The need for a replacement for expressjs (Zuul in this case) is an artifact > that our request/response cycle for REST calls is a simple matter of > forwarding with some additional headers for authentication. There's a > JSESSIONID managed by the client browser in our current architecture, for > example. You login to the alerts or the management UI which forwards a > request to REST, which looks up credentials in a backend database, and > passes the results back up the chain. All browser requests go directly to > the specific UI you're working with - this is the CORS problem. You can't, > without some effort with headers for adding other domains to the safe list > or disabling the security check for CORS, make remote calls directly to > REST. That's why we proxy. Switching over to Spring Boot leaves a gap with > expressjs having handled the proxying and filtering, since it's only > available to a NodeJs application (it's server-side javascript vs the > client side javascript deployed via our Angular applications). Enter Zuul, > which now effectively handles that. At runtime, Zuul is a part of the > Spring app that serves up our UI's. It handles the requests via filtering, > forwards them to REST, manages the response back to the client. Very > similar to what expressjs was doing, per my current understanding. The > sequence diagrams Simon added are useful, and I think some of what was less > clear was what we currently vs what the new changes are doing to the > architecture. This is no fault of Simon's - there simply wasn't any > architecture diagrams/documents around this before. Here's my impression of > the very very basic current state - someone more familiar with this > architecture please advise if I'm incorrect about anything (probably Ryan). > > https://imgur.com/f8GtSmh > > Zuul would be replacing the bit about expressjs in the diagram, and instead > of node we have spring boot. This covers 1. 2 and 3 are other issues. I'd > like to see similar exposition of those server processes with knox > involved. I imagine in that case we bump up from 3 to 4 server instances > for the additional knox endpoint. > > Mike > > > > > > On Wed, Sep 19, 2018 at 11:28 AM James Sirota wrote: > > > Thank you, Simon. The diagrams help a lot > > > > 19.09.2018, 21:27, "Simon Elliston Ball" : > > > To clarify some of this I've put some documentation into > > > https://github.com/apache/metron/pull/1203 under METRON-1755 ( > > > https://issues.apache.org/jira/browse/METRON-1755). Hopefully the > > diagrams > > > there should make it clearer. > > > > > > Simon > > > > > > On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < > > > si...@simonellistonball.com> wrote: > > > > > >> Hi Mike, > > >> > > >> Some good points here which could do with some clarification. I > suspect > > >> the architecture documentation could be clearer and fill in some of > > these > > >> gaps, and I'll have a look at working on that and providing some > > diagrams. > > >> > > >> The short version is that the Zuul proxy gateway has been added to > > replace > > >> the Nodejs express
Re: [DISCUSS] Knox SSO feature branch review and features
I've spent some more time reading through Simon's response and the added sequence diagram. This is definitely helpful - thank you Simon. I need to redact my initial list: 1. Node migrated to Spring Boot, expressjs migrated to a non-JS/non-NodeJs proxying mechanism (ie Zuul in this case) 2. JDBC removed completely in favor of LDAP 3. Knox/SSO I'm a bit conflicted on the best way to move forward and would like some thoughts from other community members on this. I think an argument can be made that 1 and 2 are independent of 3, and should/could really be independent PR's against master. The need for a replacement for expressjs (Zuul in this case) is an artifact that our request/response cycle for REST calls is a simple matter of forwarding with some additional headers for authentication. There's a JSESSIONID managed by the client browser in our current architecture, for example. You login to the alerts or the management UI which forwards a request to REST, which looks up credentials in a backend database, and passes the results back up the chain. All browser requests go directly to the specific UI you're working with - this is the CORS problem. You can't, without some effort with headers for adding other domains to the safe list or disabling the security check for CORS, make remote calls directly to REST. That's why we proxy. Switching over to Spring Boot leaves a gap with expressjs having handled the proxying and filtering, since it's only available to a NodeJs application (it's server-side javascript vs the client side javascript deployed via our Angular applications). Enter Zuul, which now effectively handles that. At runtime, Zuul is a part of the Spring app that serves up our UI's. It handles the requests via filtering, forwards them to REST, manages the response back to the client. Very similar to what expressjs was doing, per my current understanding. The sequence diagrams Simon added are useful, and I think some of what was less clear was what we currently vs what the new changes are doing to the architecture. This is no fault of Simon's - there simply wasn't any architecture diagrams/documents around this before. Here's my impression of the very very basic current state - someone more familiar with this architecture please advise if I'm incorrect about anything (probably Ryan). https://imgur.com/f8GtSmh Zuul would be replacing the bit about expressjs in the diagram, and instead of node we have spring boot. This covers 1. 2 and 3 are other issues. I'd like to see similar exposition of those server processes with knox involved. I imagine in that case we bump up from 3 to 4 server instances for the additional knox endpoint. Mike On Wed, Sep 19, 2018 at 11:28 AM James Sirota wrote: > Thank you, Simon. The diagrams help a lot > > 19.09.2018, 21:27, "Simon Elliston Ball" : > > To clarify some of this I've put some documentation into > > https://github.com/apache/metron/pull/1203 under METRON-1755 ( > > https://issues.apache.org/jira/browse/METRON-1755). Hopefully the > diagrams > > there should make it clearer. > > > > Simon > > > > On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < > > si...@simonellistonball.com> wrote: > > > >> Hi Mike, > >> > >> Some good points here which could do with some clarification. I suspect > >> the architecture documentation could be clearer and fill in some of > these > >> gaps, and I'll have a look at working on that and providing some > diagrams. > >> > >> The short version is that the Zuul proxy gateway has been added to > replace > >> the Nodejs express proxy used to gateway the REST api calls in the > current > >> hosts. This is done in both cases to avoid CORS restrictions by > allowing > >> the same host that serves the UI files to proxy call to the API. > >> > >> The choice of Zuul was partly a pragmatic one (it's the one that's > there > >> in the box as it were with Spring Boot, which we use for the REST API, > via > >> the Spring Cloud Netflix project which wraps a bunch of related pieces > into > >> Spring). The choice of Spring Boot to host the UIs themselves was > similarly > >> for parity with the REST host, to simplify the stack (we remove the > >> occasionally problematic need to install nodejs on target servers, > which is > >> outside of the regular OS and HDP stacks we support). > >> > >> Arguably, the Zuul proxy is not necessary if we force everything > through a > >> Knox instance, since Knox would provide a single endpoint. We probably > >> however don't want to force Knox and SSL, hence using Zuul to keep it > >> closer to our current architecture. Zuul does some other nice things, > which > >> might help us in future, so it's really about laying down some options > for > >> potentially doing micro-services style things at a later date. I'm not > >> saying we have to, or even should go that way, it will just make life > >> easier later if we decide to. It will also help us if we want to add > HA, >
Re: [DISCUSS] Knox SSO feature branch review and features
Thank you, Simon. The diagrams help a lot 19.09.2018, 21:27, "Simon Elliston Ball" : > To clarify some of this I've put some documentation into > https://github.com/apache/metron/pull/1203 under METRON-1755 ( > https://issues.apache.org/jira/browse/METRON-1755). Hopefully the diagrams > there should make it clearer. > > Simon > > On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> Hi Mike, >> >> Some good points here which could do with some clarification. I suspect >> the architecture documentation could be clearer and fill in some of these >> gaps, and I'll have a look at working on that and providing some diagrams. >> >> The short version is that the Zuul proxy gateway has been added to replace >> the Nodejs express proxy used to gateway the REST api calls in the current >> hosts. This is done in both cases to avoid CORS restrictions by allowing >> the same host that serves the UI files to proxy call to the API. >> >> The choice of Zuul was partly a pragmatic one (it's the one that's there >> in the box as it were with Spring Boot, which we use for the REST API, via >> the Spring Cloud Netflix project which wraps a bunch of related pieces into >> Spring). The choice of Spring Boot to host the UIs themselves was similarly >> for parity with the REST host, to simplify the stack (we remove the >> occasionally problematic need to install nodejs on target servers, which is >> outside of the regular OS and HDP stacks we support). >> >> Arguably, the Zuul proxy is not necessary if we force everything through a >> Knox instance, since Knox would provide a single endpoint. We probably >> however don't want to force Knox and SSL, hence using Zuul to keep it >> closer to our current architecture. Zuul does some other nice things, which >> might help us in future, so it's really about laying down some options for >> potentially doing micro-services style things at a later date. I'm not >> saying we have to, or even should go that way, it will just make life >> easier later if we decide to. It will also help us if we want to add HA, >> circuit breaking etc to the architecture at a later date. That said, I >> regret that I ever said the word micro-services, since it's caused >> confusion. Just think of it as a proxy to deal with the CORS problem. >> >> Zuul is implemented as a set of filters, but we are not using it for its >> authentication filtering. We're using it as a proxy. Shiro is an >> authentication framework, and could arguably used to provide the security >> piece, but frankly wrapping shiro as a replacement for Spring Security in a >> Spring application seemed like it will make life a lot harder. This could >> be done, but it's not the native happy path, and would pull in additional >> dependencies that duplicate functionality that's already embedded in Spring >> Security. >> >> The version of Knox used is the default from HDP. The link version you >> mention is a docs link. I'll update it to be the older version, which is >> the same and we can decide if we want to maintain the freshness of it when >> we look to upgrade underlying patterns. Either way, the content is the >> same. >> >> I did consider other hosting mechanisms, including Undertow a >> >> If you have a different suggestion to using the Spring default ways of >> doing things, or we want to use a framework other than Spring for this, >> then maybe we could change to that, but the route chosen here is definitely >> the easy path in the context of the decision made to use Spring in metron >> rest, and if anything opens up our choices while minimising, in fact >> reducing, our dependency management overhead. >> >> I hope that explains some of the thinking behind the choices made, but the >> guiding principals I followed were: >> * Don't fight the framework if you don't have to >> * Reduce the need for additional installation pieces and third party repos >> * Minimize dependencies we would have to manage >> * Avoid excessive change of the architecture, or forcing users to adopt >> Knox if they didn't want the SSL overhead. >> >> Simon >> >> On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >>> Thanks for the write-up Ryan, this is a great start. I have some further >>> questions based on your feedback and in addition to my initial thread. >>> >>> Just for clarification, what version of Knox are we using? HDP 2.6.5, >>> which >>> is what we currently run full dev against, supports 0.12.0. >>> >>> >>> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html >>> . >>> I see references to Knox 1.1.0 (latest) in this committed PR - >>> >>> >>> https://github.com/apache/metron/pull//files#diff-70b412194819f3cb829566f05d77c1a6R122 >>> . >>> This is probably just a super small mismatch, and it probably goes without >>> saying, but I just want to be
Re: [DISCUSS] Knox SSO feature branch review and features
To clarify some of this I've put some documentation into https://github.com/apache/metron/pull/1203 under METRON-1755 ( https://issues.apache.org/jira/browse/METRON-1755). Hopefully the diagrams there should make it clearer. Simon On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Hi Mike, > > Some good points here which could do with some clarification. I suspect > the architecture documentation could be clearer and fill in some of these > gaps, and I'll have a look at working on that and providing some diagrams. > > The short version is that the Zuul proxy gateway has been added to replace > the Nodejs express proxy used to gateway the REST api calls in the current > hosts. This is done in both cases to avoid CORS restrictions by allowing > the same host that serves the UI files to proxy call to the API. > > The choice of Zuul was partly a pragmatic one (it's the one that's there > in the box as it were with Spring Boot, which we use for the REST API, via > the Spring Cloud Netflix project which wraps a bunch of related pieces into > Spring). The choice of Spring Boot to host the UIs themselves was similarly > for parity with the REST host, to simplify the stack (we remove the > occasionally problematic need to install nodejs on target servers, which is > outside of the regular OS and HDP stacks we support). > > Arguably, the Zuul proxy is not necessary if we force everything through a > Knox instance, since Knox would provide a single endpoint. We probably > however don't want to force Knox and SSL, hence using Zuul to keep it > closer to our current architecture. Zuul does some other nice things, which > might help us in future, so it's really about laying down some options for > potentially doing micro-services style things at a later date. I'm not > saying we have to, or even should go that way, it will just make life > easier later if we decide to. It will also help us if we want to add HA, > circuit breaking etc to the architecture at a later date. That said, I > regret that I ever said the word micro-services, since it's caused > confusion. Just think of it as a proxy to deal with the CORS problem. > > Zuul is implemented as a set of filters, but we are not using it for its > authentication filtering. We're using it as a proxy. Shiro is an > authentication framework, and could arguably used to provide the security > piece, but frankly wrapping shiro as a replacement for Spring Security in a > Spring application seemed like it will make life a lot harder. This could > be done, but it's not the native happy path, and would pull in additional > dependencies that duplicate functionality that's already embedded in Spring > Security. > > The version of Knox used is the default from HDP. The link version you > mention is a docs link. I'll update it to be the older version, which is > the same and we can decide if we want to maintain the freshness of it when > we look to upgrade underlying patterns. Either way, the content is the > same. > > I did consider other hosting mechanisms, including Undertow a > > If you have a different suggestion to using the Spring default ways of > doing things, or we want to use a framework other than Spring for this, > then maybe we could change to that, but the route chosen here is definitely > the easy path in the context of the decision made to use Spring in metron > rest, and if anything opens up our choices while minimising, in fact > reducing, our dependency management overhead. > > I hope that explains some of the thinking behind the choices made, but the > guiding principals I followed were: > * Don't fight the framework if you don't have to > * Reduce the need for additional installation pieces and third party repos > * Minimize dependencies we would have to manage > * Avoid excessive change of the architecture, or forcing users to adopt > Knox if they didn't want the SSL overhead. > > Simon > > > On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Thanks for the write-up Ryan, this is a great start. I have some further >> questions based on your feedback and in addition to my initial thread. >> >> Just for clarification, what version of Knox are we using? HDP 2.6.5, >> which >> is what we currently run full dev against, supports 0.12.0. >> >> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html >> . >> I see references to Knox 1.1.0 (latest) in this committed PR - >> >> https://github.com/apache/metron/pull//files#diff-70b412194819f3cb829566f05d77c1a6R122 >> . >> This is probably just a super small mismatch, and it probably goes without >> saying, but I just want to be doubly sure that we're installing the >> default >> via the standard install mechanism as opposed to something separate and >> manual. >> >> On the subject of Zuul wrt Nodejs filters. I'd like to hear some more >> detail on: >> >>1. Why do we need filtering via
Re: [DISCUSS] Knox SSO feature branch review and features
Hi Mike, Some good points here which could do with some clarification. I suspect the architecture documentation could be clearer and fill in some of these gaps, and I'll have a look at working on that and providing some diagrams. The short version is that the Zuul proxy gateway has been added to replace the Nodejs express proxy used to gateway the REST api calls in the current hosts. This is done in both cases to avoid CORS restrictions by allowing the same host that serves the UI files to proxy call to the API. The choice of Zuul was partly a pragmatic one (it's the one that's there in the box as it were with Spring Boot, which we use for the REST API, via the Spring Cloud Netflix project which wraps a bunch of related pieces into Spring). The choice of Spring Boot to host the UIs themselves was similarly for parity with the REST host, to simplify the stack (we remove the occasionally problematic need to install nodejs on target servers, which is outside of the regular OS and HDP stacks we support). Arguably, the Zuul proxy is not necessary if we force everything through a Knox instance, since Knox would provide a single endpoint. We probably however don't want to force Knox and SSL, hence using Zuul to keep it closer to our current architecture. Zuul does some other nice things, which might help us in future, so it's really about laying down some options for potentially doing micro-services style things at a later date. I'm not saying we have to, or even should go that way, it will just make life easier later if we decide to. It will also help us if we want to add HA, circuit breaking etc to the architecture at a later date. That said, I regret that I ever said the word micro-services, since it's caused confusion. Just think of it as a proxy to deal with the CORS problem. Zuul is implemented as a set of filters, but we are not using it for its authentication filtering. We're using it as a proxy. Shiro is an authentication framework, and could arguably used to provide the security piece, but frankly wrapping shiro as a replacement for Spring Security in a Spring application seemed like it will make life a lot harder. This could be done, but it's not the native happy path, and would pull in additional dependencies that duplicate functionality that's already embedded in Spring Security. The version of Knox used is the default from HDP. The link version you mention is a docs link. I'll update it to be the older version, which is the same and we can decide if we want to maintain the freshness of it when we look to upgrade underlying patterns. Either way, the content is the same. I did consider other hosting mechanisms, including Undertow a If you have a different suggestion to using the Spring default ways of doing things, or we want to use a framework other than Spring for this, then maybe we could change to that, but the route chosen here is definitely the easy path in the context of the decision made to use Spring in metron rest, and if anything opens up our choices while minimising, in fact reducing, our dependency management overhead. I hope that explains some of the thinking behind the choices made, but the guiding principals I followed were: * Don't fight the framework if you don't have to * Reduce the need for additional installation pieces and third party repos * Minimize dependencies we would have to manage * Avoid excessive change of the architecture, or forcing users to adopt Knox if they didn't want the SSL overhead. Simon On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic wrote: > Thanks for the write-up Ryan, this is a great start. I have some further > questions based on your feedback and in addition to my initial thread. > > Just for clarification, what version of Knox are we using? HDP 2.6.5, which > is what we currently run full dev against, supports 0.12.0. > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html > . > I see references to Knox 1.1.0 (latest) in this committed PR - > > https://github.com/apache/metron/pull//files#diff-70b412194819f3cb829566f05d77c1a6R122 > . > This is probably just a super small mismatch, and it probably goes without > saying, but I just want to be doubly sure that we're installing the default > via the standard install mechanism as opposed to something separate and > manual. > > On the subject of Zuul wrt Nodejs filters. I'd like to hear some more > detail on: > >1. Why do we need filtering via Zuul? For instance, is filtering routing >not handled by Knox? From the beginner docs: "The gateway itself is a > layer >over an embedded Jetty JEE server. At the very highest level the gateway >processes requests by using request URLs to lookup specific JEE Servlet >Filter chain that is used to process the request. The gateway framework >provides extensible mechanisms to assemble chains of custom filters that >support secured access to services." [1] >2. What other
Re: [DISCUSS] Knox SSO feature branch review and features
Thanks for the write-up Ryan, this is a great start. I have some further questions based on your feedback and in addition to my initial thread. Just for clarification, what version of Knox are we using? HDP 2.6.5, which is what we currently run full dev against, supports 0.12.0. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html. I see references to Knox 1.1.0 (latest) in this committed PR - https://github.com/apache/metron/pull//files#diff-70b412194819f3cb829566f05d77c1a6R122. This is probably just a super small mismatch, and it probably goes without saying, but I just want to be doubly sure that we're installing the default via the standard install mechanism as opposed to something separate and manual. On the subject of Zuul wrt Nodejs filters. I'd like to hear some more detail on: 1. Why do we need filtering via Zuul? For instance, is filtering routing not handled by Knox? From the beginner docs: "The gateway itself is a layer over an embedded Jetty JEE server. At the very highest level the gateway processes requests by using request URLs to lookup specific JEE Servlet Filter chain that is used to process the request. The gateway framework provides extensible mechanisms to assemble chains of custom filters that support secured access to services." [1] 2. What other library options were considered for this feature and how was it chosen over the others? I search on "apache spring web filters" and it's almost all about Shiro - https://shiro.apache.org/spring.html. I also see quite a bit about filtering for Spring Boot applications along with a write-up of how to accomplish the same with Web MVC here - https://stackoverflow.com/questions/19825946/how-to-add-a-filter-class-in-spring-boot. The Knox documentation boilerplate examples are also using Shiro. "shiro.ini - The configuration file for the Shiro authentication provider’s filters. This information is derived from the information in the provider section of the topology file." [1] My assumption is that there are deliberate decisions in favor of this mix of technologies over others, and I think some additional explanation will make that clear. As it stands per the Knox documentation, it looks like we're going on a different route from the preferred/recommended idioms. [1] http://knox.apache.org/books/knox-0-12-0/dev-guide.html#Architecture+Overview Ryan, I agree about microservices. This should not derail nor be a major part of discussion around this feature, imho. I think there's quite a bit left to discuss on that subject. I want to make sure that we're not prematurely favoring architectural choices by pulling in libraries that are potentially opinionated about how to accomplish those goals. If they are, I would expect we are comfortable ripping those libraries out if the community favors a different direction. On the subject of Spring Boot vs Nodejs. I can see some rationale for making things homogenous (though, in a microservices architecture, if we go that route, that's not strictly necessary), but what is the justification for Spring Boot over Nodejs? Why would want one over the other? On Mon, Sep 17, 2018 at 3:38 PM Ryan Merriman wrote: > I have reviewed a couple different PRs so I'll add some context where I > can. Obviously Simon would be the most qualified to answer but I'll add my > thoughts. > > For question 1, while they may not all be necessary I think it does make > sense to include them in this feature branch if our primary goal is > integrating Knox SSO. We could push off removing JDBC authentication for > reasons I'll get to in my response to question 2. If we want to do one at > a time (switch to spring boot, add Zuul as a dependency, then add Knox SSO) > then that's ok but I do think there are dependencies and should be done in > order. For example, adding Knox SSO requires some work around request > filtering. If we were to do this before moving to Spring Boot we would > need to implement the filters in Nodejs which would be throwaway once we > get around to migrating away from that. For Zuul, I believe it's purpose > is to facilitate the filtering (although it does a lot more) so it doesn't > make sense to add that separate from the Knox SSO work. > > For question 2, I think you bring up a good point. We probably don't want > to just rip our current authentication method out. We might want to > consider deprecating it instead and making Knox SSO and LDAP authentication > optional. > > For question 3, this is a bigger shift than just a component upgrade. It's > more like shifting platforms, from Elasticsearch to Solr for example. Like > I alluded to in my response to question 1, I don't think we should require > throwaway work just because we want to review these parts separately. > > For question 4, I will defer to Simon. I don't believe we necessarily > require Zuul so I will let him elaborate on why he
Re: [DISCUSS] Knox SSO feature branch review and features
I have reviewed a couple different PRs so I'll add some context where I can. Obviously Simon would be the most qualified to answer but I'll add my thoughts. For question 1, while they may not all be necessary I think it does make sense to include them in this feature branch if our primary goal is integrating Knox SSO. We could push off removing JDBC authentication for reasons I'll get to in my response to question 2. If we want to do one at a time (switch to spring boot, add Zuul as a dependency, then add Knox SSO) then that's ok but I do think there are dependencies and should be done in order. For example, adding Knox SSO requires some work around request filtering. If we were to do this before moving to Spring Boot we would need to implement the filters in Nodejs which would be throwaway once we get around to migrating away from that. For Zuul, I believe it's purpose is to facilitate the filtering (although it does a lot more) so it doesn't make sense to add that separate from the Knox SSO work. For question 2, I think you bring up a good point. We probably don't want to just rip our current authentication method out. We might want to consider deprecating it instead and making Knox SSO and LDAP authentication optional. For question 3, this is a bigger shift than just a component upgrade. It's more like shifting platforms, from Elasticsearch to Solr for example. Like I alluded to in my response to question 1, I don't think we should require throwaway work just because we want to review these parts separately. For question 4, I will defer to Simon. I don't believe we necessarily require Zuul so I will let him elaborate on why he choose that library and what the potential impact is of adding it to our project. For question 5 and 6, I will also defer to Simon on this. The focus of this feature as I understand it is a consistent authentication mechanism and support for SSO. I will let him lay out his vision for micro services. Knox SSO would be a great improvement and is what I think we should focus on in this feature branch. Micro services is something we should certainly discuss but it might be a bit of a distraction and I wouldn't want to hold up the other useful parts. On Fri, Sep 14, 2018 at 8:38 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Hey all, > > I started looking through the Knox SSO feature branch (see here > https://issues.apache.org/jira/browse/METRON-1663). This is some great new > security functionality work and it looks like it will bring some important > new features to the Metron platform. I'm coming at this pretty green, so I > do have some questions regarding the proposed changes from a high level > architectural perspective. There are a few changes within the current FB > PR's that I think could use some further explanation. At first glance, it > seems we could potentially simplify this branch a great deal and get it > completed much sooner if we narrowed the focus a bit. But I could certainly > be wrong here and happy for other opinions. I searched through the mailing > list history to see if there is any additional background and the main > DISCUSS thread I could find was regarding initially setting up the feature > branch, which talked about adding Knox and LDAP. > > https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E > . > If I've missed any follow-up, please let me know. > > Looking at the broader set of Jiras associated with 1663 and the first PR > 1665, it looks like there are 4 main thrusts of this branch right now: > >1. Knox/SSO >2. Node migrated to Spring Boot >3. JDBC removed completely in favor of LDAP >4. Introduction of Zuul, also microservices? > > I strongly urge for the purpose of reviewing this feature branch that we > base much of the discussion off of > https://issues.apache.org/jira/browse/METRON-1755, the architecture > diagram. Minimally, an explanation of the current architecture along with > discussion around the additional proposed changes and rationale would be > useful for evaluation. I don't have a solid enough understanding yet of the > full scope of changes and how they differ from the existing architecture > just from looking at the PR's alone. > >1. The first question is a general one regarding the necessity of the 3 >additional features alongside Knox - migrating Node to Spring Boot, >removing JDBC altogether, adding dependencies on Netflix's Zuul > framework. >Are these necessary for adding Knox/SSO? They seem like potentially >separate features, imo. >2. It looks like LDAP will be a required component for interacting with >Metron via the UI's. I see this PR >https://github.com/apache/metron/pull/1186 which removes JDBC >authentication. Are we ready to remove it completely or would it be > better >to leave it as a minimal installation option? What is the proposed >migration path
