Ok, here is something I don’t understand, but would like to. Knox comes configured with build in services for a number of other apache products and UI’s. It would seem to me, that the best integration with Knox would be to do what these other products have done.
1. Do whatever you have to do to make your own stuff compatible. 2. Create a knox service definition and provide it or try to get it into knox itself This would make the knox integration with metron optional and pluggable wouldn’t it? Then knox with metron would just be the same as knox with anything else. Please help me if I am wrong, but we seem to be going our own way here. Why don’t we just do what these other products have done? Why don’t we try to get apache metron services accepted to the knox project? Why don’t we model our knox integration with how XYZ does it? Have we looked at how others integrate? Having all the code and being able to track stuff is kind of the point of this whole thing isn’t it? Maybe this is implied and I’m missing it, if so I apologize. I think consistency with the rest of the hadoop stack with knox helps us. On November 15, 2018 at 22:20:00, Ryan Merriman (merrim...@gmail.com) wrote: 1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI as far as I know. I put up a PR with a proposed solution here: https://github.com/apache/metron/pull/1266. 2) Yes Knox is a service you can install with Ambari, similar to Ranger or Spark. There are some things that are specifically configured in Knox and there are some things specific to Metron. I will put up a PR with the changes needed so you can see exactly what is involved. 3) I don't understand what you mean here. Is this a question? 4) I think it's a little early to predict the Ambari changes required. This will depend on how tasks 1-3 go. I imagine it's similar to other mpack work: expose some parameters in ambari and bind those to config files. My understanding from this thread so far is that we should focus on a manual, documented approach to start. On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Thanks Ryan, > > 1) Can you clarify "not a good way to do this?" Are you saying we don't > have a way to set this and need to add the config option, or that a > solution is not obvious and it's unclear what to do? It seems to me you're > saying the former, but I'd like to be sure. > 2) Is Knox not a service made available by Ambari similar to Ranger or > Spark? I'm assuming that similar to Kerberos, there are some things that > are specifically configured in Knox and others that are app-specific. Some > explanation of what this looks like would be helpful. > 3) Sounds like this follows pretty naturally from #1 > 4) Relates to #2. I think we need some guidance on what a manual vs > MPack/automated install would look like. > > Cheers, > Mike > > > On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman <merrim...@gmail.com> wrote: > > > Wanted to give an update on the context path issue. I investigated > > rewriting url links in the outgoing static assets with Knox and it was > not > > trivial. Fortunately I found a simple solution that works with or > without > > Knox. I changed the base tag in index.html from <base href="/"> to <base > > href="./">, or in other words made the base href relative. > > > > I believe I am at the point where I can task this out and provide a high > > level overview of the changes needed. I think that each task will be a > > manageable size and can stand alone so I don't think we need a feature > > branch. > > > > The first task involves a general change to the UI code. We need a way > to > > set the path to the REST service with a configuration setting because it > is > > different with and without Knox. Currently there is not a good way to do > > this in the UI. We can use the environment files but that is a build > time > > setting and is not flexible. I can see this capability being useful for > > other use cases in the future. I think we could even split this up into > 2 > > separate tasks, one for the alerts UI and one for the management UI. > > > > The second task involves adding Knox to our stack either by default as a > > dependency in the mpack or with a documented approach. We would add our > > REST service, Alerts UI, and Management UI as services in Knox. > Everything > > would continue to function as it currently does but with all > communication > > going through Knox. LDAP authentication would be required when using > Knox > > and Knox will authenticate with the REST service by passing along an > > Authorization header. Enabling Knox would be a manual process that > > involves deploying assets (Knox descriptor files) and changing > > configuration. There would be no change to how the UI functions by > default > > (without Knox) and either LDAP or JDBC authentication could still be > used.. > > > > The third task involves enabling SSO with Knox. We would update the REST > > service so that it can authenticate with a Knox SSO token. We would > > provide documentation on how to update the Knox settings in Ambari to > > enable SSO. The Alerts/Management UI would need to expose configuration > > properties for the REST url and login url since these would be different > > when Knox is enabled. We would also need to provide documentation on how > > to make these UI configuration changes, based on the work done in task 1. > > > > An optional forth task would be exposing configuration settings and > > enabling Knox with Ambari. We would eliminate the manual steps necessary > > for enabling Knox and instead automate those steps with an Ambari input > > control, similar to how LDAP is enabled. > > > > Thoughts on this plan? > > > > On Thu, Nov 15, 2018 at 10:22 AM James Sirota <jsir...@apache.org> > wrote: > > > > > In my view Knox SSO is such a minor feature when it comes to Metron's > > > capabilities than it's not worth supporting multiple scenarios where it > > > works with Knox or without Knox. Where we should be configurable (and > > are > > > configurable) is on the analytics and stream processing. But this? As > > > long as the UI authenticates securely I don't think anyone is going to > > care > > > what proxy it's using. The code itself should be written in a way that > > > it's pluggable so if we ever wanted to use another proxy or disable it > > all > > > together we could. But this should not be a configuration we pass on > to > > > the user. The added complexity is simply not worth it here. We have > to > > > start being opinionated about making sensible choices on behalf of the > > > user. A sensible choice here is to run with Knox and LDAP. The JDBC > > > component should exist for another release to allow the community to > > > migrate over to LDAP and then be deprecated. The code should still be > > > pluggable and if anyone wanted to extend it to work with JDBC they > could, > > > or if people wanted to plug in another proxy they could, but this is > not > > > something we would officially support. > > > > > > Thanks, > > > James > > > > > > 12.11.2018, 07:36, "Ryan Merriman" <merrim...@gmail.com>: > > > > Let me clarify on exposing both legacy and Knox URLs at the same > time. > > > The > > > > base urls will look something like this: > > > > > > > > Legacy REST - http://node1:8082/api/v1 > > > > Legacy Alerts UI - http://node1:4201:/alerts-list > > > > > > > > Knox REST - https://node1:8443/gateway/default/metron/api/v1 > > > > Knox Alerts UI - > > > > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > > > > > > > If Knox were turned on and the alerts UI deployed as is, it would not > > > > work. This is because static assets are referenced with > > > > http://node1:4201/assets/some-asset.js which does not include the > > > correct > > > > context path to the alerts UI in knox. To make it work, you have to > set > > > > the base ref to "/gateway/default/metron-alerts-ui" so that static > > assets > > > > are referenced at > > > > > > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js > > > . > > > > When you do that, the legacy alerts UI will no longer work. I guess > the > > > > point I'm trying to make is that we would have to switch between them > > or > > > > have 2 separate application running. I imagine most users only need > one > > > or > > > > the other running so probably not an issue. > > > > > > > > Jon, the primary upgrade consideration I see is with authentication. > To > > > be > > > > able to use Knox, you would have to upgrade to LDAP-based > > authentication > > > if > > > > you were still using JDBC-based authentication in REST. The urls > would > > > > also change obviously. > > > > > > > > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com <zeo...@gmail.com> > > > wrote: > > > > > > > >> Phew, that was quite the thread to catch up on. > > > >> > > > >> I agree that this should be optional/pluggable to start, and I'm > > > interested > > > >> to hear the issues as they relate to upgrading an existing cluster > > > (given > > > >> the suggested approach) and exposing both legacy and knox URLs at > the > > > same > > > >> time. > > > >> > > > >> Jon > > > >> > > > >> On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < > > > >> michael.miklav...@gmail.com> > > > >> wrote: > > > >> > > > >> > A couple more things, and I think this goes without saying - > > > whatever we > > > >> do > > > >> > with Knox should NOT > > > >> > > > > >> > 1. Require unit and integration tests to use Knox > > > >> > 2. Break fulldev > > > >> > > > > >> > Also, I don't know that I saw you mention this, but I'm unsure > how > > we > > > >> > should leverage Knox as a core piece of the platform. i.e. should > > we > > > make > > > >> > this required or optional? I'm open to hearing opinions on this, > > but > > > I'm > > > >> > inclined to keep this a pluggable option. > > > >> > > > > >> > Mike > > > >> > > > > >> > > > > >> > On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic < > > > >> > michael.miklav...@gmail.com> wrote: > > > >> > > > > >> > > Thanks for the update Ryan. Per my earlier comments, I thought > it > > > might > > > >> > be > > > >> > > the case that we could dramatically simplify this by leveraging > > > Knox's > > > >> > > proxy capabilities, and per your research that appears to be > the > > > case. > > > >> > This > > > >> > > is a dramatic simplification and improvement of this feature > imo, > > > +1. > > > >> I'm > > > >> > > also +1 on a couple distinct steps that you've laid out: fix > the > > UI > > > >> > issues > > > >> > > in master, then add Knox for SSO. That should help mitigate > > issues > > > with > > > >> > > merge conflicts with ongoing development. > > > >> > > > > > >> > > > I think it will be a challenge exposing the UIs through both > > the > > > Knox > > > >> > > url and legacy urls at the same time. > > > >> > > I'm not sure I understand the issue here. Are you referring to > > this > > > >> > > comment? "Added a ng build option to build the UI with base > href > > > set to > > > >> > > Knox base path." Isn't it just a matter of URL > > > rewriting/forwarding? I > > > >> > > thought we'd be exposing the URL's directly in one context, and > > > through > > > >> > > Knox in the other. Either way, it seems like we should be able > to > > > >> > provide a > > > >> > > dynamic base path through configuration in our web > applications. > > > I'd > > > >> > expect > > > >> > > to modify that property based on whether Knox is configured or > > not. > > > >> > > > > > >> > > > I'm also not clear on how one would use Knox with REST set to > > > legacy > > > >> > > JDBC-based authentication. As far as I know Knox does not > support > > > JDBC > > > >> so > > > >> > > there would be a mismatch between Knox and REST. > > > >> > > I'm OK with not having Knox work with JDBC. That's a feature of > > > Knox > > > >> and > > > >> > > probably not something we care much about. > > > >> > > > > > >> > > >We could initially make Knox an optional feature that requires > > > setup > > > >> > with > > > >> > > the help of some documentation (like Kerberos) while keeping > the > > > system > > > >> > the > > > >> > > way it is now by default. > > > >> > > Sounds good to me. > > > >> > > > > > >> > > > I imagine we'll deprecate JDBC-based authentication at some > > > point so > > > >> > > that may be a good time to switch. > > > >> > > I would like to announce deprecation in our next release and > move > > > to > > > >> > > remove it in a following release. > > > >> > > > > > >> > > Thanks for taking this on and great job laying things out. > > > >> > > > > > >> > > Thanks, > > > >> > > Mike > > > >> > > > > > >> > > On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman < > > merrim...@gmail.com> > > > >> > wrote: > > > >> > > > > > >> > >> I have spent some time recently reviewing this discussion and > > the > > > >> > feature > > > >> > >> branch that Simon put out. I think this is an important > feature > > > and > > > >> > want > > > >> > >> to move it forward. I started another discussion on adding > Knox > > to > > > >> our > > > >> > >> stack but this discussion has a lot of good context so I will > > > continue > > > >> > it > > > >> > >> here. > > > >> > >> > > > >> > >> I think the main point of contention was that this feature > > branch > > > >> > included > > > >> > >> several different architectural changes and it was unclear if > > they > > > >> were > > > >> > >> needed and if so, could be done separately. Fortunately LDAP > > > >> > >> authentication has been accepted into master so we can cross > it > > > off > > > >> the > > > >> > >> list. From my understanding of the points people have made, > that > > > >> leaves > > > >> > >> Knox related SSO changes and migrating expressjs to a > different, > > > >> > JVM-based > > > >> > >> web server that includes proxying capabilities (Zuul). > > > >> > >> > > > >> > >> I think everyone agrees that if we can limit the scope to just > > > Knox > > > >> > >> related > > > >> > >> SSO changes that would be ideal. I believe I have found a way > to > > > do > > > >> > that > > > >> > >> while working on a small POC this week. The key to this (Simon > > > >> alluded > > > >> > to > > > >> > >> it earlier) is to put both REST and our UIs behind Knox. I > > > initially > > > >> > was > > > >> > >> focused on just adding REST as a service in Knox and decided > to > > > >> > experiment > > > >> > >> with also adding our UIs. After I did this it became clear > that > > > this > > > >> > >> simplifies things considerably: > > > >> > >> > > > >> > >> - The REST app and the UIs are now served from the same host > so > > > >> CORS > > > >> > >> concerns go away. > > > >> > >> - We no longer need to worry about proxying REST requests from > > the > > > >> > UIs > > > >> > >> with express or Zuul because Knox handles that for us. This > will > > > >> > make > > > >> > >> our > > > >> > >> express configuration even simpler. In fact, all we need is a > > > >> simple > > > >> > >> way > > > >> > >> to serve static UI assets. > > > >> > >> - We no longer need to check for SSO tokens and redirect in > the > > UI > > > >> > >> web/app servers (or the REST app for that matter) because Knox > > > >> > handles > > > >> > >> that > > > >> > >> for us. > > > >> > >> - The UIs can now easily access any Knox service (not just our > > > REST > > > >> > >> app) > > > >> > >> without any extra proxy configuration. > > > >> > >> - SSO token authentication is only necessary in REST so there > is > > > no > > > >> > >> need > > > >> > >> to create shared Spring modules or split functionality out. > > > >> > >> > > > >> > >> The most significant change I had to make (borrowed from > Simon's > > > >> feature > > > >> > >> branch) was the SSO token authentication mentioned above. The > > > primary > > > >> > >> short term benefit with this approach is that outside of some > > > general > > > >> > >> deficiencies unrelated to this our UI architecture doesn't > need > > to > > > >> > >> fundamentally change. I could summarize the changes as: > > > >> > >> > > > >> > >> - Knox install and configuration (setting up REST and the > alerts > > > UI > > > >> > as > > > >> > >> Knox services) > > > >> > >> - Added Knox SSO token authentication to REST > > > >> > >> - Updated REST urls in the UI code (should be configurable) > > > >> > >> - Fixed a few UI bugs where relative paths were not being used > > > >> > >> - Added a ng build option to build the UI with base href set > to > > > >> Knox > > > >> > >> base path ( > > > >> > >> > > > >> > >> > > > >> > > > > >> > > > > > > https://github.com/angular/angular-cli/wiki/build#base-tag-handling-in-indexhtml > > > >> > >> ) > > > >> > >> > > > >> > >> Most the UI changes are preexisting, minor issues that could > be > > > fixed > > > >> > >> directly in master. We would need to think of an approach for > > the > > > >> base > > > >> > >> href build requirement but I'm sure it's not that bad. > > > >> > >> > > > >> > >> However there will be some backwards compatibility issues we > > would > > > >> need > > > >> > to > > > >> > >> think through. I think it will be a challenge exposing the UIs > > > >> through > > > >> > >> both the Knox url and legacy urls at the same time. I'm also > not > > > >> clear > > > >> > on > > > >> > >> how one would use Knox with REST set to legacy JDBC-based > > > >> > authentication. > > > >> > >> As far as I know Knox does not support JDBC so there would be > a > > > >> mismatch > > > >> > >> between Knox and REST. Knox does have the ability to pass > along > > > basic > > > >> > >> authentication headers so LDAP in REST would work. We could > > > initially > > > >> > >> make > > > >> > >> Knox an optional feature that requires setup with the help of > > some > > > >> > >> documentation (like Kerberos) while keeping the system the way > > it > > > is > > > >> now > > > >> > >> by > > > >> > >> default. I imagine we'll deprecate JDBC-based authentication > at > > > some > > > >> > >> point > > > >> > >> so that may be a good time to switch. > > > >> > >> > > > >> > >> What do people think about this approach? Concerns? Are there > > any > > > >> huge > > > >> > >> holes in this I'm not thinking about? > > > >> > >> > > > >> > >> I want to highlight that the work Simon did in his feature > > branch > > > was > > > >> > >> crucial to better understanding this. I am pretty sure we'll > end > > > up > > > >> > >> reusing a lot code from that branch. > > > >> > >> > > > >> > >> On Thu, Sep 27, 2018 at 6:30 PM Michael Miklavcic < > > > >> > >> michael.miklav...@gmail.com> wrote: > > > >> > >> > > > >> > >> > Apparently, I hit send on my last email before finishing my > > > synopsis > > > >> > >> (per > > > >> > >> > @Otto's Q in Slack). To summarize, based on my current > > > >> understanding I > > > >> > >> > believe that each of the feature branch changes I've outline > > > above > > > >> are > > > >> > >> > units of work that are related, yet should be executed on > > > >> > independently. > > > >> > >> > Knox SSO in its own feature branch. Migrating technologies > > like > > > >> NodeJs > > > >> > >> or > > > >> > >> > migrating the auth DB to LDAP seem like they belong in their > > own > > > >> > >> separate > > > >> > >> > PR's or feature branches. > > > >> > >> > > > > >> > >> > Thanks, > > > >> > >> > Mike > > > >> > >> > > > > >> > >> > On Thu, Sep 27, 2018 at 4:08 PM Casey Stella < > > > ceste...@gmail.com> > > > >> > >> wrote: > > > >> > >> > > > > >> > >> > > I'm coming in late to the game here, but for my mind a > > feature > > > >> > branch > > > >> > >> > > should involve the minimum architectural change to > > accomplish > > > a > > > >> > given > > > >> > >> > > feature. > > > >> > >> > > The feature in question is SSO integration. It seems to me > > > that > > > >> the > > > >> > >> > > operative question is can we do the feature without making > > the > > > >> OTHER > > > >> > >> > > architectural change > > > >> > >> > > (e.g. migrating from expressjs to spring boot + zuul). I > > would > > > >> > argue > > > >> > >> > that > > > >> > >> > > if we WANT to do that, then it should be a separate > feature > > > >> branch. > > > >> > >> > > > > > >> > >> > > Thus, I leave with a question: is there a way to > accomplish > > > this > > > >> > >> feature > > > >> > >> > > without ripping out expressjs? > > > >> > >> > > > > > >> > >> > > - If so and it is feasible, I would argue that we should > > > >> decouple > > > >> > >> this > > > >> > >> > > into a separate feature branch. > > > >> > >> > > - If so and it is infeasible, I'd like to hear an argument > > as > > > >> to > > > >> > >> the > > > >> > >> > > infeasibility and let's decide given that > > > >> > >> > > - If it is not possible, then I'd argue that we should > keep > > > >> them > > > >> > >> > coupled > > > >> > >> > > and move this through as-is. > > > >> > >> > > > > > >> > >> > > On a side-note, it feels a bit weird that we're narrowing > > to a > > > >> > bundled > > > >> > >> > > proxy, rather than having that be a pluggable thing. I'm > not > > > >> super > > > >> > >> > > knowledgeable in this space, so I apologize > > > >> > >> > > in advance if this is naive, but isn't this a pluggable, > > > external > > > >> > >> > component > > > >> > >> > > (e.g. nginx)? > > > >> > >> > > > > > >> > >> > > On Thu, Sep 27, 2018 at 5:05 PM Michael Miklavcic < > > > >> > >> > > michael.miklav...@gmail.com> wrote: > > > >> > >> > > > > > >> > >> > > > I've spent some more time reading through Simon's > response > > > and > > > >> the > > > >> > >> > added > > > >> > >> > > > sequence diagram. This is definitely helpful - thank you > > > Simon. > > > >> > >> > > > > > > >> > >> > > > I need to redact my initial list: > > > >> > >> > > > > > > >> > >> > > > 1. Node migrated to Spring Boot, expressjs migrated to a > > > >> > >> > > > non-JS/non-NodeJs proxying mechanism (ie Zuul in this > > case) > > > >> > >> > > > 2. JDBC removed completely in favor of LDAP > > > >> > >> > > > 3. Knox/SSO > > > >> > >> > > > > > > >> > >> > > > I'm a bit conflicted on the best way to move forward and > > > would > > > >> > like > > > >> > >> > some > > > >> > >> > > > thoughts from other community members on this. I think > an > > > >> argument > > > >> > >> can > > > >> > >> > be > > > >> > >> > > > made that 1 and 2 are independent of 3, and should/could > > > really > > > >> be > > > >> > >> > > > independent PR's against master. > > > >> > >> > > > > > > >> > >> > > > The need for a replacement for expressjs (Zuul in this > > > case) is > > > >> an > > > >> > >> > > artifact > > > >> > >> > > > that our request/response cycle for REST calls is a > simple > > > >> matter > > > >> > of > > > >> > >> > > > forwarding with some additional headers for > > authentication. > > > >> > There's > > > >> > >> a > > > >> > >> > > > JSESSIONID managed by the client browser in our current > > > >> > >> architecture, > > > >> > >> > for > > > >> > >> > > > example. You login to the alerts or the management UI > > which > > > >> > >> forwards a > > > >> > >> > > > request to REST, which looks up credentials in a backend > > > >> database, > > > >> > >> and > > > >> > >> > > > passes the results back up the chain. All browser > requests > > > go > > > >> > >> directly > > > >> > >> > to > > > >> > >> > > > the specific UI you're working with - this is the CORS > > > problem. > > > >> > You > > > >> > >> > > can't, > > > >> > >> > > > without some effort with headers for adding other > domains > > > to the > > > >> > >> safe > > > >> > >> > > list > > > >> > >> > > > or disabling the security check for CORS, make remote > > calls > > > >> > >> directly to > > > >> > >> > > > REST. That's why we proxy. Switching over to Spring Boot > > > leaves > > > >> a > > > >> > >> gap > > > >> > >> > > with > > > >> > >> > > > expressjs having handled the proxying and filtering, > since > > > it's > > > >> > only > > > >> > >> > > > available to a NodeJs application (it's server-side > > > javascript > > > >> vs > > > >> > >> the > > > >> > >> > > > client side javascript deployed via our Angular > > > applications). > > > >> > Enter > > > >> > >> > > Zuul, > > > >> > >> > > > which now effectively handles that. At runtime, Zuul is > a > > > part > > > >> of > > > >> > >> the > > > >> > >> > > > Spring app that serves up our UI's. It handles the > > requests > > > via > > > >> > >> > > filtering, > > > >> > >> > > > forwards them to REST, manages the response back to the > > > client. > > > >> > Very > > > >> > >> > > > similar to what expressjs was doing, per my current > > > >> understanding. > > > >> > >> The > > > >> > >> > > > sequence diagrams Simon added are useful, and I think > some > > > of > > > >> what > > > >> > >> was > > > >> > >> > > less > > > >> > >> > > > clear was what we currently vs what the new changes are > > > doing to > > > >> > the > > > >> > >> > > > architecture. This is no fault of Simon's - there simply > > > wasn't > > > >> > any > > > >> > >> > > > architecture diagrams/documents around this before. > Here's > > > my > > > >> > >> > impression > > > >> > >> > > of > > > >> > >> > > > the very very basic current state - someone more > familiar > > > with > > > >> > this > > > >> > >> > > > architecture please advise if I'm incorrect about > anything > > > >> > (probably > > > >> > >> > > Ryan). > > > >> > >> > > > > > > >> > >> > > > https://imgur.com/f8GtSmh > > > >> > >> > > > > > > >> > >> > > > Zuul would be replacing the bit about expressjs in the > > > diagram, > > > >> > and > > > >> > >> > > instead > > > >> > >> > > > of node we have spring boot. This covers 1. 2 and 3 are > > > other > > > >> > >> issues. > > > >> > >> > I'd > > > >> > >> > > > like to see similar exposition of those server processes > > > with > > > >> knox > > > >> > >> > > > involved. I imagine in that case we bump up from 3 to 4 > > > server > > > >> > >> > instances > > > >> > >> > > > for the additional knox endpoint. > > > >> > >> > > > > > > >> > >> > > > Mike > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > On Wed, Sep 19, 2018 at 11:28 AM James Sirota < > > > >> jsir...@apache.org > > > >> > > > > > >> > >> > > wrote: > > > >> > >> > > > > > > >> > >> > > > > Thank you, Simon. The diagrams help a lot > > > >> > >> > > > > > > > >> > >> > > > > 19.09.2018, 21:27, "Simon Elliston Ball" < > > > >> > >> > si...@simonellistonball.com > > > >> > >> > > >: > > > >> > >> > > > > > To clarify some of this I've put some documentation > > into > > > >> > >> > > > > > https://github.com/apache/metron/pull/1203 under > > > >> METRON-1755 > > > >> > ( > > > >> > >> > > > > > https://issues.apache.org/jira/browse/METRON-1755). > > > >> Hopefully > > > >> > >> the > > > >> > >> > > > > diagrams > > > >> > >> > > > > > there should make it clearer. > > > >> > >> > > > > > > > > >> > >> > > > > > Simon > > > >> > >> > > > > > > > > >> > >> > > > > > On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < > > > >> > >> > > > > > si...@simonellistonball.com> wrote: > > > >> > >> > > > > > > > > >> > >> > > > > >> Hi Mike, > > > >> > >> > > > > >> > > > >> > >> > > > > >> Some good points here which could do with some > > > >> > clarification. > > > >> > >> I > > > >> > >> > > > suspect > > > >> > >> > > > > >> the architecture documentation could be clearer and > > > fill > > > >> in > > > >> > >> some > > > >> > >> > of > > > >> > >> > > > > these > > > >> > >> > > > > >> gaps, and I'll have a look at working on that and > > > >> providing > > > >> > >> some > > > >> > >> > > > > diagrams. > > > >> > >> > > > > >> > > > >> > >> > > > > >> The short version is that the Zuul proxy gateway > has > > > been > > > >> > >> added > > > >> > >> > to > > > >> > >> > > > > replace > > > >> > >> > > > > >> the Nodejs express proxy used to gateway the REST > api > > > >> calls > > > >> > in > > > >> > >> > the > > > >> > >> > > > > current > > > >> > >> > > > > >> hosts. This is done in both cases to avoid CORS > > > >> restrictions > > > >> > >> by > > > >> > >> > > > > allowing > > > >> > >> > > > > >> the same host that serves the UI files to proxy > call > > to > > > >> the > > > >> > >> API. > > > >> > >> > > > > >> > > > >> > >> > > > > >> The choice of Zuul was partly a pragmatic one (it's > > the > > > >> one > > > >> > >> > that's > > > >> > >> > > > > there > > > >> > >> > > > > >> in the box as it were with Spring Boot, which we > use > > > for > > > >> the > > > >> > >> REST > > > >> > >> > > > API, > > > >> > >> > > > > via > > > >> > >> > > > > >> the Spring Cloud Netflix project which wraps a > bunch > > of > > > >> > >> related > > > >> > >> > > > pieces > > > >> > >> > > > > into > > > >> > >> > > > > >> Spring). The choice of Spring Boot to host the UIs > > > >> > themselves > > > >> > >> was > > > >> > >> > > > > similarly > > > >> > >> > > > > >> for parity with the REST host, to simplify the > stack > > > (we > > > >> > >> remove > > > >> > >> > the > > > >> > >> > > > > >> occasionally problematic need to install nodejs on > > > target > > > >> > >> > servers, > > > >> > >> > > > > which is > > > >> > >> > > > > >> outside of the regular OS and HDP stacks we > support). > > > >> > >> > > > > >> > > > >> > >> > > > > >> Arguably, the Zuul proxy is not necessary if we > force > > > >> > >> everything > > > >> > >> > > > > through a > > > >> > >> > > > > >> Knox instance, since Knox would provide a single > > > endpoint. > > > >> > We > > > >> > >> > > > probably > > > >> > >> > > > > >> however don't want to force Knox and SSL, hence > using > > > Zuul > > > >> > to > > > >> > >> > keep > > > >> > >> > > it > > > >> > >> > > > > >> closer to our current architecture. Zuul does some > > > other > > > >> > nice > > > >> > >> > > things, > > > >> > >> > > > > which > > > >> > >> > > > > >> might help us in future, so it's really about > laying > > > down > > > >> > some > > > >> > >> > > > options > > > >> > >> > > > > for > > > >> > >> > > > > >> potentially doing micro-services style things at a > > > later > > > >> > date. > > > >> > >> > I'm > > > >> > >> > > > not > > > >> > >> > > > > >> saying we have to, or even should go that way, it > > will > > > >> just > > > >> > >> make > > > >> > >> > > life > > > >> > >> > > > > >> easier later if we decide to. It will also help us > if > > > we > > > >> > want > > > >> > >> to > > > >> > >> > > add > > > >> > >> > > > > HA, > > > >> > >> > > > > >> circuit breaking etc to the architecture at a later > > > date. > > > >> > That > > > >> > >> > > said, > > > >> > >> > > > I > > > >> > >> > > > > >> regret that I ever said the word micro-services, > > since > > > >> it's > > > >> > >> > caused > > > >> > >> > > > > >> confusion. Just think of it as a proxy to deal with > > the > > > >> CORS > > > >> > >> > > problem. > > > >> > >> > > > > >> > > > >> > >> > > > > >> Zuul is implemented as a set of filters, but we are > > not > > > >> > using > > > >> > >> it > > > >> > >> > > for > > > >> > >> > > > > its > > > >> > >> > > > > >> authentication filtering. We're using it as a > proxy. > > > Shiro > > > >> > is > > > >> > >> an > > > >> > >> > > > > >> authentication framework, and could arguably used > to > > > >> provide > > > >> > >> the > > > >> > >> > > > > security > > > >> > >> > > > > >> piece, but frankly wrapping shiro as a replacement > > for > > > >> > Spring > > > >> > >> > > > Security > > > >> > >> > > > > in a > > > >> > >> > > > > >> Spring application seemed like it will make life a > > lot > > > >> > harder. > > > >> > >> > This > > > >> > >> > > > > could > > > >> > >> > > > > >> be done, but it's not the native happy path, and > > would > > > >> pull > > > >> > in > > > >> > >> > > > > additional > > > >> > >> > > > > >> dependencies that duplicate functionality that's > > > already > > > >> > >> embedded > > > >> > >> > > in > > > >> > >> > > > > Spring > > > >> > >> > > > > >> Security. > > > >> > >> > > > > >> > > > >> > >> > > > > >> The version of Knox used is the default from HDP. > The > > > link > > > >> > >> > version > > > >> > >> > > > you > > > >> > >> > > > > >> mention is a docs link. I'll update it to be the > > older > > > >> > >> version, > > > >> > >> > > which > > > >> > >> > > > > is > > > >> > >> > > > > >> the same and we can decide if we want to maintain > the > > > >> > >> freshness > > > >> > >> > of > > > >> > >> > > it > > > >> > >> > > > > when > > > >> > >> > > > > >> we look to upgrade underlying patterns. Either way, > > the > > > >> > >> content > > > >> > >> > is > > > >> > >> > > > the > > > >> > >> > > > > >> same. > > > >> > >> > > > > >> > > > >> > >> > > > > >> I did consider other hosting mechanisms, including > > > >> Undertow > > > >> > a > > > >> > >> > > > > >> > > > >> > >> > > > > >> If you have a different suggestion to using the > > Spring > > > >> > default > > > >> > >> > ways > > > >> > >> > > > of > > > >> > >> > > > > >> doing things, or we want to use a framework other > > than > > > >> > Spring > > > >> > >> for > > > >> > >> > > > this, > > > >> > >> > > > > >> then maybe we could change to that, but the route > > > chosen > > > >> > here > > > >> > >> is > > > >> > >> > > > > definitely > > > >> > >> > > > > >> the easy path in the context of the decision made > to > > > use > > > >> > >> Spring > > > >> > >> > in > > > >> > >> > > > > metron > > > >> > >> > > > > >> rest, and if anything opens up our choices while > > > >> minimising, > > > >> > >> in > > > >> > >> > > fact > > > >> > >> > > > > >> reducing, our dependency management overhead. > > > >> > >> > > > > >> > > > >> > >> > > > > >> I hope that explains some of the thinking behind > the > > > >> choices > > > >> > >> > made, > > > >> > >> > > > but > > > >> > >> > > > > the > > > >> > >> > > > > >> guiding principals I followed were: > > > >> > >> > > > > >> * Don't fight the framework if you don't have to > > > >> > >> > > > > >> * Reduce the need for additional installation > pieces > > > and > > > >> > third > > > >> > >> > > party > > > >> > >> > > > > repos > > > >> > >> > > > > >> * Minimize dependencies we would have to manage > > > >> > >> > > > > >> * Avoid excessive change of the architecture, or > > > forcing > > > >> > >> users to > > > >> > >> > > > adopt > > > >> > >> > > > > >> Knox if they didn't want the SSL overhead. > > > >> > >> > > > > >> > > > >> > >> > > > > >> Simon > > > >> > >> > > > > >> > > > >> > >> > > > > >> On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic < > > > >> > >> > > > > >> michael.miklav...@gmail.com> wrote: > > > >> > >> > > > > >> > > > >> > >> > > > > >>> Thanks for the write-up Ryan, this is a great > > start. I > > > >> have > > > >> > >> some > > > >> > >> > > > > further > > > >> > >> > > > > >>> questions based on your feedback and in addition > to > > my > > > >> > >> initial > > > >> > >> > > > thread. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> Just for clarification, what version of Knox are > we > > > >> using? > > > >> > >> HDP > > > >> > >> > > > 2.6.5, > > > >> > >> > > > > >>> which > > > >> > >> > > > > >>> is what we currently run full dev against, > supports > > > >> 0.12.0. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html > > > >> > >> > > > > >>> . > > > >> > >> > > > > >>> I see references to Knox 1.1.0 (latest) in this > > > committed > > > >> > PR > > > >> > >> - > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://github.com/apache/metron/pull/1111/files#diff-70b412194819f3cb829566f05d77c1a6R122 > > > >> > >> > > > > >>> . > > > >> > >> > > > > >>> This is probably just a super small mismatch, and > it > > > >> > probably > > > >> > >> > goes > > > >> > >> > > > > without > > > >> > >> > > > > >>> saying, but I just want to be doubly sure that > we're > > > >> > >> installing > > > >> > >> > > the > > > >> > >> > > > > >>> default > > > >> > >> > > > > >>> via the standard install mechanism as opposed to > > > >> something > > > >> > >> > > separate > > > >> > >> > > > > and > > > >> > >> > > > > >>> manual. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> On the subject of Zuul wrt Nodejs filters. I'd > like > > to > > > >> hear > > > >> > >> some > > > >> > >> > > > more > > > >> > >> > > > > >>> detail on: > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> 1. Why do we need filtering via Zuul? For > instance, > > is > > > >> > >> > > filtering > > > >> > >> > > > > >>> routing > > > >> > >> > > > > >>> not handled by Knox? From the beginner docs: "The > > > >> > gateway > > > >> > >> > > itself > > > >> > >> > > > > is a > > > >> > >> > > > > >>> layer > > > >> > >> > > > > >>> over an embedded Jetty JEE server. At the very > > highest > > > >> > >> level > > > >> > >> > > the > > > >> > >> > > > > >>> gateway > > > >> > >> > > > > >>> processes requests by using request URLs to lookup > > > >> > >> specific > > > >> > >> > JEE > > > >> > >> > > > > Servlet > > > >> > >> > > > > >>> Filter chain that is used to process the request. > > The > > > >> > >> gateway > > > >> > >> > > > > framework > > > >> > >> > > > > >>> provides extensible mechanisms to assemble chains > of > > > >> > >> custom > > > >> > >> > > > filters > > > >> > >> > > > > >>> that > > > >> > >> > > > > >>> support secured access to services." [1] > > > >> > >> > > > > >>> 2. What other library options were considered for > > this > > > >> > >> > feature > > > >> > >> > > > and > > > >> > >> > > > > how > > > >> > >> > > > > >>> was it chosen over the others? I search on "apache > > > >> > spring > > > >> > >> web > > > >> > >> > > > > filters" > > > >> > >> > > > > >>> and > > > >> > >> > > > > >>> it's almost all about Shiro - > > > >> > >> > > > https://shiro.apache.org/spring.html. > > > >> > >> > > > > I > > > >> > >> > > > > >>> also see quite a bit about filtering for Spring > Boot > > > >> > >> > > applications > > > >> > >> > > > > along > > > >> > >> > > > > >>> with a write-up of how to accomplish the same with > > Web > > > >> > MVC > > > >> > >> > > here - > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://stackoverflow.com/questions/19825946/how-to-add-a-filter-class-in-spring-boot > > > >> > >> > > > > >>> . > > > >> > >> > > > > >>> The Knox documentation boilerplate examples are > also > > > >> > using > > > >> > >> > > Shiro. > > > >> > >> > > > > >>> "shiro.ini - The configuration file for the Shiro > > > >> > >> > > authentication > > > >> > >> > > > > >>> provider’s > > > >> > >> > > > > >>> filters. This information is derived from the > > > >> > information > > > >> > >> in > > > >> > >> > > the > > > >> > >> > > > > >>> provider > > > >> > >> > > > > >>> section of the topology file." [1] > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> My assumption is that there are deliberate > decisions > > > in > > > >> > >> favor of > > > >> > >> > > > this > > > >> > >> > > > > mix > > > >> > >> > > > > >>> of technologies over others, and I think some > > > additional > > > >> > >> > > explanation > > > >> > >> > > > > will > > > >> > >> > > > > >>> make that clear. As it stands per the Knox > > > documentation, > > > >> > it > > > >> > >> > looks > > > >> > >> > > > > like > > > >> > >> > > > > >>> we're going on a different route from the > > > >> > >> preferred/recommended > > > >> > >> > > > > idioms. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> [1] > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > http://knox.apache.org/books/knox-0-12-0/dev-guide.html#Architecture+Overview > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> Ryan, I agree about microservices. This should not > > > derail > > > >> > nor > > > >> > >> > be a > > > >> > >> > > > > major > > > >> > >> > > > > >>> part of discussion around this feature, imho. I > > think > > > >> > there's > > > >> > >> > > quite > > > >> > >> > > > a > > > >> > >> > > > > bit > > > >> > >> > > > > >>> left to discuss on that subject. I want to make > sure > > > that > > > >> > >> we're > > > >> > >> > > not > > > >> > >> > > > > >>> prematurely favoring architectural choices by > > pulling > > > in > > > >> > >> > libraries > > > >> > >> > > > > that > > > >> > >> > > > > >>> are > > > >> > >> > > > > >>> potentially opinionated about how to accomplish > > those > > > >> > goals. > > > >> > >> If > > > >> > >> > > they > > > >> > >> > > > > are, > > > >> > >> > > > > >>> I > > > >> > >> > > > > >>> would expect we are comfortable ripping those > > > libraries > > > >> out > > > >> > >> if > > > >> > >> > the > > > >> > >> > > > > >>> community favors a different direction. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> On the subject of Spring Boot vs Nodejs. I can see > > > some > > > >> > >> > rationale > > > >> > >> > > > for > > > >> > >> > > > > >>> making things homogenous (though, in a > microservices > > > >> > >> > architecture, > > > >> > >> > > > if > > > >> > >> > > > > we > > > >> > >> > > > > >>> go > > > >> > >> > > > > >>> that route, that's not strictly necessary), but > what > > > is > > > >> the > > > >> > >> > > > > justification > > > >> > >> > > > > >>> for Spring Boot over Nodejs? Why would want one > over > > > the > > > >> > >> other? > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> On Mon, Sep 17, 2018 at 3:38 PM Ryan Merriman < > > > >> > >> > > merrim...@gmail.com> > > > >> > >> > > > > >>> wrote: > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > I have reviewed a couple different PRs so I'll > add > > > some > > > >> > >> > context > > > >> > >> > > > > where I > > > >> > >> > > > > >>> > can. Obviously Simon would be the most qualified > > to > > > >> > answer > > > >> > >> but > > > >> > >> > > > I'll > > > >> > >> > > > > >>> add my > > > >> > >> > > > > >>> > thoughts. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 1, while they may not all be > > necessary > > > I > > > >> > >> think it > > > >> > >> > > > does > > > >> > >> > > > > make > > > >> > >> > > > > >>> > sense to include them in this feature branch if > > our > > > >> > primary > > > >> > >> > goal > > > >> > >> > > > is > > > >> > >> > > > > >>> > integrating Knox SSO. We could push off removing > > > JDBC > > > >> > >> > > > authentication > > > >> > >> > > > > >>> for > > > >> > >> > > > > >>> > reasons I'll get to in my response to question > 2. > > > If we > > > >> > >> want > > > >> > >> > to > > > >> > >> > > do > > > >> > >> > > > > one > > > >> > >> > > > > >>> at > > > >> > >> > > > > >>> > a time (switch to spring boot, add Zuul as a > > > >> dependency, > > > >> > >> then > > > >> > >> > > add > > > >> > >> > > > > Knox > > > >> > >> > > > > >>> SSO) > > > >> > >> > > > > >>> > then that's ok but I do think there are > > dependencies > > > >> and > > > >> > >> > should > > > >> > >> > > be > > > >> > >> > > > > done > > > >> > >> > > > > >>> in > > > >> > >> > > > > >>> > order. For example, adding Knox SSO requires > some > > > work > > > >> > >> around > > > >> > >> > > > > request > > > >> > >> > > > > >>> > filtering. If we were to do this before moving > to > > > >> Spring > > > >> > >> Boot > > > >> > >> > we > > > >> > >> > > > > would > > > >> > >> > > > > >>> > need to implement the filters in Nodejs which > > would > > > be > > > >> > >> > throwaway > > > >> > >> > > > > once we > > > >> > >> > > > > >>> > get around to migrating away from that. For > Zuul, > > I > > > >> > believe > > > >> > >> > it's > > > >> > >> > > > > >>> purpose > > > >> > >> > > > > >>> > is to facilitate the filtering (although it > does a > > > lot > > > >> > >> more) > > > >> > >> > so > > > >> > >> > > it > > > >> > >> > > > > >>> doesn't > > > >> > >> > > > > >>> > make sense to add that separate from the Knox > SSO > > > work. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 2, I think you bring up a good > point. > > > We > > > >> > >> probably > > > >> > >> > > > don't > > > >> > >> > > > > >>> want > > > >> > >> > > > > >>> > to just rip our current authentication method > out. > > > We > > > >> > might > > > >> > >> > want > > > >> > >> > > > to > > > >> > >> > > > > >>> > consider deprecating it instead and making Knox > > SSO > > > and > > > >> > >> LDAP > > > >> > >> > > > > >>> authentication > > > >> > >> > > > > >>> > optional. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 3, this is a bigger shift than > just a > > > >> > >> component > > > >> > >> > > > > upgrade. > > > >> > >> > > > > >>> It's > > > >> > >> > > > > >>> > more like shifting platforms, from Elasticsearch > > to > > > >> Solr > > > >> > >> for > > > >> > >> > > > > example. > > > >> > >> > > > > >>> Like > > > >> > >> > > > > >>> > I alluded to in my response to question 1, I > don't > > > >> think > > > >> > we > > > >> > >> > > should > > > >> > >> > > > > >>> require > > > >> > >> > > > > >>> > throwaway work just because we want to review > > these > > > >> parts > > > >> > >> > > > > separately. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 4, I will defer to Simon. I don't > > > believe > > > >> we > > > >> > >> > > > > necessarily > > > >> > >> > > > > >>> > require Zuul so I will let him elaborate on why > he > > > >> choose > > > >> > >> that > > > >> > >> > > > > library > > > >> > >> > > > > >>> and > > > >> > >> > > > > >>> > what the potential impact is of adding it to our > > > >> project. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 5 and 6, I will also defer to Simon > > on > > > >> this. > > > >> > >> The > > > >> > >> > > > focus > > > >> > >> > > > > of > > > >> > >> > > > > >>> > this feature as I understand it is a consistent > > > >> > >> authentication > > > >> > >> > > > > mechanism > > > >> > >> > > > > >>> > and support for SSO. I will let him lay out his > > > vision > > > >> > for > > > >> > >> > micro > > > >> > >> > > > > >>> services. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > Knox SSO would be a great improvement and is > what > > I > > > >> think > > > >> > >> we > > > >> > >> > > > should > > > >> > >> > > > > >>> focus > > > >> > >> > > > > >>> > on in this feature branch. Micro services is > > > something > > > >> we > > > >> > >> > should > > > >> > >> > > > > >>> certainly > > > >> > >> > > > > >>> > discuss but it might be a bit of a distraction > > and I > > > >> > >> wouldn't > > > >> > >> > > want > > > >> > >> > > > > to > > > >> > >> > > > > >>> hold > > > >> > >> > > > > >>> > up the other useful parts. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > On Fri, Sep 14, 2018 at 8:38 PM Michael > Miklavcic > > < > > > >> > >> > > > > >>> > michael.miklav...@gmail.com> wrote: > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > > Hey all, > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > I started looking through the Knox SSO feature > > > branch > > > >> > >> (see > > > >> > >> > > here > > > >> > >> > > > > >>> > > > > https://issues.apache.org/jira/browse/METRON-1663 > > > ). > > > >> > >> This is > > > >> > >> > > > some > > > >> > >> > > > > >>> great > > > >> > >> > > > > >>> > new > > > >> > >> > > > > >>> > > security functionality work and it looks like > it > > > will > > > >> > >> bring > > > >> > >> > > some > > > >> > >> > > > > >>> > important > > > >> > >> > > > > >>> > > new features to the Metron platform. I'm > coming > > at > > > >> this > > > >> > >> > pretty > > > >> > >> > > > > green, > > > >> > >> > > > > >>> so > > > >> > >> > > > > >>> > I > > > >> > >> > > > > >>> > > do have some questions regarding the proposed > > > changes > > > >> > >> from a > > > >> > >> > > > high > > > >> > >> > > > > >>> level > > > >> > >> > > > > >>> > > architectural perspective. There are a few > > changes > > > >> > within > > > >> > >> > the > > > >> > >> > > > > current > > > >> > >> > > > > >>> FB > > > >> > >> > > > > >>> > > PR's that I think could use some further > > > explanation. > > > >> > At > > > >> > >> > first > > > >> > >> > > > > >>> glance, it > > > >> > >> > > > > >>> > > seems we could potentially simplify this > branch > > a > > > >> great > > > >> > >> deal > > > >> > >> > > and > > > >> > >> > > > > get > > > >> > >> > > > > >>> it > > > >> > >> > > > > >>> > > completed much sooner if we narrowed the > focus a > > > bit. > > > >> > >> But I > > > >> > >> > > > could > > > >> > >> > > > > >>> > certainly > > > >> > >> > > > > >>> > > be wrong here and happy for other opinions. I > > > >> searched > > > >> > >> > through > > > >> > >> > > > the > > > >> > >> > > > > >>> > mailing > > > >> > >> > > > > >>> > > list history to see if there is any additional > > > >> > background > > > >> > >> > and > > > >> > >> > > > the > > > >> > >> > > > > main > > > >> > >> > > > > >>> > > DISCUSS thread I could find was regarding > > > initially > > > >> > >> setting > > > >> > >> > up > > > >> > >> > > > the > > > >> > >> > > > > >>> > feature > > > >> > >> > > > > >>> > > branch, which talked about adding Knox and > LDAP. > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E > > > >> > >> > > > > >>> > > . > > > >> > >> > > > > >>> > > If I've missed any follow-up, please let me > > know. > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > Looking at the broader set of Jiras associated > > > with > > > >> > 1663 > > > >> > >> and > > > >> > >> > > the > > > >> > >> > > > > >>> first PR > > > >> > >> > > > > >>> > > 1665, it looks like there are 4 main thrusts > of > > > this > > > >> > >> branch > > > >> > >> > > > right > > > >> > >> > > > > now: > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > 1. Knox/SSO > > > >> > >> > > > > >>> > > 2. Node migrated to Spring Boot > > > >> > >> > > > > >>> > > 3. JDBC removed completely in favor of LDAP > > > >> > >> > > > > >>> > > 4. Introduction of Zuul, also microservices? > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > I strongly urge for the purpose of reviewing > > this > > > >> > feature > > > >> > >> > > branch > > > >> > >> > > > > that > > > >> > >> > > > > >>> we > > > >> > >> > > > > >>> > > base much of the discussion off of > > > >> > >> > > > > >>> > > > > https://issues.apache.org/jira/browse/METRON-1755 > > > , > > > >> the > > > >> > >> > > > > architecture > > > >> > >> > > > > >>> > > diagram. Minimally, an explanation of the > > current > > > >> > >> > architecture > > > >> > >> > > > > along > > > >> > >> > > > > >>> with > > > >> > >> > > > > >>> > > discussion around the additional proposed > > changes > > > and > > > >> > >> > > rationale > > > >> > >> > > > > would > > > >> > >> > > > > >>> be > > > >> > >> > > > > >>> > > useful for evaluation. I don't have a solid > > enough > > > >> > >> > > understanding > > > >> > >> > > > > yet > > > >> > >> > > > > >>> of > > > >> > >> > > > > >>> > the > > > >> > >> > > > > >>> > > full scope of changes and how they differ from > > the > > > >> > >> existing > > > >> > >> > > > > >>> architecture > > > >> > >> > > > > >>> > > just from looking at the PR's alone. > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > 1. The first question is a general one > regarding > > > the > > > >> > >> > necessity > > > >> > >> > > > of > > > >> > >> > > > > >>> the > > > >> > >> > > > > >>> > 3 > > > >> > >> > > > > >>> > > additional features alongside Knox - migrating > > > Node > > > >> to > > > >> > >> > Spring > > > >> > >> > > > > Boot, > > > >> > >> > > > > >>> > > removing JDBC altogether, adding dependencies > on > > > >> > >> Netflix's > > > >> > >> > > Zuul > > > >> > >> > > > > >>> > > framework. > > > >> > >> > > > > >>> > > Are these necessary for adding Knox/SSO? They > > seem > > > >> like > > > >> > >> > > > > potentially > > > >> > >> > > > > >>> > > separate features, imo. > > > >> > >> > > > > >>> > > 2. It looks like LDAP will be a required > > component > > > >> for > > > >> > >> > > > interacting > > > >> > >> > > > > >>> > with > > > >> > >> > > > > >>> > > Metron via the UI's. I see this PR > > > >> > >> > > > > >>> > > https://github.com/apache/metron/pull/1186 > > which > > > >> > removes > > > >> > >> > JDBC > > > >> > >> > > > > >>> > > authentication. Are we ready to remove it > > > completely > > > >> or > > > >> > >> > would > > > >> > >> > > it > > > >> > >> > > > > be > > > >> > >> > > > > >>> > > better > > > >> > >> > > > > >>> > > to leave it as a minimal installation option? > > > What is > > > >> > the > > > >> > >> > > > proposed > > > >> > >> > > > > >>> > > migration path for existing users? Do we feel > > > >> > comfortable > > > >> > >> > > > > requiring > > > >> > >> > > > > >>> > that > > > >> > >> > > > > >>> > > all installations, including full dev, install > > and > > > >> > >> configure > > > >> > >> > > > LDAP? > > > >> > >> > > > > >>> For > > > >> > >> > > > > >>> > > comparison, in the PCAP feature branch we > > > discussed > > > >> > >> removing > > > >> > >> > > the > > > >> > >> > > > > >>> > > existing > > > >> > >> > > > > >>> > > PCAP REST application in the initial > discussion, > > > got > > > >> > >> > > agreement, > > > >> > >> > > > > and > > > >> > >> > > > > >>> > > later > > > >> > >> > > > > >>> > > removed it in the course of working on the > > feature > > > >> > >> branch. > > > >> > >> > The > > > >> > >> > > > PR > > > >> > >> > > > > >>> is > > > >> > >> > > > > >>> > > fairly > > > >> > >> > > > > >>> > > clear, however I think we're just missing some > > > basic > > > >> > >> > > discussion > > > >> > >> > > > > >>> around > > > >> > >> > > > > >>> > > the > > > >> > >> > > > > >>> > > implications, as I've outlined above. Some > > > additional > > > >> > >> > relevant > > > >> > >> > > > > >>> > > discussion > > > >> > >> > > > > >>> > > occurred on this PR > > > >> > >> > > https://github.com/apache/metron/pull/1112 > > > >> > >> > > > > >>> which > > > >> > >> > > > > >>> > > would be good to summarize for purposes of > this > > > >> > >> overarching > > > >> > >> > > > > >>> > architecture > > > >> > >> > > > > >>> > > discussion. > > > >> > >> > > > > >>> > > 3. Migration from Node to Spring Boot. I > believe > > > this > > > >> > is > > > >> > >> > > already > > > >> > >> > > > > >>> used > > > >> > >> > > > > >>> > by > > > >> > >> > > > > >>> > > the REST application and if anything brings > some > > > >> > >> cohesion to > > > >> > >> > > our > > > >> > >> > > > > >>> > server > > > >> > >> > > > > >>> > > strategy. Strictly speaking, is there a reason > > > this > > > >> is > > > >> > >> > > required > > > >> > >> > > > > for > > > >> > >> > > > > >>> > > Knox? > > > >> > >> > > > > >>> > > It seems comparable to a component upgrade, > such > > > as > > > >> > >> moving > > > >> > >> > > from > > > >> > >> > > > ES > > > >> > >> > > > > >>> 2.x > > > >> > >> > > > > >>> > > to > > > >> > >> > > > > >>> > > 5.6.x and upgrading Angular 6. > > > >> > >> > > > > >>> > > 4. Introduction of Netflix's Zuul. > > > >> > >> > > > > >>> > > > > https://issues.apache.org/jira/browse/METRON-1665 > > > . > > > >> > >> > > > > >>> > > - > "The UIs currently proxy to the REST API > to > > > avoid > > > >> > >> CORS > > > >> > >> > > > > >>> issues, > > > >> > >> > > > > >>> > > this will be achieved with Zuul." > > > >> > >> > > > > >>> > > - Can we elaborate more on where or how CORS > is > > a > > > >> > problem > > > >> > >> > with > > > >> > >> > > > > >>> our > > > >> > >> > > > > >>> > > existing architecture, how Zuul will help > solve > > > that, > > > >> > and > > > >> > >> > how > > > >> > >> > > it > > > >> > >> > > > > >>> > > fits with > > > >> > >> > > > > >>> > > Knox? Wouldn't this be handled by Knox? Since > > > Larry > > > >> > McCay > > > >> > >> > > > > >>> chimed in > > > >> > >> > > > > >>> > > with > > > >> > >> > > > > >>> > > interest on the original SSO thread about the > > FB, > > > I'm > > > >> > >> hoping > > > >> > >> > > he > > > >> > >> > > > > >>> is > > > >> > >> > > > > >>> > > also > > > >> > >> > > > > >>> > > willing to chime in on this as well. > > > >> > >> > > > > >>> > > - This looks like it has the potential to be a > > > rather > > > >> > >> large > > > >> > >> > > > > >>> piece > > > >> > >> > > > > >>> > of > > > >> > >> > > > > >>> > > fundamental infrastructure (as it's also > > > pertinent to > > > >> > >> > > > > >>> > microservices) > > > >> > >> > > > > >>> > > to > > > >> > >> > > > > >>> > > pull into the platform, and I'd like to be > sure > > > the > > > >> > >> > community > > > >> > >> > > is > > > >> > >> > > > > >>> > > aware of > > > >> > >> > > > > >>> > > and is OK with the implications. > > > >> > >> > > > > >>> > > 5. > "The proposal is to use a spring boot > > > >> application, > > > >> > >> > > allowing > > > >> > >> > > > > >>> us to > > > >> > >> > > > > >>> > > harmonize the security implementation across > the > > > UI > > > >> > >> static > > > >> > >> > > > servers > > > >> > >> > > > > >>> and > > > >> > >> > > > > >>> > > the > > > >> > >> > > > > >>> > > REST layer, and to provide a routing platform > > for > > > >> later > > > >> > >> > > > > >>> > microservices." > > > >> > >> > > > > >>> > > - > > > >> > >> > > > > >>> > > > > https://issues.apache.org/jira/browse/METRON-1665 > > > . > > > >> > >> > > > > >>> > > - Microservices is a pretty loaded term. I > know > > > there > > > >> > had > > > >> > >> > been > > > >> > >> > > > > >>> some > > > >> > >> > > > > >>> > > discussion a while back during the PCAP > feature > > > >> branch > > > >> > >> > start, > > > >> > >> > > > > >>> but I > > > >> > >> > > > > >>> > > don't > > > >> > >> > > > > >>> > > recall ever reaching a consensus on it. More > > > detail > > > >> in > > > >> > >> this > > > >> > >> > > > > >>> thread > > > >> > >> > > > > >>> > - > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E > > > >> > >> > > > > >>> > > . > > > >> > >> > > > > >>> > > Can we get some clarification on what is meant > > by > > > >> > >> > > microservices > > > >> > >> > > > > >>> > > in the case > > > >> > >> > > > > >>> > > of this FB and relevant PR's, what that > > > architecture > > > >> > >> looks > > > >> > >> > > like, > > > >> > >> > > > > >>> > and > > > >> > >> > > > > >>> > > how > > > >> > >> > > > > >>> > > it's achieved with the proposed changes in > this > > > >> PR/FB? > > > >> > It > > > >> > >> > > seems > > > >> > >> > > > > >>> > Zuul > > > >> > >> > > > > >>> > > is > > > >> > >> > > > > >>> > > also pertinent to this discussion, but there > are > > > many > > > >> > >> ways > > > >> > >> > to > > > >> > >> > > > > >>> > > skin this cat > > > >> > >> > > > > >>> > > so I don't want to presume - > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > >> > > > > https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices > > > >> > >> > > > > >>> > > 6. Zuul, Spring Boot, and microservices - > > Closely > > > >> > >> related to > > > >> > >> > > > > >>> > point 5 > > > >> > >> > > > > >>> > > above. It seems that we weren't quite ready > for > > > this > > > >> > >> when it > > > >> > >> > > was > > > >> > >> > > > > >>> > > brought up > > > >> > >> > > > > >>> > > in May, or at the very least we had some > concern > > > of > > > >> > what > > > >> > >> > > > direction > > > >> > >> > > > > >>> to > > > >> > >> > > > > >>> > > go. > > > >> > >> > > > > >>> > > What is the operational impact, mpack impact, > > and > > > how > > > >> > we > > > >> > >> > > propose > > > >> > >> > > > > to > > > >> > >> > > > > >>> > > manage > > > >> > >> > > > > >>> > > it with Kerberos, etc.? > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > There is a lot to like in this feature branch, > > > imo. > > > >> > Great > > > >> > >> > > > feature > > > >> > >> > > > > >>> > addition > > > >> > >> > > > > >>> > > with Knox and SSO. Introduction of LDAP > support > > > for > > > >> > >> > > > > authentication for > > > >> > >> > > > > >>> > > Metron UI's. Simplification/unification of our > > > server > > > >> > >> > hosting > > > >> > >> > > > > >>> > > infrastructure. I'm hoping we can flesh out > some > > > of > > > >> the > > > >> > >> > > details > > > >> > >> > > > > >>> pointed > > > >> > >> > > > > >>> > out > > > >> > >> > > > > >>> > > above a bit more and get this feature through. > > > Great > > > >> > >> work so > > > >> > >> > > > far! > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > Best, > > > >> > >> > > > > >>> > > Mike Miklavcic > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > >> > >> > > > > >> > > > >> > >> > > > > >> -- > > > >> > >> > > > > >> -- > > > >> > >> > > > > >> simon elliston ball > > > >> > >> > > > > >> @sireb > > > >> > >> > > > > > > > > >> > >> > > > > > -- > > > >> > >> > > > > > -- > > > >> > >> > > > > > simon elliston ball > > > >> > >> > > > > > @sireb > > > >> > >> > > > > > > > >> > >> > > > > ------------------- > > > >> > >> > > > > Thank you, > > > >> > >> > > > > > > > >> > >> > > > > James Sirota > > > >> > >> > > > > PMC- Apache Metron > > > >> > >> > > > > jsirota AT apache DOT org > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > > >> > > > > >> -- > > > >> > > > >> Jon Zeolla > > > > > > ------------------- > > > Thank you, > > > > > > James Sirota > > > PMC- Apache Metron > > > jsirota AT apache DOT org > > > > > >
