1) Sorry I misspoke. I meant to say this is not possible in the Alerts UI as far as I know. I put up a PR with a proposed solution here: https://github.com/apache/metron/pull/1266. 2) Yes Knox is a service you can install with Ambari, similar to Ranger or Spark. There are some things that are specifically configured in Knox and there are some things specific to Metron. I will put up a PR with the changes needed so you can see exactly what is involved. 3) I don't understand what you mean here. Is this a question? 4) I think it's a little early to predict the Ambari changes required. This will depend on how tasks 1-3 go. I imagine it's similar to other mpack work: expose some parameters in ambari and bind those to config files. My understanding from this thread so far is that we should focus on a manual, documented approach to start.
On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Thanks Ryan, > > 1) Can you clarify "not a good way to do this?" Are you saying we don't > have a way to set this and need to add the config option, or that a > solution is not obvious and it's unclear what to do? It seems to me you're > saying the former, but I'd like to be sure. > 2) Is Knox not a service made available by Ambari similar to Ranger or > Spark? I'm assuming that similar to Kerberos, there are some things that > are specifically configured in Knox and others that are app-specific. Some > explanation of what this looks like would be helpful. > 3) Sounds like this follows pretty naturally from #1 > 4) Relates to #2. I think we need some guidance on what a manual vs > MPack/automated install would look like. > > Cheers, > Mike > > > On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman <merrim...@gmail.com> wrote: > > > Wanted to give an update on the context path issue. I investigated > > rewriting url links in the outgoing static assets with Knox and it was > not > > trivial. Fortunately I found a simple solution that works with or > without > > Knox. I changed the base tag in index.html from <base href="/"> to <base > > href="./">, or in other words made the base href relative. > > > > I believe I am at the point where I can task this out and provide a high > > level overview of the changes needed. I think that each task will be a > > manageable size and can stand alone so I don't think we need a feature > > branch. > > > > The first task involves a general change to the UI code. We need a way > to > > set the path to the REST service with a configuration setting because it > is > > different with and without Knox. Currently there is not a good way to do > > this in the UI. We can use the environment files but that is a build > time > > setting and is not flexible. I can see this capability being useful for > > other use cases in the future. I think we could even split this up into > 2 > > separate tasks, one for the alerts UI and one for the management UI. > > > > The second task involves adding Knox to our stack either by default as a > > dependency in the mpack or with a documented approach. We would add our > > REST service, Alerts UI, and Management UI as services in Knox. > Everything > > would continue to function as it currently does but with all > communication > > going through Knox. LDAP authentication would be required when using > Knox > > and Knox will authenticate with the REST service by passing along an > > Authorization header. Enabling Knox would be a manual process that > > involves deploying assets (Knox descriptor files) and changing > > configuration. There would be no change to how the UI functions by > default > > (without Knox) and either LDAP or JDBC authentication could still be > used.. > > > > The third task involves enabling SSO with Knox. We would update the REST > > service so that it can authenticate with a Knox SSO token. We would > > provide documentation on how to update the Knox settings in Ambari to > > enable SSO. The Alerts/Management UI would need to expose configuration > > properties for the REST url and login url since these would be different > > when Knox is enabled. We would also need to provide documentation on how > > to make these UI configuration changes, based on the work done in task 1. > > > > An optional forth task would be exposing configuration settings and > > enabling Knox with Ambari. We would eliminate the manual steps necessary > > for enabling Knox and instead automate those steps with an Ambari input > > control, similar to how LDAP is enabled. > > > > Thoughts on this plan? > > > > On Thu, Nov 15, 2018 at 10:22 AM James Sirota <jsir...@apache.org> > wrote: > > > > > In my view Knox SSO is such a minor feature when it comes to Metron's > > > capabilities than it's not worth supporting multiple scenarios where it > > > works with Knox or without Knox. Where we should be configurable (and > > are > > > configurable) is on the analytics and stream processing. But this? As > > > long as the UI authenticates securely I don't think anyone is going to > > care > > > what proxy it's using. The code itself should be written in a way that > > > it's pluggable so if we ever wanted to use another proxy or disable it > > all > > > together we could. But this should not be a configuration we pass on > to > > > the user. The added complexity is simply not worth it here. We have > to > > > start being opinionated about making sensible choices on behalf of the > > > user. A sensible choice here is to run with Knox and LDAP. The JDBC > > > component should exist for another release to allow the community to > > > migrate over to LDAP and then be deprecated. The code should still be > > > pluggable and if anyone wanted to extend it to work with JDBC they > could, > > > or if people wanted to plug in another proxy they could, but this is > not > > > something we would officially support. > > > > > > Thanks, > > > James > > > > > > 12.11.2018, 07:36, "Ryan Merriman" <merrim...@gmail.com>: > > > > Let me clarify on exposing both legacy and Knox URLs at the same > time. > > > The > > > > base urls will look something like this: > > > > > > > > Legacy REST - http://node1:8082/api/v1 > > > > Legacy Alerts UI - http://node1:4201:/alerts-list > > > > > > > > Knox REST - https://node1:8443/gateway/default/metron/api/v1 > > > > Knox Alerts UI - > > > > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > > > > > > > If Knox were turned on and the alerts UI deployed as is, it would not > > > > work. This is because static assets are referenced with > > > > http://node1:4201/assets/some-asset.js which does not include the > > > correct > > > > context path to the alerts UI in knox. To make it work, you have to > set > > > > the base ref to "/gateway/default/metron-alerts-ui" so that static > > assets > > > > are referenced at > > > > > > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js > > > . > > > > When you do that, the legacy alerts UI will no longer work. I guess > the > > > > point I'm trying to make is that we would have to switch between them > > or > > > > have 2 separate application running. I imagine most users only need > one > > > or > > > > the other running so probably not an issue. > > > > > > > > Jon, the primary upgrade consideration I see is with authentication. > To > > > be > > > > able to use Knox, you would have to upgrade to LDAP-based > > authentication > > > if > > > > you were still using JDBC-based authentication in REST. The urls > would > > > > also change obviously. > > > > > > > > On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com <zeo...@gmail.com> > > > wrote: > > > > > > > >> Phew, that was quite the thread to catch up on. > > > >> > > > >> I agree that this should be optional/pluggable to start, and I'm > > > interested > > > >> to hear the issues as they relate to upgrading an existing cluster > > > (given > > > >> the suggested approach) and exposing both legacy and knox URLs at > the > > > same > > > >> time. > > > >> > > > >> Jon > > > >> > > > >> On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < > > > >> michael.miklav...@gmail.com> > > > >> wrote: > > > >> > > > >> > A couple more things, and I think this goes without saying - > > > whatever we > > > >> do > > > >> > with Knox should NOT > > > >> > > > > >> > 1. Require unit and integration tests to use Knox > > > >> > 2. Break fulldev > > > >> > > > > >> > Also, I don't know that I saw you mention this, but I'm unsure > how > > we > > > >> > should leverage Knox as a core piece of the platform. i.e. should > > we > > > make > > > >> > this required or optional? I'm open to hearing opinions on this, > > but > > > I'm > > > >> > inclined to keep this a pluggable option. > > > >> > > > > >> > Mike > > > >> > > > > >> > > > > >> > On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic < > > > >> > michael.miklav...@gmail.com> wrote: > > > >> > > > > >> > > Thanks for the update Ryan. Per my earlier comments, I thought > it > > > might > > > >> > be > > > >> > > the case that we could dramatically simplify this by leveraging > > > Knox's > > > >> > > proxy capabilities, and per your research that appears to be > the > > > case. > > > >> > This > > > >> > > is a dramatic simplification and improvement of this feature > imo, > > > +1. > > > >> I'm > > > >> > > also +1 on a couple distinct steps that you've laid out: fix > the > > UI > > > >> > issues > > > >> > > in master, then add Knox for SSO. That should help mitigate > > issues > > > with > > > >> > > merge conflicts with ongoing development. > > > >> > > > > > >> > > > I think it will be a challenge exposing the UIs through both > > the > > > Knox > > > >> > > url and legacy urls at the same time. > > > >> > > I'm not sure I understand the issue here. Are you referring to > > this > > > >> > > comment? "Added a ng build option to build the UI with base > href > > > set to > > > >> > > Knox base path." Isn't it just a matter of URL > > > rewriting/forwarding? I > > > >> > > thought we'd be exposing the URL's directly in one context, and > > > through > > > >> > > Knox in the other. Either way, it seems like we should be able > to > > > >> > provide a > > > >> > > dynamic base path through configuration in our web > applications. > > > I'd > > > >> > expect > > > >> > > to modify that property based on whether Knox is configured or > > not. > > > >> > > > > > >> > > > I'm also not clear on how one would use Knox with REST set to > > > legacy > > > >> > > JDBC-based authentication. As far as I know Knox does not > support > > > JDBC > > > >> so > > > >> > > there would be a mismatch between Knox and REST. > > > >> > > I'm OK with not having Knox work with JDBC. That's a feature of > > > Knox > > > >> and > > > >> > > probably not something we care much about. > > > >> > > > > > >> > > >We could initially make Knox an optional feature that requires > > > setup > > > >> > with > > > >> > > the help of some documentation (like Kerberos) while keeping > the > > > system > > > >> > the > > > >> > > way it is now by default. > > > >> > > Sounds good to me. > > > >> > > > > > >> > > > I imagine we'll deprecate JDBC-based authentication at some > > > point so > > > >> > > that may be a good time to switch. > > > >> > > I would like to announce deprecation in our next release and > move > > > to > > > >> > > remove it in a following release. > > > >> > > > > > >> > > Thanks for taking this on and great job laying things out. > > > >> > > > > > >> > > Thanks, > > > >> > > Mike > > > >> > > > > > >> > > On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman < > > merrim...@gmail.com> > > > >> > wrote: > > > >> > > > > > >> > >> I have spent some time recently reviewing this discussion and > > the > > > >> > feature > > > >> > >> branch that Simon put out. I think this is an important > feature > > > and > > > >> > want > > > >> > >> to move it forward. I started another discussion on adding > Knox > > to > > > >> our > > > >> > >> stack but this discussion has a lot of good context so I will > > > continue > > > >> > it > > > >> > >> here. > > > >> > >> > > > >> > >> I think the main point of contention was that this feature > > branch > > > >> > included > > > >> > >> several different architectural changes and it was unclear if > > they > > > >> were > > > >> > >> needed and if so, could be done separately. Fortunately LDAP > > > >> > >> authentication has been accepted into master so we can cross > it > > > off > > > >> the > > > >> > >> list. From my understanding of the points people have made, > that > > > >> leaves > > > >> > >> Knox related SSO changes and migrating expressjs to a > different, > > > >> > JVM-based > > > >> > >> web server that includes proxying capabilities (Zuul). > > > >> > >> > > > >> > >> I think everyone agrees that if we can limit the scope to just > > > Knox > > > >> > >> related > > > >> > >> SSO changes that would be ideal. I believe I have found a way > to > > > do > > > >> > that > > > >> > >> while working on a small POC this week. The key to this (Simon > > > >> alluded > > > >> > to > > > >> > >> it earlier) is to put both REST and our UIs behind Knox. I > > > initially > > > >> > was > > > >> > >> focused on just adding REST as a service in Knox and decided > to > > > >> > experiment > > > >> > >> with also adding our UIs. After I did this it became clear > that > > > this > > > >> > >> simplifies things considerably: > > > >> > >> > > > >> > >> - The REST app and the UIs are now served from the same host > so > > > >> CORS > > > >> > >> concerns go away. > > > >> > >> - We no longer need to worry about proxying REST requests from > > the > > > >> > UIs > > > >> > >> with express or Zuul because Knox handles that for us. This > will > > > >> > make > > > >> > >> our > > > >> > >> express configuration even simpler. In fact, all we need is a > > > >> simple > > > >> > >> way > > > >> > >> to serve static UI assets. > > > >> > >> - We no longer need to check for SSO tokens and redirect in > the > > UI > > > >> > >> web/app servers (or the REST app for that matter) because Knox > > > >> > handles > > > >> > >> that > > > >> > >> for us. > > > >> > >> - The UIs can now easily access any Knox service (not just our > > > REST > > > >> > >> app) > > > >> > >> without any extra proxy configuration. > > > >> > >> - SSO token authentication is only necessary in REST so there > is > > > no > > > >> > >> need > > > >> > >> to create shared Spring modules or split functionality out. > > > >> > >> > > > >> > >> The most significant change I had to make (borrowed from > Simon's > > > >> feature > > > >> > >> branch) was the SSO token authentication mentioned above. The > > > primary > > > >> > >> short term benefit with this approach is that outside of some > > > general > > > >> > >> deficiencies unrelated to this our UI architecture doesn't > need > > to > > > >> > >> fundamentally change. I could summarize the changes as: > > > >> > >> > > > >> > >> - Knox install and configuration (setting up REST and the > alerts > > > UI > > > >> > as > > > >> > >> Knox services) > > > >> > >> - Added Knox SSO token authentication to REST > > > >> > >> - Updated REST urls in the UI code (should be configurable) > > > >> > >> - Fixed a few UI bugs where relative paths were not being used > > > >> > >> - Added a ng build option to build the UI with base href set > to > > > >> Knox > > > >> > >> base path ( > > > >> > >> > > > >> > >> > > > >> > > > > >> > > > > > > https://github.com/angular/angular-cli/wiki/build#base-tag-handling-in-indexhtml > > > >> > >> ) > > > >> > >> > > > >> > >> Most the UI changes are preexisting, minor issues that could > be > > > fixed > > > >> > >> directly in master. We would need to think of an approach for > > the > > > >> base > > > >> > >> href build requirement but I'm sure it's not that bad. > > > >> > >> > > > >> > >> However there will be some backwards compatibility issues we > > would > > > >> need > > > >> > to > > > >> > >> think through. I think it will be a challenge exposing the UIs > > > >> through > > > >> > >> both the Knox url and legacy urls at the same time. I'm also > not > > > >> clear > > > >> > on > > > >> > >> how one would use Knox with REST set to legacy JDBC-based > > > >> > authentication. > > > >> > >> As far as I know Knox does not support JDBC so there would be > a > > > >> mismatch > > > >> > >> between Knox and REST. Knox does have the ability to pass > along > > > basic > > > >> > >> authentication headers so LDAP in REST would work. We could > > > initially > > > >> > >> make > > > >> > >> Knox an optional feature that requires setup with the help of > > some > > > >> > >> documentation (like Kerberos) while keeping the system the way > > it > > > is > > > >> now > > > >> > >> by > > > >> > >> default. I imagine we'll deprecate JDBC-based authentication > at > > > some > > > >> > >> point > > > >> > >> so that may be a good time to switch. > > > >> > >> > > > >> > >> What do people think about this approach? Concerns? Are there > > any > > > >> huge > > > >> > >> holes in this I'm not thinking about? > > > >> > >> > > > >> > >> I want to highlight that the work Simon did in his feature > > branch > > > was > > > >> > >> crucial to better understanding this. I am pretty sure we'll > end > > > up > > > >> > >> reusing a lot code from that branch. > > > >> > >> > > > >> > >> On Thu, Sep 27, 2018 at 6:30 PM Michael Miklavcic < > > > >> > >> michael.miklav...@gmail.com> wrote: > > > >> > >> > > > >> > >> > Apparently, I hit send on my last email before finishing my > > > synopsis > > > >> > >> (per > > > >> > >> > @Otto's Q in Slack). To summarize, based on my current > > > >> understanding I > > > >> > >> > believe that each of the feature branch changes I've outline > > > above > > > >> are > > > >> > >> > units of work that are related, yet should be executed on > > > >> > independently. > > > >> > >> > Knox SSO in its own feature branch. Migrating technologies > > like > > > >> NodeJs > > > >> > >> or > > > >> > >> > migrating the auth DB to LDAP seem like they belong in their > > own > > > >> > >> separate > > > >> > >> > PR's or feature branches. > > > >> > >> > > > > >> > >> > Thanks, > > > >> > >> > Mike > > > >> > >> > > > > >> > >> > On Thu, Sep 27, 2018 at 4:08 PM Casey Stella < > > > ceste...@gmail.com> > > > >> > >> wrote: > > > >> > >> > > > > >> > >> > > I'm coming in late to the game here, but for my mind a > > feature > > > >> > branch > > > >> > >> > > should involve the minimum architectural change to > > accomplish > > > a > > > >> > given > > > >> > >> > > feature. > > > >> > >> > > The feature in question is SSO integration. It seems to me > > > that > > > >> the > > > >> > >> > > operative question is can we do the feature without making > > the > > > >> OTHER > > > >> > >> > > architectural change > > > >> > >> > > (e.g. migrating from expressjs to spring boot + zuul). I > > would > > > >> > argue > > > >> > >> > that > > > >> > >> > > if we WANT to do that, then it should be a separate > feature > > > >> branch. > > > >> > >> > > > > > >> > >> > > Thus, I leave with a question: is there a way to > accomplish > > > this > > > >> > >> feature > > > >> > >> > > without ripping out expressjs? > > > >> > >> > > > > > >> > >> > > - If so and it is feasible, I would argue that we should > > > >> decouple > > > >> > >> this > > > >> > >> > > into a separate feature branch. > > > >> > >> > > - If so and it is infeasible, I'd like to hear an argument > > as > > > >> to > > > >> > >> the > > > >> > >> > > infeasibility and let's decide given that > > > >> > >> > > - If it is not possible, then I'd argue that we should > keep > > > >> them > > > >> > >> > coupled > > > >> > >> > > and move this through as-is. > > > >> > >> > > > > > >> > >> > > On a side-note, it feels a bit weird that we're narrowing > > to a > > > >> > bundled > > > >> > >> > > proxy, rather than having that be a pluggable thing. I'm > not > > > >> super > > > >> > >> > > knowledgeable in this space, so I apologize > > > >> > >> > > in advance if this is naive, but isn't this a pluggable, > > > external > > > >> > >> > component > > > >> > >> > > (e.g. nginx)? > > > >> > >> > > > > > >> > >> > > On Thu, Sep 27, 2018 at 5:05 PM Michael Miklavcic < > > > >> > >> > > michael.miklav...@gmail.com> wrote: > > > >> > >> > > > > > >> > >> > > > I've spent some more time reading through Simon's > response > > > and > > > >> the > > > >> > >> > added > > > >> > >> > > > sequence diagram. This is definitely helpful - thank you > > > Simon. > > > >> > >> > > > > > > >> > >> > > > I need to redact my initial list: > > > >> > >> > > > > > > >> > >> > > > 1. Node migrated to Spring Boot, expressjs migrated to a > > > >> > >> > > > non-JS/non-NodeJs proxying mechanism (ie Zuul in this > > case) > > > >> > >> > > > 2. JDBC removed completely in favor of LDAP > > > >> > >> > > > 3. Knox/SSO > > > >> > >> > > > > > > >> > >> > > > I'm a bit conflicted on the best way to move forward and > > > would > > > >> > like > > > >> > >> > some > > > >> > >> > > > thoughts from other community members on this. I think > an > > > >> argument > > > >> > >> can > > > >> > >> > be > > > >> > >> > > > made that 1 and 2 are independent of 3, and should/could > > > really > > > >> be > > > >> > >> > > > independent PR's against master. > > > >> > >> > > > > > > >> > >> > > > The need for a replacement for expressjs (Zuul in this > > > case) is > > > >> an > > > >> > >> > > artifact > > > >> > >> > > > that our request/response cycle for REST calls is a > simple > > > >> matter > > > >> > of > > > >> > >> > > > forwarding with some additional headers for > > authentication. > > > >> > There's > > > >> > >> a > > > >> > >> > > > JSESSIONID managed by the client browser in our current > > > >> > >> architecture, > > > >> > >> > for > > > >> > >> > > > example. You login to the alerts or the management UI > > which > > > >> > >> forwards a > > > >> > >> > > > request to REST, which looks up credentials in a backend > > > >> database, > > > >> > >> and > > > >> > >> > > > passes the results back up the chain. All browser > requests > > > go > > > >> > >> directly > > > >> > >> > to > > > >> > >> > > > the specific UI you're working with - this is the CORS > > > problem. > > > >> > You > > > >> > >> > > can't, > > > >> > >> > > > without some effort with headers for adding other > domains > > > to the > > > >> > >> safe > > > >> > >> > > list > > > >> > >> > > > or disabling the security check for CORS, make remote > > calls > > > >> > >> directly to > > > >> > >> > > > REST. That's why we proxy. Switching over to Spring Boot > > > leaves > > > >> a > > > >> > >> gap > > > >> > >> > > with > > > >> > >> > > > expressjs having handled the proxying and filtering, > since > > > it's > > > >> > only > > > >> > >> > > > available to a NodeJs application (it's server-side > > > javascript > > > >> vs > > > >> > >> the > > > >> > >> > > > client side javascript deployed via our Angular > > > applications). > > > >> > Enter > > > >> > >> > > Zuul, > > > >> > >> > > > which now effectively handles that. At runtime, Zuul is > a > > > part > > > >> of > > > >> > >> the > > > >> > >> > > > Spring app that serves up our UI's. It handles the > > requests > > > via > > > >> > >> > > filtering, > > > >> > >> > > > forwards them to REST, manages the response back to the > > > client. > > > >> > Very > > > >> > >> > > > similar to what expressjs was doing, per my current > > > >> understanding. > > > >> > >> The > > > >> > >> > > > sequence diagrams Simon added are useful, and I think > some > > > of > > > >> what > > > >> > >> was > > > >> > >> > > less > > > >> > >> > > > clear was what we currently vs what the new changes are > > > doing to > > > >> > the > > > >> > >> > > > architecture. This is no fault of Simon's - there simply > > > wasn't > > > >> > any > > > >> > >> > > > architecture diagrams/documents around this before. > Here's > > > my > > > >> > >> > impression > > > >> > >> > > of > > > >> > >> > > > the very very basic current state - someone more > familiar > > > with > > > >> > this > > > >> > >> > > > architecture please advise if I'm incorrect about > anything > > > >> > (probably > > > >> > >> > > Ryan). > > > >> > >> > > > > > > >> > >> > > > https://imgur.com/f8GtSmh > > > >> > >> > > > > > > >> > >> > > > Zuul would be replacing the bit about expressjs in the > > > diagram, > > > >> > and > > > >> > >> > > instead > > > >> > >> > > > of node we have spring boot. This covers 1. 2 and 3 are > > > other > > > >> > >> issues. > > > >> > >> > I'd > > > >> > >> > > > like to see similar exposition of those server processes > > > with > > > >> knox > > > >> > >> > > > involved. I imagine in that case we bump up from 3 to 4 > > > server > > > >> > >> > instances > > > >> > >> > > > for the additional knox endpoint. > > > >> > >> > > > > > > >> > >> > > > Mike > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > On Wed, Sep 19, 2018 at 11:28 AM James Sirota < > > > >> jsir...@apache.org > > > >> > > > > > >> > >> > > wrote: > > > >> > >> > > > > > > >> > >> > > > > Thank you, Simon. The diagrams help a lot > > > >> > >> > > > > > > > >> > >> > > > > 19.09.2018, 21:27, "Simon Elliston Ball" < > > > >> > >> > si...@simonellistonball.com > > > >> > >> > > >: > > > >> > >> > > > > > To clarify some of this I've put some documentation > > into > > > >> > >> > > > > > https://github.com/apache/metron/pull/1203 under > > > >> METRON-1755 > > > >> > ( > > > >> > >> > > > > > https://issues.apache.org/jira/browse/METRON-1755). > > > >> Hopefully > > > >> > >> the > > > >> > >> > > > > diagrams > > > >> > >> > > > > > there should make it clearer. > > > >> > >> > > > > > > > > >> > >> > > > > > Simon > > > >> > >> > > > > > > > > >> > >> > > > > > On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < > > > >> > >> > > > > > si...@simonellistonball.com> wrote: > > > >> > >> > > > > > > > > >> > >> > > > > >> Hi Mike, > > > >> > >> > > > > >> > > > >> > >> > > > > >> Some good points here which could do with some > > > >> > clarification. > > > >> > >> I > > > >> > >> > > > suspect > > > >> > >> > > > > >> the architecture documentation could be clearer and > > > fill > > > >> in > > > >> > >> some > > > >> > >> > of > > > >> > >> > > > > these > > > >> > >> > > > > >> gaps, and I'll have a look at working on that and > > > >> providing > > > >> > >> some > > > >> > >> > > > > diagrams. > > > >> > >> > > > > >> > > > >> > >> > > > > >> The short version is that the Zuul proxy gateway > has > > > been > > > >> > >> added > > > >> > >> > to > > > >> > >> > > > > replace > > > >> > >> > > > > >> the Nodejs express proxy used to gateway the REST > api > > > >> calls > > > >> > in > > > >> > >> > the > > > >> > >> > > > > current > > > >> > >> > > > > >> hosts. This is done in both cases to avoid CORS > > > >> restrictions > > > >> > >> by > > > >> > >> > > > > allowing > > > >> > >> > > > > >> the same host that serves the UI files to proxy > call > > to > > > >> the > > > >> > >> API. > > > >> > >> > > > > >> > > > >> > >> > > > > >> The choice of Zuul was partly a pragmatic one (it's > > the > > > >> one > > > >> > >> > that's > > > >> > >> > > > > there > > > >> > >> > > > > >> in the box as it were with Spring Boot, which we > use > > > for > > > >> the > > > >> > >> REST > > > >> > >> > > > API, > > > >> > >> > > > > via > > > >> > >> > > > > >> the Spring Cloud Netflix project which wraps a > bunch > > of > > > >> > >> related > > > >> > >> > > > pieces > > > >> > >> > > > > into > > > >> > >> > > > > >> Spring). The choice of Spring Boot to host the UIs > > > >> > themselves > > > >> > >> was > > > >> > >> > > > > similarly > > > >> > >> > > > > >> for parity with the REST host, to simplify the > stack > > > (we > > > >> > >> remove > > > >> > >> > the > > > >> > >> > > > > >> occasionally problematic need to install nodejs on > > > target > > > >> > >> > servers, > > > >> > >> > > > > which is > > > >> > >> > > > > >> outside of the regular OS and HDP stacks we > support). > > > >> > >> > > > > >> > > > >> > >> > > > > >> Arguably, the Zuul proxy is not necessary if we > force > > > >> > >> everything > > > >> > >> > > > > through a > > > >> > >> > > > > >> Knox instance, since Knox would provide a single > > > endpoint. > > > >> > We > > > >> > >> > > > probably > > > >> > >> > > > > >> however don't want to force Knox and SSL, hence > using > > > Zuul > > > >> > to > > > >> > >> > keep > > > >> > >> > > it > > > >> > >> > > > > >> closer to our current architecture. Zuul does some > > > other > > > >> > nice > > > >> > >> > > things, > > > >> > >> > > > > which > > > >> > >> > > > > >> might help us in future, so it's really about > laying > > > down > > > >> > some > > > >> > >> > > > options > > > >> > >> > > > > for > > > >> > >> > > > > >> potentially doing micro-services style things at a > > > later > > > >> > date. > > > >> > >> > I'm > > > >> > >> > > > not > > > >> > >> > > > > >> saying we have to, or even should go that way, it > > will > > > >> just > > > >> > >> make > > > >> > >> > > life > > > >> > >> > > > > >> easier later if we decide to. It will also help us > if > > > we > > > >> > want > > > >> > >> to > > > >> > >> > > add > > > >> > >> > > > > HA, > > > >> > >> > > > > >> circuit breaking etc to the architecture at a later > > > date. > > > >> > That > > > >> > >> > > said, > > > >> > >> > > > I > > > >> > >> > > > > >> regret that I ever said the word micro-services, > > since > > > >> it's > > > >> > >> > caused > > > >> > >> > > > > >> confusion. Just think of it as a proxy to deal with > > the > > > >> CORS > > > >> > >> > > problem. > > > >> > >> > > > > >> > > > >> > >> > > > > >> Zuul is implemented as a set of filters, but we are > > not > > > >> > using > > > >> > >> it > > > >> > >> > > for > > > >> > >> > > > > its > > > >> > >> > > > > >> authentication filtering. We're using it as a > proxy. > > > Shiro > > > >> > is > > > >> > >> an > > > >> > >> > > > > >> authentication framework, and could arguably used > to > > > >> provide > > > >> > >> the > > > >> > >> > > > > security > > > >> > >> > > > > >> piece, but frankly wrapping shiro as a replacement > > for > > > >> > Spring > > > >> > >> > > > Security > > > >> > >> > > > > in a > > > >> > >> > > > > >> Spring application seemed like it will make life a > > lot > > > >> > harder. > > > >> > >> > This > > > >> > >> > > > > could > > > >> > >> > > > > >> be done, but it's not the native happy path, and > > would > > > >> pull > > > >> > in > > > >> > >> > > > > additional > > > >> > >> > > > > >> dependencies that duplicate functionality that's > > > already > > > >> > >> embedded > > > >> > >> > > in > > > >> > >> > > > > Spring > > > >> > >> > > > > >> Security. > > > >> > >> > > > > >> > > > >> > >> > > > > >> The version of Knox used is the default from HDP. > The > > > link > > > >> > >> > version > > > >> > >> > > > you > > > >> > >> > > > > >> mention is a docs link. I'll update it to be the > > older > > > >> > >> version, > > > >> > >> > > which > > > >> > >> > > > > is > > > >> > >> > > > > >> the same and we can decide if we want to maintain > the > > > >> > >> freshness > > > >> > >> > of > > > >> > >> > > it > > > >> > >> > > > > when > > > >> > >> > > > > >> we look to upgrade underlying patterns. Either way, > > the > > > >> > >> content > > > >> > >> > is > > > >> > >> > > > the > > > >> > >> > > > > >> same. > > > >> > >> > > > > >> > > > >> > >> > > > > >> I did consider other hosting mechanisms, including > > > >> Undertow > > > >> > a > > > >> > >> > > > > >> > > > >> > >> > > > > >> If you have a different suggestion to using the > > Spring > > > >> > default > > > >> > >> > ways > > > >> > >> > > > of > > > >> > >> > > > > >> doing things, or we want to use a framework other > > than > > > >> > Spring > > > >> > >> for > > > >> > >> > > > this, > > > >> > >> > > > > >> then maybe we could change to that, but the route > > > chosen > > > >> > here > > > >> > >> is > > > >> > >> > > > > definitely > > > >> > >> > > > > >> the easy path in the context of the decision made > to > > > use > > > >> > >> Spring > > > >> > >> > in > > > >> > >> > > > > metron > > > >> > >> > > > > >> rest, and if anything opens up our choices while > > > >> minimising, > > > >> > >> in > > > >> > >> > > fact > > > >> > >> > > > > >> reducing, our dependency management overhead. > > > >> > >> > > > > >> > > > >> > >> > > > > >> I hope that explains some of the thinking behind > the > > > >> choices > > > >> > >> > made, > > > >> > >> > > > but > > > >> > >> > > > > the > > > >> > >> > > > > >> guiding principals I followed were: > > > >> > >> > > > > >> * Don't fight the framework if you don't have to > > > >> > >> > > > > >> * Reduce the need for additional installation > pieces > > > and > > > >> > third > > > >> > >> > > party > > > >> > >> > > > > repos > > > >> > >> > > > > >> * Minimize dependencies we would have to manage > > > >> > >> > > > > >> * Avoid excessive change of the architecture, or > > > forcing > > > >> > >> users to > > > >> > >> > > > adopt > > > >> > >> > > > > >> Knox if they didn't want the SSL overhead. > > > >> > >> > > > > >> > > > >> > >> > > > > >> Simon > > > >> > >> > > > > >> > > > >> > >> > > > > >> On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic < > > > >> > >> > > > > >> michael.miklav...@gmail.com> wrote: > > > >> > >> > > > > >> > > > >> > >> > > > > >>> Thanks for the write-up Ryan, this is a great > > start. I > > > >> have > > > >> > >> some > > > >> > >> > > > > further > > > >> > >> > > > > >>> questions based on your feedback and in addition > to > > my > > > >> > >> initial > > > >> > >> > > > thread. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> Just for clarification, what version of Knox are > we > > > >> using? > > > >> > >> HDP > > > >> > >> > > > 2.6.5, > > > >> > >> > > > > >>> which > > > >> > >> > > > > >>> is what we currently run full dev against, > supports > > > >> 0.12.0. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html > > > >> > >> > > > > >>> . > > > >> > >> > > > > >>> I see references to Knox 1.1.0 (latest) in this > > > committed > > > >> > PR > > > >> > >> - > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://github.com/apache/metron/pull/1111/files#diff-70b412194819f3cb829566f05d77c1a6R122 > > > >> > >> > > > > >>> . > > > >> > >> > > > > >>> This is probably just a super small mismatch, and > it > > > >> > probably > > > >> > >> > goes > > > >> > >> > > > > without > > > >> > >> > > > > >>> saying, but I just want to be doubly sure that > we're > > > >> > >> installing > > > >> > >> > > the > > > >> > >> > > > > >>> default > > > >> > >> > > > > >>> via the standard install mechanism as opposed to > > > >> something > > > >> > >> > > separate > > > >> > >> > > > > and > > > >> > >> > > > > >>> manual. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> On the subject of Zuul wrt Nodejs filters. I'd > like > > to > > > >> hear > > > >> > >> some > > > >> > >> > > > more > > > >> > >> > > > > >>> detail on: > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> 1. Why do we need filtering via Zuul? For > instance, > > is > > > >> > >> > > filtering > > > >> > >> > > > > >>> routing > > > >> > >> > > > > >>> not handled by Knox? From the beginner docs: "The > > > >> > gateway > > > >> > >> > > itself > > > >> > >> > > > > is a > > > >> > >> > > > > >>> layer > > > >> > >> > > > > >>> over an embedded Jetty JEE server. At the very > > highest > > > >> > >> level > > > >> > >> > > the > > > >> > >> > > > > >>> gateway > > > >> > >> > > > > >>> processes requests by using request URLs to lookup > > > >> > >> specific > > > >> > >> > JEE > > > >> > >> > > > > Servlet > > > >> > >> > > > > >>> Filter chain that is used to process the request. > > The > > > >> > >> gateway > > > >> > >> > > > > framework > > > >> > >> > > > > >>> provides extensible mechanisms to assemble chains > of > > > >> > >> custom > > > >> > >> > > > filters > > > >> > >> > > > > >>> that > > > >> > >> > > > > >>> support secured access to services." [1] > > > >> > >> > > > > >>> 2. What other library options were considered for > > this > > > >> > >> > feature > > > >> > >> > > > and > > > >> > >> > > > > how > > > >> > >> > > > > >>> was it chosen over the others? I search on "apache > > > >> > spring > > > >> > >> web > > > >> > >> > > > > filters" > > > >> > >> > > > > >>> and > > > >> > >> > > > > >>> it's almost all about Shiro - > > > >> > >> > > > https://shiro.apache.org/spring.html. > > > >> > >> > > > > I > > > >> > >> > > > > >>> also see quite a bit about filtering for Spring > Boot > > > >> > >> > > applications > > > >> > >> > > > > along > > > >> > >> > > > > >>> with a write-up of how to accomplish the same with > > Web > > > >> > MVC > > > >> > >> > > here - > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://stackoverflow.com/questions/19825946/how-to-add-a-filter-class-in-spring-boot > > > >> > >> > > > > >>> . > > > >> > >> > > > > >>> The Knox documentation boilerplate examples are > also > > > >> > using > > > >> > >> > > Shiro. > > > >> > >> > > > > >>> "shiro.ini - The configuration file for the Shiro > > > >> > >> > > authentication > > > >> > >> > > > > >>> provider’s > > > >> > >> > > > > >>> filters. This information is derived from the > > > >> > information > > > >> > >> in > > > >> > >> > > the > > > >> > >> > > > > >>> provider > > > >> > >> > > > > >>> section of the topology file." [1] > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> My assumption is that there are deliberate > decisions > > > in > > > >> > >> favor of > > > >> > >> > > > this > > > >> > >> > > > > mix > > > >> > >> > > > > >>> of technologies over others, and I think some > > > additional > > > >> > >> > > explanation > > > >> > >> > > > > will > > > >> > >> > > > > >>> make that clear. As it stands per the Knox > > > documentation, > > > >> > it > > > >> > >> > looks > > > >> > >> > > > > like > > > >> > >> > > > > >>> we're going on a different route from the > > > >> > >> preferred/recommended > > > >> > >> > > > > idioms. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> [1] > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > http://knox.apache.org/books/knox-0-12-0/dev-guide.html#Architecture+Overview > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> Ryan, I agree about microservices. This should not > > > derail > > > >> > nor > > > >> > >> > be a > > > >> > >> > > > > major > > > >> > >> > > > > >>> part of discussion around this feature, imho. I > > think > > > >> > there's > > > >> > >> > > quite > > > >> > >> > > > a > > > >> > >> > > > > bit > > > >> > >> > > > > >>> left to discuss on that subject. I want to make > sure > > > that > > > >> > >> we're > > > >> > >> > > not > > > >> > >> > > > > >>> prematurely favoring architectural choices by > > pulling > > > in > > > >> > >> > libraries > > > >> > >> > > > > that > > > >> > >> > > > > >>> are > > > >> > >> > > > > >>> potentially opinionated about how to accomplish > > those > > > >> > goals. > > > >> > >> If > > > >> > >> > > they > > > >> > >> > > > > are, > > > >> > >> > > > > >>> I > > > >> > >> > > > > >>> would expect we are comfortable ripping those > > > libraries > > > >> out > > > >> > >> if > > > >> > >> > the > > > >> > >> > > > > >>> community favors a different direction. > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> On the subject of Spring Boot vs Nodejs. I can see > > > some > > > >> > >> > rationale > > > >> > >> > > > for > > > >> > >> > > > > >>> making things homogenous (though, in a > microservices > > > >> > >> > architecture, > > > >> > >> > > > if > > > >> > >> > > > > we > > > >> > >> > > > > >>> go > > > >> > >> > > > > >>> that route, that's not strictly necessary), but > what > > > is > > > >> the > > > >> > >> > > > > justification > > > >> > >> > > > > >>> for Spring Boot over Nodejs? Why would want one > over > > > the > > > >> > >> other? > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> On Mon, Sep 17, 2018 at 3:38 PM Ryan Merriman < > > > >> > >> > > merrim...@gmail.com> > > > >> > >> > > > > >>> wrote: > > > >> > >> > > > > >>> > > > >> > >> > > > > >>> > I have reviewed a couple different PRs so I'll > add > > > some > > > >> > >> > context > > > >> > >> > > > > where I > > > >> > >> > > > > >>> > can. Obviously Simon would be the most qualified > > to > > > >> > answer > > > >> > >> but > > > >> > >> > > > I'll > > > >> > >> > > > > >>> add my > > > >> > >> > > > > >>> > thoughts. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 1, while they may not all be > > necessary > > > I > > > >> > >> think it > > > >> > >> > > > does > > > >> > >> > > > > make > > > >> > >> > > > > >>> > sense to include them in this feature branch if > > our > > > >> > primary > > > >> > >> > goal > > > >> > >> > > > is > > > >> > >> > > > > >>> > integrating Knox SSO. We could push off removing > > > JDBC > > > >> > >> > > > authentication > > > >> > >> > > > > >>> for > > > >> > >> > > > > >>> > reasons I'll get to in my response to question > 2. > > > If we > > > >> > >> want > > > >> > >> > to > > > >> > >> > > do > > > >> > >> > > > > one > > > >> > >> > > > > >>> at > > > >> > >> > > > > >>> > a time (switch to spring boot, add Zuul as a > > > >> dependency, > > > >> > >> then > > > >> > >> > > add > > > >> > >> > > > > Knox > > > >> > >> > > > > >>> SSO) > > > >> > >> > > > > >>> > then that's ok but I do think there are > > dependencies > > > >> and > > > >> > >> > should > > > >> > >> > > be > > > >> > >> > > > > done > > > >> > >> > > > > >>> in > > > >> > >> > > > > >>> > order. For example, adding Knox SSO requires > some > > > work > > > >> > >> around > > > >> > >> > > > > request > > > >> > >> > > > > >>> > filtering. If we were to do this before moving > to > > > >> Spring > > > >> > >> Boot > > > >> > >> > we > > > >> > >> > > > > would > > > >> > >> > > > > >>> > need to implement the filters in Nodejs which > > would > > > be > > > >> > >> > throwaway > > > >> > >> > > > > once we > > > >> > >> > > > > >>> > get around to migrating away from that. For > Zuul, > > I > > > >> > believe > > > >> > >> > it's > > > >> > >> > > > > >>> purpose > > > >> > >> > > > > >>> > is to facilitate the filtering (although it > does a > > > lot > > > >> > >> more) > > > >> > >> > so > > > >> > >> > > it > > > >> > >> > > > > >>> doesn't > > > >> > >> > > > > >>> > make sense to add that separate from the Knox > SSO > > > work. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 2, I think you bring up a good > point. > > > We > > > >> > >> probably > > > >> > >> > > > don't > > > >> > >> > > > > >>> want > > > >> > >> > > > > >>> > to just rip our current authentication method > out. > > > We > > > >> > might > > > >> > >> > want > > > >> > >> > > > to > > > >> > >> > > > > >>> > consider deprecating it instead and making Knox > > SSO > > > and > > > >> > >> LDAP > > > >> > >> > > > > >>> authentication > > > >> > >> > > > > >>> > optional. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 3, this is a bigger shift than > just a > > > >> > >> component > > > >> > >> > > > > upgrade. > > > >> > >> > > > > >>> It's > > > >> > >> > > > > >>> > more like shifting platforms, from Elasticsearch > > to > > > >> Solr > > > >> > >> for > > > >> > >> > > > > example. > > > >> > >> > > > > >>> Like > > > >> > >> > > > > >>> > I alluded to in my response to question 1, I > don't > > > >> think > > > >> > we > > > >> > >> > > should > > > >> > >> > > > > >>> require > > > >> > >> > > > > >>> > throwaway work just because we want to review > > these > > > >> parts > > > >> > >> > > > > separately. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 4, I will defer to Simon. I don't > > > believe > > > >> we > > > >> > >> > > > > necessarily > > > >> > >> > > > > >>> > require Zuul so I will let him elaborate on why > he > > > >> choose > > > >> > >> that > > > >> > >> > > > > library > > > >> > >> > > > > >>> and > > > >> > >> > > > > >>> > what the potential impact is of adding it to our > > > >> project. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > For question 5 and 6, I will also defer to Simon > > on > > > >> this. > > > >> > >> The > > > >> > >> > > > focus > > > >> > >> > > > > of > > > >> > >> > > > > >>> > this feature as I understand it is a consistent > > > >> > >> authentication > > > >> > >> > > > > mechanism > > > >> > >> > > > > >>> > and support for SSO. I will let him lay out his > > > vision > > > >> > for > > > >> > >> > micro > > > >> > >> > > > > >>> services. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > Knox SSO would be a great improvement and is > what > > I > > > >> think > > > >> > >> we > > > >> > >> > > > should > > > >> > >> > > > > >>> focus > > > >> > >> > > > > >>> > on in this feature branch. Micro services is > > > something > > > >> we > > > >> > >> > should > > > >> > >> > > > > >>> certainly > > > >> > >> > > > > >>> > discuss but it might be a bit of a distraction > > and I > > > >> > >> wouldn't > > > >> > >> > > want > > > >> > >> > > > > to > > > >> > >> > > > > >>> hold > > > >> > >> > > > > >>> > up the other useful parts. > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > On Fri, Sep 14, 2018 at 8:38 PM Michael > Miklavcic > > < > > > >> > >> > > > > >>> > michael.miklav...@gmail.com> wrote: > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > > Hey all, > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > I started looking through the Knox SSO feature > > > branch > > > >> > >> (see > > > >> > >> > > here > > > >> > >> > > > > >>> > > > > https://issues.apache.org/jira/browse/METRON-1663 > > > ). > > > >> > >> This is > > > >> > >> > > > some > > > >> > >> > > > > >>> great > > > >> > >> > > > > >>> > new > > > >> > >> > > > > >>> > > security functionality work and it looks like > it > > > will > > > >> > >> bring > > > >> > >> > > some > > > >> > >> > > > > >>> > important > > > >> > >> > > > > >>> > > new features to the Metron platform. I'm > coming > > at > > > >> this > > > >> > >> > pretty > > > >> > >> > > > > green, > > > >> > >> > > > > >>> so > > > >> > >> > > > > >>> > I > > > >> > >> > > > > >>> > > do have some questions regarding the proposed > > > changes > > > >> > >> from a > > > >> > >> > > > high > > > >> > >> > > > > >>> level > > > >> > >> > > > > >>> > > architectural perspective. There are a few > > changes > > > >> > within > > > >> > >> > the > > > >> > >> > > > > current > > > >> > >> > > > > >>> FB > > > >> > >> > > > > >>> > > PR's that I think could use some further > > > explanation. > > > >> > At > > > >> > >> > first > > > >> > >> > > > > >>> glance, it > > > >> > >> > > > > >>> > > seems we could potentially simplify this > branch > > a > > > >> great > > > >> > >> deal > > > >> > >> > > and > > > >> > >> > > > > get > > > >> > >> > > > > >>> it > > > >> > >> > > > > >>> > > completed much sooner if we narrowed the > focus a > > > bit. > > > >> > >> But I > > > >> > >> > > > could > > > >> > >> > > > > >>> > certainly > > > >> > >> > > > > >>> > > be wrong here and happy for other opinions. I > > > >> searched > > > >> > >> > through > > > >> > >> > > > the > > > >> > >> > > > > >>> > mailing > > > >> > >> > > > > >>> > > list history to see if there is any additional > > > >> > background > > > >> > >> > and > > > >> > >> > > > the > > > >> > >> > > > > main > > > >> > >> > > > > >>> > > DISCUSS thread I could find was regarding > > > initially > > > >> > >> setting > > > >> > >> > up > > > >> > >> > > > the > > > >> > >> > > > > >>> > feature > > > >> > >> > > > > >>> > > branch, which talked about adding Knox and > LDAP. > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E > > > >> > >> > > > > >>> > > . > > > >> > >> > > > > >>> > > If I've missed any follow-up, please let me > > know. > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > Looking at the broader set of Jiras associated > > > with > > > >> > 1663 > > > >> > >> and > > > >> > >> > > the > > > >> > >> > > > > >>> first PR > > > >> > >> > > > > >>> > > 1665, it looks like there are 4 main thrusts > of > > > this > > > >> > >> branch > > > >> > >> > > > right > > > >> > >> > > > > now: > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > 1. Knox/SSO > > > >> > >> > > > > >>> > > 2. Node migrated to Spring Boot > > > >> > >> > > > > >>> > > 3. JDBC removed completely in favor of LDAP > > > >> > >> > > > > >>> > > 4. Introduction of Zuul, also microservices? > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > I strongly urge for the purpose of reviewing > > this > > > >> > feature > > > >> > >> > > branch > > > >> > >> > > > > that > > > >> > >> > > > > >>> we > > > >> > >> > > > > >>> > > base much of the discussion off of > > > >> > >> > > > > >>> > > > > https://issues.apache.org/jira/browse/METRON-1755 > > > , > > > >> the > > > >> > >> > > > > architecture > > > >> > >> > > > > >>> > > diagram. Minimally, an explanation of the > > current > > > >> > >> > architecture > > > >> > >> > > > > along > > > >> > >> > > > > >>> with > > > >> > >> > > > > >>> > > discussion around the additional proposed > > changes > > > and > > > >> > >> > > rationale > > > >> > >> > > > > would > > > >> > >> > > > > >>> be > > > >> > >> > > > > >>> > > useful for evaluation. I don't have a solid > > enough > > > >> > >> > > understanding > > > >> > >> > > > > yet > > > >> > >> > > > > >>> of > > > >> > >> > > > > >>> > the > > > >> > >> > > > > >>> > > full scope of changes and how they differ from > > the > > > >> > >> existing > > > >> > >> > > > > >>> architecture > > > >> > >> > > > > >>> > > just from looking at the PR's alone. > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > 1. The first question is a general one > regarding > > > the > > > >> > >> > necessity > > > >> > >> > > > of > > > >> > >> > > > > >>> the > > > >> > >> > > > > >>> > 3 > > > >> > >> > > > > >>> > > additional features alongside Knox - migrating > > > Node > > > >> to > > > >> > >> > Spring > > > >> > >> > > > > Boot, > > > >> > >> > > > > >>> > > removing JDBC altogether, adding dependencies > on > > > >> > >> Netflix's > > > >> > >> > > Zuul > > > >> > >> > > > > >>> > > framework. > > > >> > >> > > > > >>> > > Are these necessary for adding Knox/SSO? They > > seem > > > >> like > > > >> > >> > > > > potentially > > > >> > >> > > > > >>> > > separate features, imo. > > > >> > >> > > > > >>> > > 2. It looks like LDAP will be a required > > component > > > >> for > > > >> > >> > > > interacting > > > >> > >> > > > > >>> > with > > > >> > >> > > > > >>> > > Metron via the UI's. I see this PR > > > >> > >> > > > > >>> > > https://github.com/apache/metron/pull/1186 > > which > > > >> > removes > > > >> > >> > JDBC > > > >> > >> > > > > >>> > > authentication. Are we ready to remove it > > > completely > > > >> or > > > >> > >> > would > > > >> > >> > > it > > > >> > >> > > > > be > > > >> > >> > > > > >>> > > better > > > >> > >> > > > > >>> > > to leave it as a minimal installation option? > > > What is > > > >> > the > > > >> > >> > > > proposed > > > >> > >> > > > > >>> > > migration path for existing users? Do we feel > > > >> > comfortable > > > >> > >> > > > > requiring > > > >> > >> > > > > >>> > that > > > >> > >> > > > > >>> > > all installations, including full dev, install > > and > > > >> > >> configure > > > >> > >> > > > LDAP? > > > >> > >> > > > > >>> For > > > >> > >> > > > > >>> > > comparison, in the PCAP feature branch we > > > discussed > > > >> > >> removing > > > >> > >> > > the > > > >> > >> > > > > >>> > > existing > > > >> > >> > > > > >>> > > PCAP REST application in the initial > discussion, > > > got > > > >> > >> > > agreement, > > > >> > >> > > > > and > > > >> > >> > > > > >>> > > later > > > >> > >> > > > > >>> > > removed it in the course of working on the > > feature > > > >> > >> branch. > > > >> > >> > The > > > >> > >> > > > PR > > > >> > >> > > > > >>> is > > > >> > >> > > > > >>> > > fairly > > > >> > >> > > > > >>> > > clear, however I think we're just missing some > > > basic > > > >> > >> > > discussion > > > >> > >> > > > > >>> around > > > >> > >> > > > > >>> > > the > > > >> > >> > > > > >>> > > implications, as I've outlined above. Some > > > additional > > > >> > >> > relevant > > > >> > >> > > > > >>> > > discussion > > > >> > >> > > > > >>> > > occurred on this PR > > > >> > >> > > https://github.com/apache/metron/pull/1112 > > > >> > >> > > > > >>> which > > > >> > >> > > > > >>> > > would be good to summarize for purposes of > this > > > >> > >> overarching > > > >> > >> > > > > >>> > architecture > > > >> > >> > > > > >>> > > discussion. > > > >> > >> > > > > >>> > > 3. Migration from Node to Spring Boot. I > believe > > > this > > > >> > is > > > >> > >> > > already > > > >> > >> > > > > >>> used > > > >> > >> > > > > >>> > by > > > >> > >> > > > > >>> > > the REST application and if anything brings > some > > > >> > >> cohesion to > > > >> > >> > > our > > > >> > >> > > > > >>> > server > > > >> > >> > > > > >>> > > strategy. Strictly speaking, is there a reason > > > this > > > >> is > > > >> > >> > > required > > > >> > >> > > > > for > > > >> > >> > > > > >>> > > Knox? > > > >> > >> > > > > >>> > > It seems comparable to a component upgrade, > such > > > as > > > >> > >> moving > > > >> > >> > > from > > > >> > >> > > > ES > > > >> > >> > > > > >>> 2.x > > > >> > >> > > > > >>> > > to > > > >> > >> > > > > >>> > > 5.6.x and upgrading Angular 6. > > > >> > >> > > > > >>> > > 4. Introduction of Netflix's Zuul. > > > >> > >> > > > > >>> > > > > https://issues.apache.org/jira/browse/METRON-1665 > > > . > > > >> > >> > > > > >>> > > - > "The UIs currently proxy to the REST API > to > > > avoid > > > >> > >> CORS > > > >> > >> > > > > >>> issues, > > > >> > >> > > > > >>> > > this will be achieved with Zuul." > > > >> > >> > > > > >>> > > - Can we elaborate more on where or how CORS > is > > a > > > >> > problem > > > >> > >> > with > > > >> > >> > > > > >>> our > > > >> > >> > > > > >>> > > existing architecture, how Zuul will help > solve > > > that, > > > >> > and > > > >> > >> > how > > > >> > >> > > it > > > >> > >> > > > > >>> > > fits with > > > >> > >> > > > > >>> > > Knox? Wouldn't this be handled by Knox? Since > > > Larry > > > >> > McCay > > > >> > >> > > > > >>> chimed in > > > >> > >> > > > > >>> > > with > > > >> > >> > > > > >>> > > interest on the original SSO thread about the > > FB, > > > I'm > > > >> > >> hoping > > > >> > >> > > he > > > >> > >> > > > > >>> is > > > >> > >> > > > > >>> > > also > > > >> > >> > > > > >>> > > willing to chime in on this as well. > > > >> > >> > > > > >>> > > - This looks like it has the potential to be a > > > rather > > > >> > >> large > > > >> > >> > > > > >>> piece > > > >> > >> > > > > >>> > of > > > >> > >> > > > > >>> > > fundamental infrastructure (as it's also > > > pertinent to > > > >> > >> > > > > >>> > microservices) > > > >> > >> > > > > >>> > > to > > > >> > >> > > > > >>> > > pull into the platform, and I'd like to be > sure > > > the > > > >> > >> > community > > > >> > >> > > is > > > >> > >> > > > > >>> > > aware of > > > >> > >> > > > > >>> > > and is OK with the implications. > > > >> > >> > > > > >>> > > 5. > "The proposal is to use a spring boot > > > >> application, > > > >> > >> > > allowing > > > >> > >> > > > > >>> us to > > > >> > >> > > > > >>> > > harmonize the security implementation across > the > > > UI > > > >> > >> static > > > >> > >> > > > servers > > > >> > >> > > > > >>> and > > > >> > >> > > > > >>> > > the > > > >> > >> > > > > >>> > > REST layer, and to provide a routing platform > > for > > > >> later > > > >> > >> > > > > >>> > microservices." > > > >> > >> > > > > >>> > > - > > > >> > >> > > > > >>> > > > > https://issues.apache.org/jira/browse/METRON-1665 > > > . > > > >> > >> > > > > >>> > > - Microservices is a pretty loaded term. I > know > > > there > > > >> > had > > > >> > >> > been > > > >> > >> > > > > >>> some > > > >> > >> > > > > >>> > > discussion a while back during the PCAP > feature > > > >> branch > > > >> > >> > start, > > > >> > >> > > > > >>> but I > > > >> > >> > > > > >>> > > don't > > > >> > >> > > > > >>> > > recall ever reaching a consensus on it. More > > > detail > > > >> in > > > >> > >> this > > > >> > >> > > > > >>> thread > > > >> > >> > > > > >>> > - > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E > > > >> > >> > > > > >>> > > . > > > >> > >> > > > > >>> > > Can we get some clarification on what is meant > > by > > > >> > >> > > microservices > > > >> > >> > > > > >>> > > in the case > > > >> > >> > > > > >>> > > of this FB and relevant PR's, what that > > > architecture > > > >> > >> looks > > > >> > >> > > like, > > > >> > >> > > > > >>> > and > > > >> > >> > > > > >>> > > how > > > >> > >> > > > > >>> > > it's achieved with the proposed changes in > this > > > >> PR/FB? > > > >> > It > > > >> > >> > > seems > > > >> > >> > > > > >>> > Zuul > > > >> > >> > > > > >>> > > is > > > >> > >> > > > > >>> > > also pertinent to this discussion, but there > are > > > many > > > >> > >> ways > > > >> > >> > to > > > >> > >> > > > > >>> > > skin this cat > > > >> > >> > > > > >>> > > so I don't want to presume - > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > >> > > > > https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices > > > >> > >> > > > > >>> > > 6. Zuul, Spring Boot, and microservices - > > Closely > > > >> > >> related to > > > >> > >> > > > > >>> > point 5 > > > >> > >> > > > > >>> > > above. It seems that we weren't quite ready > for > > > this > > > >> > >> when it > > > >> > >> > > was > > > >> > >> > > > > >>> > > brought up > > > >> > >> > > > > >>> > > in May, or at the very least we had some > concern > > > of > > > >> > what > > > >> > >> > > > direction > > > >> > >> > > > > >>> to > > > >> > >> > > > > >>> > > go. > > > >> > >> > > > > >>> > > What is the operational impact, mpack impact, > > and > > > how > > > >> > we > > > >> > >> > > propose > > > >> > >> > > > > to > > > >> > >> > > > > >>> > > manage > > > >> > >> > > > > >>> > > it with Kerberos, etc.? > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > >> > >> > > > > >>> > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > >> > > > > > > https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > There is a lot to like in this feature branch, > > > imo. > > > >> > Great > > > >> > >> > > > feature > > > >> > >> > > > > >>> > addition > > > >> > >> > > > > >>> > > with Knox and SSO. Introduction of LDAP > support > > > for > > > >> > >> > > > > authentication for > > > >> > >> > > > > >>> > > Metron UI's. Simplification/unification of our > > > server > > > >> > >> > hosting > > > >> > >> > > > > >>> > > infrastructure. I'm hoping we can flesh out > some > > > of > > > >> the > > > >> > >> > > details > > > >> > >> > > > > >>> pointed > > > >> > >> > > > > >>> > out > > > >> > >> > > > > >>> > > above a bit more and get this feature through. > > > Great > > > >> > >> work so > > > >> > >> > > > far! > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > Best, > > > >> > >> > > > > >>> > > Mike Miklavcic > > > >> > >> > > > > >>> > > > > > >> > >> > > > > >>> > > > > >> > >> > > > > >> > > > >> > >> > > > > >> -- > > > >> > >> > > > > >> -- > > > >> > >> > > > > >> simon elliston ball > > > >> > >> > > > > >> @sireb > > > >> > >> > > > > > > > > >> > >> > > > > > -- > > > >> > >> > > > > > -- > > > >> > >> > > > > > simon elliston ball > > > >> > >> > > > > > @sireb > > > >> > >> > > > > > > > >> > >> > > > > ------------------- > > > >> > >> > > > > Thank you, > > > >> > >> > > > > > > > >> > >> > > > > James Sirota > > > >> > >> > > > > PMC- Apache Metron > > > >> > >> > > > > jsirota AT apache DOT org > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > > > > > >> > > > > >> -- > > > >> > > > >> Jon Zeolla > > > > > > ------------------- > > > Thank you, > > > > > > James Sirota > > > PMC- Apache Metron > > > jsirota AT apache DOT org > > > > > >
