Phew, that was quite the thread to catch up on. I agree that this should be optional/pluggable to start, and I'm interested to hear the issues as they relate to upgrading an existing cluster (given the suggested approach) and exposing both legacy and knox URLs at the same time.
Jon On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic <michael.miklav...@gmail.com> wrote: > A couple more things, and I think this goes without saying - whatever we do > with Knox should NOT > > 1. Require unit and integration tests to use Knox > 2. Break fulldev > > Also, I don't know that I saw you mention this, but I'm unsure how we > should leverage Knox as a core piece of the platform. i.e. should we make > this required or optional? I'm open to hearing opinions on this, but I'm > inclined to keep this a pluggable option. > > Mike > > > On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Thanks for the update Ryan. Per my earlier comments, I thought it might > be > > the case that we could dramatically simplify this by leveraging Knox's > > proxy capabilities, and per your research that appears to be the case. > This > > is a dramatic simplification and improvement of this feature imo, +1. I'm > > also +1 on a couple distinct steps that you've laid out: fix the UI > issues > > in master, then add Knox for SSO. That should help mitigate issues with > > merge conflicts with ongoing development. > > > > > I think it will be a challenge exposing the UIs through both the Knox > > url and legacy urls at the same time. > > I'm not sure I understand the issue here. Are you referring to this > > comment? "Added a ng build option to build the UI with base href set to > > Knox base path." Isn't it just a matter of URL rewriting/forwarding? I > > thought we'd be exposing the URL's directly in one context, and through > > Knox in the other. Either way, it seems like we should be able to > provide a > > dynamic base path through configuration in our web applications. I'd > expect > > to modify that property based on whether Knox is configured or not. > > > > > I'm also not clear on how one would use Knox with REST set to legacy > > JDBC-based authentication. As far as I know Knox does not support JDBC so > > there would be a mismatch between Knox and REST. > > I'm OK with not having Knox work with JDBC. That's a feature of Knox and > > probably not something we care much about. > > > > >We could initially make Knox an optional feature that requires setup > with > > the help of some documentation (like Kerberos) while keeping the system > the > > way it is now by default. > > Sounds good to me. > > > > > I imagine we'll deprecate JDBC-based authentication at some point so > > that may be a good time to switch. > > I would like to announce deprecation in our next release and move to > > remove it in a following release. > > > > Thanks for taking this on and great job laying things out. > > > > Thanks, > > Mike > > > > On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman <merrim...@gmail.com> > wrote: > > > >> I have spent some time recently reviewing this discussion and the > feature > >> branch that Simon put out. I think this is an important feature and > want > >> to move it forward. I started another discussion on adding Knox to our > >> stack but this discussion has a lot of good context so I will continue > it > >> here. > >> > >> I think the main point of contention was that this feature branch > included > >> several different architectural changes and it was unclear if they were > >> needed and if so, could be done separately. Fortunately LDAP > >> authentication has been accepted into master so we can cross it off the > >> list. From my understanding of the points people have made, that leaves > >> Knox related SSO changes and migrating expressjs to a different, > JVM-based > >> web server that includes proxying capabilities (Zuul). > >> > >> I think everyone agrees that if we can limit the scope to just Knox > >> related > >> SSO changes that would be ideal. I believe I have found a way to do > that > >> while working on a small POC this week. The key to this (Simon alluded > to > >> it earlier) is to put both REST and our UIs behind Knox. I initially > was > >> focused on just adding REST as a service in Knox and decided to > experiment > >> with also adding our UIs. After I did this it became clear that this > >> simplifies things considerably: > >> > >> - The REST app and the UIs are now served from the same host so CORS > >> concerns go away. > >> - We no longer need to worry about proxying REST requests from the > UIs > >> with express or Zuul because Knox handles that for us. This will > make > >> our > >> express configuration even simpler. In fact, all we need is a simple > >> way > >> to serve static UI assets. > >> - We no longer need to check for SSO tokens and redirect in the UI > >> web/app servers (or the REST app for that matter) because Knox > handles > >> that > >> for us. > >> - The UIs can now easily access any Knox service (not just our REST > >> app) > >> without any extra proxy configuration. > >> - SSO token authentication is only necessary in REST so there is no > >> need > >> to create shared Spring modules or split functionality out. > >> > >> The most significant change I had to make (borrowed from Simon's feature > >> branch) was the SSO token authentication mentioned above. The primary > >> short term benefit with this approach is that outside of some general > >> deficiencies unrelated to this our UI architecture doesn't need to > >> fundamentally change. I could summarize the changes as: > >> > >> - Knox install and configuration (setting up REST and the alerts UI > as > >> Knox services) > >> - Added Knox SSO token authentication to REST > >> - Updated REST urls in the UI code (should be configurable) > >> - Fixed a few UI bugs where relative paths were not being used > >> - Added a ng build option to build the UI with base href set to Knox > >> base path ( > >> > >> > https://github.com/angular/angular-cli/wiki/build#base-tag-handling-in-indexhtml > >> ) > >> > >> Most the UI changes are preexisting, minor issues that could be fixed > >> directly in master. We would need to think of an approach for the base > >> href build requirement but I'm sure it's not that bad. > >> > >> However there will be some backwards compatibility issues we would need > to > >> think through. I think it will be a challenge exposing the UIs through > >> both the Knox url and legacy urls at the same time. I'm also not clear > on > >> how one would use Knox with REST set to legacy JDBC-based > authentication. > >> As far as I know Knox does not support JDBC so there would be a mismatch > >> between Knox and REST. Knox does have the ability to pass along basic > >> authentication headers so LDAP in REST would work. We could initially > >> make > >> Knox an optional feature that requires setup with the help of some > >> documentation (like Kerberos) while keeping the system the way it is now > >> by > >> default. I imagine we'll deprecate JDBC-based authentication at some > >> point > >> so that may be a good time to switch. > >> > >> What do people think about this approach? Concerns? Are there any huge > >> holes in this I'm not thinking about? > >> > >> I want to highlight that the work Simon did in his feature branch was > >> crucial to better understanding this. I am pretty sure we'll end up > >> reusing a lot code from that branch. > >> > >> On Thu, Sep 27, 2018 at 6:30 PM Michael Miklavcic < > >> michael.miklav...@gmail.com> wrote: > >> > >> > Apparently, I hit send on my last email before finishing my synopsis > >> (per > >> > @Otto's Q in Slack). To summarize, based on my current understanding I > >> > believe that each of the feature branch changes I've outline above are > >> > units of work that are related, yet should be executed on > independently. > >> > Knox SSO in its own feature branch. Migrating technologies like NodeJs > >> or > >> > migrating the auth DB to LDAP seem like they belong in their own > >> separate > >> > PR's or feature branches. > >> > > >> > Thanks, > >> > Mike > >> > > >> > On Thu, Sep 27, 2018 at 4:08 PM Casey Stella <ceste...@gmail.com> > >> wrote: > >> > > >> > > I'm coming in late to the game here, but for my mind a feature > branch > >> > > should involve the minimum architectural change to accomplish a > given > >> > > feature. > >> > > The feature in question is SSO integration. It seems to me that the > >> > > operative question is can we do the feature without making the OTHER > >> > > architectural change > >> > > (e.g. migrating from expressjs to spring boot + zuul). I would > argue > >> > that > >> > > if we WANT to do that, then it should be a separate feature branch. > >> > > > >> > > Thus, I leave with a question: is there a way to accomplish this > >> feature > >> > > without ripping out expressjs? > >> > > > >> > > - If so and it is feasible, I would argue that we should decouple > >> this > >> > > into a separate feature branch. > >> > > - If so and it is infeasible, I'd like to hear an argument as to > >> the > >> > > infeasibility and let's decide given that > >> > > - If it is not possible, then I'd argue that we should keep them > >> > coupled > >> > > and move this through as-is. > >> > > > >> > > On a side-note, it feels a bit weird that we're narrowing to a > bundled > >> > > proxy, rather than having that be a pluggable thing. I'm not super > >> > > knowledgeable in this space, so I apologize > >> > > in advance if this is naive, but isn't this a pluggable, external > >> > component > >> > > (e.g. nginx)? > >> > > > >> > > On Thu, Sep 27, 2018 at 5:05 PM Michael Miklavcic < > >> > > michael.miklav...@gmail.com> wrote: > >> > > > >> > > > I've spent some more time reading through Simon's response and the > >> > added > >> > > > sequence diagram. This is definitely helpful - thank you Simon. > >> > > > > >> > > > I need to redact my initial list: > >> > > > > >> > > > 1. Node migrated to Spring Boot, expressjs migrated to a > >> > > > non-JS/non-NodeJs proxying mechanism (ie Zuul in this case) > >> > > > 2. JDBC removed completely in favor of LDAP > >> > > > 3. Knox/SSO > >> > > > > >> > > > I'm a bit conflicted on the best way to move forward and would > like > >> > some > >> > > > thoughts from other community members on this. I think an argument > >> can > >> > be > >> > > > made that 1 and 2 are independent of 3, and should/could really be > >> > > > independent PR's against master. > >> > > > > >> > > > The need for a replacement for expressjs (Zuul in this case) is an > >> > > artifact > >> > > > that our request/response cycle for REST calls is a simple matter > of > >> > > > forwarding with some additional headers for authentication. > There's > >> a > >> > > > JSESSIONID managed by the client browser in our current > >> architecture, > >> > for > >> > > > example. You login to the alerts or the management UI which > >> forwards a > >> > > > request to REST, which looks up credentials in a backend database, > >> and > >> > > > passes the results back up the chain. All browser requests go > >> directly > >> > to > >> > > > the specific UI you're working with - this is the CORS problem. > You > >> > > can't, > >> > > > without some effort with headers for adding other domains to the > >> safe > >> > > list > >> > > > or disabling the security check for CORS, make remote calls > >> directly to > >> > > > REST. That's why we proxy. Switching over to Spring Boot leaves a > >> gap > >> > > with > >> > > > expressjs having handled the proxying and filtering, since it's > only > >> > > > available to a NodeJs application (it's server-side javascript vs > >> the > >> > > > client side javascript deployed via our Angular applications). > Enter > >> > > Zuul, > >> > > > which now effectively handles that. At runtime, Zuul is a part of > >> the > >> > > > Spring app that serves up our UI's. It handles the requests via > >> > > filtering, > >> > > > forwards them to REST, manages the response back to the client. > Very > >> > > > similar to what expressjs was doing, per my current understanding. > >> The > >> > > > sequence diagrams Simon added are useful, and I think some of what > >> was > >> > > less > >> > > > clear was what we currently vs what the new changes are doing to > the > >> > > > architecture. This is no fault of Simon's - there simply wasn't > any > >> > > > architecture diagrams/documents around this before. Here's my > >> > impression > >> > > of > >> > > > the very very basic current state - someone more familiar with > this > >> > > > architecture please advise if I'm incorrect about anything > (probably > >> > > Ryan). > >> > > > > >> > > > https://imgur.com/f8GtSmh > >> > > > > >> > > > Zuul would be replacing the bit about expressjs in the diagram, > and > >> > > instead > >> > > > of node we have spring boot. This covers 1. 2 and 3 are other > >> issues. > >> > I'd > >> > > > like to see similar exposition of those server processes with knox > >> > > > involved. I imagine in that case we bump up from 3 to 4 server > >> > instances > >> > > > for the additional knox endpoint. > >> > > > > >> > > > Mike > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > On Wed, Sep 19, 2018 at 11:28 AM James Sirota <jsir...@apache.org > > > >> > > wrote: > >> > > > > >> > > > > Thank you, Simon. The diagrams help a lot > >> > > > > > >> > > > > 19.09.2018, 21:27, "Simon Elliston Ball" < > >> > si...@simonellistonball.com > >> > > >: > >> > > > > > To clarify some of this I've put some documentation into > >> > > > > > https://github.com/apache/metron/pull/1203 under METRON-1755 > ( > >> > > > > > https://issues.apache.org/jira/browse/METRON-1755). Hopefully > >> the > >> > > > > diagrams > >> > > > > > there should make it clearer. > >> > > > > > > >> > > > > > Simon > >> > > > > > > >> > > > > > On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < > >> > > > > > si...@simonellistonball.com> wrote: > >> > > > > > > >> > > > > >> Hi Mike, > >> > > > > >> > >> > > > > >> Some good points here which could do with some > clarification. > >> I > >> > > > suspect > >> > > > > >> the architecture documentation could be clearer and fill in > >> some > >> > of > >> > > > > these > >> > > > > >> gaps, and I'll have a look at working on that and providing > >> some > >> > > > > diagrams. > >> > > > > >> > >> > > > > >> The short version is that the Zuul proxy gateway has been > >> added > >> > to > >> > > > > replace > >> > > > > >> the Nodejs express proxy used to gateway the REST api calls > in > >> > the > >> > > > > current > >> > > > > >> hosts. This is done in both cases to avoid CORS restrictions > >> by > >> > > > > allowing > >> > > > > >> the same host that serves the UI files to proxy call to the > >> API. > >> > > > > >> > >> > > > > >> The choice of Zuul was partly a pragmatic one (it's the one > >> > that's > >> > > > > there > >> > > > > >> in the box as it were with Spring Boot, which we use for the > >> REST > >> > > > API, > >> > > > > via > >> > > > > >> the Spring Cloud Netflix project which wraps a bunch of > >> related > >> > > > pieces > >> > > > > into > >> > > > > >> Spring). The choice of Spring Boot to host the UIs > themselves > >> was > >> > > > > similarly > >> > > > > >> for parity with the REST host, to simplify the stack (we > >> remove > >> > the > >> > > > > >> occasionally problematic need to install nodejs on target > >> > servers, > >> > > > > which is > >> > > > > >> outside of the regular OS and HDP stacks we support). > >> > > > > >> > >> > > > > >> Arguably, the Zuul proxy is not necessary if we force > >> everything > >> > > > > through a > >> > > > > >> Knox instance, since Knox would provide a single endpoint. > We > >> > > > probably > >> > > > > >> however don't want to force Knox and SSL, hence using Zuul > to > >> > keep > >> > > it > >> > > > > >> closer to our current architecture. Zuul does some other > nice > >> > > things, > >> > > > > which > >> > > > > >> might help us in future, so it's really about laying down > some > >> > > > options > >> > > > > for > >> > > > > >> potentially doing micro-services style things at a later > date. > >> > I'm > >> > > > not > >> > > > > >> saying we have to, or even should go that way, it will just > >> make > >> > > life > >> > > > > >> easier later if we decide to. It will also help us if we > want > >> to > >> > > add > >> > > > > HA, > >> > > > > >> circuit breaking etc to the architecture at a later date. > That > >> > > said, > >> > > > I > >> > > > > >> regret that I ever said the word micro-services, since it's > >> > caused > >> > > > > >> confusion. Just think of it as a proxy to deal with the CORS > >> > > problem. > >> > > > > >> > >> > > > > >> Zuul is implemented as a set of filters, but we are not > using > >> it > >> > > for > >> > > > > its > >> > > > > >> authentication filtering. We're using it as a proxy. Shiro > is > >> an > >> > > > > >> authentication framework, and could arguably used to provide > >> the > >> > > > > security > >> > > > > >> piece, but frankly wrapping shiro as a replacement for > Spring > >> > > > Security > >> > > > > in a > >> > > > > >> Spring application seemed like it will make life a lot > harder. > >> > This > >> > > > > could > >> > > > > >> be done, but it's not the native happy path, and would pull > in > >> > > > > additional > >> > > > > >> dependencies that duplicate functionality that's already > >> embedded > >> > > in > >> > > > > Spring > >> > > > > >> Security. > >> > > > > >> > >> > > > > >> The version of Knox used is the default from HDP. The link > >> > version > >> > > > you > >> > > > > >> mention is a docs link. I'll update it to be the older > >> version, > >> > > which > >> > > > > is > >> > > > > >> the same and we can decide if we want to maintain the > >> freshness > >> > of > >> > > it > >> > > > > when > >> > > > > >> we look to upgrade underlying patterns. Either way, the > >> content > >> > is > >> > > > the > >> > > > > >> same. > >> > > > > >> > >> > > > > >> I did consider other hosting mechanisms, including Undertow > a > >> > > > > >> > >> > > > > >> If you have a different suggestion to using the Spring > default > >> > ways > >> > > > of > >> > > > > >> doing things, or we want to use a framework other than > Spring > >> for > >> > > > this, > >> > > > > >> then maybe we could change to that, but the route chosen > here > >> is > >> > > > > definitely > >> > > > > >> the easy path in the context of the decision made to use > >> Spring > >> > in > >> > > > > metron > >> > > > > >> rest, and if anything opens up our choices while minimising, > >> in > >> > > fact > >> > > > > >> reducing, our dependency management overhead. > >> > > > > >> > >> > > > > >> I hope that explains some of the thinking behind the choices > >> > made, > >> > > > but > >> > > > > the > >> > > > > >> guiding principals I followed were: > >> > > > > >> * Don't fight the framework if you don't have to > >> > > > > >> * Reduce the need for additional installation pieces and > third > >> > > party > >> > > > > repos > >> > > > > >> * Minimize dependencies we would have to manage > >> > > > > >> * Avoid excessive change of the architecture, or forcing > >> users to > >> > > > adopt > >> > > > > >> Knox if they didn't want the SSL overhead. > >> > > > > >> > >> > > > > >> Simon > >> > > > > >> > >> > > > > >> On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic < > >> > > > > >> michael.miklav...@gmail.com> wrote: > >> > > > > >> > >> > > > > >>> Thanks for the write-up Ryan, this is a great start. I have > >> some > >> > > > > further > >> > > > > >>> questions based on your feedback and in addition to my > >> initial > >> > > > thread. > >> > > > > >>> > >> > > > > >>> Just for clarification, what version of Knox are we using? > >> HDP > >> > > > 2.6.5, > >> > > > > >>> which > >> > > > > >>> is what we currently run full dev against, supports 0.12.0. > >> > > > > >>> > >> > > > > >>> > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html > >> > > > > >>> . > >> > > > > >>> I see references to Knox 1.1.0 (latest) in this committed > PR > >> - > >> > > > > >>> > >> > > > > >>> > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/metron/pull/1111/files#diff-70b412194819f3cb829566f05d77c1a6R122 > >> > > > > >>> . > >> > > > > >>> This is probably just a super small mismatch, and it > probably > >> > goes > >> > > > > without > >> > > > > >>> saying, but I just want to be doubly sure that we're > >> installing > >> > > the > >> > > > > >>> default > >> > > > > >>> via the standard install mechanism as opposed to something > >> > > separate > >> > > > > and > >> > > > > >>> manual. > >> > > > > >>> > >> > > > > >>> On the subject of Zuul wrt Nodejs filters. I'd like to hear > >> some > >> > > > more > >> > > > > >>> detail on: > >> > > > > >>> > >> > > > > >>> 1. Why do we need filtering via Zuul? For instance, is > >> > > filtering > >> > > > > >>> routing > >> > > > > >>> not handled by Knox? From the beginner docs: "The > gateway > >> > > itself > >> > > > > is a > >> > > > > >>> layer > >> > > > > >>> over an embedded Jetty JEE server. At the very highest > >> level > >> > > the > >> > > > > >>> gateway > >> > > > > >>> processes requests by using request URLs to lookup > >> specific > >> > JEE > >> > > > > Servlet > >> > > > > >>> Filter chain that is used to process the request. The > >> gateway > >> > > > > framework > >> > > > > >>> provides extensible mechanisms to assemble chains of > >> custom > >> > > > filters > >> > > > > >>> that > >> > > > > >>> support secured access to services." [1] > >> > > > > >>> 2. What other library options were considered for this > >> > feature > >> > > > and > >> > > > > how > >> > > > > >>> was it chosen over the others? I search on "apache > spring > >> web > >> > > > > filters" > >> > > > > >>> and > >> > > > > >>> it's almost all about Shiro - > >> > > > https://shiro.apache.org/spring.html. > >> > > > > I > >> > > > > >>> also see quite a bit about filtering for Spring Boot > >> > > applications > >> > > > > along > >> > > > > >>> with a write-up of how to accomplish the same with Web > MVC > >> > > here - > >> > > > > >>> > >> > > > > >>> > >> > > > > > >> > > > > >> > > > >> > > >> > https://stackoverflow.com/questions/19825946/how-to-add-a-filter-class-in-spring-boot > >> > > > > >>> . > >> > > > > >>> The Knox documentation boilerplate examples are also > using > >> > > Shiro. > >> > > > > >>> "shiro.ini - The configuration file for the Shiro > >> > > authentication > >> > > > > >>> provider’s > >> > > > > >>> filters. This information is derived from the > information > >> in > >> > > the > >> > > > > >>> provider > >> > > > > >>> section of the topology file." [1] > >> > > > > >>> > >> > > > > >>> My assumption is that there are deliberate decisions in > >> favor of > >> > > > this > >> > > > > mix > >> > > > > >>> of technologies over others, and I think some additional > >> > > explanation > >> > > > > will > >> > > > > >>> make that clear. As it stands per the Knox documentation, > it > >> > looks > >> > > > > like > >> > > > > >>> we're going on a different route from the > >> preferred/recommended > >> > > > > idioms. > >> > > > > >>> > >> > > > > >>> [1] > >> > > > > >>> > >> > > > > >>> > >> > > > > > >> > > > > >> > > > >> > > >> > http://knox.apache.org/books/knox-0-12-0/dev-guide.html#Architecture+Overview > >> > > > > >>> > >> > > > > >>> Ryan, I agree about microservices. This should not derail > nor > >> > be a > >> > > > > major > >> > > > > >>> part of discussion around this feature, imho. I think > there's > >> > > quite > >> > > > a > >> > > > > bit > >> > > > > >>> left to discuss on that subject. I want to make sure that > >> we're > >> > > not > >> > > > > >>> prematurely favoring architectural choices by pulling in > >> > libraries > >> > > > > that > >> > > > > >>> are > >> > > > > >>> potentially opinionated about how to accomplish those > goals. > >> If > >> > > they > >> > > > > are, > >> > > > > >>> I > >> > > > > >>> would expect we are comfortable ripping those libraries out > >> if > >> > the > >> > > > > >>> community favors a different direction. > >> > > > > >>> > >> > > > > >>> On the subject of Spring Boot vs Nodejs. I can see some > >> > rationale > >> > > > for > >> > > > > >>> making things homogenous (though, in a microservices > >> > architecture, > >> > > > if > >> > > > > we > >> > > > > >>> go > >> > > > > >>> that route, that's not strictly necessary), but what is the > >> > > > > justification > >> > > > > >>> for Spring Boot over Nodejs? Why would want one over the > >> other? > >> > > > > >>> > >> > > > > >>> On Mon, Sep 17, 2018 at 3:38 PM Ryan Merriman < > >> > > merrim...@gmail.com> > >> > > > > >>> wrote: > >> > > > > >>> > >> > > > > >>> > I have reviewed a couple different PRs so I'll add some > >> > context > >> > > > > where I > >> > > > > >>> > can. Obviously Simon would be the most qualified to > answer > >> but > >> > > > I'll > >> > > > > >>> add my > >> > > > > >>> > thoughts. > >> > > > > >>> > > >> > > > > >>> > For question 1, while they may not all be necessary I > >> think it > >> > > > does > >> > > > > make > >> > > > > >>> > sense to include them in this feature branch if our > primary > >> > goal > >> > > > is > >> > > > > >>> > integrating Knox SSO. We could push off removing JDBC > >> > > > authentication > >> > > > > >>> for > >> > > > > >>> > reasons I'll get to in my response to question 2. If we > >> want > >> > to > >> > > do > >> > > > > one > >> > > > > >>> at > >> > > > > >>> > a time (switch to spring boot, add Zuul as a dependency, > >> then > >> > > add > >> > > > > Knox > >> > > > > >>> SSO) > >> > > > > >>> > then that's ok but I do think there are dependencies and > >> > should > >> > > be > >> > > > > done > >> > > > > >>> in > >> > > > > >>> > order. For example, adding Knox SSO requires some work > >> around > >> > > > > request > >> > > > > >>> > filtering. If we were to do this before moving to Spring > >> Boot > >> > we > >> > > > > would > >> > > > > >>> > need to implement the filters in Nodejs which would be > >> > throwaway > >> > > > > once we > >> > > > > >>> > get around to migrating away from that. For Zuul, I > believe > >> > it's > >> > > > > >>> purpose > >> > > > > >>> > is to facilitate the filtering (although it does a lot > >> more) > >> > so > >> > > it > >> > > > > >>> doesn't > >> > > > > >>> > make sense to add that separate from the Knox SSO work. > >> > > > > >>> > > >> > > > > >>> > For question 2, I think you bring up a good point. We > >> probably > >> > > > don't > >> > > > > >>> want > >> > > > > >>> > to just rip our current authentication method out. We > might > >> > want > >> > > > to > >> > > > > >>> > consider deprecating it instead and making Knox SSO and > >> LDAP > >> > > > > >>> authentication > >> > > > > >>> > optional. > >> > > > > >>> > > >> > > > > >>> > For question 3, this is a bigger shift than just a > >> component > >> > > > > upgrade. > >> > > > > >>> It's > >> > > > > >>> > more like shifting platforms, from Elasticsearch to Solr > >> for > >> > > > > example. > >> > > > > >>> Like > >> > > > > >>> > I alluded to in my response to question 1, I don't think > we > >> > > should > >> > > > > >>> require > >> > > > > >>> > throwaway work just because we want to review these parts > >> > > > > separately. > >> > > > > >>> > > >> > > > > >>> > For question 4, I will defer to Simon. I don't believe we > >> > > > > necessarily > >> > > > > >>> > require Zuul so I will let him elaborate on why he choose > >> that > >> > > > > library > >> > > > > >>> and > >> > > > > >>> > what the potential impact is of adding it to our project. > >> > > > > >>> > > >> > > > > >>> > For question 5 and 6, I will also defer to Simon on this. > >> The > >> > > > focus > >> > > > > of > >> > > > > >>> > this feature as I understand it is a consistent > >> authentication > >> > > > > mechanism > >> > > > > >>> > and support for SSO. I will let him lay out his vision > for > >> > micro > >> > > > > >>> services. > >> > > > > >>> > > >> > > > > >>> > Knox SSO would be a great improvement and is what I think > >> we > >> > > > should > >> > > > > >>> focus > >> > > > > >>> > on in this feature branch. Micro services is something we > >> > should > >> > > > > >>> certainly > >> > > > > >>> > discuss but it might be a bit of a distraction and I > >> wouldn't > >> > > want > >> > > > > to > >> > > > > >>> hold > >> > > > > >>> > up the other useful parts. > >> > > > > >>> > > >> > > > > >>> > On Fri, Sep 14, 2018 at 8:38 PM Michael Miklavcic < > >> > > > > >>> > michael.miklav...@gmail.com> wrote: > >> > > > > >>> > > >> > > > > >>> > > Hey all, > >> > > > > >>> > > > >> > > > > >>> > > I started looking through the Knox SSO feature branch > >> (see > >> > > here > >> > > > > >>> > > https://issues.apache.org/jira/browse/METRON-1663). > >> This is > >> > > > some > >> > > > > >>> great > >> > > > > >>> > new > >> > > > > >>> > > security functionality work and it looks like it will > >> bring > >> > > some > >> > > > > >>> > important > >> > > > > >>> > > new features to the Metron platform. I'm coming at this > >> > pretty > >> > > > > green, > >> > > > > >>> so > >> > > > > >>> > I > >> > > > > >>> > > do have some questions regarding the proposed changes > >> from a > >> > > > high > >> > > > > >>> level > >> > > > > >>> > > architectural perspective. There are a few changes > within > >> > the > >> > > > > current > >> > > > > >>> FB > >> > > > > >>> > > PR's that I think could use some further explanation. > At > >> > first > >> > > > > >>> glance, it > >> > > > > >>> > > seems we could potentially simplify this branch a great > >> deal > >> > > and > >> > > > > get > >> > > > > >>> it > >> > > > > >>> > > completed much sooner if we narrowed the focus a bit. > >> But I > >> > > > could > >> > > > > >>> > certainly > >> > > > > >>> > > be wrong here and happy for other opinions. I searched > >> > through > >> > > > the > >> > > > > >>> > mailing > >> > > > > >>> > > list history to see if there is any additional > background > >> > and > >> > > > the > >> > > > > main > >> > > > > >>> > > DISCUSS thread I could find was regarding initially > >> setting > >> > up > >> > > > the > >> > > > > >>> > feature > >> > > > > >>> > > branch, which talked about adding Knox and LDAP. > >> > > > > >>> > > > >> > > > > >>> > > > >> > > > > >>> > > >> > > > > >>> > >> > > > > > >> > > > > >> > > > >> > > >> > https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E > >> > > > > >>> > > . > >> > > > > >>> > > If I've missed any follow-up, please let me know. > >> > > > > >>> > > > >> > > > > >>> > > Looking at the broader set of Jiras associated with > 1663 > >> and > >> > > the > >> > > > > >>> first PR > >> > > > > >>> > > 1665, it looks like there are 4 main thrusts of this > >> branch > >> > > > right > >> > > > > now: > >> > > > > >>> > > > >> > > > > >>> > > 1. Knox/SSO > >> > > > > >>> > > 2. Node migrated to Spring Boot > >> > > > > >>> > > 3. JDBC removed completely in favor of LDAP > >> > > > > >>> > > 4. Introduction of Zuul, also microservices? > >> > > > > >>> > > > >> > > > > >>> > > I strongly urge for the purpose of reviewing this > feature > >> > > branch > >> > > > > that > >> > > > > >>> we > >> > > > > >>> > > base much of the discussion off of > >> > > > > >>> > > https://issues.apache.org/jira/browse/METRON-1755, the > >> > > > > architecture > >> > > > > >>> > > diagram. Minimally, an explanation of the current > >> > architecture > >> > > > > along > >> > > > > >>> with > >> > > > > >>> > > discussion around the additional proposed changes and > >> > > rationale > >> > > > > would > >> > > > > >>> be > >> > > > > >>> > > useful for evaluation. I don't have a solid enough > >> > > understanding > >> > > > > yet > >> > > > > >>> of > >> > > > > >>> > the > >> > > > > >>> > > full scope of changes and how they differ from the > >> existing > >> > > > > >>> architecture > >> > > > > >>> > > just from looking at the PR's alone. > >> > > > > >>> > > > >> > > > > >>> > > 1. The first question is a general one regarding the > >> > necessity > >> > > > of > >> > > > > >>> the > >> > > > > >>> > 3 > >> > > > > >>> > > additional features alongside Knox - migrating Node to > >> > Spring > >> > > > > Boot, > >> > > > > >>> > > removing JDBC altogether, adding dependencies on > >> Netflix's > >> > > Zuul > >> > > > > >>> > > framework. > >> > > > > >>> > > Are these necessary for adding Knox/SSO? They seem like > >> > > > > potentially > >> > > > > >>> > > separate features, imo. > >> > > > > >>> > > 2. It looks like LDAP will be a required component for > >> > > > interacting > >> > > > > >>> > with > >> > > > > >>> > > Metron via the UI's. I see this PR > >> > > > > >>> > > https://github.com/apache/metron/pull/1186 which > removes > >> > JDBC > >> > > > > >>> > > authentication. Are we ready to remove it completely or > >> > would > >> > > it > >> > > > > be > >> > > > > >>> > > better > >> > > > > >>> > > to leave it as a minimal installation option? What is > the > >> > > > proposed > >> > > > > >>> > > migration path for existing users? Do we feel > comfortable > >> > > > > requiring > >> > > > > >>> > that > >> > > > > >>> > > all installations, including full dev, install and > >> configure > >> > > > LDAP? > >> > > > > >>> For > >> > > > > >>> > > comparison, in the PCAP feature branch we discussed > >> removing > >> > > the > >> > > > > >>> > > existing > >> > > > > >>> > > PCAP REST application in the initial discussion, got > >> > > agreement, > >> > > > > and > >> > > > > >>> > > later > >> > > > > >>> > > removed it in the course of working on the feature > >> branch. > >> > The > >> > > > PR > >> > > > > >>> is > >> > > > > >>> > > fairly > >> > > > > >>> > > clear, however I think we're just missing some basic > >> > > discussion > >> > > > > >>> around > >> > > > > >>> > > the > >> > > > > >>> > > implications, as I've outlined above. Some additional > >> > relevant > >> > > > > >>> > > discussion > >> > > > > >>> > > occurred on this PR > >> > > https://github.com/apache/metron/pull/1112 > >> > > > > >>> which > >> > > > > >>> > > would be good to summarize for purposes of this > >> overarching > >> > > > > >>> > architecture > >> > > > > >>> > > discussion. > >> > > > > >>> > > 3. Migration from Node to Spring Boot. I believe this > is > >> > > already > >> > > > > >>> used > >> > > > > >>> > by > >> > > > > >>> > > the REST application and if anything brings some > >> cohesion to > >> > > our > >> > > > > >>> > server > >> > > > > >>> > > strategy. Strictly speaking, is there a reason this is > >> > > required > >> > > > > for > >> > > > > >>> > > Knox? > >> > > > > >>> > > It seems comparable to a component upgrade, such as > >> moving > >> > > from > >> > > > ES > >> > > > > >>> 2.x > >> > > > > >>> > > to > >> > > > > >>> > > 5.6.x and upgrading Angular 6. > >> > > > > >>> > > 4. Introduction of Netflix's Zuul. > >> > > > > >>> > > https://issues.apache.org/jira/browse/METRON-1665. > >> > > > > >>> > > - > "The UIs currently proxy to the REST API to avoid > >> CORS > >> > > > > >>> issues, > >> > > > > >>> > > this will be achieved with Zuul." > >> > > > > >>> > > - Can we elaborate more on where or how CORS is a > problem > >> > with > >> > > > > >>> our > >> > > > > >>> > > existing architecture, how Zuul will help solve that, > and > >> > how > >> > > it > >> > > > > >>> > > fits with > >> > > > > >>> > > Knox? Wouldn't this be handled by Knox? Since Larry > McCay > >> > > > > >>> chimed in > >> > > > > >>> > > with > >> > > > > >>> > > interest on the original SSO thread about the FB, I'm > >> hoping > >> > > he > >> > > > > >>> is > >> > > > > >>> > > also > >> > > > > >>> > > willing to chime in on this as well. > >> > > > > >>> > > - This looks like it has the potential to be a rather > >> large > >> > > > > >>> piece > >> > > > > >>> > of > >> > > > > >>> > > fundamental infrastructure (as it's also pertinent to > >> > > > > >>> > microservices) > >> > > > > >>> > > to > >> > > > > >>> > > pull into the platform, and I'd like to be sure the > >> > community > >> > > is > >> > > > > >>> > > aware of > >> > > > > >>> > > and is OK with the implications. > >> > > > > >>> > > 5. > "The proposal is to use a spring boot application, > >> > > allowing > >> > > > > >>> us to > >> > > > > >>> > > harmonize the security implementation across the UI > >> static > >> > > > servers > >> > > > > >>> and > >> > > > > >>> > > the > >> > > > > >>> > > REST layer, and to provide a routing platform for later > >> > > > > >>> > microservices." > >> > > > > >>> > > - > >> > > > > >>> > > https://issues.apache.org/jira/browse/METRON-1665. > >> > > > > >>> > > - Microservices is a pretty loaded term. I know there > had > >> > been > >> > > > > >>> some > >> > > > > >>> > > discussion a while back during the PCAP feature branch > >> > start, > >> > > > > >>> but I > >> > > > > >>> > > don't > >> > > > > >>> > > recall ever reaching a consensus on it. More detail in > >> this > >> > > > > >>> thread > >> > > > > >>> > - > >> > > > > >>> > > > >> > > > > >>> > > > >> > > > > >>> > > >> > > > > >>> > >> > > > > > >> > > > > >> > > > >> > > >> > https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E > >> > > > > >>> > > . > >> > > > > >>> > > Can we get some clarification on what is meant by > >> > > microservices > >> > > > > >>> > > in the case > >> > > > > >>> > > of this FB and relevant PR's, what that architecture > >> looks > >> > > like, > >> > > > > >>> > and > >> > > > > >>> > > how > >> > > > > >>> > > it's achieved with the proposed changes in this PR/FB? > It > >> > > seems > >> > > > > >>> > Zuul > >> > > > > >>> > > is > >> > > > > >>> > > also pertinent to this discussion, but there are many > >> ways > >> > to > >> > > > > >>> > > skin this cat > >> > > > > >>> > > so I don't want to presume - > >> > > > > >>> > > > >> > > > > >>> > > > >> > > > > >>> > >> > > > > > >> > > https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices > >> > > > > >>> > > 6. Zuul, Spring Boot, and microservices - Closely > >> related to > >> > > > > >>> > point 5 > >> > > > > >>> > > above. It seems that we weren't quite ready for this > >> when it > >> > > was > >> > > > > >>> > > brought up > >> > > > > >>> > > in May, or at the very least we had some concern of > what > >> > > > direction > >> > > > > >>> to > >> > > > > >>> > > go. > >> > > > > >>> > > What is the operational impact, mpack impact, and how > we > >> > > propose > >> > > > > to > >> > > > > >>> > > manage > >> > > > > >>> > > it with Kerberos, etc.? > >> > > > > >>> > > > >> > > > > >>> > > > >> > > > > >>> > > >> > > > > >>> > >> > > > > > >> > > > > >> > > > >> > > >> > https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E > >> > > > > >>> > > > >> > > > > >>> > > There is a lot to like in this feature branch, imo. > Great > >> > > > feature > >> > > > > >>> > addition > >> > > > > >>> > > with Knox and SSO. Introduction of LDAP support for > >> > > > > authentication for > >> > > > > >>> > > Metron UI's. Simplification/unification of our server > >> > hosting > >> > > > > >>> > > infrastructure. I'm hoping we can flesh out some of the > >> > > details > >> > > > > >>> pointed > >> > > > > >>> > out > >> > > > > >>> > > above a bit more and get this feature through. Great > >> work so > >> > > > far! > >> > > > > >>> > > > >> > > > > >>> > > Best, > >> > > > > >>> > > Mike Miklavcic > >> > > > > >>> > > > >> > > > > >>> > > >> > > > > >> > >> > > > > >> -- > >> > > > > >> -- > >> > > > > >> simon elliston ball > >> > > > > >> @sireb > >> > > > > > > >> > > > > > -- > >> > > > > > -- > >> > > > > > simon elliston ball > >> > > > > > @sireb > >> > > > > > >> > > > > ------------------- > >> > > > > Thank you, > >> > > > > > >> > > > > James Sirota > >> > > > > PMC- Apache Metron > >> > > > > jsirota AT apache DOT org > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > -- Jon Zeolla
