Let me clarify on exposing both legacy and Knox URLs at the same time. The base urls will look something like this:
Legacy REST - http://node1:8082/api/v1 Legacy Alerts UI - http://node1:4201:/alerts-list Knox REST - https://node1:8443/gateway/default/metron/api/v1 Knox Alerts UI - https://node1:8443/gateway/default/metron-alerts-ui/alerts-list If Knox were turned on and the alerts UI deployed as is, it would not work. This is because static assets are referenced with http://node1:4201/assets/some-asset.js which does not include the correct context path to the alerts UI in knox. To make it work, you have to set the base ref to "/gateway/default/metron-alerts-ui" so that static assets are referenced at https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js. When you do that, the legacy alerts UI will no longer work. I guess the point I'm trying to make is that we would have to switch between them or have 2 separate application running. I imagine most users only need one or the other running so probably not an issue. Jon, the primary upgrade consideration I see is with authentication. To be able to use Knox, you would have to upgrade to LDAP-based authentication if you were still using JDBC-based authentication in REST. The urls would also change obviously. On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com <zeo...@gmail.com> wrote: > Phew, that was quite the thread to catch up on. > > I agree that this should be optional/pluggable to start, and I'm interested > to hear the issues as they relate to upgrading an existing cluster (given > the suggested approach) and exposing both legacy and knox URLs at the same > time. > > Jon > > On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < > michael.miklav...@gmail.com> > wrote: > > > A couple more things, and I think this goes without saying - whatever we > do > > with Knox should NOT > > > > 1. Require unit and integration tests to use Knox > > 2. Break fulldev > > > > Also, I don't know that I saw you mention this, but I'm unsure how we > > should leverage Knox as a core piece of the platform. i.e. should we make > > this required or optional? I'm open to hearing opinions on this, but I'm > > inclined to keep this a pluggable option. > > > > Mike > > > > > > On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > Thanks for the update Ryan. Per my earlier comments, I thought it might > > be > > > the case that we could dramatically simplify this by leveraging Knox's > > > proxy capabilities, and per your research that appears to be the case. > > This > > > is a dramatic simplification and improvement of this feature imo, +1. > I'm > > > also +1 on a couple distinct steps that you've laid out: fix the UI > > issues > > > in master, then add Knox for SSO. That should help mitigate issues with > > > merge conflicts with ongoing development. > > > > > > > I think it will be a challenge exposing the UIs through both the Knox > > > url and legacy urls at the same time. > > > I'm not sure I understand the issue here. Are you referring to this > > > comment? "Added a ng build option to build the UI with base href set to > > > Knox base path." Isn't it just a matter of URL rewriting/forwarding? I > > > thought we'd be exposing the URL's directly in one context, and through > > > Knox in the other. Either way, it seems like we should be able to > > provide a > > > dynamic base path through configuration in our web applications. I'd > > expect > > > to modify that property based on whether Knox is configured or not. > > > > > > > I'm also not clear on how one would use Knox with REST set to legacy > > > JDBC-based authentication. As far as I know Knox does not support JDBC > so > > > there would be a mismatch between Knox and REST. > > > I'm OK with not having Knox work with JDBC. That's a feature of Knox > and > > > probably not something we care much about. > > > > > > >We could initially make Knox an optional feature that requires setup > > with > > > the help of some documentation (like Kerberos) while keeping the system > > the > > > way it is now by default. > > > Sounds good to me. > > > > > > > I imagine we'll deprecate JDBC-based authentication at some point so > > > that may be a good time to switch. > > > I would like to announce deprecation in our next release and move to > > > remove it in a following release. > > > > > > Thanks for taking this on and great job laying things out. > > > > > > Thanks, > > > Mike > > > > > > On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman <merrim...@gmail.com> > > wrote: > > > > > >> I have spent some time recently reviewing this discussion and the > > feature > > >> branch that Simon put out. I think this is an important feature and > > want > > >> to move it forward. I started another discussion on adding Knox to > our > > >> stack but this discussion has a lot of good context so I will continue > > it > > >> here. > > >> > > >> I think the main point of contention was that this feature branch > > included > > >> several different architectural changes and it was unclear if they > were > > >> needed and if so, could be done separately. Fortunately LDAP > > >> authentication has been accepted into master so we can cross it off > the > > >> list. From my understanding of the points people have made, that > leaves > > >> Knox related SSO changes and migrating expressjs to a different, > > JVM-based > > >> web server that includes proxying capabilities (Zuul). > > >> > > >> I think everyone agrees that if we can limit the scope to just Knox > > >> related > > >> SSO changes that would be ideal. I believe I have found a way to do > > that > > >> while working on a small POC this week. The key to this (Simon > alluded > > to > > >> it earlier) is to put both REST and our UIs behind Knox. I initially > > was > > >> focused on just adding REST as a service in Knox and decided to > > experiment > > >> with also adding our UIs. After I did this it became clear that this > > >> simplifies things considerably: > > >> > > >> - The REST app and the UIs are now served from the same host so > CORS > > >> concerns go away. > > >> - We no longer need to worry about proxying REST requests from the > > UIs > > >> with express or Zuul because Knox handles that for us. This will > > make > > >> our > > >> express configuration even simpler. In fact, all we need is a > simple > > >> way > > >> to serve static UI assets. > > >> - We no longer need to check for SSO tokens and redirect in the UI > > >> web/app servers (or the REST app for that matter) because Knox > > handles > > >> that > > >> for us. > > >> - The UIs can now easily access any Knox service (not just our REST > > >> app) > > >> without any extra proxy configuration. > > >> - SSO token authentication is only necessary in REST so there is no > > >> need > > >> to create shared Spring modules or split functionality out. > > >> > > >> The most significant change I had to make (borrowed from Simon's > feature > > >> branch) was the SSO token authentication mentioned above. The primary > > >> short term benefit with this approach is that outside of some general > > >> deficiencies unrelated to this our UI architecture doesn't need to > > >> fundamentally change. I could summarize the changes as: > > >> > > >> - Knox install and configuration (setting up REST and the alerts UI > > as > > >> Knox services) > > >> - Added Knox SSO token authentication to REST > > >> - Updated REST urls in the UI code (should be configurable) > > >> - Fixed a few UI bugs where relative paths were not being used > > >> - Added a ng build option to build the UI with base href set to > Knox > > >> base path ( > > >> > > >> > > > https://github.com/angular/angular-cli/wiki/build#base-tag-handling-in-indexhtml > > >> ) > > >> > > >> Most the UI changes are preexisting, minor issues that could be fixed > > >> directly in master. We would need to think of an approach for the > base > > >> href build requirement but I'm sure it's not that bad. > > >> > > >> However there will be some backwards compatibility issues we would > need > > to > > >> think through. I think it will be a challenge exposing the UIs > through > > >> both the Knox url and legacy urls at the same time. I'm also not > clear > > on > > >> how one would use Knox with REST set to legacy JDBC-based > > authentication. > > >> As far as I know Knox does not support JDBC so there would be a > mismatch > > >> between Knox and REST. Knox does have the ability to pass along basic > > >> authentication headers so LDAP in REST would work. We could initially > > >> make > > >> Knox an optional feature that requires setup with the help of some > > >> documentation (like Kerberos) while keeping the system the way it is > now > > >> by > > >> default. I imagine we'll deprecate JDBC-based authentication at some > > >> point > > >> so that may be a good time to switch. > > >> > > >> What do people think about this approach? Concerns? Are there any > huge > > >> holes in this I'm not thinking about? > > >> > > >> I want to highlight that the work Simon did in his feature branch was > > >> crucial to better understanding this. I am pretty sure we'll end up > > >> reusing a lot code from that branch. > > >> > > >> On Thu, Sep 27, 2018 at 6:30 PM Michael Miklavcic < > > >> michael.miklav...@gmail.com> wrote: > > >> > > >> > Apparently, I hit send on my last email before finishing my synopsis > > >> (per > > >> > @Otto's Q in Slack). To summarize, based on my current > understanding I > > >> > believe that each of the feature branch changes I've outline above > are > > >> > units of work that are related, yet should be executed on > > independently. > > >> > Knox SSO in its own feature branch. Migrating technologies like > NodeJs > > >> or > > >> > migrating the auth DB to LDAP seem like they belong in their own > > >> separate > > >> > PR's or feature branches. > > >> > > > >> > Thanks, > > >> > Mike > > >> > > > >> > On Thu, Sep 27, 2018 at 4:08 PM Casey Stella <ceste...@gmail.com> > > >> wrote: > > >> > > > >> > > I'm coming in late to the game here, but for my mind a feature > > branch > > >> > > should involve the minimum architectural change to accomplish a > > given > > >> > > feature. > > >> > > The feature in question is SSO integration. It seems to me that > the > > >> > > operative question is can we do the feature without making the > OTHER > > >> > > architectural change > > >> > > (e.g. migrating from expressjs to spring boot + zuul). I would > > argue > > >> > that > > >> > > if we WANT to do that, then it should be a separate feature > branch. > > >> > > > > >> > > Thus, I leave with a question: is there a way to accomplish this > > >> feature > > >> > > without ripping out expressjs? > > >> > > > > >> > > - If so and it is feasible, I would argue that we should > decouple > > >> this > > >> > > into a separate feature branch. > > >> > > - If so and it is infeasible, I'd like to hear an argument as > to > > >> the > > >> > > infeasibility and let's decide given that > > >> > > - If it is not possible, then I'd argue that we should keep > them > > >> > coupled > > >> > > and move this through as-is. > > >> > > > > >> > > On a side-note, it feels a bit weird that we're narrowing to a > > bundled > > >> > > proxy, rather than having that be a pluggable thing. I'm not > super > > >> > > knowledgeable in this space, so I apologize > > >> > > in advance if this is naive, but isn't this a pluggable, external > > >> > component > > >> > > (e.g. nginx)? > > >> > > > > >> > > On Thu, Sep 27, 2018 at 5:05 PM Michael Miklavcic < > > >> > > michael.miklav...@gmail.com> wrote: > > >> > > > > >> > > > I've spent some more time reading through Simon's response and > the > > >> > added > > >> > > > sequence diagram. This is definitely helpful - thank you Simon. > > >> > > > > > >> > > > I need to redact my initial list: > > >> > > > > > >> > > > 1. Node migrated to Spring Boot, expressjs migrated to a > > >> > > > non-JS/non-NodeJs proxying mechanism (ie Zuul in this case) > > >> > > > 2. JDBC removed completely in favor of LDAP > > >> > > > 3. Knox/SSO > > >> > > > > > >> > > > I'm a bit conflicted on the best way to move forward and would > > like > > >> > some > > >> > > > thoughts from other community members on this. I think an > argument > > >> can > > >> > be > > >> > > > made that 1 and 2 are independent of 3, and should/could really > be > > >> > > > independent PR's against master. > > >> > > > > > >> > > > The need for a replacement for expressjs (Zuul in this case) is > an > > >> > > artifact > > >> > > > that our request/response cycle for REST calls is a simple > matter > > of > > >> > > > forwarding with some additional headers for authentication. > > There's > > >> a > > >> > > > JSESSIONID managed by the client browser in our current > > >> architecture, > > >> > for > > >> > > > example. You login to the alerts or the management UI which > > >> forwards a > > >> > > > request to REST, which looks up credentials in a backend > database, > > >> and > > >> > > > passes the results back up the chain. All browser requests go > > >> directly > > >> > to > > >> > > > the specific UI you're working with - this is the CORS problem. > > You > > >> > > can't, > > >> > > > without some effort with headers for adding other domains to the > > >> safe > > >> > > list > > >> > > > or disabling the security check for CORS, make remote calls > > >> directly to > > >> > > > REST. That's why we proxy. Switching over to Spring Boot leaves > a > > >> gap > > >> > > with > > >> > > > expressjs having handled the proxying and filtering, since it's > > only > > >> > > > available to a NodeJs application (it's server-side javascript > vs > > >> the > > >> > > > client side javascript deployed via our Angular applications). > > Enter > > >> > > Zuul, > > >> > > > which now effectively handles that. At runtime, Zuul is a part > of > > >> the > > >> > > > Spring app that serves up our UI's. It handles the requests via > > >> > > filtering, > > >> > > > forwards them to REST, manages the response back to the client. > > Very > > >> > > > similar to what expressjs was doing, per my current > understanding. > > >> The > > >> > > > sequence diagrams Simon added are useful, and I think some of > what > > >> was > > >> > > less > > >> > > > clear was what we currently vs what the new changes are doing to > > the > > >> > > > architecture. This is no fault of Simon's - there simply wasn't > > any > > >> > > > architecture diagrams/documents around this before. Here's my > > >> > impression > > >> > > of > > >> > > > the very very basic current state - someone more familiar with > > this > > >> > > > architecture please advise if I'm incorrect about anything > > (probably > > >> > > Ryan). > > >> > > > > > >> > > > https://imgur.com/f8GtSmh > > >> > > > > > >> > > > Zuul would be replacing the bit about expressjs in the diagram, > > and > > >> > > instead > > >> > > > of node we have spring boot. This covers 1. 2 and 3 are other > > >> issues. > > >> > I'd > > >> > > > like to see similar exposition of those server processes with > knox > > >> > > > involved. I imagine in that case we bump up from 3 to 4 server > > >> > instances > > >> > > > for the additional knox endpoint. > > >> > > > > > >> > > > Mike > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Wed, Sep 19, 2018 at 11:28 AM James Sirota < > jsir...@apache.org > > > > > >> > > wrote: > > >> > > > > > >> > > > > Thank you, Simon. The diagrams help a lot > > >> > > > > > > >> > > > > 19.09.2018, 21:27, "Simon Elliston Ball" < > > >> > si...@simonellistonball.com > > >> > > >: > > >> > > > > > To clarify some of this I've put some documentation into > > >> > > > > > https://github.com/apache/metron/pull/1203 under > METRON-1755 > > ( > > >> > > > > > https://issues.apache.org/jira/browse/METRON-1755). > Hopefully > > >> the > > >> > > > > diagrams > > >> > > > > > there should make it clearer. > > >> > > > > > > > >> > > > > > Simon > > >> > > > > > > > >> > > > > > On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < > > >> > > > > > si...@simonellistonball.com> wrote: > > >> > > > > > > > >> > > > > >> Hi Mike, > > >> > > > > >> > > >> > > > > >> Some good points here which could do with some > > clarification. > > >> I > > >> > > > suspect > > >> > > > > >> the architecture documentation could be clearer and fill > in > > >> some > > >> > of > > >> > > > > these > > >> > > > > >> gaps, and I'll have a look at working on that and > providing > > >> some > > >> > > > > diagrams. > > >> > > > > >> > > >> > > > > >> The short version is that the Zuul proxy gateway has been > > >> added > > >> > to > > >> > > > > replace > > >> > > > > >> the Nodejs express proxy used to gateway the REST api > calls > > in > > >> > the > > >> > > > > current > > >> > > > > >> hosts. This is done in both cases to avoid CORS > restrictions > > >> by > > >> > > > > allowing > > >> > > > > >> the same host that serves the UI files to proxy call to > the > > >> API. > > >> > > > > >> > > >> > > > > >> The choice of Zuul was partly a pragmatic one (it's the > one > > >> > that's > > >> > > > > there > > >> > > > > >> in the box as it were with Spring Boot, which we use for > the > > >> REST > > >> > > > API, > > >> > > > > via > > >> > > > > >> the Spring Cloud Netflix project which wraps a bunch of > > >> related > > >> > > > pieces > > >> > > > > into > > >> > > > > >> Spring). The choice of Spring Boot to host the UIs > > themselves > > >> was > > >> > > > > similarly > > >> > > > > >> for parity with the REST host, to simplify the stack (we > > >> remove > > >> > the > > >> > > > > >> occasionally problematic need to install nodejs on target > > >> > servers, > > >> > > > > which is > > >> > > > > >> outside of the regular OS and HDP stacks we support). > > >> > > > > >> > > >> > > > > >> Arguably, the Zuul proxy is not necessary if we force > > >> everything > > >> > > > > through a > > >> > > > > >> Knox instance, since Knox would provide a single endpoint. > > We > > >> > > > probably > > >> > > > > >> however don't want to force Knox and SSL, hence using Zuul > > to > > >> > keep > > >> > > it > > >> > > > > >> closer to our current architecture. Zuul does some other > > nice > > >> > > things, > > >> > > > > which > > >> > > > > >> might help us in future, so it's really about laying down > > some > > >> > > > options > > >> > > > > for > > >> > > > > >> potentially doing micro-services style things at a later > > date. > > >> > I'm > > >> > > > not > > >> > > > > >> saying we have to, or even should go that way, it will > just > > >> make > > >> > > life > > >> > > > > >> easier later if we decide to. It will also help us if we > > want > > >> to > > >> > > add > > >> > > > > HA, > > >> > > > > >> circuit breaking etc to the architecture at a later date. > > That > > >> > > said, > > >> > > > I > > >> > > > > >> regret that I ever said the word micro-services, since > it's > > >> > caused > > >> > > > > >> confusion. Just think of it as a proxy to deal with the > CORS > > >> > > problem. > > >> > > > > >> > > >> > > > > >> Zuul is implemented as a set of filters, but we are not > > using > > >> it > > >> > > for > > >> > > > > its > > >> > > > > >> authentication filtering. We're using it as a proxy. Shiro > > is > > >> an > > >> > > > > >> authentication framework, and could arguably used to > provide > > >> the > > >> > > > > security > > >> > > > > >> piece, but frankly wrapping shiro as a replacement for > > Spring > > >> > > > Security > > >> > > > > in a > > >> > > > > >> Spring application seemed like it will make life a lot > > harder. > > >> > This > > >> > > > > could > > >> > > > > >> be done, but it's not the native happy path, and would > pull > > in > > >> > > > > additional > > >> > > > > >> dependencies that duplicate functionality that's already > > >> embedded > > >> > > in > > >> > > > > Spring > > >> > > > > >> Security. > > >> > > > > >> > > >> > > > > >> The version of Knox used is the default from HDP. The link > > >> > version > > >> > > > you > > >> > > > > >> mention is a docs link. I'll update it to be the older > > >> version, > > >> > > which > > >> > > > > is > > >> > > > > >> the same and we can decide if we want to maintain the > > >> freshness > > >> > of > > >> > > it > > >> > > > > when > > >> > > > > >> we look to upgrade underlying patterns. Either way, the > > >> content > > >> > is > > >> > > > the > > >> > > > > >> same. > > >> > > > > >> > > >> > > > > >> I did consider other hosting mechanisms, including > Undertow > > a > > >> > > > > >> > > >> > > > > >> If you have a different suggestion to using the Spring > > default > > >> > ways > > >> > > > of > > >> > > > > >> doing things, or we want to use a framework other than > > Spring > > >> for > > >> > > > this, > > >> > > > > >> then maybe we could change to that, but the route chosen > > here > > >> is > > >> > > > > definitely > > >> > > > > >> the easy path in the context of the decision made to use > > >> Spring > > >> > in > > >> > > > > metron > > >> > > > > >> rest, and if anything opens up our choices while > minimising, > > >> in > > >> > > fact > > >> > > > > >> reducing, our dependency management overhead. > > >> > > > > >> > > >> > > > > >> I hope that explains some of the thinking behind the > choices > > >> > made, > > >> > > > but > > >> > > > > the > > >> > > > > >> guiding principals I followed were: > > >> > > > > >> * Don't fight the framework if you don't have to > > >> > > > > >> * Reduce the need for additional installation pieces and > > third > > >> > > party > > >> > > > > repos > > >> > > > > >> * Minimize dependencies we would have to manage > > >> > > > > >> * Avoid excessive change of the architecture, or forcing > > >> users to > > >> > > > adopt > > >> > > > > >> Knox if they didn't want the SSL overhead. > > >> > > > > >> > > >> > > > > >> Simon > > >> > > > > >> > > >> > > > > >> On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic < > > >> > > > > >> michael.miklav...@gmail.com> wrote: > > >> > > > > >> > > >> > > > > >>> Thanks for the write-up Ryan, this is a great start. I > have > > >> some > > >> > > > > further > > >> > > > > >>> questions based on your feedback and in addition to my > > >> initial > > >> > > > thread. > > >> > > > > >>> > > >> > > > > >>> Just for clarification, what version of Knox are we > using? > > >> HDP > > >> > > > 2.6.5, > > >> > > > > >>> which > > >> > > > > >>> is what we currently run full dev against, supports > 0.12.0. > > >> > > > > >>> > > >> > > > > >>> > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html > > >> > > > > >>> . > > >> > > > > >>> I see references to Knox 1.1.0 (latest) in this committed > > PR > > >> - > > >> > > > > >>> > > >> > > > > >>> > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://github.com/apache/metron/pull/1111/files#diff-70b412194819f3cb829566f05d77c1a6R122 > > >> > > > > >>> . > > >> > > > > >>> This is probably just a super small mismatch, and it > > probably > > >> > goes > > >> > > > > without > > >> > > > > >>> saying, but I just want to be doubly sure that we're > > >> installing > > >> > > the > > >> > > > > >>> default > > >> > > > > >>> via the standard install mechanism as opposed to > something > > >> > > separate > > >> > > > > and > > >> > > > > >>> manual. > > >> > > > > >>> > > >> > > > > >>> On the subject of Zuul wrt Nodejs filters. I'd like to > hear > > >> some > > >> > > > more > > >> > > > > >>> detail on: > > >> > > > > >>> > > >> > > > > >>> 1. Why do we need filtering via Zuul? For instance, is > > >> > > filtering > > >> > > > > >>> routing > > >> > > > > >>> not handled by Knox? From the beginner docs: "The > > gateway > > >> > > itself > > >> > > > > is a > > >> > > > > >>> layer > > >> > > > > >>> over an embedded Jetty JEE server. At the very highest > > >> level > > >> > > the > > >> > > > > >>> gateway > > >> > > > > >>> processes requests by using request URLs to lookup > > >> specific > > >> > JEE > > >> > > > > Servlet > > >> > > > > >>> Filter chain that is used to process the request. The > > >> gateway > > >> > > > > framework > > >> > > > > >>> provides extensible mechanisms to assemble chains of > > >> custom > > >> > > > filters > > >> > > > > >>> that > > >> > > > > >>> support secured access to services." [1] > > >> > > > > >>> 2. What other library options were considered for this > > >> > feature > > >> > > > and > > >> > > > > how > > >> > > > > >>> was it chosen over the others? I search on "apache > > spring > > >> web > > >> > > > > filters" > > >> > > > > >>> and > > >> > > > > >>> it's almost all about Shiro - > > >> > > > https://shiro.apache.org/spring.html. > > >> > > > > I > > >> > > > > >>> also see quite a bit about filtering for Spring Boot > > >> > > applications > > >> > > > > along > > >> > > > > >>> with a write-up of how to accomplish the same with Web > > MVC > > >> > > here - > > >> > > > > >>> > > >> > > > > >>> > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://stackoverflow.com/questions/19825946/how-to-add-a-filter-class-in-spring-boot > > >> > > > > >>> . > > >> > > > > >>> The Knox documentation boilerplate examples are also > > using > > >> > > Shiro. > > >> > > > > >>> "shiro.ini - The configuration file for the Shiro > > >> > > authentication > > >> > > > > >>> provider’s > > >> > > > > >>> filters. This information is derived from the > > information > > >> in > > >> > > the > > >> > > > > >>> provider > > >> > > > > >>> section of the topology file." [1] > > >> > > > > >>> > > >> > > > > >>> My assumption is that there are deliberate decisions in > > >> favor of > > >> > > > this > > >> > > > > mix > > >> > > > > >>> of technologies over others, and I think some additional > > >> > > explanation > > >> > > > > will > > >> > > > > >>> make that clear. As it stands per the Knox documentation, > > it > > >> > looks > > >> > > > > like > > >> > > > > >>> we're going on a different route from the > > >> preferred/recommended > > >> > > > > idioms. > > >> > > > > >>> > > >> > > > > >>> [1] > > >> > > > > >>> > > >> > > > > >>> > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > http://knox.apache.org/books/knox-0-12-0/dev-guide.html#Architecture+Overview > > >> > > > > >>> > > >> > > > > >>> Ryan, I agree about microservices. This should not derail > > nor > > >> > be a > > >> > > > > major > > >> > > > > >>> part of discussion around this feature, imho. I think > > there's > > >> > > quite > > >> > > > a > > >> > > > > bit > > >> > > > > >>> left to discuss on that subject. I want to make sure that > > >> we're > > >> > > not > > >> > > > > >>> prematurely favoring architectural choices by pulling in > > >> > libraries > > >> > > > > that > > >> > > > > >>> are > > >> > > > > >>> potentially opinionated about how to accomplish those > > goals. > > >> If > > >> > > they > > >> > > > > are, > > >> > > > > >>> I > > >> > > > > >>> would expect we are comfortable ripping those libraries > out > > >> if > > >> > the > > >> > > > > >>> community favors a different direction. > > >> > > > > >>> > > >> > > > > >>> On the subject of Spring Boot vs Nodejs. I can see some > > >> > rationale > > >> > > > for > > >> > > > > >>> making things homogenous (though, in a microservices > > >> > architecture, > > >> > > > if > > >> > > > > we > > >> > > > > >>> go > > >> > > > > >>> that route, that's not strictly necessary), but what is > the > > >> > > > > justification > > >> > > > > >>> for Spring Boot over Nodejs? Why would want one over the > > >> other? > > >> > > > > >>> > > >> > > > > >>> On Mon, Sep 17, 2018 at 3:38 PM Ryan Merriman < > > >> > > merrim...@gmail.com> > > >> > > > > >>> wrote: > > >> > > > > >>> > > >> > > > > >>> > I have reviewed a couple different PRs so I'll add some > > >> > context > > >> > > > > where I > > >> > > > > >>> > can. Obviously Simon would be the most qualified to > > answer > > >> but > > >> > > > I'll > > >> > > > > >>> add my > > >> > > > > >>> > thoughts. > > >> > > > > >>> > > > >> > > > > >>> > For question 1, while they may not all be necessary I > > >> think it > > >> > > > does > > >> > > > > make > > >> > > > > >>> > sense to include them in this feature branch if our > > primary > > >> > goal > > >> > > > is > > >> > > > > >>> > integrating Knox SSO. We could push off removing JDBC > > >> > > > authentication > > >> > > > > >>> for > > >> > > > > >>> > reasons I'll get to in my response to question 2. If we > > >> want > > >> > to > > >> > > do > > >> > > > > one > > >> > > > > >>> at > > >> > > > > >>> > a time (switch to spring boot, add Zuul as a > dependency, > > >> then > > >> > > add > > >> > > > > Knox > > >> > > > > >>> SSO) > > >> > > > > >>> > then that's ok but I do think there are dependencies > and > > >> > should > > >> > > be > > >> > > > > done > > >> > > > > >>> in > > >> > > > > >>> > order. For example, adding Knox SSO requires some work > > >> around > > >> > > > > request > > >> > > > > >>> > filtering. If we were to do this before moving to > Spring > > >> Boot > > >> > we > > >> > > > > would > > >> > > > > >>> > need to implement the filters in Nodejs which would be > > >> > throwaway > > >> > > > > once we > > >> > > > > >>> > get around to migrating away from that. For Zuul, I > > believe > > >> > it's > > >> > > > > >>> purpose > > >> > > > > >>> > is to facilitate the filtering (although it does a lot > > >> more) > > >> > so > > >> > > it > > >> > > > > >>> doesn't > > >> > > > > >>> > make sense to add that separate from the Knox SSO work. > > >> > > > > >>> > > > >> > > > > >>> > For question 2, I think you bring up a good point. We > > >> probably > > >> > > > don't > > >> > > > > >>> want > > >> > > > > >>> > to just rip our current authentication method out. We > > might > > >> > want > > >> > > > to > > >> > > > > >>> > consider deprecating it instead and making Knox SSO and > > >> LDAP > > >> > > > > >>> authentication > > >> > > > > >>> > optional. > > >> > > > > >>> > > > >> > > > > >>> > For question 3, this is a bigger shift than just a > > >> component > > >> > > > > upgrade. > > >> > > > > >>> It's > > >> > > > > >>> > more like shifting platforms, from Elasticsearch to > Solr > > >> for > > >> > > > > example. > > >> > > > > >>> Like > > >> > > > > >>> > I alluded to in my response to question 1, I don't > think > > we > > >> > > should > > >> > > > > >>> require > > >> > > > > >>> > throwaway work just because we want to review these > parts > > >> > > > > separately. > > >> > > > > >>> > > > >> > > > > >>> > For question 4, I will defer to Simon. I don't believe > we > > >> > > > > necessarily > > >> > > > > >>> > require Zuul so I will let him elaborate on why he > choose > > >> that > > >> > > > > library > > >> > > > > >>> and > > >> > > > > >>> > what the potential impact is of adding it to our > project. > > >> > > > > >>> > > > >> > > > > >>> > For question 5 and 6, I will also defer to Simon on > this. > > >> The > > >> > > > focus > > >> > > > > of > > >> > > > > >>> > this feature as I understand it is a consistent > > >> authentication > > >> > > > > mechanism > > >> > > > > >>> > and support for SSO. I will let him lay out his vision > > for > > >> > micro > > >> > > > > >>> services. > > >> > > > > >>> > > > >> > > > > >>> > Knox SSO would be a great improvement and is what I > think > > >> we > > >> > > > should > > >> > > > > >>> focus > > >> > > > > >>> > on in this feature branch. Micro services is something > we > > >> > should > > >> > > > > >>> certainly > > >> > > > > >>> > discuss but it might be a bit of a distraction and I > > >> wouldn't > > >> > > want > > >> > > > > to > > >> > > > > >>> hold > > >> > > > > >>> > up the other useful parts. > > >> > > > > >>> > > > >> > > > > >>> > On Fri, Sep 14, 2018 at 8:38 PM Michael Miklavcic < > > >> > > > > >>> > michael.miklav...@gmail.com> wrote: > > >> > > > > >>> > > > >> > > > > >>> > > Hey all, > > >> > > > > >>> > > > > >> > > > > >>> > > I started looking through the Knox SSO feature branch > > >> (see > > >> > > here > > >> > > > > >>> > > https://issues.apache.org/jira/browse/METRON-1663). > > >> This is > > >> > > > some > > >> > > > > >>> great > > >> > > > > >>> > new > > >> > > > > >>> > > security functionality work and it looks like it will > > >> bring > > >> > > some > > >> > > > > >>> > important > > >> > > > > >>> > > new features to the Metron platform. I'm coming at > this > > >> > pretty > > >> > > > > green, > > >> > > > > >>> so > > >> > > > > >>> > I > > >> > > > > >>> > > do have some questions regarding the proposed changes > > >> from a > > >> > > > high > > >> > > > > >>> level > > >> > > > > >>> > > architectural perspective. There are a few changes > > within > > >> > the > > >> > > > > current > > >> > > > > >>> FB > > >> > > > > >>> > > PR's that I think could use some further explanation. > > At > > >> > first > > >> > > > > >>> glance, it > > >> > > > > >>> > > seems we could potentially simplify this branch a > great > > >> deal > > >> > > and > > >> > > > > get > > >> > > > > >>> it > > >> > > > > >>> > > completed much sooner if we narrowed the focus a bit. > > >> But I > > >> > > > could > > >> > > > > >>> > certainly > > >> > > > > >>> > > be wrong here and happy for other opinions. I > searched > > >> > through > > >> > > > the > > >> > > > > >>> > mailing > > >> > > > > >>> > > list history to see if there is any additional > > background > > >> > and > > >> > > > the > > >> > > > > main > > >> > > > > >>> > > DISCUSS thread I could find was regarding initially > > >> setting > > >> > up > > >> > > > the > > >> > > > > >>> > feature > > >> > > > > >>> > > branch, which talked about adding Knox and LDAP. > > >> > > > > >>> > > > > >> > > > > >>> > > > > >> > > > > >>> > > > >> > > > > >>> > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E > > >> > > > > >>> > > . > > >> > > > > >>> > > If I've missed any follow-up, please let me know. > > >> > > > > >>> > > > > >> > > > > >>> > > Looking at the broader set of Jiras associated with > > 1663 > > >> and > > >> > > the > > >> > > > > >>> first PR > > >> > > > > >>> > > 1665, it looks like there are 4 main thrusts of this > > >> branch > > >> > > > right > > >> > > > > now: > > >> > > > > >>> > > > > >> > > > > >>> > > 1. Knox/SSO > > >> > > > > >>> > > 2. Node migrated to Spring Boot > > >> > > > > >>> > > 3. JDBC removed completely in favor of LDAP > > >> > > > > >>> > > 4. Introduction of Zuul, also microservices? > > >> > > > > >>> > > > > >> > > > > >>> > > I strongly urge for the purpose of reviewing this > > feature > > >> > > branch > > >> > > > > that > > >> > > > > >>> we > > >> > > > > >>> > > base much of the discussion off of > > >> > > > > >>> > > https://issues.apache.org/jira/browse/METRON-1755, > the > > >> > > > > architecture > > >> > > > > >>> > > diagram. Minimally, an explanation of the current > > >> > architecture > > >> > > > > along > > >> > > > > >>> with > > >> > > > > >>> > > discussion around the additional proposed changes and > > >> > > rationale > > >> > > > > would > > >> > > > > >>> be > > >> > > > > >>> > > useful for evaluation. I don't have a solid enough > > >> > > understanding > > >> > > > > yet > > >> > > > > >>> of > > >> > > > > >>> > the > > >> > > > > >>> > > full scope of changes and how they differ from the > > >> existing > > >> > > > > >>> architecture > > >> > > > > >>> > > just from looking at the PR's alone. > > >> > > > > >>> > > > > >> > > > > >>> > > 1. The first question is a general one regarding the > > >> > necessity > > >> > > > of > > >> > > > > >>> the > > >> > > > > >>> > 3 > > >> > > > > >>> > > additional features alongside Knox - migrating Node > to > > >> > Spring > > >> > > > > Boot, > > >> > > > > >>> > > removing JDBC altogether, adding dependencies on > > >> Netflix's > > >> > > Zuul > > >> > > > > >>> > > framework. > > >> > > > > >>> > > Are these necessary for adding Knox/SSO? They seem > like > > >> > > > > potentially > > >> > > > > >>> > > separate features, imo. > > >> > > > > >>> > > 2. It looks like LDAP will be a required component > for > > >> > > > interacting > > >> > > > > >>> > with > > >> > > > > >>> > > Metron via the UI's. I see this PR > > >> > > > > >>> > > https://github.com/apache/metron/pull/1186 which > > removes > > >> > JDBC > > >> > > > > >>> > > authentication. Are we ready to remove it completely > or > > >> > would > > >> > > it > > >> > > > > be > > >> > > > > >>> > > better > > >> > > > > >>> > > to leave it as a minimal installation option? What is > > the > > >> > > > proposed > > >> > > > > >>> > > migration path for existing users? Do we feel > > comfortable > > >> > > > > requiring > > >> > > > > >>> > that > > >> > > > > >>> > > all installations, including full dev, install and > > >> configure > > >> > > > LDAP? > > >> > > > > >>> For > > >> > > > > >>> > > comparison, in the PCAP feature branch we discussed > > >> removing > > >> > > the > > >> > > > > >>> > > existing > > >> > > > > >>> > > PCAP REST application in the initial discussion, got > > >> > > agreement, > > >> > > > > and > > >> > > > > >>> > > later > > >> > > > > >>> > > removed it in the course of working on the feature > > >> branch. > > >> > The > > >> > > > PR > > >> > > > > >>> is > > >> > > > > >>> > > fairly > > >> > > > > >>> > > clear, however I think we're just missing some basic > > >> > > discussion > > >> > > > > >>> around > > >> > > > > >>> > > the > > >> > > > > >>> > > implications, as I've outlined above. Some additional > > >> > relevant > > >> > > > > >>> > > discussion > > >> > > > > >>> > > occurred on this PR > > >> > > https://github.com/apache/metron/pull/1112 > > >> > > > > >>> which > > >> > > > > >>> > > would be good to summarize for purposes of this > > >> overarching > > >> > > > > >>> > architecture > > >> > > > > >>> > > discussion. > > >> > > > > >>> > > 3. Migration from Node to Spring Boot. I believe this > > is > > >> > > already > > >> > > > > >>> used > > >> > > > > >>> > by > > >> > > > > >>> > > the REST application and if anything brings some > > >> cohesion to > > >> > > our > > >> > > > > >>> > server > > >> > > > > >>> > > strategy. Strictly speaking, is there a reason this > is > > >> > > required > > >> > > > > for > > >> > > > > >>> > > Knox? > > >> > > > > >>> > > It seems comparable to a component upgrade, such as > > >> moving > > >> > > from > > >> > > > ES > > >> > > > > >>> 2.x > > >> > > > > >>> > > to > > >> > > > > >>> > > 5.6.x and upgrading Angular 6. > > >> > > > > >>> > > 4. Introduction of Netflix's Zuul. > > >> > > > > >>> > > https://issues.apache.org/jira/browse/METRON-1665. > > >> > > > > >>> > > - > "The UIs currently proxy to the REST API to avoid > > >> CORS > > >> > > > > >>> issues, > > >> > > > > >>> > > this will be achieved with Zuul." > > >> > > > > >>> > > - Can we elaborate more on where or how CORS is a > > problem > > >> > with > > >> > > > > >>> our > > >> > > > > >>> > > existing architecture, how Zuul will help solve that, > > and > > >> > how > > >> > > it > > >> > > > > >>> > > fits with > > >> > > > > >>> > > Knox? Wouldn't this be handled by Knox? Since Larry > > McCay > > >> > > > > >>> chimed in > > >> > > > > >>> > > with > > >> > > > > >>> > > interest on the original SSO thread about the FB, I'm > > >> hoping > > >> > > he > > >> > > > > >>> is > > >> > > > > >>> > > also > > >> > > > > >>> > > willing to chime in on this as well. > > >> > > > > >>> > > - This looks like it has the potential to be a rather > > >> large > > >> > > > > >>> piece > > >> > > > > >>> > of > > >> > > > > >>> > > fundamental infrastructure (as it's also pertinent to > > >> > > > > >>> > microservices) > > >> > > > > >>> > > to > > >> > > > > >>> > > pull into the platform, and I'd like to be sure the > > >> > community > > >> > > is > > >> > > > > >>> > > aware of > > >> > > > > >>> > > and is OK with the implications. > > >> > > > > >>> > > 5. > "The proposal is to use a spring boot > application, > > >> > > allowing > > >> > > > > >>> us to > > >> > > > > >>> > > harmonize the security implementation across the UI > > >> static > > >> > > > servers > > >> > > > > >>> and > > >> > > > > >>> > > the > > >> > > > > >>> > > REST layer, and to provide a routing platform for > later > > >> > > > > >>> > microservices." > > >> > > > > >>> > > - > > >> > > > > >>> > > https://issues.apache.org/jira/browse/METRON-1665. > > >> > > > > >>> > > - Microservices is a pretty loaded term. I know there > > had > > >> > been > > >> > > > > >>> some > > >> > > > > >>> > > discussion a while back during the PCAP feature > branch > > >> > start, > > >> > > > > >>> but I > > >> > > > > >>> > > don't > > >> > > > > >>> > > recall ever reaching a consensus on it. More detail > in > > >> this > > >> > > > > >>> thread > > >> > > > > >>> > - > > >> > > > > >>> > > > > >> > > > > >>> > > > > >> > > > > >>> > > > >> > > > > >>> > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E > > >> > > > > >>> > > . > > >> > > > > >>> > > Can we get some clarification on what is meant by > > >> > > microservices > > >> > > > > >>> > > in the case > > >> > > > > >>> > > of this FB and relevant PR's, what that architecture > > >> looks > > >> > > like, > > >> > > > > >>> > and > > >> > > > > >>> > > how > > >> > > > > >>> > > it's achieved with the proposed changes in this > PR/FB? > > It > > >> > > seems > > >> > > > > >>> > Zuul > > >> > > > > >>> > > is > > >> > > > > >>> > > also pertinent to this discussion, but there are many > > >> ways > > >> > to > > >> > > > > >>> > > skin this cat > > >> > > > > >>> > > so I don't want to presume - > > >> > > > > >>> > > > > >> > > > > >>> > > > > >> > > > > >>> > > >> > > > > > > >> > > > https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices > > >> > > > > >>> > > 6. Zuul, Spring Boot, and microservices - Closely > > >> related to > > >> > > > > >>> > point 5 > > >> > > > > >>> > > above. It seems that we weren't quite ready for this > > >> when it > > >> > > was > > >> > > > > >>> > > brought up > > >> > > > > >>> > > in May, or at the very least we had some concern of > > what > > >> > > > direction > > >> > > > > >>> to > > >> > > > > >>> > > go. > > >> > > > > >>> > > What is the operational impact, mpack impact, and how > > we > > >> > > propose > > >> > > > > to > > >> > > > > >>> > > manage > > >> > > > > >>> > > it with Kerberos, etc.? > > >> > > > > >>> > > > > >> > > > > >>> > > > > >> > > > > >>> > > > >> > > > > >>> > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E > > >> > > > > >>> > > > > >> > > > > >>> > > There is a lot to like in this feature branch, imo. > > Great > > >> > > > feature > > >> > > > > >>> > addition > > >> > > > > >>> > > with Knox and SSO. Introduction of LDAP support for > > >> > > > > authentication for > > >> > > > > >>> > > Metron UI's. Simplification/unification of our server > > >> > hosting > > >> > > > > >>> > > infrastructure. I'm hoping we can flesh out some of > the > > >> > > details > > >> > > > > >>> pointed > > >> > > > > >>> > out > > >> > > > > >>> > > above a bit more and get this feature through. Great > > >> work so > > >> > > > far! > > >> > > > > >>> > > > > >> > > > > >>> > > Best, > > >> > > > > >>> > > Mike Miklavcic > > >> > > > > >>> > > > > >> > > > > >>> > > > >> > > > > >> > > >> > > > > >> -- > > >> > > > > >> -- > > >> > > > > >> simon elliston ball > > >> > > > > >> @sireb > > >> > > > > > > > >> > > > > > -- > > >> > > > > > -- > > >> > > > > > simon elliston ball > > >> > > > > > @sireb > > >> > > > > > > >> > > > > ------------------- > > >> > > > > Thank you, > > >> > > > > > > >> > > > > James Sirota > > >> > > > > PMC- Apache Metron > > >> > > > > jsirota AT apache DOT org > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > -- > > Jon Zeolla >
