Hi Mike, Some good points here which could do with some clarification. I suspect the architecture documentation could be clearer and fill in some of these gaps, and I'll have a look at working on that and providing some diagrams.
The short version is that the Zuul proxy gateway has been added to replace the Nodejs express proxy used to gateway the REST api calls in the current hosts. This is done in both cases to avoid CORS restrictions by allowing the same host that serves the UI files to proxy call to the API. The choice of Zuul was partly a pragmatic one (it's the one that's there in the box as it were with Spring Boot, which we use for the REST API, via the Spring Cloud Netflix project which wraps a bunch of related pieces into Spring). The choice of Spring Boot to host the UIs themselves was similarly for parity with the REST host, to simplify the stack (we remove the occasionally problematic need to install nodejs on target servers, which is outside of the regular OS and HDP stacks we support). Arguably, the Zuul proxy is not necessary if we force everything through a Knox instance, since Knox would provide a single endpoint. We probably however don't want to force Knox and SSL, hence using Zuul to keep it closer to our current architecture. Zuul does some other nice things, which might help us in future, so it's really about laying down some options for potentially doing micro-services style things at a later date. I'm not saying we have to, or even should go that way, it will just make life easier later if we decide to. It will also help us if we want to add HA, circuit breaking etc to the architecture at a later date. That said, I regret that I ever said the word micro-services, since it's caused confusion. Just think of it as a proxy to deal with the CORS problem. Zuul is implemented as a set of filters, but we are not using it for its authentication filtering. We're using it as a proxy. Shiro is an authentication framework, and could arguably used to provide the security piece, but frankly wrapping shiro as a replacement for Spring Security in a Spring application seemed like it will make life a lot harder. This could be done, but it's not the native happy path, and would pull in additional dependencies that duplicate functionality that's already embedded in Spring Security. The version of Knox used is the default from HDP. The link version you mention is a docs link. I'll update it to be the older version, which is the same and we can decide if we want to maintain the freshness of it when we look to upgrade underlying patterns. Either way, the content is the same. I did consider other hosting mechanisms, including Undertow a If you have a different suggestion to using the Spring default ways of doing things, or we want to use a framework other than Spring for this, then maybe we could change to that, but the route chosen here is definitely the easy path in the context of the decision made to use Spring in metron rest, and if anything opens up our choices while minimising, in fact reducing, our dependency management overhead. I hope that explains some of the thinking behind the choices made, but the guiding principals I followed were: * Don't fight the framework if you don't have to * Reduce the need for additional installation pieces and third party repos * Minimize dependencies we would have to manage * Avoid excessive change of the architecture, or forcing users to adopt Knox if they didn't want the SSL overhead. Simon On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic <michael.miklav...@gmail.com> wrote: > Thanks for the write-up Ryan, this is a great start. I have some further > questions based on your feedback and in addition to my initial thread. > > Just for clarification, what version of Knox are we using? HDP 2.6.5, which > is what we currently run full dev against, supports 0.12.0. > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html > . > I see references to Knox 1.1.0 (latest) in this committed PR - > > https://github.com/apache/metron/pull/1111/files#diff-70b412194819f3cb829566f05d77c1a6R122 > . > This is probably just a super small mismatch, and it probably goes without > saying, but I just want to be doubly sure that we're installing the default > via the standard install mechanism as opposed to something separate and > manual. > > On the subject of Zuul wrt Nodejs filters. I'd like to hear some more > detail on: > > 1. Why do we need filtering via Zuul? For instance, is filtering routing > not handled by Knox? From the beginner docs: "The gateway itself is a > layer > over an embedded Jetty JEE server. At the very highest level the gateway > processes requests by using request URLs to lookup specific JEE Servlet > Filter chain that is used to process the request. The gateway framework > provides extensible mechanisms to assemble chains of custom filters that > support secured access to services." [1] > 2. What other library options were considered for this feature and how > was it chosen over the others? I search on "apache spring web filters" > and > it's almost all about Shiro - https://shiro.apache.org/spring.html. I > also see quite a bit about filtering for Spring Boot applications along > with a write-up of how to accomplish the same with Web MVC here - > > https://stackoverflow.com/questions/19825946/how-to-add-a-filter-class-in-spring-boot > . > The Knox documentation boilerplate examples are also using Shiro. > "shiro.ini - The configuration file for the Shiro authentication > provider’s > filters. This information is derived from the information in the > provider > section of the topology file." [1] > > My assumption is that there are deliberate decisions in favor of this mix > of technologies over others, and I think some additional explanation will > make that clear. As it stands per the Knox documentation, it looks like > we're going on a different route from the preferred/recommended idioms. > > [1] > > http://knox.apache.org/books/knox-0-12-0/dev-guide.html#Architecture+Overview > > Ryan, I agree about microservices. This should not derail nor be a major > part of discussion around this feature, imho. I think there's quite a bit > left to discuss on that subject. I want to make sure that we're not > prematurely favoring architectural choices by pulling in libraries that are > potentially opinionated about how to accomplish those goals. If they are, I > would expect we are comfortable ripping those libraries out if the > community favors a different direction. > > On the subject of Spring Boot vs Nodejs. I can see some rationale for > making things homogenous (though, in a microservices architecture, if we go > that route, that's not strictly necessary), but what is the justification > for Spring Boot over Nodejs? Why would want one over the other? > > On Mon, Sep 17, 2018 at 3:38 PM Ryan Merriman <merrim...@gmail.com> wrote: > > > I have reviewed a couple different PRs so I'll add some context where I > > can. Obviously Simon would be the most qualified to answer but I'll add > my > > thoughts. > > > > For question 1, while they may not all be necessary I think it does make > > sense to include them in this feature branch if our primary goal is > > integrating Knox SSO. We could push off removing JDBC authentication for > > reasons I'll get to in my response to question 2. If we want to do one > at > > a time (switch to spring boot, add Zuul as a dependency, then add Knox > SSO) > > then that's ok but I do think there are dependencies and should be done > in > > order. For example, adding Knox SSO requires some work around request > > filtering. If we were to do this before moving to Spring Boot we would > > need to implement the filters in Nodejs which would be throwaway once we > > get around to migrating away from that. For Zuul, I believe it's purpose > > is to facilitate the filtering (although it does a lot more) so it > doesn't > > make sense to add that separate from the Knox SSO work. > > > > For question 2, I think you bring up a good point. We probably don't > want > > to just rip our current authentication method out. We might want to > > consider deprecating it instead and making Knox SSO and LDAP > authentication > > optional. > > > > For question 3, this is a bigger shift than just a component upgrade. > It's > > more like shifting platforms, from Elasticsearch to Solr for example. > Like > > I alluded to in my response to question 1, I don't think we should > require > > throwaway work just because we want to review these parts separately. > > > > For question 4, I will defer to Simon. I don't believe we necessarily > > require Zuul so I will let him elaborate on why he choose that library > and > > what the potential impact is of adding it to our project. > > > > For question 5 and 6, I will also defer to Simon on this. The focus of > > this feature as I understand it is a consistent authentication mechanism > > and support for SSO. I will let him lay out his vision for micro > services. > > > > Knox SSO would be a great improvement and is what I think we should focus > > on in this feature branch. Micro services is something we should > certainly > > discuss but it might be a bit of a distraction and I wouldn't want to > hold > > up the other useful parts. > > > > On Fri, Sep 14, 2018 at 8:38 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > Hey all, > > > > > > I started looking through the Knox SSO feature branch (see here > > > https://issues.apache.org/jira/browse/METRON-1663). This is some great > > new > > > security functionality work and it looks like it will bring some > > important > > > new features to the Metron platform. I'm coming at this pretty green, > so > > I > > > do have some questions regarding the proposed changes from a high level > > > architectural perspective. There are a few changes within the current > FB > > > PR's that I think could use some further explanation. At first glance, > it > > > seems we could potentially simplify this branch a great deal and get it > > > completed much sooner if we narrowed the focus a bit. But I could > > certainly > > > be wrong here and happy for other opinions. I searched through the > > mailing > > > list history to see if there is any additional background and the main > > > DISCUSS thread I could find was regarding initially setting up the > > feature > > > branch, which talked about adding Knox and LDAP. > > > > > > > > > https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E > > > . > > > If I've missed any follow-up, please let me know. > > > > > > Looking at the broader set of Jiras associated with 1663 and the first > PR > > > 1665, it looks like there are 4 main thrusts of this branch right now: > > > > > > 1. Knox/SSO > > > 2. Node migrated to Spring Boot > > > 3. JDBC removed completely in favor of LDAP > > > 4. Introduction of Zuul, also microservices? > > > > > > I strongly urge for the purpose of reviewing this feature branch that > we > > > base much of the discussion off of > > > https://issues.apache.org/jira/browse/METRON-1755, the architecture > > > diagram. Minimally, an explanation of the current architecture along > with > > > discussion around the additional proposed changes and rationale would > be > > > useful for evaluation. I don't have a solid enough understanding yet of > > the > > > full scope of changes and how they differ from the existing > architecture > > > just from looking at the PR's alone. > > > > > > 1. The first question is a general one regarding the necessity of > the > > 3 > > > additional features alongside Knox - migrating Node to Spring Boot, > > > removing JDBC altogether, adding dependencies on Netflix's Zuul > > > framework. > > > Are these necessary for adding Knox/SSO? They seem like potentially > > > separate features, imo. > > > 2. It looks like LDAP will be a required component for interacting > > with > > > Metron via the UI's. I see this PR > > > https://github.com/apache/metron/pull/1186 which removes JDBC > > > authentication. Are we ready to remove it completely or would it be > > > better > > > to leave it as a minimal installation option? What is the proposed > > > migration path for existing users? Do we feel comfortable requiring > > that > > > all installations, including full dev, install and configure LDAP? > For > > > comparison, in the PCAP feature branch we discussed removing the > > > existing > > > PCAP REST application in the initial discussion, got agreement, and > > > later > > > removed it in the course of working on the feature branch. The PR is > > > fairly > > > clear, however I think we're just missing some basic discussion > around > > > the > > > implications, as I've outlined above. Some additional relevant > > > discussion > > > occurred on this PR https://github.com/apache/metron/pull/1112 > which > > > would be good to summarize for purposes of this overarching > > architecture > > > discussion. > > > 3. Migration from Node to Spring Boot. I believe this is already > used > > by > > > the REST application and if anything brings some cohesion to our > > server > > > strategy. Strictly speaking, is there a reason this is required for > > > Knox? > > > It seems comparable to a component upgrade, such as moving from ES > 2.x > > > to > > > 5.6.x and upgrading Angular 6. > > > 4. Introduction of Netflix's Zuul. > > > https://issues.apache.org/jira/browse/METRON-1665. > > > - > "The UIs currently proxy to the REST API to avoid CORS > issues, > > > this will be achieved with Zuul." > > > - Can we elaborate more on where or how CORS is a problem with > our > > > existing architecture, how Zuul will help solve that, and how it > > > fits with > > > Knox? Wouldn't this be handled by Knox? Since Larry McCay chimed > in > > > with > > > interest on the original SSO thread about the FB, I'm hoping he > is > > > also > > > willing to chime in on this as well. > > > - This looks like it has the potential to be a rather large piece > > of > > > fundamental infrastructure (as it's also pertinent to > > microservices) > > > to > > > pull into the platform, and I'd like to be sure the community is > > > aware of > > > and is OK with the implications. > > > 5. > "The proposal is to use a spring boot application, allowing us > to > > > harmonize the security implementation across the UI static servers > and > > > the > > > REST layer, and to provide a routing platform for later > > microservices." > > > - > > > https://issues.apache.org/jira/browse/METRON-1665. > > > - Microservices is a pretty loaded term. I know there had been > some > > > discussion a while back during the PCAP feature branch start, > but I > > > don't > > > recall ever reaching a consensus on it. More detail in this > thread > > - > > > > > > > > > https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E > > > . > > > Can we get some clarification on what is meant by microservices > > > in the case > > > of this FB and relevant PR's, what that architecture looks like, > > and > > > how > > > it's achieved with the proposed changes in this PR/FB? It seems > > Zuul > > > is > > > also pertinent to this discussion, but there are many ways to > > > skin this cat > > > so I don't want to presume - > > > > > > https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices > > > 6. Zuul, Spring Boot, and microservices - Closely related to > > point 5 > > > above. It seems that we weren't quite ready for this when it was > > > brought up > > > in May, or at the very least we had some concern of what direction > to > > > go. > > > What is the operational impact, mpack impact, and how we propose to > > > manage > > > it with Kerberos, etc.? > > > > > > > > > https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E > > > > > > There is a lot to like in this feature branch, imo. Great feature > > addition > > > with Knox and SSO. Introduction of LDAP support for authentication for > > > Metron UI's. Simplification/unification of our server hosting > > > infrastructure. I'm hoping we can flesh out some of the details pointed > > out > > > above a bit more and get this feature through. Great work so far! > > > > > > Best, > > > Mike Miklavcic > > > > > > -- -- simon elliston ball @sireb
