Thanks for the write-up Ryan, this is a great start. I have some further questions based on your feedback and in addition to my initial thread.
Just for clarification, what version of Knox are we using? HDP 2.6.5, which is what we currently run full dev against, supports 0.12.0. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html. I see references to Knox 1.1.0 (latest) in this committed PR - https://github.com/apache/metron/pull/1111/files#diff-70b412194819f3cb829566f05d77c1a6R122. This is probably just a super small mismatch, and it probably goes without saying, but I just want to be doubly sure that we're installing the default via the standard install mechanism as opposed to something separate and manual. On the subject of Zuul wrt Nodejs filters. I'd like to hear some more detail on: 1. Why do we need filtering via Zuul? For instance, is filtering routing not handled by Knox? From the beginner docs: "The gateway itself is a layer over an embedded Jetty JEE server. At the very highest level the gateway processes requests by using request URLs to lookup specific JEE Servlet Filter chain that is used to process the request. The gateway framework provides extensible mechanisms to assemble chains of custom filters that support secured access to services." [1] 2. What other library options were considered for this feature and how was it chosen over the others? I search on "apache spring web filters" and it's almost all about Shiro - https://shiro.apache.org/spring.html. I also see quite a bit about filtering for Spring Boot applications along with a write-up of how to accomplish the same with Web MVC here - https://stackoverflow.com/questions/19825946/how-to-add-a-filter-class-in-spring-boot. The Knox documentation boilerplate examples are also using Shiro. "shiro.ini - The configuration file for the Shiro authentication provider’s filters. This information is derived from the information in the provider section of the topology file." [1] My assumption is that there are deliberate decisions in favor of this mix of technologies over others, and I think some additional explanation will make that clear. As it stands per the Knox documentation, it looks like we're going on a different route from the preferred/recommended idioms. [1] http://knox.apache.org/books/knox-0-12-0/dev-guide.html#Architecture+Overview Ryan, I agree about microservices. This should not derail nor be a major part of discussion around this feature, imho. I think there's quite a bit left to discuss on that subject. I want to make sure that we're not prematurely favoring architectural choices by pulling in libraries that are potentially opinionated about how to accomplish those goals. If they are, I would expect we are comfortable ripping those libraries out if the community favors a different direction. On the subject of Spring Boot vs Nodejs. I can see some rationale for making things homogenous (though, in a microservices architecture, if we go that route, that's not strictly necessary), but what is the justification for Spring Boot over Nodejs? Why would want one over the other? On Mon, Sep 17, 2018 at 3:38 PM Ryan Merriman <merrim...@gmail.com> wrote: > I have reviewed a couple different PRs so I'll add some context where I > can. Obviously Simon would be the most qualified to answer but I'll add my > thoughts. > > For question 1, while they may not all be necessary I think it does make > sense to include them in this feature branch if our primary goal is > integrating Knox SSO. We could push off removing JDBC authentication for > reasons I'll get to in my response to question 2. If we want to do one at > a time (switch to spring boot, add Zuul as a dependency, then add Knox SSO) > then that's ok but I do think there are dependencies and should be done in > order. For example, adding Knox SSO requires some work around request > filtering. If we were to do this before moving to Spring Boot we would > need to implement the filters in Nodejs which would be throwaway once we > get around to migrating away from that. For Zuul, I believe it's purpose > is to facilitate the filtering (although it does a lot more) so it doesn't > make sense to add that separate from the Knox SSO work. > > For question 2, I think you bring up a good point. We probably don't want > to just rip our current authentication method out. We might want to > consider deprecating it instead and making Knox SSO and LDAP authentication > optional. > > For question 3, this is a bigger shift than just a component upgrade. It's > more like shifting platforms, from Elasticsearch to Solr for example. Like > I alluded to in my response to question 1, I don't think we should require > throwaway work just because we want to review these parts separately. > > For question 4, I will defer to Simon. I don't believe we necessarily > require Zuul so I will let him elaborate on why he choose that library and > what the potential impact is of adding it to our project. > > For question 5 and 6, I will also defer to Simon on this. The focus of > this feature as I understand it is a consistent authentication mechanism > and support for SSO. I will let him lay out his vision for micro services. > > Knox SSO would be a great improvement and is what I think we should focus > on in this feature branch. Micro services is something we should certainly > discuss but it might be a bit of a distraction and I wouldn't want to hold > up the other useful parts. > > On Fri, Sep 14, 2018 at 8:38 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Hey all, > > > > I started looking through the Knox SSO feature branch (see here > > https://issues.apache.org/jira/browse/METRON-1663). This is some great > new > > security functionality work and it looks like it will bring some > important > > new features to the Metron platform. I'm coming at this pretty green, so > I > > do have some questions regarding the proposed changes from a high level > > architectural perspective. There are a few changes within the current FB > > PR's that I think could use some further explanation. At first glance, it > > seems we could potentially simplify this branch a great deal and get it > > completed much sooner if we narrowed the focus a bit. But I could > certainly > > be wrong here and happy for other opinions. I searched through the > mailing > > list history to see if there is any additional background and the main > > DISCUSS thread I could find was regarding initially setting up the > feature > > branch, which talked about adding Knox and LDAP. > > > > > https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E > > . > > If I've missed any follow-up, please let me know. > > > > Looking at the broader set of Jiras associated with 1663 and the first PR > > 1665, it looks like there are 4 main thrusts of this branch right now: > > > > 1. Knox/SSO > > 2. Node migrated to Spring Boot > > 3. JDBC removed completely in favor of LDAP > > 4. Introduction of Zuul, also microservices? > > > > I strongly urge for the purpose of reviewing this feature branch that we > > base much of the discussion off of > > https://issues.apache.org/jira/browse/METRON-1755, the architecture > > diagram. Minimally, an explanation of the current architecture along with > > discussion around the additional proposed changes and rationale would be > > useful for evaluation. I don't have a solid enough understanding yet of > the > > full scope of changes and how they differ from the existing architecture > > just from looking at the PR's alone. > > > > 1. The first question is a general one regarding the necessity of the > 3 > > additional features alongside Knox - migrating Node to Spring Boot, > > removing JDBC altogether, adding dependencies on Netflix's Zuul > > framework. > > Are these necessary for adding Knox/SSO? They seem like potentially > > separate features, imo. > > 2. It looks like LDAP will be a required component for interacting > with > > Metron via the UI's. I see this PR > > https://github.com/apache/metron/pull/1186 which removes JDBC > > authentication. Are we ready to remove it completely or would it be > > better > > to leave it as a minimal installation option? What is the proposed > > migration path for existing users? Do we feel comfortable requiring > that > > all installations, including full dev, install and configure LDAP? For > > comparison, in the PCAP feature branch we discussed removing the > > existing > > PCAP REST application in the initial discussion, got agreement, and > > later > > removed it in the course of working on the feature branch. The PR is > > fairly > > clear, however I think we're just missing some basic discussion around > > the > > implications, as I've outlined above. Some additional relevant > > discussion > > occurred on this PR https://github.com/apache/metron/pull/1112 which > > would be good to summarize for purposes of this overarching > architecture > > discussion. > > 3. Migration from Node to Spring Boot. I believe this is already used > by > > the REST application and if anything brings some cohesion to our > server > > strategy. Strictly speaking, is there a reason this is required for > > Knox? > > It seems comparable to a component upgrade, such as moving from ES 2.x > > to > > 5.6.x and upgrading Angular 6. > > 4. Introduction of Netflix's Zuul. > > https://issues.apache.org/jira/browse/METRON-1665. > > - > "The UIs currently proxy to the REST API to avoid CORS issues, > > this will be achieved with Zuul." > > - Can we elaborate more on where or how CORS is a problem with our > > existing architecture, how Zuul will help solve that, and how it > > fits with > > Knox? Wouldn't this be handled by Knox? Since Larry McCay chimed in > > with > > interest on the original SSO thread about the FB, I'm hoping he is > > also > > willing to chime in on this as well. > > - This looks like it has the potential to be a rather large piece > of > > fundamental infrastructure (as it's also pertinent to > microservices) > > to > > pull into the platform, and I'd like to be sure the community is > > aware of > > and is OK with the implications. > > 5. > "The proposal is to use a spring boot application, allowing us to > > harmonize the security implementation across the UI static servers and > > the > > REST layer, and to provide a routing platform for later > microservices." > > - > > https://issues.apache.org/jira/browse/METRON-1665. > > - Microservices is a pretty loaded term. I know there had been some > > discussion a while back during the PCAP feature branch start, but I > > don't > > recall ever reaching a consensus on it. More detail in this thread > - > > > > > https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E > > . > > Can we get some clarification on what is meant by microservices > > in the case > > of this FB and relevant PR's, what that architecture looks like, > and > > how > > it's achieved with the proposed changes in this PR/FB? It seems > Zuul > > is > > also pertinent to this discussion, but there are many ways to > > skin this cat > > so I don't want to presume - > > > > https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices > > 6. Zuul, Spring Boot, and microservices - Closely related to > point 5 > > above. It seems that we weren't quite ready for this when it was > > brought up > > in May, or at the very least we had some concern of what direction to > > go. > > What is the operational impact, mpack impact, and how we propose to > > manage > > it with Kerberos, etc.? > > > > > https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E > > > > There is a lot to like in this feature branch, imo. Great feature > addition > > with Knox and SSO. Introduction of LDAP support for authentication for > > Metron UI's. Simplification/unification of our server hosting > > infrastructure. I'm hoping we can flesh out some of the details pointed > out > > above a bit more and get this feature through. Great work so far! > > > > Best, > > Mike Miklavcic > > >
