Thank you, Simon. The diagrams help a lot 19.09.2018, 21:27, "Simon Elliston Ball" <si...@simonellistonball.com>: > To clarify some of this I've put some documentation into > https://github.com/apache/metron/pull/1203 under METRON-1755 ( > https://issues.apache.org/jira/browse/METRON-1755). Hopefully the diagrams > there should make it clearer. > > Simon > > On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> Hi Mike, >> >> Some good points here which could do with some clarification. I suspect >> the architecture documentation could be clearer and fill in some of these >> gaps, and I'll have a look at working on that and providing some diagrams. >> >> The short version is that the Zuul proxy gateway has been added to replace >> the Nodejs express proxy used to gateway the REST api calls in the current >> hosts. This is done in both cases to avoid CORS restrictions by allowing >> the same host that serves the UI files to proxy call to the API. >> >> The choice of Zuul was partly a pragmatic one (it's the one that's there >> in the box as it were with Spring Boot, which we use for the REST API, via >> the Spring Cloud Netflix project which wraps a bunch of related pieces into >> Spring). The choice of Spring Boot to host the UIs themselves was similarly >> for parity with the REST host, to simplify the stack (we remove the >> occasionally problematic need to install nodejs on target servers, which is >> outside of the regular OS and HDP stacks we support). >> >> Arguably, the Zuul proxy is not necessary if we force everything through a >> Knox instance, since Knox would provide a single endpoint. We probably >> however don't want to force Knox and SSL, hence using Zuul to keep it >> closer to our current architecture. Zuul does some other nice things, which >> might help us in future, so it's really about laying down some options for >> potentially doing micro-services style things at a later date. I'm not >> saying we have to, or even should go that way, it will just make life >> easier later if we decide to. It will also help us if we want to add HA, >> circuit breaking etc to the architecture at a later date. That said, I >> regret that I ever said the word micro-services, since it's caused >> confusion. Just think of it as a proxy to deal with the CORS problem. >> >> Zuul is implemented as a set of filters, but we are not using it for its >> authentication filtering. We're using it as a proxy. Shiro is an >> authentication framework, and could arguably used to provide the security >> piece, but frankly wrapping shiro as a replacement for Spring Security in a >> Spring application seemed like it will make life a lot harder. This could >> be done, but it's not the native happy path, and would pull in additional >> dependencies that duplicate functionality that's already embedded in Spring >> Security. >> >> The version of Knox used is the default from HDP. The link version you >> mention is a docs link. I'll update it to be the older version, which is >> the same and we can decide if we want to maintain the freshness of it when >> we look to upgrade underlying patterns. Either way, the content is the >> same. >> >> I did consider other hosting mechanisms, including Undertow a >> >> If you have a different suggestion to using the Spring default ways of >> doing things, or we want to use a framework other than Spring for this, >> then maybe we could change to that, but the route chosen here is definitely >> the easy path in the context of the decision made to use Spring in metron >> rest, and if anything opens up our choices while minimising, in fact >> reducing, our dependency management overhead. >> >> I hope that explains some of the thinking behind the choices made, but the >> guiding principals I followed were: >> * Don't fight the framework if you don't have to >> * Reduce the need for additional installation pieces and third party repos >> * Minimize dependencies we would have to manage >> * Avoid excessive change of the architecture, or forcing users to adopt >> Knox if they didn't want the SSL overhead. >> >> Simon >> >> On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >>> Thanks for the write-up Ryan, this is a great start. I have some further >>> questions based on your feedback and in addition to my initial thread. >>> >>> Just for clarification, what version of Knox are we using? HDP 2.6.5, >>> which >>> is what we currently run full dev against, supports 0.12.0. >>> >>> >>> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html >>> . >>> I see references to Knox 1.1.0 (latest) in this committed PR - >>> >>> >>> https://github.com/apache/metron/pull/1111/files#diff-70b412194819f3cb829566f05d77c1a6R122 >>> . >>> This is probably just a super small mismatch, and it probably goes without >>> saying, but I just want to be doubly sure that we're installing the >>> default >>> via the standard install mechanism as opposed to something separate and >>> manual. >>> >>> On the subject of Zuul wrt Nodejs filters. I'd like to hear some more >>> detail on: >>> >>> 1. Why do we need filtering via Zuul? For instance, is filtering >>> routing >>> not handled by Knox? From the beginner docs: "The gateway itself is a >>> layer >>> over an embedded Jetty JEE server. At the very highest level the >>> gateway >>> processes requests by using request URLs to lookup specific JEE Servlet >>> Filter chain that is used to process the request. The gateway framework >>> provides extensible mechanisms to assemble chains of custom filters >>> that >>> support secured access to services." [1] >>> 2. What other library options were considered for this feature and how >>> was it chosen over the others? I search on "apache spring web filters" >>> and >>> it's almost all about Shiro - https://shiro.apache.org/spring.html. I >>> also see quite a bit about filtering for Spring Boot applications along >>> with a write-up of how to accomplish the same with Web MVC here - >>> >>> >>> https://stackoverflow.com/questions/19825946/how-to-add-a-filter-class-in-spring-boot >>> . >>> The Knox documentation boilerplate examples are also using Shiro. >>> "shiro.ini - The configuration file for the Shiro authentication >>> provider’s >>> filters. This information is derived from the information in the >>> provider >>> section of the topology file." [1] >>> >>> My assumption is that there are deliberate decisions in favor of this mix >>> of technologies over others, and I think some additional explanation will >>> make that clear. As it stands per the Knox documentation, it looks like >>> we're going on a different route from the preferred/recommended idioms. >>> >>> [1] >>> >>> >>> http://knox.apache.org/books/knox-0-12-0/dev-guide.html#Architecture+Overview >>> >>> Ryan, I agree about microservices. This should not derail nor be a major >>> part of discussion around this feature, imho. I think there's quite a bit >>> left to discuss on that subject. I want to make sure that we're not >>> prematurely favoring architectural choices by pulling in libraries that >>> are >>> potentially opinionated about how to accomplish those goals. If they are, >>> I >>> would expect we are comfortable ripping those libraries out if the >>> community favors a different direction. >>> >>> On the subject of Spring Boot vs Nodejs. I can see some rationale for >>> making things homogenous (though, in a microservices architecture, if we >>> go >>> that route, that's not strictly necessary), but what is the justification >>> for Spring Boot over Nodejs? Why would want one over the other? >>> >>> On Mon, Sep 17, 2018 at 3:38 PM Ryan Merriman <merrim...@gmail.com> >>> wrote: >>> >>> > I have reviewed a couple different PRs so I'll add some context where I >>> > can. Obviously Simon would be the most qualified to answer but I'll >>> add my >>> > thoughts. >>> > >>> > For question 1, while they may not all be necessary I think it does make >>> > sense to include them in this feature branch if our primary goal is >>> > integrating Knox SSO. We could push off removing JDBC authentication >>> for >>> > reasons I'll get to in my response to question 2. If we want to do one >>> at >>> > a time (switch to spring boot, add Zuul as a dependency, then add Knox >>> SSO) >>> > then that's ok but I do think there are dependencies and should be done >>> in >>> > order. For example, adding Knox SSO requires some work around request >>> > filtering. If we were to do this before moving to Spring Boot we would >>> > need to implement the filters in Nodejs which would be throwaway once we >>> > get around to migrating away from that. For Zuul, I believe it's >>> purpose >>> > is to facilitate the filtering (although it does a lot more) so it >>> doesn't >>> > make sense to add that separate from the Knox SSO work. >>> > >>> > For question 2, I think you bring up a good point. We probably don't >>> want >>> > to just rip our current authentication method out. We might want to >>> > consider deprecating it instead and making Knox SSO and LDAP >>> authentication >>> > optional. >>> > >>> > For question 3, this is a bigger shift than just a component upgrade. >>> It's >>> > more like shifting platforms, from Elasticsearch to Solr for example. >>> Like >>> > I alluded to in my response to question 1, I don't think we should >>> require >>> > throwaway work just because we want to review these parts separately. >>> > >>> > For question 4, I will defer to Simon. I don't believe we necessarily >>> > require Zuul so I will let him elaborate on why he choose that library >>> and >>> > what the potential impact is of adding it to our project. >>> > >>> > For question 5 and 6, I will also defer to Simon on this. The focus of >>> > this feature as I understand it is a consistent authentication mechanism >>> > and support for SSO. I will let him lay out his vision for micro >>> services. >>> > >>> > Knox SSO would be a great improvement and is what I think we should >>> focus >>> > on in this feature branch. Micro services is something we should >>> certainly >>> > discuss but it might be a bit of a distraction and I wouldn't want to >>> hold >>> > up the other useful parts. >>> > >>> > On Fri, Sep 14, 2018 at 8:38 PM Michael Miklavcic < >>> > michael.miklav...@gmail.com> wrote: >>> > >>> > > Hey all, >>> > > >>> > > I started looking through the Knox SSO feature branch (see here >>> > > https://issues.apache.org/jira/browse/METRON-1663). This is some >>> great >>> > new >>> > > security functionality work and it looks like it will bring some >>> > important >>> > > new features to the Metron platform. I'm coming at this pretty green, >>> so >>> > I >>> > > do have some questions regarding the proposed changes from a high >>> level >>> > > architectural perspective. There are a few changes within the current >>> FB >>> > > PR's that I think could use some further explanation. At first >>> glance, it >>> > > seems we could potentially simplify this branch a great deal and get >>> it >>> > > completed much sooner if we narrowed the focus a bit. But I could >>> > certainly >>> > > be wrong here and happy for other opinions. I searched through the >>> > mailing >>> > > list history to see if there is any additional background and the main >>> > > DISCUSS thread I could find was regarding initially setting up the >>> > feature >>> > > branch, which talked about adding Knox and LDAP. >>> > > >>> > > >>> > >>> >>> https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E >>> > > . >>> > > If I've missed any follow-up, please let me know. >>> > > >>> > > Looking at the broader set of Jiras associated with 1663 and the >>> first PR >>> > > 1665, it looks like there are 4 main thrusts of this branch right now: >>> > > >>> > > 1. Knox/SSO >>> > > 2. Node migrated to Spring Boot >>> > > 3. JDBC removed completely in favor of LDAP >>> > > 4. Introduction of Zuul, also microservices? >>> > > >>> > > I strongly urge for the purpose of reviewing this feature branch that >>> we >>> > > base much of the discussion off of >>> > > https://issues.apache.org/jira/browse/METRON-1755, the architecture >>> > > diagram. Minimally, an explanation of the current architecture along >>> with >>> > > discussion around the additional proposed changes and rationale would >>> be >>> > > useful for evaluation. I don't have a solid enough understanding yet >>> of >>> > the >>> > > full scope of changes and how they differ from the existing >>> architecture >>> > > just from looking at the PR's alone. >>> > > >>> > > 1. The first question is a general one regarding the necessity of >>> the >>> > 3 >>> > > additional features alongside Knox - migrating Node to Spring Boot, >>> > > removing JDBC altogether, adding dependencies on Netflix's Zuul >>> > > framework. >>> > > Are these necessary for adding Knox/SSO? They seem like potentially >>> > > separate features, imo. >>> > > 2. It looks like LDAP will be a required component for interacting >>> > with >>> > > Metron via the UI's. I see this PR >>> > > https://github.com/apache/metron/pull/1186 which removes JDBC >>> > > authentication. Are we ready to remove it completely or would it be >>> > > better >>> > > to leave it as a minimal installation option? What is the proposed >>> > > migration path for existing users? Do we feel comfortable requiring >>> > that >>> > > all installations, including full dev, install and configure LDAP? >>> For >>> > > comparison, in the PCAP feature branch we discussed removing the >>> > > existing >>> > > PCAP REST application in the initial discussion, got agreement, and >>> > > later >>> > > removed it in the course of working on the feature branch. The PR >>> is >>> > > fairly >>> > > clear, however I think we're just missing some basic discussion >>> around >>> > > the >>> > > implications, as I've outlined above. Some additional relevant >>> > > discussion >>> > > occurred on this PR https://github.com/apache/metron/pull/1112 >>> which >>> > > would be good to summarize for purposes of this overarching >>> > architecture >>> > > discussion. >>> > > 3. Migration from Node to Spring Boot. I believe this is already >>> used >>> > by >>> > > the REST application and if anything brings some cohesion to our >>> > server >>> > > strategy. Strictly speaking, is there a reason this is required for >>> > > Knox? >>> > > It seems comparable to a component upgrade, such as moving from ES >>> 2.x >>> > > to >>> > > 5.6.x and upgrading Angular 6. >>> > > 4. Introduction of Netflix's Zuul. >>> > > https://issues.apache.org/jira/browse/METRON-1665. >>> > > - > "The UIs currently proxy to the REST API to avoid CORS >>> issues, >>> > > this will be achieved with Zuul." >>> > > - Can we elaborate more on where or how CORS is a problem with >>> our >>> > > existing architecture, how Zuul will help solve that, and how it >>> > > fits with >>> > > Knox? Wouldn't this be handled by Knox? Since Larry McCay >>> chimed in >>> > > with >>> > > interest on the original SSO thread about the FB, I'm hoping he >>> is >>> > > also >>> > > willing to chime in on this as well. >>> > > - This looks like it has the potential to be a rather large >>> piece >>> > of >>> > > fundamental infrastructure (as it's also pertinent to >>> > microservices) >>> > > to >>> > > pull into the platform, and I'd like to be sure the community is >>> > > aware of >>> > > and is OK with the implications. >>> > > 5. > "The proposal is to use a spring boot application, allowing >>> us to >>> > > harmonize the security implementation across the UI static servers >>> and >>> > > the >>> > > REST layer, and to provide a routing platform for later >>> > microservices." >>> > > - >>> > > https://issues.apache.org/jira/browse/METRON-1665. >>> > > - Microservices is a pretty loaded term. I know there had been >>> some >>> > > discussion a while back during the PCAP feature branch start, >>> but I >>> > > don't >>> > > recall ever reaching a consensus on it. More detail in this >>> thread >>> > - >>> > > >>> > > >>> > >>> >>> https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E >>> > > . >>> > > Can we get some clarification on what is meant by microservices >>> > > in the case >>> > > of this FB and relevant PR's, what that architecture looks like, >>> > and >>> > > how >>> > > it's achieved with the proposed changes in this PR/FB? It seems >>> > Zuul >>> > > is >>> > > also pertinent to this discussion, but there are many ways to >>> > > skin this cat >>> > > so I don't want to presume - >>> > > >>> > > >>> https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices >>> > > 6. Zuul, Spring Boot, and microservices - Closely related to >>> > point 5 >>> > > above. It seems that we weren't quite ready for this when it was >>> > > brought up >>> > > in May, or at the very least we had some concern of what direction >>> to >>> > > go. >>> > > What is the operational impact, mpack impact, and how we propose to >>> > > manage >>> > > it with Kerberos, etc.? >>> > > >>> > > >>> > >>> >>> https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E >>> > > >>> > > There is a lot to like in this feature branch, imo. Great feature >>> > addition >>> > > with Knox and SSO. Introduction of LDAP support for authentication for >>> > > Metron UI's. Simplification/unification of our server hosting >>> > > infrastructure. I'm hoping we can flesh out some of the details >>> pointed >>> > out >>> > > above a bit more and get this feature through. Great work so far! >>> > > >>> > > Best, >>> > > Mike Miklavcic >>> > > >>> > >> >> -- >> -- >> simon elliston ball >> @sireb > > -- > -- > simon elliston ball > @sireb
------------------- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
