I did not change the internals of rewriter implementation or dispatcher implementation for the POC. So, it should be getting these from wherever it was getting it from without my POC change. I need some time and help from you/Kevin to comprehend what goes on in the internals of rewriter.
My proposal attempts to expose the filters and filter chain to administrator. gateway.xml hides the filters and chain from the administrator. They have their pros and cons. Would leave it to the choice of the community. Thanks Dilli On Fri, Oct 25, 2013 at 9:36 AM, larry mccay <[email protected]> wrote: > hmmmm, so, where does the dispatcher get the endpoint info from - the > topology file? > That isn't even in WEB-INF. > > > On Fri, Oct 25, 2013 at 12:30 PM, Dilli Arumugam > <[email protected]>wrote: > > > The POC is done starting with a knox web app already deployed. > > Then, > > > > 1. remove WEB-INF/gateway.xml > > 2. update WEB-INF/web.xml > > 3. update WEB-INF/shiro.ini > > > > So, the web app was first deployed using current knox contributor > > framework. > > Have not spent time to simulate the equivalent of contribution > framework. > > > > Thanks > > Dilli > > > > > > On Fri, Oct 25, 2013 at 9:23 AM, larry mccay <[email protected]> > > wrote: > > > > > On Fri, Oct 25, 2013 at 12:03 PM, Dilli Arumugam > > > <[email protected]>wrote: > > > > > > > Thanks Larry for the review and comments. > > > > > > > > The proposed knox.ini is meant to be edited by Knox system > > administrator > > > > until we get to have a admin console. This is not meant to be edited > by > > > end > > > > user. > > > > > > > > > > Well, by end user here, I meant the same person that had to edit > > > topology.xml. > > > Topology.xml is much simpler than this file and most of what is in > > > shiro.ini here should never need to be seen - even by the > administrators. > > > Giving the ability to mess with the order of the chain and the like is > > only > > > asking for trouble and user problems. > > > I think that we should limit the amount of time we have to support > Shiro > > > config file problems by keeping folks out as much as we can. > > > > > > > > > > > > > > To integrate additional back end hadoop service, following pieces of > > > > information have to come from pointers from knox.ini to external > > > > configuration files. > > > > > > > > 1. end point of the new hadoop servie back end > > > > 2. rewriting rules for the new service > > > > > > > > > > These can be specified in one file or split between 2 files with one > of > > > > them being topology file or its equivalent. > > > > I have not shown all those details in the current shiro.ini. > > > > But, you would get the idea. > > > > > > > > > > But if you don't show all of that then it isn't showing the whole > > picture. > > > What are you currently doing in your POC for the services? > > > > > > > > > > > > > > Dilli > > > > > > > > > > > > On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <[email protected]> > > wrote: > > > > > > > > > Hi Dilli - > > > > > > > > > > Good work here! > > > > > > > > > > Question: > > > > > 1. Is this complicated ini hand edited by the enduser in order to > > > deploy > > > > a > > > > > given topology or have you changed the shiro contributor to write > > this > > > > out > > > > > from a much more understandable config within the topology? > > > > > > > > > > 2. Where are the services configured for a given topology? > > > > > 3. Did you rename all the existing filters to start with knox? > > > > > > > > > > A couple observations: > > > > > > > > > > 1. removing the use of gateway.xml means that we *require* Shiro > for > > > our > > > > > servlet filter chaining > > > > > 2. While comparing this to the current format of gateway.xml I can > > > agree > > > > > that this is easier to understand - however, I don't actually find > > ini > > > > > format simpler to comprehend and administer than XML or JSON. In > > fact, > > > > > comparing this to topology.xml which is all you really need to do > for > > > > > admin, I think the topology file is very easy to understand and > > > > administer. > > > > > 3. moving the rewrite rules out to not require coding is good but > > > should > > > > > probably not require that they be in this format and file > > > > > 4. Someone that wants to leverage their own or another third party > > > > solution > > > > > is stuck squeezing it into Shiro > > > > > 5. This would violate a principle of mine of "Is-a" vs "Has-a" - > Knox > > > is > > > > > not a Shiro provider it can however *have* a Shiro provider > > > > > 6. End users should never have to see rewrite rules and maybe a lot > > of > > > > what > > > > > is in that ini. Principal mapping, authz policy, etc will > eventually > > > come > > > > > from Zookeeper which should remove their need to be in this file > for > > > > those > > > > > things. Maybe we can even get the LDAP config separately from there > > as > > > > well > > > > > - though I still think it is best provided in a topology file with > a > > > much > > > > > smaller config requirements and get pushed into shiro.ini. Then the > > > user > > > > > may never need to see - or at least open - this file. > > > > > > > > > > I continue to believe that we are best served by enabling a > > deployment > > > > that > > > > > looks like this and encouraging the use of it as the preferred > > > solution. > > > > > However, we should continue to support the use of arbitrary > providers > > > > > through the use of gateway.xml. In most cases, gateway.xml can be > > > > leveraged > > > > > to only include the Shiro provider. In addition, all knowledge of > > Shiro > > > > > specifics should remain inside of that provider. If it leaks out > then > > > we > > > > > are violating the separation of Is-a vs Has-a again. > > > > > > > > > > This also allows us the flexibility to add something before or > after > > > that > > > > > provider for things that we find are not easily or appropriately > done > > > > > inside the security provider. > > > > > > > > > > "I think we can rename shiro.ini as knox.ini to make it explicit > this > > > is > > > > > more about knox configuration than shiro library configuration." > > > > > > > > > > While I can understand the thinking of renaming the shiro.ini file > to > > > > > knox.ini, also violates the Is-a vs Has-a principle for me. We are > > not > > > a > > > > > shiro security provider extension. We are a REST API Gateway for > > Hadoop > > > > > that has security providers - one of which is Shiro. > > > > > > > > > > "We are using shiro config file mechanism as simple, lightweight > > > > depenency > > > > > injection." > > > > > > > > > > Where is there dependency injection here? If you are referring to > the > > > > > configuration of the filter chain that is not really injection > though > > > it > > > > is > > > > > an admittedly simpler mechanism than the current state of the > > > > contributors > > > > > and our deployment machinery - which could be refactored and > > improved. > > > > > > > > > > "This does not really tie us to using only Shiro authentication > > > mechanism > > > > > or > > > > > authorization mechanisms. > > > > > > > > > > We have the choice of writing all our authentication or > authorization > > > in > > > > > our own servlet filters or leverage filters from Shiro library, > > Realms > > > > from > > > > > Shiro library or write your own Shiro filters or Shiro realms." > > > > > > > > > > The only change that this proposal makes to our flexibility is the > > > > > inability to take Shiro out of the picture. > > > > > We could have leveraged Shiro to do all these things with the > current > > > > > design as well. > > > > > BUT, we have control over our own interception channel in our > chains > > > > today. > > > > > This proposal takes a specific *security provider's* proprietary > > filter > > > > > chain and uses it as the only interceptor channel for the entire > > > server. > > > > > Just because they also had a need for one - for their security > work - > > > > > doesn't mean that we shouldn't have our own. The tail should not > wag > > > the > > > > > dog. Shiro is not an application or server framework it is a > security > > > > > provider. > > > > > > > > > > "In most of the cases, when we want to integrate with a new Hadoop > > back > > > > end > > > > > service, all that we have to do is specify the path to a file > having > > > > > rewrite rules for the service. Rest of the things in > > > knox.ini(=shiro.ini) > > > > > would remain same." > > > > > > > > > > This isn't really very clear to me. Do we still have services in a > > > > > topology.xml file? > > > > > I don't see where you specify the url for the backend services - > so I > > > > > assume there is still a topology file. > > > > > How do we know where to dispatch the requests to? > > > > > > > > > > I do agree that this is a very good step forward in defining a > > > preferred > > > > > security provider model for Knox and that we should continue down > > this > > > > > road. We just need to do so carefully. If, over time, we have found > > no > > > > > reason to use the existing solution that we can consider removing > it > > > > > entirely. > > > > > > > > > > Thanks for all this work and insight, Dilli! > > > > > > > > > > --larry > > > > > > > > > > > > > > > On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam > > > > > <[email protected]>wrote: > > > > > > > > > > > Have been trying out a few ideas with Knox and managed to get > Knox > > > > > running > > > > > > with > > > > > > > > > > > > 1. WEB-INF/gateway.xml is completely removed > > > > > > 2. WEB-INF/web.xml declares only shiro filter and a defautl > servlet > > > > > > 2. all filters are defined and injected using WEB-INF/shiro.ini > > > > > > > > > > > > To me that looks much simpler to comprehend and administer. > > > > > > Agreed, this could be subjective. > > > > > > Hence, seeking comments from community. > > > > > > > > > > > > Pasting web.xml and shiro.ini inline. > > > > > > > > > > > > web.xml > > > > > > -------------- > > > > > > > > > > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > > > > > > <web-app xmlns="http://java.sun.com/xml/ns/javaee"; xmlns:xsi=" > > > > > > http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" > > > > > > http://java.sun.com/xml/ns/javaee > > > > > > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd";> > > > > > > > > > > > > <listener> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class> > > > > > > </listener> > > > > > > > > > > > > <listener> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class> > > > > > > </listener> > > > > > > > > > > > > <listener> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class> > > > > > > </listener> > > > > > > > > > > > > <context-param> > > > > > > <param-name>rewriteDescriptorLocation</param-name> > > > > > > <param-value>/WEB-INF/rewrite.xml</param-value> > > > > > > </context-param> > > > > > > > > > > > > <filter> > > > > > > <filter-name>ShiroFilter</filter-name> > > > > > > > > > > > > <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class> > > > > > > </filter> > > > > > > > > > > > > <filter-mapping> > > > > > > <filter-name>ShiroFilter</filter-name> > > > > > > <url-pattern>/*</url-pattern> > > > > > > </filter-mapping> > > > > > > > > > > > > <session-config> > > > > > > <session-timeout>30</session-timeout> > > > > > > </session-config> > > > > > > > > > > > > <servlet> > > > > > > <servlet-name>errorservlet</servlet-name> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class> > > > > > > </servlet> > > > > > > > > > > > > <servlet-mapping> > > > > > > <servlet-name>errorservlet</servlet-name> > > > > > > <url-pattern>/*</url-pattern> > > > > > > </servlet-mapping> > > > > > > > > > > > > </web-app> > > > > > > > > > > > > > > > > > > shiro.ini > > > > > > ------------- > > > > > > > > > > > > [main] > > > > > > > > > > > > # define ldapRealm > > > > > > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm > > > > > > ldapRealm.contextFactory.authenticationMechanism=simple > > > > > > ldapRealm.contextFactory.url=ldap://localhost:33389 > > > > > > > > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org > > > > > > > > > > > > > > > > > > # define filter: knoxResponseCookieFilter > > > > > > knoxResponseCookieFilter = > > > > > > org.apache.hadoop.gateway.filter.ResponseCookieFilter > > > > > > knoxResponseCookieFilter.enabled = true > > > > > > knoxResponseCookieFilter.filterHeaders = rememberMe, > > > hadoop.auth.cookie > > > > > > > > > > > > # define filter: knoxPrincipalMapper > > > > > > knoxPrincipalMapper = > > > > > org.apache.hadoop.gateway.filter.KnoxPrincipalMapper > > > > > > knoxPrincipalMapper.enabled = true > > > > > > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob > > > > > > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin > > > > > > > > > > > > # define filter: knoxIPTracker > > > > > > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker > > > > > > knoxIPTracker.enabled = true > > > > > > > > > > > > # define filter: knoxAclAuthzFilter > > > > > > knoxAclAuthzFilter = > > > > org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter > > > > > > knoxAclAuthzFilter.enabled = true > > > > > > knoxAclAuthzFilter.globalGroupAclMode = OR > > > > > > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, > > > > /templeton/:OR > > > > > > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users > > > admin; > > > > > > 127.*.*.*, /templeton/:user11 user12; users admin > > > > > > > > > > > > # define filter: javaSubjectMapper > > > > > > javaSubjectMapper = > > > > > > org.apache.hadoop.gateway.filter.PostAuthenticationFilter > > > > > > > > > > > > # define filter: knoxIdentityAsserter > > > > > > knoxIdentityAsserter = > > > > > > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter > > > > > > knoxIdentityAsserter.enabled = true > > > > > > > > > > > > # define filter: knoxUrlRewriter > > > > > > knoxUrlRewriter = > > > > > > > > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter > > > > > > > > > > > > knoxUrlRewriter.requestUrlMap = > > > > > > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root, > > > > > > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file, > > > > > > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home, > > > > > > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file, > > > > > > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode > > > > > > > > > > > > knoxUrlRewriter.requestBodyMap = > /oozie/:OOZIE/oozie/configuration, > > > > > > /oozie/v1/.*:OOZIE/oozie/configuration, > > > > > > /oozie/v2/.*:OOZIE/oozie/configuration > > > > > > > > > > > > knoxUrlRewriter.responseHeadersMap = > > > > > > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers, > > > > > > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers, > > > > > > /hbase/:WEBHBASE/webhbase/headers/outbound, > > > > > > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound, > > > > > > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound, > > > > > > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound > > > > > > > > > > > > # define filter: knoxHttpDispatcher > > > > > > knoxHttpDispatcher = > > > > > org.apache.hadoop.gateway.dispatch.HttpClientDispatch > > > > > > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, > > > > oozie:4 > > > > > > > > > > > > [urls] > > > > > > # you could choose to have a different chain of filter for > > different > > > > url > > > > > > patterns > > > > > > # so far Knox did not need it > > > > > > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper, > > > > > > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper, > > > > > knoxIdentityAsserter, > > > > > > knoxUrlRewriter, knoxHttpDispatcher > > > > > > > > > > > > # end of shiro.ini > > > > > > > > > > > > I think we can rename shiro.ini as knox.ini to make it explicit > > this > > > is > > > > > > more about knox configuration than shiro library configuration. > > > > > > > > > > > > > > > > > > We are using shiro config file mechanism as simple, lightweight > > > > depenency > > > > > > injection. > > > > > > > > > > > > This does not really tie us to using only Shiro authentication > > > > mechanism > > > > > or > > > > > > authorization mechanisms. > > > > > > > > > > > > We have the choice of writing all our authentication or > > authorization > > > > in > > > > > > our own servlet filters or leverage filters from Shiro library, > > > Realms > > > > > from > > > > > > Shiro library or write your own Shiro filters or Shiro realms. > > > > > > > > > > > > In most of the cases, when we want to integrate with a new Hadoop > > > back > > > > > end > > > > > > service, all that we have to do is specify the path to a file > > having > > > > > > rewrite rules for the service. Rest of the things in > > > > knox.ini(=shiro.ini) > > > > > > would remain same. > > > > > > > > > > > > Please review and comment. > > > > > > > > > > > > > > > > > > Thanks > > > > > > Dilli > > > > > > > > > > > > -- > > > > > > CONFIDENTIALITY NOTICE > > > > > > NOTICE: This message is intended for the use of the individual or > > > > entity > > > > > to > > > > > > which it is addressed and may contain information that is > > > confidential, > > > > > > privileged and exempt from disclosure under applicable law. If > the > > > > reader > > > > > > of this message is not the intended recipient, you are hereby > > > notified > > > > > that > > > > > > any printing, copying, dissemination, distribution, disclosure or > > > > > > forwarding of this communication is strictly prohibited. If you > > have > > > > > > received this communication in error, please contact the sender > > > > > immediately > > > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > > > > > > > -- > > > > CONFIDENTIALITY NOTICE > > > > NOTICE: This message is intended for the use of the individual or > > entity > > > to > > > > which it is addressed and may contain information that is > confidential, > > > > privileged and exempt from disclosure under applicable law. If the > > reader > > > > of this message is not the intended recipient, you are hereby > notified > > > that > > > > any printing, copying, dissemination, distribution, disclosure or > > > > forwarding of this communication is strictly prohibited. If you have > > > > received this communication in error, please contact the sender > > > immediately > > > > and delete it from your system. Thank You. > > > > > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
