hmmmm, so, where does the dispatcher get the endpoint info from - the topology file? That isn't even in WEB-INF.
On Fri, Oct 25, 2013 at 12:30 PM, Dilli Arumugam <[email protected]>wrote: > The POC is done starting with a knox web app already deployed. > Then, > > 1. remove WEB-INF/gateway.xml > 2. update WEB-INF/web.xml > 3. update WEB-INF/shiro.ini > > So, the web app was first deployed using current knox contributor > framework. > Have not spent time to simulate the equivalent of contribution framework. > > Thanks > Dilli > > > On Fri, Oct 25, 2013 at 9:23 AM, larry mccay <[email protected]> > wrote: > > > On Fri, Oct 25, 2013 at 12:03 PM, Dilli Arumugam > > <[email protected]>wrote: > > > > > Thanks Larry for the review and comments. > > > > > > The proposed knox.ini is meant to be edited by Knox system > administrator > > > until we get to have a admin console. This is not meant to be edited by > > end > > > user. > > > > > > > Well, by end user here, I meant the same person that had to edit > > topology.xml. > > Topology.xml is much simpler than this file and most of what is in > > shiro.ini here should never need to be seen - even by the administrators. > > Giving the ability to mess with the order of the chain and the like is > only > > asking for trouble and user problems. > > I think that we should limit the amount of time we have to support Shiro > > config file problems by keeping folks out as much as we can. > > > > > > > > > > To integrate additional back end hadoop service, following pieces of > > > information have to come from pointers from knox.ini to external > > > configuration files. > > > > > > 1. end point of the new hadoop servie back end > > > 2. rewriting rules for the new service > > > > > > > These can be specified in one file or split between 2 files with one of > > > them being topology file or its equivalent. > > > I have not shown all those details in the current shiro.ini. > > > But, you would get the idea. > > > > > > > But if you don't show all of that then it isn't showing the whole > picture. > > What are you currently doing in your POC for the services? > > > > > > > > > > Dilli > > > > > > > > > On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <[email protected]> > wrote: > > > > > > > Hi Dilli - > > > > > > > > Good work here! > > > > > > > > Question: > > > > 1. Is this complicated ini hand edited by the enduser in order to > > deploy > > > a > > > > given topology or have you changed the shiro contributor to write > this > > > out > > > > from a much more understandable config within the topology? > > > > > > > > 2. Where are the services configured for a given topology? > > > > 3. Did you rename all the existing filters to start with knox? > > > > > > > > A couple observations: > > > > > > > > 1. removing the use of gateway.xml means that we *require* Shiro for > > our > > > > servlet filter chaining > > > > 2. While comparing this to the current format of gateway.xml I can > > agree > > > > that this is easier to understand - however, I don't actually find > ini > > > > format simpler to comprehend and administer than XML or JSON. In > fact, > > > > comparing this to topology.xml which is all you really need to do for > > > > admin, I think the topology file is very easy to understand and > > > administer. > > > > 3. moving the rewrite rules out to not require coding is good but > > should > > > > probably not require that they be in this format and file > > > > 4. Someone that wants to leverage their own or another third party > > > solution > > > > is stuck squeezing it into Shiro > > > > 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox > > is > > > > not a Shiro provider it can however *have* a Shiro provider > > > > 6. End users should never have to see rewrite rules and maybe a lot > of > > > what > > > > is in that ini. Principal mapping, authz policy, etc will eventually > > come > > > > from Zookeeper which should remove their need to be in this file for > > > those > > > > things. Maybe we can even get the LDAP config separately from there > as > > > well > > > > - though I still think it is best provided in a topology file with a > > much > > > > smaller config requirements and get pushed into shiro.ini. Then the > > user > > > > may never need to see - or at least open - this file. > > > > > > > > I continue to believe that we are best served by enabling a > deployment > > > that > > > > looks like this and encouraging the use of it as the preferred > > solution. > > > > However, we should continue to support the use of arbitrary providers > > > > through the use of gateway.xml. In most cases, gateway.xml can be > > > leveraged > > > > to only include the Shiro provider. In addition, all knowledge of > Shiro > > > > specifics should remain inside of that provider. If it leaks out then > > we > > > > are violating the separation of Is-a vs Has-a again. > > > > > > > > This also allows us the flexibility to add something before or after > > that > > > > provider for things that we find are not easily or appropriately done > > > > inside the security provider. > > > > > > > > "I think we can rename shiro.ini as knox.ini to make it explicit this > > is > > > > more about knox configuration than shiro library configuration." > > > > > > > > While I can understand the thinking of renaming the shiro.ini file to > > > > knox.ini, also violates the Is-a vs Has-a principle for me. We are > not > > a > > > > shiro security provider extension. We are a REST API Gateway for > Hadoop > > > > that has security providers - one of which is Shiro. > > > > > > > > "We are using shiro config file mechanism as simple, lightweight > > > depenency > > > > injection." > > > > > > > > Where is there dependency injection here? If you are referring to the > > > > configuration of the filter chain that is not really injection though > > it > > > is > > > > an admittedly simpler mechanism than the current state of the > > > contributors > > > > and our deployment machinery - which could be refactored and > improved. > > > > > > > > "This does not really tie us to using only Shiro authentication > > mechanism > > > > or > > > > authorization mechanisms. > > > > > > > > We have the choice of writing all our authentication or authorization > > in > > > > our own servlet filters or leverage filters from Shiro library, > Realms > > > from > > > > Shiro library or write your own Shiro filters or Shiro realms." > > > > > > > > The only change that this proposal makes to our flexibility is the > > > > inability to take Shiro out of the picture. > > > > We could have leveraged Shiro to do all these things with the current > > > > design as well. > > > > BUT, we have control over our own interception channel in our chains > > > today. > > > > This proposal takes a specific *security provider's* proprietary > filter > > > > chain and uses it as the only interceptor channel for the entire > > server. > > > > Just because they also had a need for one - for their security work - > > > > doesn't mean that we shouldn't have our own. The tail should not wag > > the > > > > dog. Shiro is not an application or server framework it is a security > > > > provider. > > > > > > > > "In most of the cases, when we want to integrate with a new Hadoop > back > > > end > > > > service, all that we have to do is specify the path to a file having > > > > rewrite rules for the service. Rest of the things in > > knox.ini(=shiro.ini) > > > > would remain same." > > > > > > > > This isn't really very clear to me. Do we still have services in a > > > > topology.xml file? > > > > I don't see where you specify the url for the backend services - so I > > > > assume there is still a topology file. > > > > How do we know where to dispatch the requests to? > > > > > > > > I do agree that this is a very good step forward in defining a > > preferred > > > > security provider model for Knox and that we should continue down > this > > > > road. We just need to do so carefully. If, over time, we have found > no > > > > reason to use the existing solution that we can consider removing it > > > > entirely. > > > > > > > > Thanks for all this work and insight, Dilli! > > > > > > > > --larry > > > > > > > > > > > > On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam > > > > <[email protected]>wrote: > > > > > > > > > Have been trying out a few ideas with Knox and managed to get Knox > > > > running > > > > > with > > > > > > > > > > 1. WEB-INF/gateway.xml is completely removed > > > > > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet > > > > > 2. all filters are defined and injected using WEB-INF/shiro.ini > > > > > > > > > > To me that looks much simpler to comprehend and administer. > > > > > Agreed, this could be subjective. > > > > > Hence, seeking comments from community. > > > > > > > > > > Pasting web.xml and shiro.ini inline. > > > > > > > > > > web.xml > > > > > -------------- > > > > > > > > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > > > > > <web-app xmlns="http://java.sun.com/xml/ns/javaee"; xmlns:xsi=" > > > > > http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" > > > > > http://java.sun.com/xml/ns/javaee > > > > > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd";> > > > > > > > > > > <listener> > > > > > > > > > > > > > > > > > > > > > > > > > <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class> > > > > > </listener> > > > > > > > > > > <listener> > > > > > > > > > > > > > > > > > > > > > > > > > <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class> > > > > > </listener> > > > > > > > > > > <listener> > > > > > > > > > > > > > > > > > > > > > > > > > <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class> > > > > > </listener> > > > > > > > > > > <context-param> > > > > > <param-name>rewriteDescriptorLocation</param-name> > > > > > <param-value>/WEB-INF/rewrite.xml</param-value> > > > > > </context-param> > > > > > > > > > > <filter> > > > > > <filter-name>ShiroFilter</filter-name> > > > > > > > > > <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class> > > > > > </filter> > > > > > > > > > > <filter-mapping> > > > > > <filter-name>ShiroFilter</filter-name> > > > > > <url-pattern>/*</url-pattern> > > > > > </filter-mapping> > > > > > > > > > > <session-config> > > > > > <session-timeout>30</session-timeout> > > > > > </session-config> > > > > > > > > > > <servlet> > > > > > <servlet-name>errorservlet</servlet-name> > > > > > > > > > > > > > > > > > > > > > > > > > <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class> > > > > > </servlet> > > > > > > > > > > <servlet-mapping> > > > > > <servlet-name>errorservlet</servlet-name> > > > > > <url-pattern>/*</url-pattern> > > > > > </servlet-mapping> > > > > > > > > > > </web-app> > > > > > > > > > > > > > > > shiro.ini > > > > > ------------- > > > > > > > > > > [main] > > > > > > > > > > # define ldapRealm > > > > > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm > > > > > ldapRealm.contextFactory.authenticationMechanism=simple > > > > > ldapRealm.contextFactory.url=ldap://localhost:33389 > > > > > > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org > > > > > > > > > > > > > > > # define filter: knoxResponseCookieFilter > > > > > knoxResponseCookieFilter = > > > > > org.apache.hadoop.gateway.filter.ResponseCookieFilter > > > > > knoxResponseCookieFilter.enabled = true > > > > > knoxResponseCookieFilter.filterHeaders = rememberMe, > > hadoop.auth.cookie > > > > > > > > > > # define filter: knoxPrincipalMapper > > > > > knoxPrincipalMapper = > > > > org.apache.hadoop.gateway.filter.KnoxPrincipalMapper > > > > > knoxPrincipalMapper.enabled = true > > > > > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob > > > > > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin > > > > > > > > > > # define filter: knoxIPTracker > > > > > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker > > > > > knoxIPTracker.enabled = true > > > > > > > > > > # define filter: knoxAclAuthzFilter > > > > > knoxAclAuthzFilter = > > > org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter > > > > > knoxAclAuthzFilter.enabled = true > > > > > knoxAclAuthzFilter.globalGroupAclMode = OR > > > > > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, > > > /templeton/:OR > > > > > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users > > admin; > > > > > 127.*.*.*, /templeton/:user11 user12; users admin > > > > > > > > > > # define filter: javaSubjectMapper > > > > > javaSubjectMapper = > > > > > org.apache.hadoop.gateway.filter.PostAuthenticationFilter > > > > > > > > > > # define filter: knoxIdentityAsserter > > > > > knoxIdentityAsserter = > > > > > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter > > > > > knoxIdentityAsserter.enabled = true > > > > > > > > > > # define filter: knoxUrlRewriter > > > > > knoxUrlRewriter = > > > > > > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter > > > > > > > > > > knoxUrlRewriter.requestUrlMap = > > > > > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root, > > > > > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file, > > > > > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home, > > > > > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file, > > > > > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode > > > > > > > > > > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration, > > > > > /oozie/v1/.*:OOZIE/oozie/configuration, > > > > > /oozie/v2/.*:OOZIE/oozie/configuration > > > > > > > > > > knoxUrlRewriter.responseHeadersMap = > > > > > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers, > > > > > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers, > > > > > /hbase/:WEBHBASE/webhbase/headers/outbound, > > > > > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound, > > > > > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound, > > > > > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound > > > > > > > > > > # define filter: knoxHttpDispatcher > > > > > knoxHttpDispatcher = > > > > org.apache.hadoop.gateway.dispatch.HttpClientDispatch > > > > > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, > > > oozie:4 > > > > > > > > > > [urls] > > > > > # you could choose to have a different chain of filter for > different > > > url > > > > > patterns > > > > > # so far Knox did not need it > > > > > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper, > > > > > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper, > > > > knoxIdentityAsserter, > > > > > knoxUrlRewriter, knoxHttpDispatcher > > > > > > > > > > # end of shiro.ini > > > > > > > > > > I think we can rename shiro.ini as knox.ini to make it explicit > this > > is > > > > > more about knox configuration than shiro library configuration. > > > > > > > > > > > > > > > We are using shiro config file mechanism as simple, lightweight > > > depenency > > > > > injection. > > > > > > > > > > This does not really tie us to using only Shiro authentication > > > mechanism > > > > or > > > > > authorization mechanisms. > > > > > > > > > > We have the choice of writing all our authentication or > authorization > > > in > > > > > our own servlet filters or leverage filters from Shiro library, > > Realms > > > > from > > > > > Shiro library or write your own Shiro filters or Shiro realms. > > > > > > > > > > In most of the cases, when we want to integrate with a new Hadoop > > back > > > > end > > > > > service, all that we have to do is specify the path to a file > having > > > > > rewrite rules for the service. Rest of the things in > > > knox.ini(=shiro.ini) > > > > > would remain same. > > > > > > > > > > Please review and comment. > > > > > > > > > > > > > > > Thanks > > > > > Dilli > > > > > > > > > > -- > > > > > CONFIDENTIALITY NOTICE > > > > > NOTICE: This message is intended for the use of the individual or > > > entity > > > > to > > > > > which it is addressed and may contain information that is > > confidential, > > > > > privileged and exempt from disclosure under applicable law. If the > > > reader > > > > > of this message is not the intended recipient, you are hereby > > notified > > > > that > > > > > any printing, copying, dissemination, distribution, disclosure or > > > > > forwarding of this communication is strictly prohibited. If you > have > > > > > received this communication in error, please contact the sender > > > > immediately > > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
