Hi, As the j2e-pac4j library now supports the required features for Knox, I restarted working on pac4j - Knox integration.
Though, I have a very strange error. The deployment of the WARs fails with the following error: *java.lang.IllegalArgumentException: Not supported: indent-number* I tried to comment out the line: *transformerFactory.setAttribute( "indent-number", 2 );* in the XmlGatewayDescriptorExporter class, but it leads me to another error (org.apache.hadoop.gateway.deploy.DeploymentException: Failed to finalize contribution) Any idea of what's wrong? Thanks. Best regards, Jérôme 2015-11-19 15:21 GMT+01:00 larry mccay <larry.mc...@gmail.com>: > Ahhh - in that case use: > > https://localhost:8443/gateway/idp/api/v1/websso > > The original resource path for the knoxsso service would have made it more > like: > > https://localhost:8443/gateway/idp/knoxsso/api/v1/websso > > Which seems redundant and arbitrary. > We need to evolve such services to be distinct from typical topologies > since they aren't really Hadoop cluster specific like typical topologies > are. > > Anyway, try the URL above for your current configuration - that should > resolve the 404. > > > On Thu, Nov 19, 2015 at 9:09 AM, Jérôme LELEU <lel...@gmail.com> wrote: > > > Hi, > > > > I'm looking forward to read the new documentation you'll write on the > REST > > API support. > > > > About the 404, I'm not sure to understand why you want me to remove > "idp/" > > from the url as the KnoxSSO service is defined in the idp.xml topology: > > does its url not have a /idp? > > > > About the OpenID Connect support, I added it, it's now in pac4j: > > > > > https://github.com/pac4j/pac4j/blob/master/pac4j-config/src/main/java/org/pac4j/config/client/ConfigPropertiesFactory.java#L60 > > > > Thanks. > > Best regards, > > Jérôme > > > > > > > > 2015-11-18 14:35 GMT+01:00 larry mccay <larry.mc...@gmail.com>: > > > > > Hi Jérôme - > > > > > > Great progress! > > > You have the flow down perfectly. > > > > > > You should be also aware that the SSOCookieProvider is a mechanism for > > > consuming the SSO cookie for the REST APIs. > > > There are also integrations for the various Hadoop UIs - most > components > > > within the core Hadoop common project have a UI - ie. NameNode, > DataNode, > > > YARN ResourceManager, etc. Additionally, ecosystem components such as > > > Oozie, HBase, Hive, Falcon, Atlas, Ambari, Ranger, etc all have UIs and > > are > > > adding or already have support for the SSO cookie being created in > > KnoxSSO. > > > Many of these are able to leverage the support added to Hadoop common > and > > > get it largely for free. > > > > > > REST API support is interesting and immediately relevant to Knox but > may > > > actually be secondary to the WebSSO flow for the UIs. > > > When deployments do want to provide REST API support SSOCookieProvider > > > enables CORS for allowing javascript API calls from browsers in other > > > domains to access Hadoop resources. It is also great for development > > > scenarios like this where we need some consumer to test integrations. > > > > > > I am in the process of writing up more docs on these very aspects of > > > KnoxSSO and JWT cookie support across Hadoop. > > > > > > The 404 is related to the configured provider url on the > > SSOCookieProvider > > > - remove the "idp/" in the URL. > > > > > > That used to be the correct URL but we have since simplified it for the > > > time being to just be knoxsso. > > > There is a thread [1] on the dev list that describes that decision and > > what > > > we should consider moving forward. > > > > > > We will have generic OpenId Connect support as well? This is something > > that > > > would be of immediate value. > > > > > > [1] > > > > > > > > > http://mail-archives.apache.org/mod_mbox/knox-dev/201511.mbox/%3CCACRbFygkWLJ-tFW612ha_Cr82u%2B41PwDqGx_TWbijbP0g34g%3DQ%40mail.gmail.com%3E > > > > > > Thanks! > > > > > > --larry > > > > > > On Wed, Nov 18, 2015 at 6:11 AM, Jérôme LELEU <lel...@gmail.com> > wrote: > > > > > > > Hi, > > > > > > > > I've read the documentation and things seem fairly clear to me. Let > me > > > > share my understanding and progress: > > > > > > > > - the KnoxSSO service is mandatory and the > > > gateway-provider-security-pac4j > > > > won't work without it: the KnoxSSO service is responsible for > creating > > a > > > > hadoop-jwt cookie from the current authenticated user (J2E subject). > > > > - the SSOCookieProvider is also mandatory to protect services and > > request > > > > the hadoop-jwt cookie to exist by forwarding to the KnoxSSO endpoint. > > > > > > > > I have created two gateways: one for regular Hadoop services which is > > > > protected by the SSOCookieProvider -> > > > > > > > > > > > > > > https://github.com/apache/knox/pull/2/files#diff-128ca80baa2dfd6c2d25de7b8160b9d9R23 > > > > the other one for the KnoxSSO service protected by pac4j -> > > > > > > > > > > > > > > https://github.com/apache/knox/pull/2/files#diff-4ea9a9a5ee5968f29982478512a63c54R23 > > > > > > > > Here is my webflow : > > > > > > > > $ curl -ivk " > > > > https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS"; > > > > 302 Found > > > > Location: > > > > > > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > > > > > > > > > > Calling a regular Hadoop service through the gateway, it is > redirected > > to > > > > the KnoxSSO endpoint (as no hadoop-jwt cookie exists) > > > > > > > > $ curl -ivk " > > > > > > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > > " > > > > 302 Found > > > > Set-Cookie: pac4j.session.pac4jRequestedUrl=" > > > > > > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > > " > > > > Location: > > > > > > > > > > > > > > https://casserverpac4j.herokuapp.com?service=https%3A%2F%2F127.0.0.1%3A8443%2Fgateway%2Fidp%2Fknoxsso%2Fapi%2Fv1%3Fclient_name%3DCasClient > > > > > > > > > > > > Calling the KnosSSO endpoint, the originally requested url is saved > > into > > > a > > > > cookie (the naive implementation I currently use) and it is > redirected > > > to a > > > > CAS server login for authentication (my internal configuration). > > > > > > > > After login at the CAS server, the user is redirected back to the > > > callback > > > > url, which is the KnoxSSO endpoint. > > > > > > > > $ curl -ivk --cookie "pac4j.session.pac4jRequestedUrl= > > > > > > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhbhdfs/v1/tmp?op=LISTSTATUS > > > > " > > > > " > > > > > > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?client_name=CasClient&ticket=ST-2-LQ2QLLEdAhBmrCPE1MZA-cas01.example.org > > > > " > > > > 302 Found > > > > Set-Cookie: pac4j.session.pac4jUserProfile=CasProfile#jleleu > > > > Location: > > > > > > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > > > > > > > > > > The fact it's a callback (to finish the login process) is now based > on > > > the* > > > > client_name* parameter (instead of a /callback url). The > authentication > > > is > > > > finished by validating the CAS service ticket and a user profile is > > > created > > > > into session and saved in a cookie. It is redirected to the > originally > > > > requested url. > > > > > > > > $ curl -ivk --cookie > > "pac4j.session.pac4jUserProfile=CasProfile#jleleu" " > > > > > > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > > " > > > > 404 > > > > > > > > > > > > The call to the KnoxSSO endpoint triggers the pac4j filter, which > gets > > > the > > > > user profile from the cookie, calls the identity adapter and sets the > > J2E > > > > subject with the pac4j user profile. Here I get a 404 I don't > > understand > > > > and on which I'm currently investigating. > > > > > > > > So the callback url issue is solved: the KnoxSSO endpoint is the > static > > > url > > > > used for callback, for any identity provider and the *client_name* > > > > parameter is the way to know if it is a callback or not. > > > > > > > > To save the data in session, I used non-secure cookies for now, it's > my > > > > naive implementation. It currently doesn't work properly because as > we > > > rely > > > > on j2e-pac4j, response.sendRedirect occurs and after that, nothing > can > > be > > > > done on the response. So I need to change pac4j v1.8.1-SNAPSHOT and > > > > j2e-pac4j v1.2.1-SNAPSHOT to be able to plugin a specific session > > storage > > > > mechanism (the need has arisen for Play and Shiro as well) -> > > > > https://github.com/pac4j/pac4j/issues/359 > > > > > > > > As for configuration, we said that the client configuration will be > > > defined > > > > based on properties. As the same mechanism is available for the CAS > > > server, > > > > this capability has been moved to pac4j -> > > > > https://github.com/pac4j/pac4j/issues/357 > > > > It means that currently, Facebook, Twitter, a SAML IdP and a CAS > server > > > can > > > > be created via properties: if you need other identity providers, > please > > > > speak up so that I can integrate them. > > > > > > > > So I have some more work to do in pac4j and j2e-pac4j before being > able > > > to > > > > go further with Knox. I'll keep you posted. > > > > > > > > Thanks. > > > > Best regards, > > > > Jérôme > > > > > > > > > > > > > > > > > > > > 2015-11-14 19:03 GMT+01:00 larry mccay <lmc...@apache.org>: > > > > > > > > > Here is a *very* early version of an integration guide for KnoxSSO: > > > > > > > > > > http://knox.apache.org/books/knox-0-7-0/knoxsso_integration.html > > > > > > > > > > It will walk you through setting up an environment to for using > > KnoxSSO > > > > to > > > > > secure the REST APIs using KnoxSSO cookies and a webflow. > > > > > The initial environment uses simple browser HTTP basic > authentication > > > to > > > > > protect the WebSSO service which hands out the cookies. > > > > > This allows us to ensure that KnoxSSO is setup properly without > > > > introducing > > > > > a 3rd party provider into the mix first. > > > > > > > > > > Once you have it working with Basic Auth then you can switch to > your > > > > pac4j > > > > > provider instead of Shiro to protect the websso service and we can > > > start > > > > to > > > > > work through the integration of your provider. > > > > > > > > > > Please understand that this is still a document that is very much a > > > work > > > > in > > > > > progress and has content, formatting and other issues. > > > > > It should serve as a general guide for you in the short term > though. > > > > > > > > > > Hope it is useful for you! > > > > > > > > > > > > > > > > > > > > On Fri, Nov 13, 2015 at 9:51 AM, Kevin Minder < > > > > > kevin.min...@hortonworks.com> > > > > > wrote: > > > > > > > > > > > I’ll add a bit to the logging answer. I created that for several > > > > > reasons. > > > > > > 1) Localization as Larry mentions. > > > > > > 2) It abstract from actual logging provider and you can see for > > > example > > > > > > gateway-i18n-logging-sl4j as an alternate integration. > > > > > > 3) Another important aspect is centralization. It centralizes > all > > > > > aspects > > > > > > of a given message. For example the level for a given message is > > > kept > > > > > with > > > > > > the message to hopefully prevent the message from being > duplicated > > > with > > > > > > potentially different levels. It is also intended to support the > > > > notion > > > > > of > > > > > > Ids and potentially cause/action annotations which are very > common > > as > > > > > > products mature. > > > > > > 4) Lastly and possibly the most interesting/important from a > > > developer > > > > > > perspective is traceability. Once you find the log method for a > > > given > > > > > > message you can use the IDE to find all of the palaces where a > > given > > > > > > message is used. You can also have the IDE tell you if the > message > > > > isn’t > > > > > > used anymore. > > > > > > > > > > > > All that being said the external components we integrate with > > > certainly > > > > > > don’t use it. So from that perspective, direct use of log4j if > > that > > > is > > > > > > your choice isn’t going to cause any major problems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 11/13/15, 8:40 AM, "larry mccay" <lmc...@apache.org> wrote: > > > > > > > > > > > > >Hi Jérôme - > > > > > > > > > > > > > >Great questions and I'm so glad that you are here to ask them. > > > > > > > > > > > > > >I think that the set of documentation that I am working on for a > > > > KnoxSSO > > > > > > >integration guide will probably answer most of your questions. > > > > > > >I need to spend a bit more time thinking about the some of it so > > > that > > > > I > > > > > > can > > > > > > >incorporate that information as well. > > > > > > > > > > > > > >The cookie approach is what we use in KnoxSSO. Cookies obviously > > > have > > > > > some > > > > > > >disadvantages but is probably the best alternative for KnoxSSO. > > > > > > >Something that we need to consider is that KnoxSSO is already > > > keeping > > > > > > track > > > > > > >of the original URL for redirecting back to the requested > > resource. > > > > > > >We have to reconcile whether that is separate from the callback > > > needs > > > > or > > > > > > >actually the same thing. > > > > > > >Having a little trouble wrapping my head around it right now - > > must > > > > have > > > > > > >more coffee. :) > > > > > > > > > > > > > >The Messages layer allows for the localization of the actual > > > messages > > > > by > > > > > > >decoupling the messages from the actual runtime code. > > > > > > > > > > > > > >I will try and get some docs published as quickly as possible > for > > > > > KnoxSSO > > > > > > >integration. > > > > > > > > > > > > > >thanks, > > > > > > > > > > > > > >--larry > > > > > > > > > > > > > >On Fri, Nov 13, 2015 at 3:27 AM, Jérôme LELEU <lel...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > >> Hi, > > > > > > >> > > > > > > >> 1) Webflow: I think I get the idea with the KnoxSSO service: > how > > > > can I > > > > > > test > > > > > > >> everything to ensure pac4j works correctly with it and will be > > > > usable > > > > > in > > > > > > >> Hadoop UIs? > > > > > > >> > > > > > > >> > > > > > > >> 2) Callback url: > > > > > > >> > > > > > > >> For indirect clients, pac4j is designed to be called on any > url, > > > to > > > > > save > > > > > > >> it, to call the identity provider providing a static callback > > url, > > > > to > > > > > be > > > > > > >> called back on this static callback url and to restore the > > > > originally > > > > > > >> requested url. > > > > > > >> > > > > > > >> From point 1), I conclude that pac4j will always be used with > > the > > > > > > KnoxSSO > > > > > > >> service (for indirect clients): will KnoxSSO call the pac4j > > > provider > > > > > > always > > > > > > >> on the same url? Can you describe your first flow with urls? > > > > > > >> > > > > > > >> Because in that case, I could find a solution where I check > for > > > the > > > > > > >> client_name parameter to define if it's a callback or a first > > > call. > > > > > > >> > > > > > > >> Otherwise, I will need a specific pac4j service. > > > > > > >> > > > > > > >> > > > > > > >> 3) Configuration: no problem here, I will use the > AliasService. > > > > > > >> > > > > > > >> > > > > > > >> 4) Web session: I really need to put some information in > session > > > and > > > > > > check > > > > > > >> them on callback: I think I can take the data put in session, > > save > > > > > them > > > > > > in > > > > > > >> a cookie and restore them before the callback. > > > > > > >> > > > > > > >> Is it the recommended solution? Can I re-use the components > > > > available > > > > > > for > > > > > > >> JWT tokens and cookies? And how? > > > > > > >> > > > > > > >> > > > > > > >> 5) Logs: I see in every descriptor the use of Messages and > > > > > > MessagesFactory: > > > > > > >> I can't use log4j directly, can I? What's the expected > benefits > > > > using > > > > > > this > > > > > > >> Messages layer? > > > > > > >> > > > > > > >> > > > > > > >> Thanks. > > > > > > >> Best regards, > > > > > > >> Jérôme > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> 2015-11-12 18:14 GMT+01:00 larry mccay <lmc...@apache.org>: > > > > > > >> > > > > > > >> > Terrific! > > > > > > >> > > > > > > > >> > Okay... original questions: > > > > > > >> > > > > > > > >> > 1) Webflow: is it normal to get a redirection to Facebook or > > > > another > > > > > > IdP > > > > > > >> > when I made the request (for such a configuration of > course)? > > Is > > > > > this > > > > > > >> > request meant to happen in a browser? (because it's the use > > case > > > > it > > > > > > was > > > > > > >> > made for) > > > > > > >> > > > > > > > >> > It is normal. > > > > > > >> > The usecases in which this mechanism would be used would > > involve > > > > the > > > > > > >> > KnoxSSO service/feature. > > > > > > >> > What we are doing with that is providing a normalization of > > > > > > >> authentication > > > > > > >> > mechanisms by federating each auth event through a JWT token > > > that > > > > > gets > > > > > > >> set > > > > > > >> > as a cookie. > > > > > > >> > These usecases allow us to provide WebSSO flows for Hadoop > UIs > > > and > > > > > > other > > > > > > >> > applications and those applications to only ever have to > know > > > > about > > > > > > >> KnoxSSO > > > > > > >> > tokens. > > > > > > >> > > > > > > > >> > What is slightly less normal is that the flow needs to have > > > > KnoxSSO > > > > > in > > > > > > >> the > > > > > > >> > middle. > > > > > > >> > For instance: > > > > > > >> > 1. a browser requests a resource from a KnoxSSO > participating > > > > > service > > > > > > >> > provider > > > > > > >> > 2. the service provider redirects the browser to KnoxSSO > since > > > > there > > > > > > is > > > > > > >> no > > > > > > >> > cookie > > > > > > >> > 3. KnoxSSO is protected with pac4j provider and redirects > the > > > > > browser > > > > > > to > > > > > > >> FB > > > > > > >> > 4. user authenticates to FB > > > > > > >> > 5. pac4j accepts the FB authentication and normalizes the > > > security > > > > > > >> context > > > > > > >> > to a J2E Subject > > > > > > >> > 6. WebSSO service then creates a signed JWT token, sets the > > > cookie > > > > > and > > > > > > >> > redirects to the originally requested resource > > > > > > >> > 7. originally requested resource finds the hadoop-jwt > cookie, > > > > > > validates > > > > > > >> the > > > > > > >> > token and verifies the signature and grants access to the > > > resource > > > > > > >> > > > > > > > >> > 2) Callback url: > > > https://localhost:8443/gateway/sandbox/callback > > > > > > >> currently > > > > > > >> > returns a 404, so I must declare this url somewhere to be > > taken > > > > into > > > > > > >> > account and pass to the pac4j filter. It would be even > better > > if > > > > it > > > > > > could > > > > > > >> > be done by the pac4j deployment contributor. > > > > > > >> > > > > > > > >> > How and where to do that? > > > > > > >> > > > > > > > >> > This is an interesting question and one that may have > multiple > > > > > > solutions: > > > > > > >> > > > > > > > >> > * we may be able to add the callback filter as part of the > > > > original > > > > > > >> > provider and just point them to the same URL. This would > > require > > > > the > > > > > > >> > ability to chain them where one knows to not handle a > request > > > that > > > > > is > > > > > > >> > really meant for the other. > > > > > > >> > * If we truly need a separate URL the way you depict it > there > > > then > > > > > we > > > > > > >> will > > > > > > >> > need to add a new service which would be unfortunate because > > it > > > > > > couples > > > > > > >> the > > > > > > >> > use of a certain provider to the use of a particular > service. > > > > > > >> > > > > > > > >> > 3) Configuration > > > > > > >> > > > > > > > >> > Currently, I hardcoded a FacebookClient for Facebook > > > > authentication, > > > > > > but > > > > > > >> we > > > > > > >> > should be able to pass the appropriate client like Facebook > or > > > > SAML. > > > > > > >> > Basically, we could do that using filter properties: > > > facebook.key > > > > + > > > > > > >> > facebook.secret means we use Facebook authentication with > the > > > > > > appropriate > > > > > > >> > key and secret for example. Any recommendation? > > > > > > >> > > > > > > > >> > This is fine. You will add your properties to the provider > > > > > properties > > > > > > in > > > > > > >> > the topology for the pac4j provider. > > > > > > >> > You can find example code in the article to blindly add all > > > > > > properties to > > > > > > >> > the filter init params. > > > > > > >> > > > > > > > >> > Incidentally, the storage of credentials directly in config > > > files > > > > > > should > > > > > > >> be > > > > > > >> > avoided. > > > > > > >> > See the use of the AliasService where such credentials can > be > > > > > > persisted > > > > > > >> in > > > > > > >> > a credential store and resolved at runtime. > > > > > > >> > I will find examples of this for you - we can go forward > with > > > > config > > > > > > >> based > > > > > > >> > values but will need to fix that later. > > > > > > >> > > > > > > > >> > 4) Web session: I seem to be able to use the web session to > > sore > > > > > data. > > > > > > >> > Can you confirm that point? > > > > > > >> > > > > > > > >> > This is probably not going to work. > > > > > > >> > Keep in mind that there may be a cluster of Knox instances > > > running > > > > > and > > > > > > >> the > > > > > > >> > session is not replicated across all instances. > > > > > > >> > Additionally, keep in mind that your provider will only be > > > engaged > > > > > on > > > > > > the > > > > > > >> > first request to KnoxSSO and then not until the token/cookie > > > > > expires. > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
