Ahhh - in that case use: https://localhost:8443/gateway/idp/api/v1/websso
The original resource path for the knoxsso service would have made it more like: https://localhost:8443/gateway/idp/knoxsso/api/v1/websso Which seems redundant and arbitrary. We need to evolve such services to be distinct from typical topologies since they aren't really Hadoop cluster specific like typical topologies are. Anyway, try the URL above for your current configuration - that should resolve the 404. On Thu, Nov 19, 2015 at 9:09 AM, Jérôme LELEU <lel...@gmail.com> wrote: > Hi, > > I'm looking forward to read the new documentation you'll write on the REST > API support. > > About the 404, I'm not sure to understand why you want me to remove "idp/" > from the url as the KnoxSSO service is defined in the idp.xml topology: > does its url not have a /idp? > > About the OpenID Connect support, I added it, it's now in pac4j: > > https://github.com/pac4j/pac4j/blob/master/pac4j-config/src/main/java/org/pac4j/config/client/ConfigPropertiesFactory.java#L60 > > Thanks. > Best regards, > Jérôme > > > > 2015-11-18 14:35 GMT+01:00 larry mccay <larry.mc...@gmail.com>: > > > Hi Jérôme - > > > > Great progress! > > You have the flow down perfectly. > > > > You should be also aware that the SSOCookieProvider is a mechanism for > > consuming the SSO cookie for the REST APIs. > > There are also integrations for the various Hadoop UIs - most components > > within the core Hadoop common project have a UI - ie. NameNode, DataNode, > > YARN ResourceManager, etc. Additionally, ecosystem components such as > > Oozie, HBase, Hive, Falcon, Atlas, Ambari, Ranger, etc all have UIs and > are > > adding or already have support for the SSO cookie being created in > KnoxSSO. > > Many of these are able to leverage the support added to Hadoop common and > > get it largely for free. > > > > REST API support is interesting and immediately relevant to Knox but may > > actually be secondary to the WebSSO flow for the UIs. > > When deployments do want to provide REST API support SSOCookieProvider > > enables CORS for allowing javascript API calls from browsers in other > > domains to access Hadoop resources. It is also great for development > > scenarios like this where we need some consumer to test integrations. > > > > I am in the process of writing up more docs on these very aspects of > > KnoxSSO and JWT cookie support across Hadoop. > > > > The 404 is related to the configured provider url on the > SSOCookieProvider > > - remove the "idp/" in the URL. > > > > That used to be the correct URL but we have since simplified it for the > > time being to just be knoxsso. > > There is a thread [1] on the dev list that describes that decision and > what > > we should consider moving forward. > > > > We will have generic OpenId Connect support as well? This is something > that > > would be of immediate value. > > > > [1] > > > > > http://mail-archives.apache.org/mod_mbox/knox-dev/201511.mbox/%3CCACRbFygkWLJ-tFW612ha_Cr82u%2B41PwDqGx_TWbijbP0g34g%3DQ%40mail.gmail.com%3E > > > > Thanks! > > > > --larry > > > > On Wed, Nov 18, 2015 at 6:11 AM, Jérôme LELEU <lel...@gmail.com> wrote: > > > > > Hi, > > > > > > I've read the documentation and things seem fairly clear to me. Let me > > > share my understanding and progress: > > > > > > - the KnoxSSO service is mandatory and the > > gateway-provider-security-pac4j > > > won't work without it: the KnoxSSO service is responsible for creating > a > > > hadoop-jwt cookie from the current authenticated user (J2E subject). > > > - the SSOCookieProvider is also mandatory to protect services and > request > > > the hadoop-jwt cookie to exist by forwarding to the KnoxSSO endpoint. > > > > > > I have created two gateways: one for regular Hadoop services which is > > > protected by the SSOCookieProvider -> > > > > > > > > > https://github.com/apache/knox/pull/2/files#diff-128ca80baa2dfd6c2d25de7b8160b9d9R23 > > > the other one for the KnoxSSO service protected by pac4j -> > > > > > > > > > https://github.com/apache/knox/pull/2/files#diff-4ea9a9a5ee5968f29982478512a63c54R23 > > > > > > Here is my webflow : > > > > > > $ curl -ivk " > > > https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS"; > > > 302 Found > > > Location: > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > > > > > > > Calling a regular Hadoop service through the gateway, it is redirected > to > > > the KnoxSSO endpoint (as no hadoop-jwt cookie exists) > > > > > > $ curl -ivk " > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > " > > > 302 Found > > > Set-Cookie: pac4j.session.pac4jRequestedUrl=" > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > " > > > Location: > > > > > > > > > https://casserverpac4j.herokuapp.com?service=https%3A%2F%2F127.0.0.1%3A8443%2Fgateway%2Fidp%2Fknoxsso%2Fapi%2Fv1%3Fclient_name%3DCasClient > > > > > > > > > Calling the KnosSSO endpoint, the originally requested url is saved > into > > a > > > cookie (the naive implementation I currently use) and it is redirected > > to a > > > CAS server login for authentication (my internal configuration). > > > > > > After login at the CAS server, the user is redirected back to the > > callback > > > url, which is the KnoxSSO endpoint. > > > > > > $ curl -ivk --cookie "pac4j.session.pac4jRequestedUrl= > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhbhdfs/v1/tmp?op=LISTSTATUS > > > " > > > " > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?client_name=CasClient&ticket=ST-2-LQ2QLLEdAhBmrCPE1MZA-cas01.example.org > > > " > > > 302 Found > > > Set-Cookie: pac4j.session.pac4jUserProfile=CasProfile#jleleu > > > Location: > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > > > > > > > The fact it's a callback (to finish the login process) is now based on > > the* > > > client_name* parameter (instead of a /callback url). The authentication > > is > > > finished by validating the CAS service ticket and a user profile is > > created > > > into session and saved in a cookie. It is redirected to the originally > > > requested url. > > > > > > $ curl -ivk --cookie > "pac4j.session.pac4jUserProfile=CasProfile#jleleu" " > > > > > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > " > > > 404 > > > > > > > > > The call to the KnoxSSO endpoint triggers the pac4j filter, which gets > > the > > > user profile from the cookie, calls the identity adapter and sets the > J2E > > > subject with the pac4j user profile. Here I get a 404 I don't > understand > > > and on which I'm currently investigating. > > > > > > So the callback url issue is solved: the KnoxSSO endpoint is the static > > url > > > used for callback, for any identity provider and the *client_name* > > > parameter is the way to know if it is a callback or not. > > > > > > To save the data in session, I used non-secure cookies for now, it's my > > > naive implementation. It currently doesn't work properly because as we > > rely > > > on j2e-pac4j, response.sendRedirect occurs and after that, nothing can > be > > > done on the response. So I need to change pac4j v1.8.1-SNAPSHOT and > > > j2e-pac4j v1.2.1-SNAPSHOT to be able to plugin a specific session > storage > > > mechanism (the need has arisen for Play and Shiro as well) -> > > > https://github.com/pac4j/pac4j/issues/359 > > > > > > As for configuration, we said that the client configuration will be > > defined > > > based on properties. As the same mechanism is available for the CAS > > server, > > > this capability has been moved to pac4j -> > > > https://github.com/pac4j/pac4j/issues/357 > > > It means that currently, Facebook, Twitter, a SAML IdP and a CAS server > > can > > > be created via properties: if you need other identity providers, please > > > speak up so that I can integrate them. > > > > > > So I have some more work to do in pac4j and j2e-pac4j before being able > > to > > > go further with Knox. I'll keep you posted. > > > > > > Thanks. > > > Best regards, > > > Jérôme > > > > > > > > > > > > > > > 2015-11-14 19:03 GMT+01:00 larry mccay <lmc...@apache.org>: > > > > > > > Here is a *very* early version of an integration guide for KnoxSSO: > > > > > > > > http://knox.apache.org/books/knox-0-7-0/knoxsso_integration.html > > > > > > > > It will walk you through setting up an environment to for using > KnoxSSO > > > to > > > > secure the REST APIs using KnoxSSO cookies and a webflow. > > > > The initial environment uses simple browser HTTP basic authentication > > to > > > > protect the WebSSO service which hands out the cookies. > > > > This allows us to ensure that KnoxSSO is setup properly without > > > introducing > > > > a 3rd party provider into the mix first. > > > > > > > > Once you have it working with Basic Auth then you can switch to your > > > pac4j > > > > provider instead of Shiro to protect the websso service and we can > > start > > > to > > > > work through the integration of your provider. > > > > > > > > Please understand that this is still a document that is very much a > > work > > > in > > > > progress and has content, formatting and other issues. > > > > It should serve as a general guide for you in the short term though. > > > > > > > > Hope it is useful for you! > > > > > > > > > > > > > > > > On Fri, Nov 13, 2015 at 9:51 AM, Kevin Minder < > > > > kevin.min...@hortonworks.com> > > > > wrote: > > > > > > > > > I’ll add a bit to the logging answer. I created that for several > > > > reasons. > > > > > 1) Localization as Larry mentions. > > > > > 2) It abstract from actual logging provider and you can see for > > example > > > > > gateway-i18n-logging-sl4j as an alternate integration. > > > > > 3) Another important aspect is centralization. It centralizes all > > > > aspects > > > > > of a given message. For example the level for a given message is > > kept > > > > with > > > > > the message to hopefully prevent the message from being duplicated > > with > > > > > potentially different levels. It is also intended to support the > > > notion > > > > of > > > > > Ids and potentially cause/action annotations which are very common > as > > > > > products mature. > > > > > 4) Lastly and possibly the most interesting/important from a > > developer > > > > > perspective is traceability. Once you find the log method for a > > given > > > > > message you can use the IDE to find all of the palaces where a > given > > > > > message is used. You can also have the IDE tell you if the message > > > isn’t > > > > > used anymore. > > > > > > > > > > All that being said the external components we integrate with > > certainly > > > > > don’t use it. So from that perspective, direct use of log4j if > that > > is > > > > > your choice isn’t going to cause any major problems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 11/13/15, 8:40 AM, "larry mccay" <lmc...@apache.org> wrote: > > > > > > > > > > >Hi Jérôme - > > > > > > > > > > > >Great questions and I'm so glad that you are here to ask them. > > > > > > > > > > > >I think that the set of documentation that I am working on for a > > > KnoxSSO > > > > > >integration guide will probably answer most of your questions. > > > > > >I need to spend a bit more time thinking about the some of it so > > that > > > I > > > > > can > > > > > >incorporate that information as well. > > > > > > > > > > > >The cookie approach is what we use in KnoxSSO. Cookies obviously > > have > > > > some > > > > > >disadvantages but is probably the best alternative for KnoxSSO. > > > > > >Something that we need to consider is that KnoxSSO is already > > keeping > > > > > track > > > > > >of the original URL for redirecting back to the requested > resource. > > > > > >We have to reconcile whether that is separate from the callback > > needs > > > or > > > > > >actually the same thing. > > > > > >Having a little trouble wrapping my head around it right now - > must > > > have > > > > > >more coffee. :) > > > > > > > > > > > >The Messages layer allows for the localization of the actual > > messages > > > by > > > > > >decoupling the messages from the actual runtime code. > > > > > > > > > > > >I will try and get some docs published as quickly as possible for > > > > KnoxSSO > > > > > >integration. > > > > > > > > > > > >thanks, > > > > > > > > > > > >--larry > > > > > > > > > > > >On Fri, Nov 13, 2015 at 3:27 AM, Jérôme LELEU <lel...@gmail.com> > > > wrote: > > > > > > > > > > > >> Hi, > > > > > >> > > > > > >> 1) Webflow: I think I get the idea with the KnoxSSO service: how > > > can I > > > > > test > > > > > >> everything to ensure pac4j works correctly with it and will be > > > usable > > > > in > > > > > >> Hadoop UIs? > > > > > >> > > > > > >> > > > > > >> 2) Callback url: > > > > > >> > > > > > >> For indirect clients, pac4j is designed to be called on any url, > > to > > > > save > > > > > >> it, to call the identity provider providing a static callback > url, > > > to > > > > be > > > > > >> called back on this static callback url and to restore the > > > originally > > > > > >> requested url. > > > > > >> > > > > > >> From point 1), I conclude that pac4j will always be used with > the > > > > > KnoxSSO > > > > > >> service (for indirect clients): will KnoxSSO call the pac4j > > provider > > > > > always > > > > > >> on the same url? Can you describe your first flow with urls? > > > > > >> > > > > > >> Because in that case, I could find a solution where I check for > > the > > > > > >> client_name parameter to define if it's a callback or a first > > call. > > > > > >> > > > > > >> Otherwise, I will need a specific pac4j service. > > > > > >> > > > > > >> > > > > > >> 3) Configuration: no problem here, I will use the AliasService. > > > > > >> > > > > > >> > > > > > >> 4) Web session: I really need to put some information in session > > and > > > > > check > > > > > >> them on callback: I think I can take the data put in session, > save > > > > them > > > > > in > > > > > >> a cookie and restore them before the callback. > > > > > >> > > > > > >> Is it the recommended solution? Can I re-use the components > > > available > > > > > for > > > > > >> JWT tokens and cookies? And how? > > > > > >> > > > > > >> > > > > > >> 5) Logs: I see in every descriptor the use of Messages and > > > > > MessagesFactory: > > > > > >> I can't use log4j directly, can I? What's the expected benefits > > > using > > > > > this > > > > > >> Messages layer? > > > > > >> > > > > > >> > > > > > >> Thanks. > > > > > >> Best regards, > > > > > >> Jérôme > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> 2015-11-12 18:14 GMT+01:00 larry mccay <lmc...@apache.org>: > > > > > >> > > > > > >> > Terrific! > > > > > >> > > > > > > >> > Okay... original questions: > > > > > >> > > > > > > >> > 1) Webflow: is it normal to get a redirection to Facebook or > > > another > > > > > IdP > > > > > >> > when I made the request (for such a configuration of course)? > Is > > > > this > > > > > >> > request meant to happen in a browser? (because it's the use > case > > > it > > > > > was > > > > > >> > made for) > > > > > >> > > > > > > >> > It is normal. > > > > > >> > The usecases in which this mechanism would be used would > involve > > > the > > > > > >> > KnoxSSO service/feature. > > > > > >> > What we are doing with that is providing a normalization of > > > > > >> authentication > > > > > >> > mechanisms by federating each auth event through a JWT token > > that > > > > gets > > > > > >> set > > > > > >> > as a cookie. > > > > > >> > These usecases allow us to provide WebSSO flows for Hadoop UIs > > and > > > > > other > > > > > >> > applications and those applications to only ever have to know > > > about > > > > > >> KnoxSSO > > > > > >> > tokens. > > > > > >> > > > > > > >> > What is slightly less normal is that the flow needs to have > > > KnoxSSO > > > > in > > > > > >> the > > > > > >> > middle. > > > > > >> > For instance: > > > > > >> > 1. a browser requests a resource from a KnoxSSO participating > > > > service > > > > > >> > provider > > > > > >> > 2. the service provider redirects the browser to KnoxSSO since > > > there > > > > > is > > > > > >> no > > > > > >> > cookie > > > > > >> > 3. KnoxSSO is protected with pac4j provider and redirects the > > > > browser > > > > > to > > > > > >> FB > > > > > >> > 4. user authenticates to FB > > > > > >> > 5. pac4j accepts the FB authentication and normalizes the > > security > > > > > >> context > > > > > >> > to a J2E Subject > > > > > >> > 6. WebSSO service then creates a signed JWT token, sets the > > cookie > > > > and > > > > > >> > redirects to the originally requested resource > > > > > >> > 7. originally requested resource finds the hadoop-jwt cookie, > > > > > validates > > > > > >> the > > > > > >> > token and verifies the signature and grants access to the > > resource > > > > > >> > > > > > > >> > 2) Callback url: > > https://localhost:8443/gateway/sandbox/callback > > > > > >> currently > > > > > >> > returns a 404, so I must declare this url somewhere to be > taken > > > into > > > > > >> > account and pass to the pac4j filter. It would be even better > if > > > it > > > > > could > > > > > >> > be done by the pac4j deployment contributor. > > > > > >> > > > > > > >> > How and where to do that? > > > > > >> > > > > > > >> > This is an interesting question and one that may have multiple > > > > > solutions: > > > > > >> > > > > > > >> > * we may be able to add the callback filter as part of the > > > original > > > > > >> > provider and just point them to the same URL. This would > require > > > the > > > > > >> > ability to chain them where one knows to not handle a request > > that > > > > is > > > > > >> > really meant for the other. > > > > > >> > * If we truly need a separate URL the way you depict it there > > then > > > > we > > > > > >> will > > > > > >> > need to add a new service which would be unfortunate because > it > > > > > couples > > > > > >> the > > > > > >> > use of a certain provider to the use of a particular service. > > > > > >> > > > > > > >> > 3) Configuration > > > > > >> > > > > > > >> > Currently, I hardcoded a FacebookClient for Facebook > > > authentication, > > > > > but > > > > > >> we > > > > > >> > should be able to pass the appropriate client like Facebook or > > > SAML. > > > > > >> > Basically, we could do that using filter properties: > > facebook.key > > > + > > > > > >> > facebook.secret means we use Facebook authentication with the > > > > > appropriate > > > > > >> > key and secret for example. Any recommendation? > > > > > >> > > > > > > >> > This is fine. You will add your properties to the provider > > > > properties > > > > > in > > > > > >> > the topology for the pac4j provider. > > > > > >> > You can find example code in the article to blindly add all > > > > > properties to > > > > > >> > the filter init params. > > > > > >> > > > > > > >> > Incidentally, the storage of credentials directly in config > > files > > > > > should > > > > > >> be > > > > > >> > avoided. > > > > > >> > See the use of the AliasService where such credentials can be > > > > > persisted > > > > > >> in > > > > > >> > a credential store and resolved at runtime. > > > > > >> > I will find examples of this for you - we can go forward with > > > config > > > > > >> based > > > > > >> > values but will need to fix that later. > > > > > >> > > > > > > >> > 4) Web session: I seem to be able to use the web session to > sore > > > > data. > > > > > >> > Can you confirm that point? > > > > > >> > > > > > > >> > This is probably not going to work. > > > > > >> > Keep in mind that there may be a cluster of Knox instances > > running > > > > and > > > > > >> the > > > > > >> > session is not replicated across all instances. > > > > > >> > Additionally, keep in mind that your provider will only be > > engaged > > > > on > > > > > the > > > > > >> > first request to KnoxSSO and then not until the token/cookie > > > > expires. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > >
