A Quick update – I fixed the below issue by adding server dns host name inside the knoxsso.redirect.whitelist.regex parameter under knoxsso.xml. Now the application is getting landed to the original url but it’s getting looped back infinite times even after authentication. Posted the log message below for better understanding.
Could anyone help me in resolving this issue?. **************** << gateway.log >> *************** 2018-07-18 05:54:14,391 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /yarn/ 2018-07-18 05:54:14,391 DEBUG federation.jwt (SSOCookieFederationFilter.java:doFilter(114)) - Sending redirect to: https://<dnsname>:8446/gateway/knoxsso/api/v1/websso?originalUrl=https://<dnsname>:8446/gateway/gate1/yarn/ 2018-07-18 05:54:14,530 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /api/v1/websso 2018-07-18 05:54:16,954 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: POST /api/v1/websso 2018-07-18 05:54:17,109 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /api/v1/websso 2018-07-18 05:54:17,114 DEBUG filter.Pac4jIdentityAdapter (Pac4jIdentityAdapter.java:doFilter(85)) - User authenticated as: #SAML2Profile# | id: [email protected] | attributes: {Mail=[[email protected]], notOnOrAfter=2018-07-18T05:59:16.748Z, sessionindex=D_sb_9ZK7ml9Gj72RKQsx7JqW.d, notBefore=2018-07-18T05:49:16.748Z} | roles: [] | permissions: [] | isRemembered: false | clientName: SAML2Client | linkedId: null | 2018-07-18 05:54:17,895 INFO service.knoxsso (WebSSOResource.java:getCookieValue(330)) - Unable to find cookie with name: original-url 2018-07-18 05:54:17,952 DEBUG service.knoxsso (WebSSOResource.java:addJWTHadoopCookie(288)) - Adding the following JWT token as a cookie: eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJQcmF2ZWVuX1JhdmlrdW1hckBpbnR1aXQuY29tIiwiaXNzIjoiS05PWFNTTyIsImV4cCI6MTUzMTg5MzM1N30.NC_AbhW-hbS6L1t5vd6Xu55djybP6DNzA6XfPbIQp_M9B29JSXsrfShh4UFnWSAGYj2NegRTOoanhKogEkmFUIHszcU_9-LxlDP5E-ZTr0NVSOLJ3ksfGDlnTy7UXekUMn77a7Agx8WAWhdCiRYKGBtS2gmkRb5-OogA2-sg-h4 2018-07-18 05:54:17,953 INFO service.knoxsso (WebSSOResource.java:addJWTHadoopCookie(304)) - JWT cookie successfully added. 2018-07-18 05:54:17,953 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(214)) - About to redirect to original URL: https://emr-knox-webui-dev-1021294088.us-west-2.elb.amazonaws.com:8446/gateway/gate1/yarn/ 2018-07-18 05:54:18,014 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /yarn/ 2018-07-18 05:54:18,015 DEBUG federation.jwt (SSOCookieFederationFilter.java:doFilter(114)) - Sending redirect to: https://<dnsname>:8446/gateway/knoxsso/api/v1/websso?originalUrl=https://<dnsname>:8446/gateway/gate1/yarn/ 2018-07-18 05:54:18,075 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /api/v1/websso 2018-07-18 05:54:19,678 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: POST /api/v1/websso 2018-07-18 05:54:19,835 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /api/v1/websso 2018-07-18 05:54:19,836 DEBUG filter.Pac4jIdentityAdapter (Pac4jIdentityAdapter.java:doFilter(85)) - User authenticated as: #SAML2Profile# | id: [email protected] | attributes: {Mail=[[email protected]], notOnOrAfter=2018-07-18T05:59:19.514Z, sessionindex=tmVchIT5Ov3cPOhlOgD2_nN_qwl, notBefore=2018-07-18T05:49:19.514Z} | roles: [] | permissions: [] | isRemembered: false | clientName: SAML2Client | linkedId: null | 2018-07-18 05:54:19,840 INFO service.knoxsso (WebSSOResource.java:getCookieValue(330)) - Unable to find cookie with name: original-url 2018-07-18 05:54:19,844 DEBUG service.knoxsso (WebSSOResource.java:addJWTHadoopCookie(288)) - Adding the following JWT token as a cookie: eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJQcmF2ZWVuX1JhdmlrdW1hckBpbnR1aXQuY29tIiwiaXNzIjoiS05PWFNTTyIsImV4cCI6MTUzMTg5MzM1OX0.byWZF1XQB3y29mfi5FfLUwMl1_F5EPm2FylDbvCkG9QFiI30_7AbYV_cwYjBUUKXiRvFNvpmO8yT1T-tW-8btaMrNcV1gupzwJoZz3RSdVWxm1MUHhKC3UkwSQ9zFzAH73KfTVAczn8w4JTQi9_ZT4HNtFETX5ayTxdD3fC9ZTc 2018-07-18 05:54:19,845 INFO service.knoxsso (WebSSOResource.java:addJWTHadoopCookie(304)) - JWT cookie successfully added. 2018-07-18 05:54:19,845 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(214)) - About to redirect to original URL: https://emr-knox-webui-dev-1021294088.us-west-2.elb.amazonaws.com:8446/gateway/gate1/yarn/ 2018-07-18 05:54:18,014 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /yarn/ ……. ……. Thanks, Praveen. From: "Ravikumar, Praveen Krishnamoorthy" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, July 17, 2018 at 10:42 PM To: "[email protected]" <[email protected]> Cc: "[email protected]" <[email protected]>, "Mohanan, Mahesh" <[email protected]> Subject: Re: Need help in enabling SAML auth in Apache Knox Just a quick update. I have added the below property in knoxsso.xml – suggested in mail archives <param> <name>pac4j.session.store</name> <value>J2ESessionStore</value> </param> After this config, it’s getting redirected to the expected URL after authentication. But I’m facing other issue related to regex whitelist. Attached the log below. **************** << gateway.log >> *************** 2018-07-18 04:29:39,917 DEBUG filter.Pac4jIdentityAdapter (Pac4jIdentityAdapter.java:doFilter(85)) - User authenticated as: #SAML2Profile# | id: [email protected] | attributes: {Mail=[[email protected]], notOnOrAfter=2018-07-18T04:34:39.599Z, sessionindex=mmocFXXZSWoAxo4el4O1KdbThm-, notBefore=2018-07-18T04:24:39.599Z} | roles: [] | permissions: [] | isRemembered: false | clientName: SAML2Client | linkedId: null | 2018-07-18 04:29:40,729 INFO service.knoxsso (WebSSOResource.java:getCookieValue(330)) - Unable to find cookie with name: original-url 2018-07-18 04:29:40,730 ERROR service.knoxsso (WebSSOResource.java:getAuthenticationToken(188)) - The original URL: https://<dnsname>:8446/gateway/gate1/yarn/ for redirecting back after authentication is not valid according to the configured whitelist: ^https?:\/\/(localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$. See documentation for KnoxSSO Whitelisting. Thanks, Praveen.R From: "Ravikumar, Praveen Krishnamoorthy" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, July 17, 2018 at 7:47 PM To: "[email protected]" <[email protected]> Cc: "[email protected]" <[email protected]>, "Mohanan, Mahesh" <[email protected]> Subject: Re: Need help in enabling SAML auth in Apache Knox Larry, Thanks a lot for your swift response. As you listed below, first 4 steps happening in order when I try to access the YARNUI through knox but I’m seeing the below error in logs after the SAML assertion is posted. 2018-07-18 01:41:08,978 ERROR engine.DefaultCallbackLogic (DefaultCallbackLogic.java:renewSession(123)) - Unable to renew the session. The session store may not support this feature Will this be a reason for this issue. Could you please provide your thoughts? I have posted the log messages below for better understanding. Please let me know, If you need any more inputs from my end. **************** << gateway.log >> *************** 2018-07-18 01:41:06,046 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /yarn/ 2018-07-18 01:41:06,047 DEBUG federation.jwt (SSOCookieFederationFilter.java:doFilter(114)) - Sending redirect to: https://<dnsName>:8446/gateway/knoxsso/api/v1/websso?originalUrl=https://<dnsName>:8446/gateway/gate1/yarn/ 2018-07-18 01:41:06,111 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /api/v1/websso 2018-07-18 01:41:06,111 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(101)) - Get from session: pac4jUserProfiles = null 2018-07-18 01:41:06,111 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(130)) - Save in session: pac4jRequestedUrl = https://<dnsName>:8446/gateway/knoxsso/api/v1/websso?originalUrl=https://<dnsName>:8446/gateway/gate1/yarn/ 2018-07-18 01:41:06,345 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(101)) - Get from session: SAML2Client$attemptedAuthentication = null 2018-07-18 01:41:06,346 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(101)) - Get from session: samlRelayState = null 2018-07-18 01:41:06,346 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(130)) - Save in session: samlRelayState = 2018-07-18 01:41:08,679 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: POST /api/v1/websso 2018-07-18 01:41:08,748 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(130)) - Save in session: SAML2Client$attemptedAuthentication = 2018-07-18 01:41:08,749 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:set(130)) - Save in session: pac4jUserProfiles = {SAML2Client=#SAML2Profile# | id: [email protected] | attributes: {Mail=[[email protected]], notOnOrAfter=2018-07-18T01:46:08.427Z, sessionindex=X3R7kH9zunD1jn9CP5eN9sm2N0M, notBefore=2018-07-18T01:36:08.427Z} | roles: [] | permissions: [] | isRemembered: false | clientName: SAML2Client | linkedId: null |} 2018-07-18 01:41:08,978 ERROR engine.DefaultCallbackLogic (DefaultCallbackLogic.java:renewSession(123)) - Unable to renew the session. The session store may not support this feature 2018-07-18 01:41:08,979 DEBUG session.KnoxSessionStore (KnoxSessionStore.java:get(101)) - Get from session: pac4jRequestedUrl = null ******************** << gateway-audit.log >> ******************** 18/07/18 01:41:06 ||be6bf57b-7b96-4292-93ce-00ed574ecd6e|audit|10.89.78.49|YARNUI||||access|uri|/gateway/gate1/yarn/|unavailable|Request method: GET 18/07/18 01:41:06 |||audit|10.89.78.49|YARNUI||||access|uri|/gateway/gate1/yarn/|success|Response status: 302 18/07/18 01:41:06 ||668bd6c6-664f-499c-97e5-6c2294b98f04|audit|10.89.78.49|KNOXSSO||||access|uri|/gateway/knoxsso/api/v1/websso?originalUrl=https://<dnsname>:8446/gateway/gate1/yarn/|unavailable|Request method: GET 18/07/18 01:41:06 |||audit|10.89.78.49|KNOXSSO||||access|uri|/gateway/knoxsso/api/v1/websso?originalUrl=https://<dnsname>:8446/gateway/gate1/yarn/|success|Response status: 200 18/07/18 01:41:08 ||fdbcaedb-75a6-46a5-8a7d-811f194142d1|audit|10.89.78.49|KNOXSSO||||access|uri|/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client|unavailable|Request method: POST 18/07/18 01:41:08 |||audit|10.89.78.49|KNOXSSO||||access|uri|/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client|success|Response status: 302 Thanks, Praveen.R From: larry mccay <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, July 17, 2018 at 5:56 PM To: "[email protected]" <[email protected]> Cc: "[email protected]" <[email protected]>, "Mohanan, Mahesh" <[email protected]> Subject: Re: Need help in enabling SAML auth in Apache Knox Whitelist - this has nothing to do with determining where to redirect - it may not allow you to redirect somewhere if it doesn't match the expression but it is not used to determine where to redirect to. Not sure why the URL would have to be rewritten when proxying. * try to access YARNUI through Knox * SSOCookie finds no cookie so redirects to KnoxSSO and provides originalUrl query param which points to the YARNUI through Knox * KnoxSSO engages SAML IDP and user logs in * SAML assertion is POSTED to KnoxSSO endpoint with proper callback URL * KnoxSSO creates hadoop-jwt cookie and redirects to originalUrl query param I can't tell if you are saying that the POS to KnoxSSO is going to the wrong place or the originalUrl redirect is. On Tue, Jul 17, 2018 at 3:31 PM, Ravikumar, Praveen Krishnamoorthy <[email protected]<mailto:[email protected]>> wrote: Thanks a lot for your inputs Sandeep. I genuinely appreciate for your thoughts. As you mentioned I followed the below docs and enabled SAML. As of now IDP related issues are fixed, · Knox is now able to contact the IDP · POST the SAML request to IDP. · IDP responds with the window allowing user to provide the credentials. · Users are getting authenticated. · I could see the SAML response in the SAML tracer. · But after authentication the page is getting redirected to http://<dnsname>:8446<http://%3cdnsname%3e:8446> and pops message as 404 not found. I suspect, this is an issue on the knox side. I have 2 thoughts, either knoxsso.redirect.whitelist.regex param highlighted below in knoxsso.xml topology is redirecting to invalid URL or do we need to add any rewrite rules in the application to direct to a valid URL. Could anyone please help me in this. KnoxSSO.xml ------------ <topology> <gateway> <provider> <role>federation</role> <name>pac4j</name> <enabled>true</enabled> <param> <name>pac4j.callbackUrl</name> <value>https://<dnsName>:8446/gateway/knoxsso/api/v1/websso</value><https://%3cdnsName%3e:8446/gateway/knoxsso/api/v1/websso%3c/value%3e> </param> <param> <name>clientName</name> <value>SAML2Client</value> </param> <param> <name>saml.identityProviderMetadataPath</name> <value>/tmp/preprod_metadata_SP.xml</value> </param> <param> <name>saml.serviceProviderMetadataPath</name> <value>/tmp/preprod_metadata_SP.xml</value> </param> <param> <name>saml.serviceProviderEntityId</name> <value>https://<dnsName>:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client</value><https://%3cdnsName%3e:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client%3c/value%3e> </param> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> </gateway> <service> <role>KNOXSSO</role> <param> <name>knoxsso.cookie.secure.only</name> <value>true</value> </param> <param> <name>knoxsso.token.ttl</name> <value>100000</value> </param> <param> <name>knoxsso.redirect.whitelist.regex</name> <value>^https?:\/\/(<dnsName>|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value> </param> </service> </topology> Thanks, Praveen. From: Sandeep Moré <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Monday, July 16, 2018 at 6:25 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Need help in enabling SAML auth in Apache Knox Hello Praveen, Following are some of the blogs that talk about Knox SAML integration https://cwiki.apache.org/confluence/display/KNOX/KnoxSSO+and+Okta https://cwiki.apache.org/confluence/display/KNOX/Configuring+Apache+Knox+SSO+with+Ipsilon+using+SAML2 Knox user guide also has some information on how to configure SAML https://knox.apache.org/books/knox-1-0-0/user-guide.html#Pac4j+Provider+-+CAS+/+OAuth+/+SAML+/+OpenID+Connect For more details you can take a look at Pac4J provider https://github.com/pac4j/pac4j/wiki/Clients#saml-support Now coming back to your options, I could not find info on Ping IDP for SAML so I cannot comment on the options, this is just my educated guess. 1. Entity ID - I am guessing this is your "saml.serviceProviderEntityId" value, for me it is https://www.myhost.com:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client 2. Baseurl - no idea what this is, you probably have to refer to ping IDP docs 3. ACSURL (where target assertion is Sent) - this is the same as above for https://www.myhost.com:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client On Mon, Jul 16, 2018 at 5:47 PM Ravikumar, Praveen Krishnamoorthy <[email protected]<mailto:[email protected]>> wrote: Hi, Also, Could anyone please help me what should I configure for 1. Entity ID 2. BaseURL (that is used for the entire site) 3. ACSURL (where target assertion is Sent) On the Identity provider side, to enable SSO in Knox. Thanks, Praveen. From: "Ravikumar, Praveen Krishnamoorthy" <[email protected]<mailto:[email protected]>> Date: Monday, July 16, 2018 at 10:36 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Need help in enabling SAML auth in Apache Knox Hi, I'm Praveen. I'm working on POC to setup Apache Knox on the master node of an EMR cluster for our client. With the help of documentations I was able to install KNOX successfully and was able to run few tests. Currently I'm facing an issue on enabling SAML authentication, which I'm kind of blocked and I don’t know, how to proceed or troubleshoot the issue. I have provided few details regarding the issue and I would love to provide more if needed. Could anyone help me in this, would be very helpful for me to proceed further. TASK: ----- To enable SAML authentication for Apache Knox. NOTE: Apache Knox is installed and running in port 8446 STEP 1: SSO request initiation. ******************************* - Our client uses PING Federate Identity provider. - raised a request to register the application for SSO access. Entity ID - https://<dnsName>:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client<https://%3cdnsName%3e:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client> Target URL - https://<dnsName>:8446(I'm<https://%3cdnsName%3e:8446(I'm> not sure the target URL is valid, I suspect the page is getting redirected to this link after auth) - I received a IDP metadata.xml and certificate. STEP 2: Topology config *********************** KnoxSSO.xml ------------ <topology> <gateway> <provider> <role>federation</role> <name>pac4j</name> <enabled>true</enabled> <param> <name>pac4j.callbackUrl</name> <value>https://<dnsName>:8446/gateway/knoxsso/api/v1/websso</value><https://%3cdnsName%3e:8446/gateway/knoxsso/api/v1/websso%3c/value%3e> </param> <param> <name>clientName</name> <value>SAML2Client</value> </param> <param> <name>saml.identityProviderMetadataPath</name> <value>/tmp/preprod_metadata_SP.xml</value> </param> <param> <name>saml.serviceProviderMetadataPath</name> <value>/tmp/preprod_metadata_SP.xml</value> </param> <param> <name>saml.serviceProviderEntityId</name> <value>https://<dnsName>:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client</value><https://%3cdnsName%3e:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client%3c/value%3e> </param> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> </gateway> <service> <role>KNOXSSO</role> <param> <name>knoxsso.cookie.secure.only</name> <value>true</value> </param> <param> <name>knoxsso.token.ttl</name> <value>100000</value> </param> <param> <name>knoxsso.redirect.whitelist.regex</name> <value>^https?:\/\/(emr-knox-webui-dev\.us-west-2\.elb\.amazonaws\.com|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value> </param> </service> </topology> gate1.xml --------- <?xml version="1.0" encoding="utf-8"?> <topology> <gateway> <provider> <role>federation</role> <name>SSOCookieProvider</name> <enabled>true</enabled> <param> <name>sso.authentication.provider.url</name> <value>https://<dns-name>:8446/gateway/knoxsso/api/v1/websso</value><https://%3cdns-name%3e:8446/gateway/knoxsso/api/v1/websso%3c/value%3e> </param> </provider> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> </gateway> <service> <role>YARNUI</role> <url>http://<dnsname>:8088</url><http://%3cdnsname%3e:8088%3c/url%3e> </service> </topology> PROBLEM: ******** on accessing the YarnUI (firefox browser) after starting the gateway, The browser gets redirected to the Identity provider URL -> asks for the login credentials -> on submitting the user is getting authenticated but the application gets landed to https://<DNSDomain>:8446<https://%3cDNSDomain%3e:8446> and throws page not found error. I'm seeing the SAML request sent and SAML response getting received but it gets landed to an invalid page after authentication. I'm unable to figure out the page to land after authentication. Hope I have provided the required details. please do let me know if you need any additional details. Thanks, Praveen.
