Hi, we're running HDP 2.6.4 with Ranger 0.7 at a customer. We have an issue when getting Delegation Tokens from KMS.
Ranger KMS is set up in HA mode according to the docs over here < https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_security/content/ranger_kms_multi_kms.html >. What happens is this: Kind: kms-dt, Service: kms1:9292, Ident: (owner=foobar, renewer=foobar, realUser=, issueDate=1530706632603, maxDate=1531311432603, sequenceNumber=39, masterKeyId=2) 18/07/04 14:17:12 DEBUG KMSClientProvider$KMSTokenRenewer: Renewing delegation token Kind: kms-dt, Service: kms1:9292, Ident: (owner=foobar, renewer=foobar, realUser=, issueDate=1530706632603, maxDate=1531311432603, sequenceNumber=39, masterKeyId=2) 18/07/04 14:17:12 DEBUG KMSUtil: Creating key provider with config key hadoop.security.key.provider.path 18/07/04 14:17:12 DEBUG KMSClientProvider: Renewing delegation token null with url:http://kms2:9292/kms/v1/, as:null 18/07/04 14:17:12 DEBUG UserGroupInformation: UGI: foo...@foobar.com (auth:KERBEROS) 18/07/04 14:17:12 DEBUG UserGroupInformation: +LoginUGI: foo...@foobar.com (auth:KERBEROS) 18/07/04 14:17:12 DEBUG UserGroupInformation: PrivilegedAction as:foo...@foobar.com (auth:KERBEROS) from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.renewDelegationToken(KMSClientProvider.java:916) 18/07/04 14:17:12 DEBUG KerberosAuthenticator: JDK performed authentication on our behalf. 18/07/04 14:17:12 DEBUG UserGroupInformation: PrivilegedActionException as:foo...@foobar.com (auth:KERBEROS) cause:java.io.IOException: HTTP status [403], message [Forbidden] 18/07/04 14:17:12 WARN LoadBalancingKMSClientProvider: KMS provider at [ http://kms2:9292/kms/v1/] threw an IOException!! java.io.IOException: HTTP status [403], message [Forbidden] at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:301) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:217) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:415) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:920) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$2.run(KMSClientProvider.java:917) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.renewDelegationToken(KMSClientProvider.java:916) at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$2.call(LoadBalancingKMSClientProvider.java:143) at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$2.call(LoadBalancingKMSClientProvider.java:140) at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:95) at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.renewDelegationToken(LoadBalancingKMSClientProvider.java:140) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$KMSTokenRenewer.renew(KMSClientProvider.java:184) at org.apache.hadoop.security.token.Token.renew(Token.java:414) at App.main(App.java:32) 18/07/04 14:17:12 DEBUG KMSClientProvider: Renewing delegation token null with url:http://kms1:9292/kms/v1/, as:null 18/07/04 14:17:12 DEBUG UserGroupInformation: UGI: foo...@foobar.com (auth:KERBEROS) 18/07/04 14:17:12 DEBUG UserGroupInformation: +LoginUGI: foo...@foobar.com (auth:KERBEROS) 18/07/04 14:17:12 DEBUG UserGroupInformation: PrivilegedAction as:foo...@foobar.com (auth:KERBEROS) from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.renewDelegationToken(KMSClientProvider.java:916) 18/07/04 14:17:12 DEBUG KerberosAuthenticator: JDK performed authentication on our behalf. New expiration: 1530793032703 Kind: kms-dt, Service: 140.11.73.46:9292, Ident: (owner=foobar, renewer=foobar, realUser=, issueDate=1530706632603, maxDate=1531311432603, sequenceNumber=39, masterKeyId=2) We have written a super simple Java App to replicate this issue that basically does this: for (Token<? extends TokenIdentifier> token : creds.getAllTokens()) { System.out.println("Token Renew"); System.out.println(token); long expiration = token.renew(conf); System.out.println("New expiration: " + expiration); System.out.println(token); } As you can see: All we're trying to do is to renew a Token. Every once in a while (~50%) we see this warning. The problem is when the initial delegation token was awarded by KMS Server 1 but the renew goes to KMS Server 2 we see the 403 Error from Server 2 before the LoadBalancingKMSClientProvider switches to KMS Server 1 where the renew works. That seems weird to me. Should KMS2 not be able to renew a token issued by KMS1? I have a feeling that we might have an issue in our KMS HA setup. Or is this normal behavior? Thank you! Cheers, Lars
