Hi,
I think WritableRpcEngine$Invoker should not cache UGI. The intent behind this
design?
We encounter 1 problem due to client called User#login more than 1 times. The
error log:
14/05/08 15:44:40 FATAL ipc.SecureClient: SASL authentication failed. The most
likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138)
at
org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:184)
at
org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$4(SecureClient.java:180)
at
org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:302)
at
org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:1)
at java.security.AccessController.doPrivileged(Native Method)
The reason of this exception as below:
(1). Client calls login.
(2). Calls Scan. So client side caches the proxy instance for HRegionInterface,
and also caches the UGI.
(3). Client calls login again(It should not happen, but none protections we
have).
(4). Wait until the TGT expire.
(5). Calls Scan again. So it will use the cached old UGI to create the
SecureConnection.
(6). Gets a SaslException due to the invalid credential. So triggers
SecureClient# handleSaslConnectionFailure
(7). Checks whether need to re-login by SecureClient# shouldAuthenticateOverKrb:
private synchronized boolean shouldAuthenticateOverKrb() throws IOException
{
UserGroupInformation loginUser = UserGroupInformation.getLoginUser();
UserGroupInformation currentUser = UserGroupInformation.getCurrentUser();
UserGroupInformation realUser = currentUser.getRealUser();
return relogin = authMethod == AuthMethod.KERBEROS && loginUser != null
&&
// Make sure user logged in using Kerberos either keytab or TGT
loginUser.hasKerberosCredentials() &&
// relogin only in case it is the login user (e.g. JT)
// or superuser (like oozie).
(loginUser.equals(currentUser) || loginUser.equals(realUser));
}
The currentUser is the cached old one, but the login user is the new one
created by step (3). So this check returns false and causes the re-login
skipped.
The solution of this problem includes:
(1). Provide suggestion that the client side should only login once.
(2). Modify the logic in WritableRpcEngine$Invoker that does not cache UGI.
Does this make sense?
Regards,
Jieshan.
