Sent again since I got no reply. That's 1 problem we encountered due to calling User#login more than 1 times. For your reference.
Jieshan. -----Original Message----- From: Bijieshan Sent: Thursday, May 08, 2014 6:54 PM To: dev@hbase.apache.org Cc: Fanghao Subject: WritableRpcEngine$Invoker caches UGI Hi, I think WritableRpcEngine$Invoker should not cache UGI. The intent behind this design? We encounter 1 problem due to client called User#login more than 1 times. The error log: 14/05/08 15:44:40 FATAL ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:184) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$4(SecureClient.java:180) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:302) at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:1) at java.security.AccessController.doPrivileged(Native Method) The reason of this exception as below: (1). Client calls login. (2). Calls Scan. So client side caches the proxy instance for HRegionInterface, and also caches the UGI. (3). Client calls login again(It should not happen, but none protections we have). (4). Wait until the TGT expire. (5). Calls Scan again. So it will use the cached old UGI to create the SecureConnection. (6). Gets a SaslException due to the invalid credential. So triggers SecureClient# handleSaslConnectionFailure (7). Checks whether need to re-login by SecureClient# shouldAuthenticateOverKrb: private synchronized boolean shouldAuthenticateOverKrb() throws IOException { UserGroupInformation loginUser = UserGroupInformation.getLoginUser(); UserGroupInformation currentUser = UserGroupInformation.getCurrentUser(); UserGroupInformation realUser = currentUser.getRealUser(); return relogin = authMethod == AuthMethod.KERBEROS && loginUser != null && // Make sure user logged in using Kerberos either keytab or TGT loginUser.hasKerberosCredentials() && // relogin only in case it is the login user (e.g. JT) // or superuser (like oozie). (loginUser.equals(currentUser) || loginUser.equals(realUser)); } The currentUser is the cached old one, but the login user is the new one created by step (3). So this check returns false and causes the re-login skipped. The solution of this problem includes: (1). Provide suggestion that the client side should only login once. (2). Modify the logic in WritableRpcEngine$Invoker that does not cache UGI. Does this make sense? Regards, Jieshan.