[
https://issues.apache.org/jira/browse/HDFS-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-6536:
--------------------------------
Description:
With a small client program below, when running as user "root" which doesn't
have kerberos credential, exception is thrown at the end of the client run. The
config is HA security enabled, with client config setting
{code}
<property>
<name>fs.defaultFS</name>
<value>webhdfs://ns1</value>
</property>
{code}
The client program:
{code}
public class kclient1 {
public static void main(String[] args) throws IOException {
final Configuration conf = new Configuration();
//a non-root user
final UserGroupInformation ugi =
UserGroupInformation.getUGIFromTicketCache("/tmp/krb5cc_496", "[email protected]");
System.out.println("Starting");
ugi.doAs(new PrivilegedAction<Object>() {
@Override
public Object run() {
try {
FileSystem fs = FileSystem.get(conf);
String renewer = "abcdefg";
fs.addDelegationTokens(
renewer, ugi.getCredentials());
// Just to prove that we connected with right
credentials.
fs.getFileStatus(new Path("/"));
return fs.getDelegationToken(renewer);
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
});
System.out.println("THE END");
}
}
{code}
Output:
{code}
[root@yjzc5w-1 tmp2]# hadoop --config /tmp2/conf jar kclient1.jar
kclient1.kclient1
Starting
14/06/14 20:38:51 WARN ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
14/06/14 20:38:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode:
yjzc5w-2.xyz.com/172.26.3.87:20101. Already tried 0 time(s); retry policy is
org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@1a92210,
delay 0ms.
To prove that connection with right credentials
to get file status updated updated 7
THE END
14/06/14 20:38:53 WARN ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
14/06/14 20:38:53 WARN security.UserGroupInformation:
PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
14/06/14 20:38:53 INFO fs.FileSystem: FileSystem.Cache.closeAll() threw an
exception:
java.io.IOException: Authentication failed,
url=http://yjzc5w-2.xyz.com:20101/webhdfs/v1/?op=CANCELDELEGATIONTOKEN&user.name=root&token=HAAEaGRmcwRoZGZzAIoBRp2bNByKAUbBp7gcbBQUD6vWmRYJRv03XZj7Jajf8PU8CB8SV0VCSERGUyBkZWxlZ2F0aW9uC2hhLWhkZnM6bnMx
[root@yjzc5w-1 tmp2]#
{code}
We can see the the exception is thrown in the end of the client run.
I found that the problem is that at the end of client run, the
FileSystem$Cache$ClientFinalizer is run, in which process the tokens stored in
the filesystem cache is get cancelled with the following all:
{code}
final class TokenAspect<T extends FileSystem & Renewable> {
@InterfaceAudience.Private
public static class TokenManager extends TokenRenewer {
@Override
public void cancel(Token<?> token, Configuration conf) throws IOException {
getInstance(token, conf).cancelDelegationToken(token); <==
}
{code}
where getInstance(token, conf) create a FileSystem as user "root", then call
cancelDelegationToken to server side. However, server doesn't have "root"
kerberos credential, so throw this exceptoin.
When I run the same program as user "hdfs" which has the kerberos credential,
then it's fine.
In this case, the client program doesn't own the delegation token, it should
not try to cancel the token. However, the token does not get cancelled because
of the exception I described is thrown, which is good.
The remaining question is, do we need to check token ownership before trying to
cancel the token. This is a minor issue here, and I'm dropping the severity.
Hi [~daryn], I wonder if you could give a quick comment, really appreciate it!
was:
With a small client program below, when running as user "root", exception is
thrown at the end of the client run. The config is HA security enabled, with
client config setting
{code}
<property>
<name>fs.defaultFS</name>
<value>webhdfs://ns1</value>
</property>
{code}
The client program:
{code}
public class kclient1 {
public static void main(String[] args) throws IOException {
final Configuration conf = new Configuration();
//a non-root user
final UserGroupInformation ugi =
UserGroupInformation.getUGIFromTicketCache("/tmp/krb5cc_496", "[email protected]");
System.out.println("Starting");
ugi.doAs(new PrivilegedAction<Object>() {
@Override
public Object run() {
try {
FileSystem fs = FileSystem.get(conf);
String renewer = "abcdefg";
fs.addDelegationTokens(
renewer, ugi.getCredentials());
// Just to prove that we connected with right
credentials.
fs.getFileStatus(new Path("/"));
return fs.getDelegationToken(renewer);
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
});
System.out.println("THE END");
}
}
{code}
Output:
{code}
[root@yjzc5w-1 tmp2]# hadoop --config /tmp2/conf jar kclient1.jar
kclient1.kclient1
Starting
14/06/14 20:38:51 WARN ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
14/06/14 20:38:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode:
yjzc5w-2.xyz.com/172.26.3.87:20101. Already tried 0 time(s); retry policy is
org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@1a92210,
delay 0ms.
To prove that connection with right credentials
to get file status updated updated 7
THE END
14/06/14 20:38:53 WARN ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
14/06/14 20:38:53 WARN security.UserGroupInformation:
PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
14/06/14 20:38:53 INFO fs.FileSystem: FileSystem.Cache.closeAll() threw an
exception:
java.io.IOException: Authentication failed,
url=http://yjzc5w-2.xyz.com:20101/webhdfs/v1/?op=CANCELDELEGATIONTOKEN&user.name=root&token=HAAEaGRmcwRoZGZzAIoBRp2bNByKAUbBp7gcbBQUD6vWmRYJRv03XZj7Jajf8PU8CB8SV0VCSERGUyBkZWxlZ2F0aW9uC2hhLWhkZnM6bnMx
[root@yjzc5w-1 tmp2]#
{code}
We can see the the exception is thrown in the end of the client run.
I found that the problem is that at the end of client run, like the C++
destructor is called at the end of object scope, the tokens stored in the
filesystem cache is get cancelled with the following all:
{code}
final class TokenAspect<T extends FileSystem & Renewable> {
@InterfaceAudience.Private
public static class TokenManager extends TokenRenewer {
@Override
public void cancel(Token<?> token, Configuration conf) throws IOException {
getInstance(token, conf).cancelDelegationToken(token); <==
}
{code}
where getInstance(token, conf) create a FileSystem as user "root", then call
cancelDelegationToken to server side. However, server doesn't have "root"
credential, so throw this exceptoin.
When I run the same program as user "hdfs", then it's fine.
I think if we run the call to cancelDelegationToken as the user who created the
token intially ("hdfs" in this case), then it should work fine. However, the
information of the user who created the token is not available at that point.
Hi [~daryn], I wonder if you could give a quick comment, really appreciate it!
> FileSystem.Cache.closeAll() throws authentication exception at the end of a
> webhdfs client
> ------------------------------------------------------------------------------------------
>
> Key: HDFS-6536
> URL: https://issues.apache.org/jira/browse/HDFS-6536
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, webhdfs
> Affects Versions: 2.4.0
> Reporter: Yongjun Zhang
>
> With a small client program below, when running as user "root" which doesn't
> have kerberos credential, exception is thrown at the end of the client run.
> The config is HA security enabled, with client config setting
> {code}
> <property>
> <name>fs.defaultFS</name>
> <value>webhdfs://ns1</value>
> </property>
> {code}
> The client program:
> {code}
> public class kclient1 {
> public static void main(String[] args) throws IOException {
> final Configuration conf = new Configuration();
> //a non-root user
> final UserGroupInformation ugi =
> UserGroupInformation.getUGIFromTicketCache("/tmp/krb5cc_496",
> "[email protected]");
> System.out.println("Starting");
> ugi.doAs(new PrivilegedAction<Object>() {
> @Override
> public Object run() {
> try {
> FileSystem fs = FileSystem.get(conf);
> String renewer = "abcdefg";
> fs.addDelegationTokens(
> renewer, ugi.getCredentials());
> // Just to prove that we connected with right
> credentials.
> fs.getFileStatus(new Path("/"));
> return fs.getDelegationToken(renewer);
> } catch (Exception e) {
> e.printStackTrace();
> return null;
> }
> }
> });
> System.out.println("THE END");
> }
> }
> {code}
> Output:
> {code}
> [root@yjzc5w-1 tmp2]# hadoop --config /tmp2/conf jar kclient1.jar
> kclient1.kclient1
> Starting
> 14/06/14 20:38:51 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/06/14 20:38:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode:
> yjzc5w-2.xyz.com/172.26.3.87:20101. Already tried 0 time(s); retry policy is
> org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@1a92210,
> delay 0ms.
> To prove that connection with right credentials
> to get file status updated updated 7
> THE END
> 14/06/14 20:38:53 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/06/14 20:38:53 WARN security.UserGroupInformation:
> PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> 14/06/14 20:38:53 INFO fs.FileSystem: FileSystem.Cache.closeAll() threw an
> exception:
> java.io.IOException: Authentication failed,
> url=http://yjzc5w-2.xyz.com:20101/webhdfs/v1/?op=CANCELDELEGATIONTOKEN&user.name=root&token=HAAEaGRmcwRoZGZzAIoBRp2bNByKAUbBp7gcbBQUD6vWmRYJRv03XZj7Jajf8PU8CB8SV0VCSERGUyBkZWxlZ2F0aW9uC2hhLWhkZnM6bnMx
> [root@yjzc5w-1 tmp2]#
> {code}
> We can see the the exception is thrown in the end of the client run.
> I found that the problem is that at the end of client run, the
> FileSystem$Cache$ClientFinalizer is run, in which process the tokens stored
> in the filesystem cache is get cancelled with the following all:
> {code}
> final class TokenAspect<T extends FileSystem & Renewable> {
> @InterfaceAudience.Private
> public static class TokenManager extends TokenRenewer {
> @Override
> public void cancel(Token<?> token, Configuration conf) throws IOException
> {
> getInstance(token, conf).cancelDelegationToken(token); <==
> }
> {code}
> where getInstance(token, conf) create a FileSystem as user "root", then call
> cancelDelegationToken to server side. However, server doesn't have "root"
> kerberos credential, so throw this exceptoin.
> When I run the same program as user "hdfs" which has the kerberos credential,
> then it's fine.
> In this case, the client program doesn't own the delegation token, it should
> not try to cancel the token. However, the token does not get cancelled
> because of the exception I described is thrown, which is good.
> The remaining question is, do we need to check token ownership before trying
> to cancel the token. This is a minor issue here, and I'm dropping the
> severity.
> Hi [~daryn], I wonder if you could give a quick comment, really appreciate it!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
