[
https://issues.apache.org/jira/browse/HBASE-24403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wuchang updated HBASE-24403:
----------------------------
Description:
When doing Bulkload, we find that FsDelegationToken will acquire token of
NameNode everytime for a single file, although, from the comment of
acquireDelegationToken(), it claims that it is trying to find token in cache
firstly, but the newly requested token is never put to cache and thus the cache
is still empty for the following request;
In cases there are many files to do the bulk load, the token request will cause
big short to NameNode.
{code:java}
public void acquireDelegationToken(final FileSystem fs)
throws IOException {
if (userProvider.isHadoopSecurityEnabled()) {
this.fs = fs;
userToken = userProvider.getCurrent().getToken("HDFS_DELEGATION_TOKEN",
fs.getCanonicalServiceName());
if (userToken == null) {
hasForwardedToken = false;
try {
userToken = fs.getDelegationToken(renewer);
} catch (NullPointerException npe) {
// we need to handle NullPointerException in case HADOOP-10009 is missing
LOG.error("Failed to get token for " + renewer);
}
} else {
hasForwardedToken = true;
LOG.info("Use the existing token: " + userToken);
}
}
}{code}
was:
When doing Bulkload, we find that FsDelegationToken will acquire token of
NameNode everytime for a single file, although, from the comment of
acquireDelegationToken(), it claims that it is trying to find token in cache
firstly, but the newly requested token is never put to cache and thus the cache
is always empty;
In cases there are many files to do the bulk load, the token request will cause
big short to NameNode.
> FsDelegationToken should cache Token
> ------------------------------------
>
> Key: HBASE-24403
> URL: https://issues.apache.org/jira/browse/HBASE-24403
> Project: HBase
> Issue Type: Bug
> Reporter: wuchang
> Priority: Major
>
> When doing Bulkload, we find that FsDelegationToken will acquire token of
> NameNode everytime for a single file, although, from the comment of
> acquireDelegationToken(), it claims that it is trying to find token in cache
> firstly, but the newly requested token is never put to cache and thus the
> cache is still empty for the following request;
> In cases there are many files to do the bulk load, the token request will
> cause big short to NameNode.
>
> {code:java}
> public void acquireDelegationToken(final FileSystem fs)
> throws IOException {
> if (userProvider.isHadoopSecurityEnabled()) {
> this.fs = fs;
> userToken = userProvider.getCurrent().getToken("HDFS_DELEGATION_TOKEN",
> fs.getCanonicalServiceName());
> if (userToken == null) {
> hasForwardedToken = false;
> try {
> userToken = fs.getDelegationToken(renewer);
> } catch (NullPointerException npe) {
> // we need to handle NullPointerException in case HADOOP-10009 is missing
> LOG.error("Failed to get token for " + renewer);
> }
> } else {
> hasForwardedToken = true;
> LOG.info("Use the existing token: " + userToken);
> }
> }
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
