[
https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468968#comment-13468968
]
Karthik Kambatla commented on HADOOP-8852:
------------------------------------------
After spending some time on this, I noticed a couple of other related issues:
- WebHdfsFileSystem and HftpFileSystem have their own implemenations of
TokenRenewer that are not used at all.
- In DelegationTokenRenewer#renew() (see snippet below), we fetch new tokens in
case the renew fails. However, we use only the first token and ignore the rest.
Later, if we can't renew this token, we re-fetch new tokens instead of using
the previously fetched ones. Also, we might not be able to fetch new ones, in
which case we give up. Is this valid behavior? Or, should we be first using the
initially fetched tokens, and re-fetch only when we run out of them.
{code}
Token<?>[] tokens = fs.addDelegationTokens(null, null);
if (tokens.length == 0) {
throw new IOException("addDelegationTokens returned no tokens");
}
fs.setDelegationToken(tokens[0]);
{code}
Taking the above into consideration along with the goal of further decoupling
DelegationTokenRenewer from actual FS-specific details of
renewal/refetching/cancellation, I propose the following. Please advise
appropriately.
# TokenRenewer should be an interface (it is currently a fully abstract class).
# Create a new abstract class TokenManager that holds a list (queue) of tokens,
their kind, renewal period, a TokenRenewer implementation, and an abstract
method #manage() to manage (renew/re-fetch/cancel) the tokens as appropriate.
# Each FileSystem (WebHdfs and Hftp) has its own implemenation (local extended
class) of the TokenManager.
# DelegationTokenRenewer allows registering and de-registering TokenManagers.
FileSystems call register() at init(), and de-register() at close().
# DelegationTokenRenewer stores the registered TokenManagers in its DelayQueue.
On a TokenManager's turn, the manage() method is invoked. If the manage fails,
the TokenManager is automatically de-registered.
> DelegationTokenRenewer should be Singleton
> ------------------------------------------
>
> Key: HADOOP-8852
> URL: https://issues.apache.org/jira/browse/HADOOP-8852
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Karthik Kambatla
> Attachments: hadoop-8852.patch, hadoop-8852.patch,
> hadoop-8852-v1.patch
>
>
> Updated description:
> DelegationTokenRenewer should be Singleton - the instance and renewer threads
> should be created/started lazily. The filesystems using the renewer shouldn't
> need to explicity start/stop the renewer, and only register/de-register for
> token renewal.
> Original issue:
> HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer
> thread when they are closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
