[
https://issues.apache.org/jira/browse/MAPREDUCE-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609003#comment-13609003
]
Daryn Sharp commented on MAPREDUCE-3979:
----------------------------------------
[~sseth] I almost have a patch ready that tracks tokens against apps and
cancels when the last app completes (honoring the keepalive of course). I am
contemplating whether this patch negates the need for
{{MRJobConfig.JOB_CANCEL_DELEGATION_TOKEN}}? It appears to be a "hack" for
oozie & pig? If so, I think we can deprecate it, and possibly even noop it.
Oozie should be unaffected because as long as the launcher job or any child job
is running the tokens will remain valid. [~tucu00], can you confirm? Likewise
for pig run under oozie, and pig run directly should already be getting new
tokens during top-level job submissions. The only case where I can think of
where it would potentially cause problems is if something manually gets its own
tokens, and then explicitly stuffs them into multiple job submissions.
> DelegationTokens will be renewed forever if multiple jobs share tokens and
> the first one sets JOB_CANCEL_DELEGATION_TOKEN to false
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3979
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3979
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Daryn Sharp
>
> The first Job/App to register a token is the one which DelegationTokenRenewer
> associates with a a specific Token. An attempt to remove/cancel these shared
> tokens by subsequent jobs doesn't work - since the JobId will not match.
> As a result, Even if subsequent jobs have
> MRJobConfig.JOB_CANCEL_DELEGATION_TOKEN set to true - tokens will not be
> cancelled when those jobs complete.
> Tokens will eventually be removed from the RM / JT when the service that
> issued them considers them to have expired or via an explicit
> cancelDelegationTokens call (not implemented yet in 23).
> A side affect of this is that the same delegation token will end up being
> renewed multiple times (a separate TimerTask for each job which uses the
> token).
> DelegationTokenRenewer could maintain a reference count/list of jobIds for
> shared tokens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
