[
https://issues.apache.org/jira/browse/YARN-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248470#comment-14248470
]
Jason Lowe commented on YARN-2964:
----------------------------------
bq. AFAIR, this code never had the concept of a first job. An app submits
tokens, there was a flat list of tokens, everytime an app finishes, RM will
check if the CancelTokensWhenComplete flag is set, and ignore the cancelation
of this app if the flag is set.
As I understand it, the orignial code implicitly had the concept of a first job
because the tokens were stored in a Set instead of a Map. Once the token was
stashed in the set, subsequent attempts from sub-jobs to store the token would
silently be ignored because the token was already in the set. Since the
DelegationTokenToRenew only hashes and checks the underlying token, the
difference between shouldCancelAtEnd is ignored and therefore lost when the
first job's token is already in the set. In the new code, the
DelegationTokenToRenew objects are kept in a map instead of a set, so we no
longer are implicitly ignoring the same tokens from sub-jobs as we did in the
past. This is what allows a sub-job to "override" the request of the launcher
job to avoid canceling the token.
bq. Are you seeing it on a cluster or is it a theory?
This is occurring on our 2.6 clusters. Our 2.5-based clusters do not exhibit
the problem.
> RM prematurely cancels tokens for jobs that submit jobs (oozie)
> ---------------------------------------------------------------
>
> Key: YARN-2964
> URL: https://issues.apache.org/jira/browse/YARN-2964
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.6.0
> Reporter: Daryn Sharp
> Priority: Blocker
>
> The RM used to globally track the unique set of tokens for all apps. It
> remembered the first job that was submitted with the token. The first job
> controlled the cancellation of the token. This prevented completion of
> sub-jobs from canceling tokens used by the main job.
> As of YARN-2704, the RM now tracks tokens on a per-app basis. There is no
> notion of the first/main job. This results in sub-jobs canceling tokens and
> failing the main job and other sub-jobs. It also appears to schedule
> multiple redundant renewals.
> The issue is not immediately obvious because the RM will cancel tokens ~10
> min (NM livelyness interval) after log aggregation completes. The result is
> an oozie job, ex. pig, that will launch many sub-jobs over time will fail if
> any sub-jobs are launched >10 min after any sub-job completes. If all other
> sub-jobs complete within that 10 min window, then the issue goes unnoticed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
