[
https://issues.apache.org/jira/browse/YARN-3439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Advertising
Jason Lowe updated YARN-3439:
-----------------------------
Attachment: YARN-3439.001.patch
Daryn is out so posting a prototype patch he developed to get some early
feedback. Note that this patch can't go in as-is, as it's a work-in-progress
that hacks out the automatic HDFS delegation token logic that was added as part
of YARN-2704.
Essentially the idea is to reference count the tokens and only attempt to
cancel them when the token is no longer referenced. Since the launcher job
won't complete until it has successfully submitted the sub-job(s), the token
will remain referenced throughout the lifespan of the workflow even if the
launcher job exits early.
> RM fails to renew token when Oozie launcher leaves before sub-job finishes
> --------------------------------------------------------------------------
>
> Key: YARN-3439
> URL: https://issues.apache.org/jira/browse/YARN-3439
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.7.0
> Reporter: Jason Lowe
> Assignee: Daryn Sharp
> Priority: Blocker
> Attachments: YARN-3439.001.patch
>
>
> When the Oozie launcher runs a standard MapReduce job (not Pig) it doesn't
> linger waiting for the sub-job to finish. At that point the RM stops
> renewing delegation tokens for the launcher job which wreaks havoc on the
> sub-job if the sub-job runs long enough for the tokens to expire.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
