Jason Lowe updated YARN-3439:
    Attachment: YARN-3439.001.patch

Daryn is out so posting a prototype patch he developed to get some early 
feedback.  Note that this patch can't go in as-is, as it's a work-in-progress 
that hacks out the automatic HDFS delegation token logic that was added as part 
of YARN-2704.

Essentially the idea is to reference count the tokens and only attempt to 
cancel them when the token is no longer referenced.  Since the launcher job 
won't complete until it has successfully submitted the sub-job(s), the token 
will remain referenced throughout the lifespan of the workflow even if the 
launcher job exits early.

> RM fails to renew token when Oozie launcher leaves before sub-job finishes
> --------------------------------------------------------------------------
>                 Key: YARN-3439
>                 URL: https://issues.apache.org/jira/browse/YARN-3439
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.7.0
>            Reporter: Jason Lowe
>            Assignee: Daryn Sharp
>            Priority: Blocker
>         Attachments: YARN-3439.001.patch
> When the Oozie launcher runs a standard MapReduce job (not Pig) it doesn't 
> linger waiting for the sub-job to finish.  At that point the RM stops 
> renewing delegation tokens for the launcher job which wreaks havoc on the 
> sub-job if the sub-job runs long enough for the tokens to expire.

This message was sent by Atlassian JIRA

Reply via email to