[
https://issues.apache.org/jira/browse/YARN-7720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723477#comment-17723477
]
walhl.liu edited comment on YARN-7720 at 5/17/23 2:11 PM:
----------------------------------------------------------
I have met similar scene. Home app attempt failed , then home app secondary
attempt start. AM container is allocated by rm after 20 min because lack of
resource. then secondary am is timeout after 10min. When home AM started,
secondary home app can not start success becasuse of previous app expired
failed. MR shuffle can not reuse shuffle data from previous secondary home
app. [~botong] can you give me some advice handle the problem?
was (Author: walhl):
I have met similar scene. Home app attempt failed , then home app secondary
attempt start. AM container is allocated by rm after 20 min because lack of
resource. then secondary am is timeout after 10min. When home AM started,
secondary home app can not start success becasuse of previous app expired
failed. MR shuffle can not reuse shuffle data from previous secondary home
app.
> Race condition between second app attempt and UAM timeout when first attempt
> node is down
> -----------------------------------------------------------------------------------------
>
> Key: YARN-7720
> URL: https://issues.apache.org/jira/browse/YARN-7720
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Botong Huang
> Assignee: Shilun Fan
> Priority: Major
> Attachments: YARN-7720.v1.patch, YARN-7720.v2.patch
>
>
> In Federation, multiple attempts of an application share the same UAM in each
> secondary sub-cluster. When first attempt fails, we reply on the fact that
> secondary RM won't kill the existing UAM before the AM heartbeat timeout
> (default at 10 min). When second attempt comes up in the home sub-cluster, it
> will pick up the UAM token from Yarn Registry and resume the UAM heartbeat to
> secondary RMs.
> The default heartbeat timeout for NM and AM are both 10 mins. The problem is
> that when the first attempt node goes down or out of connection, only after
> 10 mins will the home RM mark the first attempt as failed, and then schedule
> the 2nd attempt in some other node. By then the UAMs in secondaries are
> already timing out, and they might not survive until the second attempt comes
> up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
