[
https://issues.apache.org/jira/browse/YARN-9948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970038#comment-16970038
]
Hu Ziqian commented on YARN-9948:
---------------------------------
Hi [~hex108], in our cluster, one attempt will use more than 1k memory in
RMAppImpl. And it will also increase the memory size and response size when
user call the restful api. In our cluster, most of the apps are streaming apps
and some of them may try more than 10 thousand attempts. It's really a big
memory cost.
in comment 2, it says that
_app will always retry if there are some attempts that does not count towards
max attempt retry in the attempts we kept._
I think it won't happen because we only delete attempt which finish time <
(endTime - attemptFailuresValidityInterval). Although we delete some old
attempts, it will not change the result of (numberOfFailure <
app.maxAppAttempts) and won't change the result of app state. If app try more
than app.maxAppAttempts during attemptFailuresValidityInterval, it will failed
as expected.
I also add a config to keep them in memory by default.
> Remove attempts that are beyond max-attempt limit from RMAppImpl
> ----------------------------------------------------------------
>
> Key: YARN-9948
> URL: https://issues.apache.org/jira/browse/YARN-9948
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
> Affects Versions: 3.1.3
> Reporter: Hu Ziqian
> Priority: Major
> Attachments: YARN-9948.001.patch
>
>
> RM will store app attempt in both state store and RMAppImpl. YARN-3480
> removes attempts that are beyond max-attempt limit from state store. In this
> issue we delete those attempts in RMAppImpl the reduce decrease memory usage
> of RM.
> We introduce flag yarn.resourcemanager.am.delete-old-attempts.enabled to
> enable this logic, default value is false.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
