[
https://issues.apache.org/jira/browse/YARN-9571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Craig Condit resolved YARN-9571.
--------------------------------
Resolution: Won't Fix
Closing as won't fix since it appears we will revert YARN-4946 instead.
> RM state store purging should be configurable if log aggregation does not
> terminate
> -----------------------------------------------------------------------------------
>
> Key: YARN-9571
> URL: https://issues.apache.org/jira/browse/YARN-9571
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 3.2.0
> Reporter: Craig Condit
> Assignee: Craig Condit
> Priority: Major
> Attachments: YARN-9571.001.patch
>
>
> YARN-4946 introduced logic which prevents applications from being removed
> from the RM State Store unless log aggregation has reached a terminal state.
> However, there are cases where log aggregation may not always complete or
> fail cleanly (we've seen instances where it shows as NOT_STARTED on
> production clusters), and in this case the state store will continue to build
> up large numbers of applications.
> We should make this behavior configurable. I propose to add a new
> configuration:
> {code:java}
> yarn.resourcemanager.completed-application-ttl-secs
> {code}
> This can be used to force removal of an application if it would otherwise be
> purged from the state store if not for log aggregation status. If this
> configuration is set to a positive value, it will be used in conjunction with
> the application finish time to determine whether or not to purge the app. The
> app would be removed if the max # of completed applications has been reached
> AND either the app has completed more than ttl-secs seconds ago OR the app
> has completed log aggregation. For backwards compatibility we can set the
> default value of this parameter to zero, which would disable the new logic.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
