[
https://issues.apache.org/jira/browse/YARN-9634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870015#comment-16870015
]
Tao Yang commented on YARN-9634:
--------------------------------
Hi, [~zhuqi].
Just noticed this issue.
The staging directories used for submission should be managed by application
frameworks instead of YARN, requirements are different for frameworks and they
should maintain the staging directories themselves, as far as I know, Flink and
Spark both can cleanup the staging directories after app totally terminate.
For log-aggregation directories, there is a cleanup thread in JobHistoryServer,
the TTL of these directories can be configured by
"yarn.log-aggregation.retain-seconds", -1 by default means cleanup disabled,
for example, you can set 604800 to retain them for only a week to avoid
overflowing the quota.
> Make yarn submit dir and log aggregation dir more evenly distributed
> --------------------------------------------------------------------
>
> Key: YARN-9634
> URL: https://issues.apache.org/jira/browse/YARN-9634
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 3.2.0
> Reporter: zhuqi
> Assignee: zhuqi
> Priority: Major
>
> When the cluster size is large, the dir which user submits the job, and the
> dir which container log aggregate, and other information will fill the HDFS
> directory, because the HDFS directory has a default storage limit. In
> response to this situation, we can change these dirs more distributed, with
> some policy to choose, such as hash policy and round robin policy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
