[
https://issues.apache.org/jira/browse/YARN-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771593#comment-16771593
]
Prabhu Joseph commented on YARN-9080:
-------------------------------------
[~Rakesh_Shah] ATS done directory is created under buckets
RM_start_timestamp/bucket1/bucket2/app_dir. For every restart of RM say at t1,
t2, t3 (current) - below structure will be present
{code}
t1/m number of bucket1/n number of bucket2/x number of app dirs
t2/m number of bucket1/n number of bucket2/x number of app dirs
t3/m number of bucket1/n number of bucket2/x number of app dirs
{code}
As part of retention, {{EntityLogCleaner}} clears only the app directories. The
bucket directories are accumulated and can cause Hdfs Directory Limit Exceeded
exception. The fix clears the bucket directories which are older than retention
leaving current timestamp t3.
> Bucket Directories as part of ATS done accumulates
> --------------------------------------------------
>
> Key: YARN-9080
> URL: https://issues.apache.org/jira/browse/YARN-9080
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
> Attachments: 0001-YARN-9080.patch, 0002-YARN-9080.patch,
> 0003-YARN-9080.patch
>
>
> Have observed older bucket directories cluster_timestamp, bucket1 and bucket2
> as part of ATS done accumulates. The cleanLogs part of EntityLogCleaner
> removes only the app directories and not the bucket directories.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
