[
https://issues.apache.org/jira/browse/YARN-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785398#comment-16785398
]
Prabhu Joseph commented on YARN-9080:
-------------------------------------
Hi [~Rakesh_Shah], Have tested on my test cluster which had bucket directories
older than a month.
Before Patch there are clustertimestamp, bucket directories which are older
than retention
yarn.timeline-service.entity-group-fs-store.retain-seconds (604800 i.e 7 days)
are present.
{code}
[yarn@tparimi-tarunhdp26-3 ~]$ hadoop fs -ls -R /ats/done
drwxr-xr-x - yarn hadoop 0 2018-10-24 15:03 /ats/done/1540393151330
drwxr-xr-x - yarn hadoop 0 2018-10-24 15:03
/ats/done/1540393151330/0000
drwxr-xr-x - yarn hadoop 0 2018-10-31 15:15
/ats/done/1540393151330/0000/000
drwxr-xr-x - yarn hadoop 0 2018-10-26 09:41 /ats/done/1540478614379
drwxr-xr-x - yarn hadoop 0 2018-10-26 09:41
/ats/done/1540478614379/0000
drwxr-xr-x - yarn hadoop 0 2018-11-02 11:02
/ats/done/1540478614379/0000/000
drwxr-xr-x - yarn hadoop 0 2019-03-05 04:44
/ats/done/1551701066597
drwxr-xr-x - yarn hadoop 0 2019-03-05 04:44
/ats/done/1551701066597/0000
drwxr-xr-x - yarn hadoop 0 2019-03-05 04:46
/ats/done/1551701066597/0000/000
drwxrwx--- - ambari-qa hadoop 0 2019-03-05 04:44
/ats/done/1551701066597/0000/000/application_1551701066597_0001
{code}
After Patch with cleaner interval set to minute.
yarn.timeline-service.entity-group-fs-store.cleaner-interval-seconds = 60
After restart of ATS and a minute, the older clustertimestamp, bucket
directories are removed as expected.
{code}
[yarn@tparimi-tarunhdp26-3 yarn]$ hadoop fs -ls -R /ats/done
drwxr-xr-x - yarn hadoop 0 2019-03-05 04:44
/ats/done/1551701066597
drwxr-xr-x - yarn hadoop 0 2019-03-05 04:44
/ats/done/1551701066597/0000
drwxr-xr-x - yarn hadoop 0 2019-03-05 04:46
/ats/done/1551701066597/0000/000
drwxrwx--- - ambari-qa hadoop 0 2019-03-05 04:44
/ats/done/1551701066597/0000/000/application_1551701066597_0001
{code}
[~rohithsharma] Can you review the v4 patch for this jira.
> Bucket Directories as part of ATS done accumulates
> --------------------------------------------------
>
> Key: YARN-9080
> URL: https://issues.apache.org/jira/browse/YARN-9080
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
> Attachments: 0001-YARN-9080.patch, 0002-YARN-9080.patch,
> 0003-YARN-9080.patch, YARN-9080-004.patch
>
>
> Have observed older bucket directories cluster_timestamp, bucket1 and bucket2
> as part of ATS done accumulates. The cleanLogs part of EntityLogCleaner
> removes only the app directories and not the bucket directories.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
