[
https://issues.apache.org/jira/browse/MAPREDUCE-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scott Chen updated MAPREDUCE-1568:
----------------------------------
Attachment: MAPREDUCE-1568-v2.txt
I have made the change based on Amareshwari's suggestion.
1. Create a class called BaseDir and move the check and cleanup cache into this
class.
2. Create a thread called cleanup thread. It periodically check if any of the
base directories requires needs cleanup and cleans it.
3. The cleanup thread will clean only the base directory which exceeds the
limit. In the old case, if one directory is full, it will cleanup all the
caches no mater which base directory it is in.
> TrackerDistributedCacheManager should do deleteLocalPath asynchronously
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-1568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1568
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-1568-v2.txt, MAPREDUCE-1568.txt
>
>
> TrackerDistributedCacheManager.deleteCache() has been improved:
> MAPREDUCE-1302 makes TrackerDistributedCacheManager rename the caches in the
> main thread and then delete them in the background
> MAPREDUCE-1098 avoids global locking while do the renaming (renaming lots of
> directories can also takes a long time)
> But the deleteLocalCache is still in the main thread of TaskRunner.run(). So
> it will still slow down the task which triggers the deletion (originally this
> will blocks all tasks, but it is fixed by MAPREDUCE-1098). Other tasks do not
> wait for the deletion. The task which triggers the deletion should not wait
> for this either. TrackerDistributedCacheManager should do deleteLocalPath()
> asynchronously.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
