[
https://issues.apache.org/jira/browse/MAPREDUCE-896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Gummadi updated MAPREDUCE-896:
-----------------------------------
Attachment: y896.v2.patch
> Was there a need to change CleanupQueue.addToQueue to take a FileSystem as
> argument instead of Configuration ? It has caused more changes than required
> by this patch - like in JobTracker and JobInProgress. Can we retain the
> original API and pass in a Configuration as before ?
With the new code you suggested, those changes are anyway needed in JobTracker
and JobInProgress and also it will be consistent with trunk's code. So sending
FileSystem instead of Configuration.
> When adding a task directory to delete, we are adding paths from all the
> local directories instead of just the one where files for the task are
> actually created. At a minimum, this is more work being done than necessary.
> More importantly, I don't know if there are any side effects this will cause.
> We can check which among the local directories the path belongs to (by doing
> a startsWith on the path) and all only that I think.
As some of the files under workDir can be on different local directories,
deletion of all the paths is to be done.
Incorporated all other review comments. Also changed the permissions being set
recursively on the path to be deleted is changed to 777 as that is the default
permission setting done by TaskTracker when taskDir and workDir are
created(just to be consistent with that).
A testcase of this patch depends on HADOOP-5771. So this patch needs to be
applied on top of HADOOP-5771.
Attaching new patch for Y! 20.
Please review and provide your comments.
> Users can set non-writable permissions on temporary files for TT and can
> abuse disk usage.
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-896
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-896
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.21.0
> Reporter: Vinod K V
> Assignee: Ravi Gummadi
> Fix For: 0.21.0
>
> Attachments: MR-896.patch, MR-896.v1.patch, y896.v1.patch,
> y896.v2.patch
>
>
> As of now, irrespective of the TaskController in use, TT itself does a full
> delete on local files created by itself or job tasks. This step, depending
> upon TT's umask and the permissions set by files by the user, for e.g in
> job-work/task-work or child.tmp directories, may or may not go through
> successful completion fully. Thus is left an opportunity for abusing disk
> space usage either accidentally or intentionally by TT/users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
