[
https://issues.apache.org/jira/browse/HADOOP-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606575#action_12606575
]
Doug Cutting commented on HADOOP-3598:
--------------------------------------
Every call to createWorkDir is in a sequence like:
{code}
Path taskOutputDir = createWorkDir(job);
if (taskOutputDir == null) {
throw new IOException("Could not create task output directory");
Path file = new Path(taskOutputDir, name);
{code}
At a minimum, the null test and exception can be added to createWorkDir, no?
And perhaps the method can also create the path, e.g., the equivalent of:
{code}
protected Path createOutputPath(job, name) {
return new Path(createWorkDir(job), name);
}
{code}
> Map-Reduce framework needlessly creates temporary _${taskid} directories for
> Maps
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3598
> URL: https://issues.apache.org/jira/browse/HADOOP-3598
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: HADOOP-3598_0_20080619.patch,
> HADOOP-3598_0_20080619.patch
>
>
> The staging directory for task-outputs (i.e.
> ${mapred.out.dir}/_temporary/_${taskid}) should only be created when Maps
> produce output on HDFS, which usually isn't the case. This plays very badly
> with HDFS quotas and may lead to thousands of temp names in the FS namespace,
> there-by overhauling the quotas. IAC, it isn't good to needlessly create
> these directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
