[
https://issues.apache.org/jira/browse/HADOOP-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606458#action_12606458
]
Devaraj Das commented on HADOOP-3598:
-------------------------------------
This makes sense. The thing that we need to take care of is the map/reduce side
files. But yes, with the implicit creation of task directories things will
continue to work. If and when that implicit creation is addressed, we need to
introduce an API that the users *must* call to create side files and that api
could in turn create the directory tree.
> Map-Reduce framework needlessly creates temporary _${taskid} directories for
> Maps
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-3598
> URL: https://issues.apache.org/jira/browse/HADOOP-3598
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: HADOOP-3598_0_20080619.patch
>
>
> The staging directory for task-outputs (i.e.
> ${mapred.out.dir}/_temporary/_${taskid}) should only be created when Maps
> produce output on HDFS, which usually isn't the case. This plays very badly
> with HDFS quotas and may lead to thousands of temp names in the FS namespace,
> there-by overhauling the quotas. IAC, it isn't good to needlessly create
> these directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
