[
https://issues.apache.org/jira/browse/HADOOP-4654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652319#action_12652319
]
Amar Kamat commented on HADOOP-4654:
------------------------------------
Patch looks good. Its good to use {{TaskCommitThread}} for doing the
{{FileSystem}} operations. The only concern is that {{PENDING}} tasks that are
waiting for commit will now contend with KILLED/FAILED tasks and we might run
into the risk of slowing down the commit rate. The case will be visible when
one job (job1) finishes while others (job2) are in middle where job1's
speculative tasks might now fill up the commit queue. Hence I feel this should
be benchmarked or tested to see if the interference level is less and
{{PENDING}} tasks are not kept waiting for long slowing down the commit rate.
The other question is whether or not we want to do it for 0.18?
> remove temporary output directory of failed tasks
> -------------------------------------------------
>
> Key: HADOOP-4654
> URL: https://issues.apache.org/jira/browse/HADOOP-4654
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.2, 0.18.1
> Reporter: Christian Kunz
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.20.0
>
> Attachments: patch-4654-0.18.txt
>
>
> When dfs is getting full (80+% of reserved space), the rate of write failures
> increases, such that more map-reduce tasks can fail. By not cleaning up the
> temporary output directory of tasks the situation worsens over the lifetime
> of a job, increasing the probability of the whole job failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
