[
https://issues.apache.org/jira/browse/HADOOP-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546535
]
srikantk edited comment on HADOOP-2119 at 11/28/07 6:28 PM:
-------------------------------------------------------------------
Batching up commits in the commit task. Greatly improves commit rate. This
patch does not completely fix the problem. In order to fix the problem we need
to enhance 1. fix findnewmaptasks to be O(1) 2. locking in jobtracker
Tested the patch, it works for a job with 100,000 mappers. With almost 0 delay
between the last complete map tasks and completing the job.
was (Author: srikantk):
Batching up commits in the commit task. Greatly improves commit rate.
> JobTracker becomes non-responsive if the task trackers finish task too fast
> ---------------------------------------------------------------------------
>
> Key: HADOOP-2119
> URL: https://issues.apache.org/jira/browse/HADOOP-2119
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.16.0
> Reporter: Runping Qi
> Fix For: 0.16.0
>
> Attachments: hadoop-2119.patch, hadoop-jobtracker-thread-dump.txt
>
>
> I ran a job with 0 reducer on a cluster with 390 nodes.
> The mappers ran very fast.
> The jobtracker lacks behind on committing completed mapper tasks.
> The number of running mappers displayed on web UI getting bigger and bigger.
> The jos tracker eventually stopped responding to web UI.
> No progress is reported afterwards.
> Job tracker is running on a separate node.
> The job tracker process consumed 100% cpu, with vm size 1.01g (reach the heap
> space limit).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
