JobTracker lockup due to JobInProgress.initTasks taking significant time for
large jobs on large clusters
---------------------------------------------------------------------------------------------------------
Key: HADOOP-3864
URL: https://issues.apache.org/jira/browse/HADOOP-3864
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.18.0
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.19.0
JobInProgress.initTasks takes significant amount of time on a large cluster for
large jobs (55k maps * 3 splits), during which the JobInProgress object is
locked up.
Simultaneously the JobClient is calling JobTracker.getTaskCompletionEvents
which locks the JobTracker & tries to lock the JobInProgress, there-by it
starves all heartbeats which are trying to lock the JobTracker - resulting in a
lockup.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
