MapEventsFetcherThread doesn't wait for the heartbeat-interval if it doesn't
have events to fetch
-------------------------------------------------------------------------------------------------
Key: HADOOP-3849
URL: https://issues.apache.org/jira/browse/HADOOP-3849
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.18.0
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Fix For: 0.19.0
The 'notify' done by FetchStatus.getMapEvents causes the MapEventsFetcherThread
to immediately run to the JobTracker (getTaskCompletionEvents).
On a 3500 node cluster, I saw that each TaskTracker calls
JobTracker.getTaskCompletionEvents multiple times per-second. This caused the
JobTracker's RPC queues to back-up resulting in each RPC spending more than
120s in the queue - leading to shuffle proceeding very very slowly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
