[
https://issues.apache.org/jira/browse/MAPREDUCE-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2047:
-------------------------------------
Status: Open (was: Patch Available)
Sorry to come in late, the patch has gone stale. Can you please rebase? Thanks.
Given this is not an issue with MRv2 should we still commit this? I'm happy to,
but not sure it's useful. Thanks.
> reduce overhead of findSpeculativeTask
> --------------------------------------
>
> Key: MAPREDUCE-2047
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2047
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Environment: hadoop-20 with HADOOP-2141
> Reporter: Joydeep Sen Sarma
> Assignee: Joydeep Sen Sarma
> Attachments: 2047.3.patch, mapreduce-2047.1.patch,
> mapreduce-2047.2.patch
>
>
> We are bottlenecked (in the JT) on the jobtracker lock and calls to
> findSpeculativeTask frequently show up as one of the top routines (by time)
> called holding this lock.
> this routine calls canBeSpeculated() and hasRunOnMachine() for each task in a
> candidate job. Both these routines are reasonably expensive when invoked
> repeatedly for thousands of tasks. The top candidates for speculation from a
> job only need to be refreshed periodically (and not once every heartbeat) -
> and we can can avoid most of these invocations this way.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
