[
https://issues.apache.org/jira/browse/MAPREDUCE-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729714#action_12729714
]
rahul k singh commented on MAPREDUCE-725:
-----------------------------------------
Looked at the JobInProgress's findSpeculativeTask , it doesn't change any state
of TaskInProgress. We can refactor this method in order to find out if jip has
any speculative tasks.
We are using hasSpeculative to block a slot on the task tracker. As
hasSpeculative is very dynamic in nature , there are chances that by the time
tasktracker comes back for the task , we dont have any speculativeTasks to
assign, this would mean we would waste some heartbeat and also would lead to
under utilization.
This problem is already there with the capacity-scheduler though.
One way of reducing this is to have MIN_SPEC_CAP(in JobInProgress) kind of
variable in capacity scheduler.
Any thoughts?
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask bypasses HADOOP-2141
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-725
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-725
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Arun C Murthy
> Priority: Critical
>
> CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask has a duplicate of the
> old speculation code (pre HADOOP-2141) which needs to be fixed in-order for
> speculation to work correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
