[
https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728125#action_12728125
]
Hemanth Yamijala commented on MAPREDUCE-722:
--------------------------------------------
I am not entirely sure this is a bad decision at all. I think we can assume
that at the point the high RAM job decides there are speculative tasks to
execute, there will be certain good tasks that will not need speculation, that
are still running and will come back to the job tracker. At that point, we can
certainly run the speculative tasks of high RAM jobs, and there will be no
starvation. I certainly don't believe there is going to be guaranteed
starvation in any case.
Also, our speculation heuristics at this point are only now improving (with
HADOOP-2141) and there is a good chance users of older versions of Hadoop (0.20
and before) will not rely on speculation to work that nicely anyway.
Also, in the extreme case that this condition of starvation is indeed hit,
there is the workaround that the user could kill slow running tasks if he
thinks the job is taking too long to finish. This is not the greatest
workaround available, but will work in the short - medium term.
So, I would definitely recommend we favor cluster utilization as I suppose is
done in the last patch.
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and
> reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
> Attachments: MAPREDUCE-722.1.txt, MAPREDUCE-722.txt
>
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800
> reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which
> much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
