[
https://issues.apache.org/jira/browse/MAPREDUCE-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13658317#comment-13658317
]
Sergey Tryuber commented on MAPREDUCE-3859:
-------------------------------------------
Mike, your test results look a little bit strange even for 2 slots per reducer.
Because you've said that max capacity is 60. So I would expect that all 60
slots are used in this case. Try to play with "user limit factor". Also try to
set up initial capacity to a little be higher value that 4 slots. I'm afraid
there is another, not related to this, bug when "slots per task" > "initial
capacity".
Arun, Matt, today I have a look into trunk (I believe this is what you call
"1.3.0" release, because there is no 1.3 brunch). And I found there fully
reworked capacity scheduler (YARN). There is another abstraction now which is
called "Resource" instead of "slot/task". I was digging into it for a couple of
hours and got to the feeling that this bug is gone there. I even found a test
which tests something similar and tried to create my own test, but test case
(TestLeafQueue.java) organized very poorly and, basically, tests nothing (mocks
over mocks, no human readable logic and so on). Sorry, I've spent couple of
hours trying to rewrite it and understood that it would take several more days
for me. So I give it up. But, once again, the bug seems to be gone in YARN
version of CS, so nothing to fix there.
For everyone else who is affected by this bug (old Capacity Scheduler), please,
use a hot fix from my first comment. Or, Arun, you can commit that fix and
attached test case (yep, old CapacityScheduler were covered by test cases much
better than in yarn) to appropriate brunch - I just don't know which brunch to
use and I didn't found "contrib" module in trunk.
> CapacityScheduler incorrectly utilizes extra-resources of queue for
> high-memory jobs
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3859
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3859
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: capacity-sched
> Affects Versions: 1.0.0
> Environment: CDH3u1
> Reporter: Sergey Tryuber
> Assignee: Sergey Tryuber
> Attachments: test-to-fail.patch.txt
>
>
> Imagine, we have a queue A with capacity 10 slots and 20 as extra-capacity,
> jobs which use 3 map slots will never consume more than 9 slots, regardless
> how many free slots on a cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
