[
https://issues.apache.org/jira/browse/MAPREDUCE-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767236#action_12767236
]
Hemanth Yamijala commented on MAPREDUCE-1105:
---------------------------------------------
Some comments on the patch for trunk:
- QueueSchedulingContext's constructor is not indented correctly.
- The check that the capacity of a queue is <= maxCapacity is done in
CapacitySchedulerConf.getMaxCapacity, when the queue hierarchy is being
constructed. This happens before spare capacity is distributed for queues whose
capacity is undefined. If the capacity of a queue is undefined, but it's
maxCapacity is defined, this leads to a case where the validation will pass,
but after the spare capacity is distributed, this may no longer be valid.
- It doesn't seem like TaskSchedulingContext.LIMIT_NORMALIZED_CAPACITY_STRING
is required at all. The only purpose it was there was to tell administrators
that they are getting less than their capacities because of limits. Since this
should never happen now, we should remove this. Likewise, the
TaskSchedulingContext.toString() method will need modification to not have any
checks for maxCapacity limits.
- Javadoc of TaskSchedulingContext.getCapacity needs update.
- TaskSchedulingContext.toString is not showing maxCapacity value for the
queue. This seems an existing bug - not introduced in this patch.
- CapacityTaskScheduler.areTasksInQueueOverLimit is still referring to
maxTaskLimit in a comment.
- In TestCapacityScheduler, rather than removing the test cases related to
limits, I would think it is better to convert them into test cases for
maxCapacities. There will be some change required, because in these tests we
were setting the maxLimit to be less than the capacity. But, in general, these
tests will ensure that features like handling high RAM jobs and user limits is
working with maxCapacities.
- The current implementation of implementing limits on capacity may have a bug.
Consider if a queue is one slot below it's maxCapacity.
CapacityTaskScheduler.areTasksInQueueOverLimit will return false. Now suppose
the job being scheduled is a high RAM job, then it could actually go beyond its
capacity. Maybe when we are assigning a task from a job in a queue, before the
assignment we should again check if the currently used slots + number of slots
required for this job will cause the limit to exceed.
- TestCapacityScheduler.checkOccupiedSlots is adding zeros to some indices.
This should be removed.
> CapacityScheduler: It should be possible to set queue hard-limit beyond it's
> actual capacity
> --------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1105
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1105
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.21.0
> Reporter: Arun C Murthy
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPRED-1105-21-1.patch,
> MAPREDUCE-1105-version20.patch.txt
>
>
> Currently the CS caps a queue's capacity to it's actual capacity if a
> hard-limit is specified to be greater than it's actual capacity. We should
> allow the queue to go upto the hard-limit if specified.
> Also, I propose we change the hard-limit unit to be percentage rather than
> #slots.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
