[
https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15155792#comment-15155792
]
Karthik Kambatla commented on YARN-1011:
----------------------------------------
Looked more closely into how we would implement this for the FairScheduler.
[~wangda], [~elgoiri] - could you guys please look if this is similar to what
is needed in the CapacityScheduler. I would like for us to make changes in the
common parts as much as possible, so the schedulers don't diverge further.
While at it, there might be value in moving more of the common logic to common
data structures.
# SchedulerNode needs to track utilized/unutilized resources in addition to
allocated (called used today) and unallocated (called available today). It
might make sense to update the names of variables that exist today in trunk
itself.
# It would be handy to have a single class (may be, NodeTracker or
NodeListManager) that tracks properties about all nodes and exposes some
convenient APIs - return nodes matching a particular condition (filter) or
returning a sorted list of NodeIds based on a custom comparator. We need the
latter to iterate through nodes in the right order for continuous/asynchronous
scheduling and the newly added opportunistic scheduling. The former could be
used for locality and label-matching in the future.
# SchedulerApplicationAttempt should expose unmetDemandGuaranteed and
unmetDemandOpportunistic to be used by the two schedulers.
# There will be two queue hierarchies - one each for guaranteed and
opportunistic allocation. This translates to adding a second list of child
queues/apps in each queue. We could build on what we have in each scheduler, or
reconcile some of this code into common queue.
> [Umbrella] Schedule containers based on utilization of currently allocated
> containers
> -------------------------------------------------------------------------------------
>
> Key: YARN-1011
> URL: https://issues.apache.org/jira/browse/YARN-1011
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Arun C Murthy
> Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf,
> yarn-1011-design-v2.pdf
>
>
> Currently RM allocates containers and assumes resources allocated are
> utilized.
> RM can, and should, get to a point where it measures utilization of allocated
> containers and, if appropriate, allocate more (speculative?) containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
