Wangda Tan created YARN-10380:
---------------------------------
Summary: Import logic of multi-node allocation in CapacityScheduler
Key: YARN-10380
URL: https://issues.apache.org/jira/browse/YARN-10380
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Wangda Tan
*1) Entry point:*
When we do multi-node allocation, we're using the same logic of async
scheduling:
{code:java}
// Allocate containers of node [start, end)
for (FiCaSchedulerNode node : nodes) {
if (current++ >= start) {
if (shouldSkipNodeSchedule(node, cs, printSkipedNodeLogging)) {
continue;
}
cs.allocateContainersToNode(node.getNodeID(), false);
}
} {code}
Is it the most effective way to do multi-node scheduling? Should we allocate
based on partitions? In above logic, if we have thousands of node in one
partition, we will repeatly access all nodes of the partition thousands of
times.
I would suggest looking at making entry-point for node-heartbeat,
async-scheduling (single node), and async-scheduling (multi-node) to be
different.
Node-heartbeat and async-scheduling (single node) can be still similar and
share most of the code.
async-scheduling (multi-node): should iterate partition first, using pseudo
code like:
{code:java}
for (partition : all partitions) {
allocateContainersOnMultiNodes(getCandidate(partition))
} {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
