Yuqi Wang created YARN-7872:
-------------------------------
Summary: labeled node cannot be used to satisfy locality specified
request
Key: YARN-7872
URL: https://issues.apache.org/jira/browse/YARN-7872
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler, capacityscheduler, resourcemanager
Affects Versions: 2.7.2
Reporter: Yuqi Wang
Assignee: Yuqi Wang
Fix For: 2.7.2, 2.8.0
labeled node (i.e. node with 'not empty' node label) cannot be used to satisfy
locality specified request (i.e. container request with 'not ANY' resource name
and the relax locality is false).
For example:
The node with available resource:
[Resource: [MemoryMB: [100] CpuNumber: [12]] {color:#14892c}NodeLabel:
[persistent]{color} {color:#f79232}HostName: \{SRG}{color} RackName:
\{/default-rack}]
The container request:
[Priority: [1] Resource: [MemoryMB: [1] CpuNumber: [1]]
{color:#14892c}NodeLabel: [null]{color} {color:#f79232}HostNames: \{SRG}{color}
RackNames: {} {color:#59afe1}RelaxLocality: [false]{color}]
Current RM capacity scheduler's behaiour is:
The node cannot allocate container for the request because of the node label
not matched in the leaf queue assign container.
However, node locality and node label should be two orthogonal dimensions to
select candidate nodes for container request. And the node label matching
should only be executed for container request with ANY resource name, since
only this kind of container request is allowed to have 'not empty' node label.
So, for container request with 'not ANY' resource name (besides, it should not
have node label), we should use resource name to match with the node instead of
node label to match with the node. And it should be safe, since the node which
is not accessible for the queue will not be sent in the leaf queue.
Attachment is the fix according to this principle, please help to review.
Without it, we cannot use locality to request container within these labeled
nodes.
If the fix is acceptable, we should also recheck whether the same issue happens
in trunk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
