[
https://issues.apache.org/jira/browse/YARN-7587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-7587:
------------------------------
Attachment: YARN-7587.002.patch
bq. We can explain a bit more in comments the different max queue lengths
I have added some more comments, and changed the variable name from term
{{maxQueueLength}} to {{queueCapacity}}. Hopefully this helps to avoid
confusion.
bq. Rather than performing an extra iteration to filter the list, maybe it
might be better to modify the getMetric method in the comparator to return
MAX_INT / MIN_INT where appropriate if the isQueueFull is true for the
clusternode
I am not sure if that's gonna work. The objective is to exclude some
unqualified nodes, if only modify the getMetric method in comparator, it will
still return all nodes. But I agree, an extra iteration is not necessary. I
have removed the filter code in v2 patch, and replaced that by the check before
adding the sorted nodes to the {{retList}}.
bq. we should do the same for de-comissioned nodes as well
Do you mean we should skip dispatching any O containers to the de-commissioned
nodes? If so I totally agree with that (I would be surprised if it is not done
so). This can be tracked in another JIRA.
Thanks.
> Skip dispatching opportunistic containers to nodes whose queue is already full
> ------------------------------------------------------------------------------
>
> Key: YARN-7587
> URL: https://issues.apache.org/jira/browse/YARN-7587
> Project: Hadoop YARN
> Issue Type: Sub-task
> Affects Versions: 2.9.0
> Reporter: Weiwei Yang
> Assignee: Weiwei Yang
> Attachments: YARN-7587.001.patch, YARN-7587.002.patch
>
>
> When there is a lot of opportunistic container requests, even if NM queue is
> full, RM continues to dispatch O containers to that node. Such allocations
> will be killed immediately by NM causing a waste of resource. Propose to
> exclude such nodes in the load monitor.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
