[
https://issues.apache.org/jira/browse/YARN-10738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17340599#comment-17340599
]
Qi Zhu commented on YARN-10738:
-------------------------------
Thanks a lot [~bibinchundatt] for reply and value information.
For above hot spots case, if we can allocate based the dominated resource
utilization, if the vcore is full, the vcore is dominated, we will allocate
other nodes whose dominated resource utilization is not full.
Based the dominated resource utilization, i think we still need to shuffle but
the shuffle gap may be consistent with the cluster size.
> When multi thread scheduling with multi node, we should shuffle with a gap to
> prevent hot accessing nodes.
> ----------------------------------------------------------------------------------------------------------
>
> Key: YARN-10738
> URL: https://issues.apache.org/jira/browse/YARN-10738
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Qi Zhu
> Assignee: Qi Zhu
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Now the multi threading scheduling with multi node is not reasonable.
> In large clusters, it will cause the hot accessing nodes, which will lead the
> abnormal boom node.
> Solution:
> I think we should shuffle the sorted node (such the available resource sort
> policy) with an interval.
> I will solve the above problem, and avoid the hot accessing node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
