[
https://issues.apache.org/jira/browse/YARN-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820685#comment-16820685
]
Tao Yang commented on YARN-9472:
--------------------------------
Thanks [~zhuqi] for confirming that.
Currently most components of global scheduler( like placement-allocator,
pluggable node sorting policies and activities etc.) are decoupled with
scheduler so that they can be used for fair scheduler without big changes. Take
complex resource placement requirements for example, if fair scheduler got
scheduling request instead of resource request, it will use
SingleConstraintAppPlacementAllocator inside and we can calling
AppSchedulingInfo#preCheckNode in property place of scheduling process to
support it. I have just go through fair scheduler quickly, perhaps there are
more things should be changed, but I think they won't be too much.
To improve scheduling process of fair scheduler, I think we should extract
parallel stages like sorting and choosing nodes to decrease lock contention,
maybe need parallel assignment phase and synchronous commit phase too, reusing
request-based approach in capacity scheduler is more efficient but I think
there are still some places need to be improved, such as support batch
assignments based on request and try to allocate on multiple nodes for the same
request in multiple threads.
> Add multi-thread asynchronous scheduling to fair scheduler
> ----------------------------------------------------------
>
> Key: YARN-9472
> URL: https://issues.apache.org/jira/browse/YARN-9472
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, resourcemanager
> Reporter: zhuqi
> Priority: Major
>
> Now the capacity scheduler has multi-thread asynchronous scheduling, i think
> the fair scheduler also need to support it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
