[jira] [Commented] (YARN-9472) Add multi-thread asynchronous scheduling to fair scheduler
[ https://issues.apache.org/jira/browse/YARN-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820685#comment-16820685 ] Tao Yang commented on YARN-9472: Thanks [~zhuqi] for confirming that. Currently most components of global scheduler( like placement-allocator, pluggable node sorting policies and activities etc.) are decoupled with scheduler so that they can be used for fair scheduler without big changes. Take complex resource placement requirements for example, if fair scheduler got scheduling request instead of resource request, it will use SingleConstraintAppPlacementAllocator inside and we can calling AppSchedulingInfo#preCheckNode in property place of scheduling process to support it. I have just go through fair scheduler quickly, perhaps there are more things should be changed, but I think they won't be too much. To improve scheduling process of fair scheduler, I think we should extract parallel stages like sorting and choosing nodes to decrease lock contention, maybe need parallel assignment phase and synchronous commit phase too, reusing request-based approach in capacity scheduler is more efficient but I think there are still some places need to be improved, such as support batch assignments based on request and try to allocate on multiple nodes for the same request in multiple threads. > Add multi-thread asynchronous scheduling to fair scheduler > -- > > Key: YARN-9472 > URL: https://issues.apache.org/jira/browse/YARN-9472 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: zhuqi >Priority: Major > > Now the capacity scheduler has multi-thread asynchronous scheduling, i think > the fair scheduler also need to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9472) Add multi-thread asynchronous scheduling to fair scheduler
[ https://issues.apache.org/jira/browse/YARN-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820149#comment-16820149 ] Adam Antal commented on YARN-9472: -- CC [~wilfreds], [~snemeth] > Add multi-thread asynchronous scheduling to fair scheduler > -- > > Key: YARN-9472 > URL: https://issues.apache.org/jira/browse/YARN-9472 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: zhuqi >Priority: Major > > Now the capacity scheduler has multi-thread asynchronous scheduling, i think > the fair scheduler also need to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9472) Add multi-thread asynchronous scheduling to fair scheduler
[ https://issues.apache.org/jira/browse/YARN-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819765#comment-16819765 ] zhuqi commented on YARN-9472: - Hi, [~Tao Yang] . Thanks for your reply. Yes, i mean the final multi-thread asynchronous scheduling can be supported in global scheduler base request-based. And the non-HB driven approach that continuous scheduling has be removed because of the lock contention in large scale (YARN-6486), the development of global scheduling for fair scheduler should be a long time . If we can first add a RM service for scheduling performance improvements only, but not consider all the complex resource placement requirements, which can be coupled with some simple localilty relaxation strategy? > Add multi-thread asynchronous scheduling to fair scheduler > -- > > Key: YARN-9472 > URL: https://issues.apache.org/jira/browse/YARN-9472 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: zhuqi >Priority: Major > > Now the capacity scheduler has multi-thread asynchronous scheduling, i think > the fair scheduler also need to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9472) Add multi-thread asynchronous scheduling to fair scheduler
[ https://issues.apache.org/jira/browse/YARN-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818695#comment-16818695 ] Tao Yang commented on YARN-9472: Hi, [~zhuqi]. Multi-thread asynchronous scheduling only means support scheduling in multiple threads, scheduling approaches can be classified as node-based and request-based, currently fair scheduler only support node-based and capacity scheduler can support both, global scheduler is request-based and aims for enhance scheduling performance with considering complex resource placement requirements. I think your purpose is to support global scheduler including multi-thread asynchronous scheduling, multi-nodes lookup mechanism, placement constraints etc for fair scheduler, Right? > Add multi-thread asynchronous scheduling to fair scheduler > -- > > Key: YARN-9472 > URL: https://issues.apache.org/jira/browse/YARN-9472 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: zhuqi >Priority: Major > > Now the capacity scheduler has multi-thread asynchronous scheduling, i think > the fair scheduler also need to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9472) Add multi-thread asynchronous scheduling to fair scheduler
[ https://issues.apache.org/jira/browse/YARN-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815238#comment-16815238 ] zhuqi commented on YARN-9472: - [~leftnoteasy] [~Tao Yang] If any suggestions. Thanks. > Add multi-thread asynchronous scheduling to fair scheduler > -- > > Key: YARN-9472 > URL: https://issues.apache.org/jira/browse/YARN-9472 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler, resourcemanager >Reporter: zhuqi >Priority: Major > > Now the capacity scheduler has multi-thread asynchronous scheduling, i think > the fair scheduler also need to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
