[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935113#comment-16935113 ] zhoukang commented on YARN-5139: 4. Add PlacementSet and score nodes implementation. this has not been implemented? [~leftnoteasy] > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599324#comment-16599324 ] Mike commented on YARN-5139: Hi [~leftnoteasy], I am currently looking at trying to reproduce the simulation results provided in {{YARN-5139-Concurrent-scheduling-performance-report.pdf}} in YARN 3.1.1. Based on the information provided, I created my own environment and ran the test, but the results I obtained do not match what was obtained earlier. Below is my environment, please advise me if there is something else I can try?. Thanks. OS - CentOS 6.6, 40 cores, 256 GB RAM, Java 1.8_101 YARN 3.1.1 Using capacity scheduler, with a single queue, maximum-applications=10, maximum-am-resource-percent=0.1, DefaultResourceCalculator, node-locality-delay=-1, rack-locality-additional-delay=-1, schedule-asynchronously-enable=true, schedule-asynchronously.maximum-threads=4-- SLS - yarn.sls.runner.pool.size=4000, yarn.sls.nm.memory.mb=131072, yarn.sls.nm.vcores=128, nm/am heartbeat 1000ms, metrics=true I had a more complex workload that more closely matched what was given, but for simplicity, I'll add an average one here (I ran this too, and got similar results to my more complex workload): { "num.nodes": 2, "num.racks": 1000 } { "job.start.ms": 0, "job.queue.name": "my_queue", "job.count": 47000, "job.tasks": [ { "count": 400, "container.duration.ms": 12, "container.type": "map" } ] } > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583331#comment-16583331 ] zhuqi commented on YARN-5139: - Hi [~yufeigu] : Thanks for adding me to contributers. I will try to contribute later when i am free. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583318#comment-16583318 ] Yufei Gu commented on YARN-5139: [~zhuqi], done. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583299#comment-16583299 ] zhuqi commented on YARN-5139: - Could you please add me to the contributers, so that i can contribute something about the fairscheduler later :D. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502076#comment-16502076 ] Yufei Gu commented on YARN-5139: [~zhuqi], you are welcome to contribute to Fair Scheduler. It's not a trivial effort to bring global scheduling to FS even with these jiras in. I strongly believe it is the right direction though. Let me know if you need any help. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501947#comment-16501947 ] Wangda Tan commented on YARN-5139: -- [~zhuqi], It is committed to 2.9.0 and after. Welcome to help :), not sure if any FS committers can help review this. cc: [~haibochen] > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501548#comment-16501548 ] zhuqi commented on YARN-5139: - Hi [~wangda] , what the version of hadoop this patch based, and i want to try to help the fairscheduler to move to gloabal scheduling , because our company use the fairscheduler and we are looking forward to the global scheduling to solve our problems. Thanks. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283265#comment-16283265 ] Tao Yang commented on YARN-5139: Thanks [~leftnoteasy] for detailed introduction above and apologize for my late reply. {quote} please share the use cases of multiple nodes look up from your POV. We can incorporate it once working on implementations. {quote} I think YARN-6592 which can support lots of customized requirements is good enough for our use cases. We will keep watching these issues and give some feedbacks from our use cases. {quote} If you have interests/bandwidth, you may take a crack at YARN-7457, which is also crucial to make a clean separation of allocation algorithm. {quote} I would like to look into YARN-7457 and make a try, Thanks. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274720#comment-16274720 ] Wangda Tan commented on YARN-5139: -- Thanks [~Tao Yang] for reporting this. I'm glad to see this runs in your prod environment for half a year! Yeah, please share the use cases of multiple nodes look up from your POV. We can incorporate it once working on implementations. In terms of the future work, despite bug fixes, they will be: 1) Global Scheduler Related Refactorings: (YARN-7438). Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm. Wangda Tan (YARN-7457). Delay scheduling should be an individual policy instead of part of scheduler implementation. (YARN-7496). Add muti node lookup support for better placement. The main purpose is to make a more self-contained per-app placement allocator algorithms. We're trying to close YARN-7438 soon, and [~sunilg] is working on YARN-7496. If you have interests/bandwidth, you may take a crack at YARN-7457, which is also crucial to make a clean separation of allocation algorithm. 2) Additional placement algorithms: They're all located under YARN-6592 branch. Currently, most API patches are get committed. And we're trying to finish simple intra-app affinity/anti-affinity feature as the first use case of the scheduling constraint in the short term. [~asuresh]/[~kkaranasos] are working on more advanced allocation algorithm which can aggregate requests from different apps/services and run LP solver to better place services with picky scheduling constraints. Please let me know your suggestions and welcome to participate this work if you have interest! > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274131#comment-16274131 ] Tao Yang commented on YARN-5139: Async-scheduling decouples scheduling from nm heartbeats and use multiple allocation threads to make scheduling more efficient. It's very important for us and we have enabled async-scheduling mode of global scheduler on our production cluster which has thousands of nodes for half a year. All the problems we met were submitted to community and most of them were already resolved. Glad to hear that this feature are moving on and will pay more attention to multiple nodes lookup (YARN-7494) and node scorer (in design) for better placement, And we would like to participate in this work if needed. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855496#comment-15855496 ] Tao Yang commented on YARN-5139: Hi, [~leftnoteasy]. YARN-5716 seems not to be able to lookup multiple nodes for each resource request. Is there another sub-task to handle this and what is the plan? > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch, > YARN-5139.000.patch, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649829#comment-15649829 ] Carlo Curino commented on YARN-5139: Makes sense... I might be traveling in 3-4 weeks, but we can continue asynch discussions. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649253#comment-15649253 ] Wangda Tan commented on YARN-5139: -- [~curino], Good point, yeah I think we should have a better structured logic to support different levels of "fairness". They may come from different places: 1) How/when to sort queues / apps, we can re-sort queues/apps for each allocated containers, or we can delay the re-sorting 2) Which I mentioned above: maximum number of pending to-be-committed resource allocations. 3) Lower level fairness such as user-limit, etc. So basically, instead of putting them into one place (such as "pluggable fairness policy"), we may need to have a couple of configurable places to update the scheduler to be more fairness or less fairness. Since I will take vacation soon, I think we could have some discussions 3-4 weeks later. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15645782#comment-15645782 ] Carlo Curino commented on YARN-5139: [~wangda] first of all, yes it is ok to do this in a later patch, provided the refactorings here do account for that (i.e., make it easier than it is today). The point I am making is a bit more structural, I agree that the code you quote does achieve the same (ignore partially fairness). My point however, was that if we decide to not do fairness we can short-circuit lots of the scheduler complexity (the above check doesn't really simplify the scheduler, just affect some of its decisions). The above is useful functionally, but I don't think substantially simplifies the scheduler nor affect performance drastically. So the point is let's organize the code so that if we decide to run "without fairness" we can employ a much simplified (and more parallel) scheduler. Makes sense? > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637617#comment-15637617 ] Wangda Tan commented on YARN-5139: -- Thanks [~curino] for the comment, very good point. Basically I agree with all of what you mentioned. Existing patch attached to YARN-5716 considers part of this. The {{ResourceCommitterService}} has a check: {code} 493 // Don't run schedule if we have some pending backlogs already 494 if (cs.getAsyncSchedulingPendingBacklogs() > 100) { 495 Thread.sleep(1); 496 } else{ 497 schedule(cs); 498 } {code} The {{getAsyncSchedulingPendingBacklogs}} can directly affect to fairness. We could make this from 100 to a configurable value. And of course, we can do better than this, for example, we can calculate allocated + potential_allocated for each queue and stop allocating container to a queue which is too much above the expected fair share. So considering size of the patch attached to YARN-5716, it might be better to have a separate patch to improve this part. Sounds like a reasonable plan? And please feel free to let me know if you have any other thoughts. I plan to commit YARN-5716 if you think it is fine. Thanks, > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637462#comment-15637462 ] Carlo Curino commented on YARN-5139: [~wangda] Thanks for working on this, it looks very promising. One thing I was considering was the following: if we were not concerned with fair-sharing of over-capacity I believe we could push this design even much further, with even lower need for coordination, and focusing more on speed/quality of the scheduling decisions. My hunch is that fairness is a very costly property to enforce, and from my conversations with many many folks and an analysis of escalation tickets and user discussion threads, I believe it is not what people really want. If you ask someone "do you want a fair cluster?" they will answer yes, but in practice they can't see unfairness, nor seem to affect users in practice much (people care about meeting deadlines or low-latency, whether this is achieve in a fair way or not seems largely secondary). To ground this mumbling a bit more, the question is: Do you think we can structure this code so that if we were not concerned with fairness (or fair over-capacity) we could squeeze even more parallelism/speed out of it? > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15620709#comment-15620709 ] Wangda Tan commented on YARN-5139: -- Thanks [~asuresh] for the comments, all great points, let me try to explain: 1) Regarding to uniform distribution of applications: First I want to make sure if I understand your question correctly, there're two different kinds of allocations: 1. Look at one node at a time for each resource request 2. Look at multiple nodes at a time for each resource request For #1, it is same as today's async scheduling or scheduling with node heartbeat, we can get uniform distribution of allocations for free. So I assume your question is for #2. For #2, we need to add node's utilization into consideration, If you look at the implementation notes (I uploaded it to https://github.com/leftnoteasy/public-hadoop-repo/blob/global-scheduling-3/global-scheduling-explaination.md), there's a pesudo code snippet about how to match nodes to resource request inside application: {code} // Filter clusterPlacementSet by given resource request, for example: // - Hard locality // - Anti-affinity / Affinity PlacementSet filteredPlacementSet = filter(clusterPlacementSet, request); // Sort filteredPlacement according to resouce-request for (node in sort(filteredPlacementSet, request)) { if (node.has_enough_available_resource()) { // If node has enough available resource to allocate this request // Return a proposal for allocate this container } else { // If node doesn't have enough available resource // Return a proposal for reserve the container } // Also, what could happen: // - Container released, for example, unnecessary reserved container // - Cannot find node, return NOTHING_ALLOCATED } } {code} The sort(filteredPlacementSet, request) could feed the nodes in an order which considers utilization. 2) bq. Another thing that came to mind is that, given that you are kind of 'late-binding' the request to a group of nodes ... Great point, which we somehow missed in today's async scheduling implementation as well. One (relatively) simple thing we can do is to maintain an internal "scheduling state" for NodeManagers: if a NM doesn't do heartbeat for X seconds (e.g. X=10*NM-heartbeat-internal), we could stop allocating new containers to such NMs. And we can also recall allocated but not-yet acquired continers on such NMs. One question from my side, IIRC, FS enables async scheduling by default (CS supports async scheduling but it has lots of issues so I didn't see anybody enables it in production). So I'm curios about in your estimation, by average, how many nodes could fail/lost in every hour for a 10K nodes cluster? If it happens very offen, any user complains about this issue for today's async scheduling? > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15620108#comment-15620108 ] Arun Suresh commented on YARN-5139: --- [~leftnoteasy], Was just going thru the design. Was wondering how you tackle uniform distribution of allocations. This was one nice thing the existing Node Heartbeat based implementation gives you for free. For example, assuming you have just a single default queue and you have a cluster of say 1 nodes. Say we have around 100 apps running. Since the ClusterNodeTracker will always give the same ordering of the 1 nodes, It is possible this new scheduling logic fill 'front-load' all allocations to the Node that appears in the front of the PlacementSet (Since the placement set provided to each application would be fundamentally the same). In the NodeHeartbeat driven case, the node that has just 'heartebeat-ed' will be preferred for allocation, and since heartbeats from all nodes are distributed uniformly, you will generally never see this issue. This is probably not too much of an issue in a fully pegged cluster, but for clusters that are running at around 50% utilization, you will probably see half the nodes fully pegged and the other half mostly sitting idle. Another thing that came to mind is that, given that you are kind of 'late-binding' the request to a group of nodes. In large clusters of sizes > 10K, it is very common to have around 5% of nodes to keep going up and down. In which case, you might have to re-do you allocation if the Node you had selected for an allocation had gone down. In a node heartbeat driven scheme, the chances of that happening are less, since you are allocating on a node that just 'heartbeat-ed' so you can be fairly certain that the node should be healthy. Let me know what you think. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15556577#comment-15556577 ] Vinod Kumar Vavilapalli commented on YARN-5139: --- Great work, [~leftnoteasy]! > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15556378#comment-15556378 ] Wangda Tan commented on YARN-5139: -- To avoid distracting global scheduler overall design/discussion, opened sub ticket: YARN-5716 to track patch reviews. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: Explanantions of Global Scheduling (YARN-5139) > Implementation.pdf, YARN-5139-Concurrent-scheduling-performance-report.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes-v2.pdf, > YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf, > YARN-5139.000.patch, wip-1.YARN-5139.patch, wip-2.YARN-5139.patch, > wip-3.YARN-5139.patch, wip-4.YARN-5139.patch, wip-5.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15550501#comment-15550501 ] Hadoop QA commented on YARN-5139: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 11 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 32s {color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 2 new + 1 unchanged - 2 fixed = 3 total (was 3) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 148 new + 1421 unchanged - 157 fixed = 1569 total (was 1578) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 10s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 11 new + 0 unchanged - 0 fixed = 11 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s {color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 3 new + 937 unchanged - 1 fixed = 940 total (was 938) {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 57s {color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 51m 20s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | | Nullcheck of node at line 1414 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(PlacementSet, boolean) At CapacityScheduler.java:1414 of value previously dereferenced in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(PlacementSet, boolean) At CapacityScheduler.java:[line 1414] | | | org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$ResourceCommitterService.run() does not release lock on all exception paths At CapacityScheduler.java:on all exception paths At CapacityScheduler.java:[line 532] | | | Unread field:ContainerAllocation.java:[line 61] | | | Unused field:ContainerAllocation.java | | | Read of unwritten field demandingHostLocalNodes in org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.LocalityPlacementSet.getPreferredNodeIterator(Place
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15315223#comment-15315223 ] Subru Krishnan commented on YARN-5139: -- +1 for the proposal. We have had multiple discussions offline about this with [~leftnoteasy], [~kasha], [~vinodkv], [~asuresh], etc Thanks to [~leftnoteasy] for initiating this. Are you planning to add _node labels_ (actually the GUTS API) to {{NodeCandidates}} and {{SchedulerNodesScorer}} for filtering? I am also in agreement with [~kasha], [~asuresh] that we should take this opportunity to consider separating *Cluster State* from *Application Scheduling*. This should allows to scale/optimize them individually. At the risk of over-simplification, we can potentially evolve the current scheduler specific policies to be limited to the ordering of applications (a la priority heap sort algorithm). I am happy to contribute, let me know if you need any help. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: wip-1.YARN-5139.patch, wip-2.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300630#comment-15300630 ] Wangda Tan commented on YARN-5139: -- [~kasha], totally agree, we should make code can be easily reused across schedulers. We're on the same page for this :). [~rohithsharma], bq. How are the nodes are grouped for each applications? Is it based on RR for each applications? In my currently thoughts, it is based on RR for each applications. bq. If so doesn't is increase sorting time for each applications every time especially in large cluster deployment? This depends, for each queue.assignContainers call, scheduler can add the latest heartbeat NM to nextAvailableNode of {{NodeCandidates}}. If application doesn't need to look up all the hosts, it can directly use the nextAvailableNode. And we can leverage caching, etc. to reduce search time. bq. Does allocation is fully independent of node heartbeat after this? I mean asynchronous allocation? It can be either related or not related to node heartbeat. Resource allocation can be triggered by node heartbeat or asynchronizedly. We should be able to support them at the same time in the longer term. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: wip-1.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299501#comment-15299501 ] Rohith Sharma K S commented on YARN-5139: - Thanks [~leftnoteasy] for initiating this major change in allocation. +1 for proposal. I believe this definitely improves in majority of 2 factors. Firstly node locality hit rate. Secondly container allocation rate. Some couple of doubts # How are the nodes are grouped for each applications? Is it based on RR for each applications? If so doesn't is increase sorting time for each applications every time especially in large cluster deployment? # Does allocation is fully independent of node heartbeat after this? I mean asynchronous allocation? > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: wip-1.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299456#comment-15299456 ] Karthik Kambatla commented on YARN-5139: I am less particular about using ClusterNodeTracker. My general hope is that we would build a library around this. I understand the power of caching and/or approximate sorting, but that needs to be backed by something more concrete. And, we could make those abstractions configurable by themselves outside of individual schedulers. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: wip-1.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299146#comment-15299146 ] Wangda Tan commented on YARN-5139: -- [~asuresh], [~kasha], The major reason of adding the {{NodeCandidates}} instead of using APIs in ClusterNodeTracker is we can cache results instead of doing O(n) filtering at ClusterNodeTracker. I'm not sure if we can merge the two filters, it will become more clear while doing the POC. Similarly, using {{SchedulerNodesScorer}} instead of ClusterNodeTracker.sortedNodeList is we don't have to do full sort of the list. For the constraints between applications, YARN-4902 can definitely cover it. However, I'm not sure what's the best way to describe it before we have YARN-4902. I don't feel strongly that we shouldn't put it into resource request, contradicting constraints should be detectable. We can continue discuss about it in YARN-1042. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: wip-1.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299084#comment-15299084 ] Arun Suresh commented on YARN-5139: --- Thanks for starting this [~leftnoteasy] !! would love to help out / contribute.. bq. Because only application knows which node is best for its pending resource requests, so we can sort and filter node candidates based on application's resource-requests. I agree.. generally.. but I am thinking that *relational* constraints (constraints that govern placement / grouping of containers belonging to multiple apps. for. eg : containers of HBase and Storm app to be grouped together. *together* can be a relaxed constraint like : within same node, rack or ANY) should maybe be expressed via an API decoupled from the applications ResourceRequest. It will also help solve problems where two apps give contradicting constraints.. for eg. RR from HBase app allocate() call says "allocate container with affinity to containers from Storm App" and RR from Storm app says "allocate container with ANTI-affinity to Hbase containers". But in any case, even if what I stated above were a separate API, I still like your concept of NodeCandidates. Since it essentially is a filter, and we should be able to compose the ResourceRequests constraints with relational constraints. Also, as [~kasha] mentioned, it would be nice to have the {{ClusterNodeTracker}} expose an API that takes a {{NodeCandidate}} and returns a list of nodes. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: wip-1.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299051#comment-15299051 ] Sangjin Lee commented on YARN-5139: --- Yes, we would love to see this for more reasons than one! > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: wip-1.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299044#comment-15299044 ] Karthik Kambatla commented on YARN-5139: Oh, and can we see how we can evolved ClusterNodeTracker and add more tooling around it to come up with NodeCandidates. I believe we can parallelize this when required for higher throughput. This has come up in previous community discussions. /cc [~subru], [~asuresh], [~sjlee0] > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: wip-1.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15299024#comment-15299024 ] Karthik Kambatla commented on YARN-5139: [~leftnoteasy] - thanks for filing this. In our experience with continuous scheduling, this approach will help for faster scheduling especially for high-priority applications. That said, once you gain more experience with your prototype, can we actually decide on the right abstracts so in the new world the schedulers share more code. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: wip-1.YARN-5139.patch > > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5139) [Umbrella] Move YARN scheduler towards global scheduler
[ https://issues.apache.org/jira/browse/YARN-5139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298988#comment-15298988 ] Wangda Tan commented on YARN-5139: -- One possibly doable approch in my mind is to move the top level loop: bq. for node in allNodes: Into application, new logic looks like: {code} Go to parentQueue Go to leafQueue for application in leafQueue.applications: for resource-request in application.resource-requests for node in nodes (node candidates sorted according to resource-request) try to schedule on node {code} Because only application knows which node is best for its pending resource requests, so we can sort and filter node candidates based on application's resource-requests. > [Umbrella] Move YARN scheduler towards global scheduler > --- > > Key: YARN-5139 > URL: https://issues.apache.org/jira/browse/YARN-5139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wangda Tan >Assignee: Wangda Tan > > Existing YARN scheduler is based on node heartbeat. This can lead to > sub-optimal decisions because scheduler can only look at one node at the time > when scheduling resources. > Pseudo code of existing scheduling logic looks like: > {code} > for node in allNodes: >Go to parentQueue > Go to leafQueue > for application in leafQueue.applications: >for resource-request in application.resource-requests > try to schedule on node > {code} > Considering future complex resource placement requirements, such as node > constraints (give me "a && b || c") or anti-affinity (do not allocate HBase > regionsevers and Storm workers on the same host), we may need to consider > moving YARN scheduler towards global scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org