[ https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221745#comment-14221745 ]
Wangda Tan commented on YARN-2877: ---------------------------------- Thanks [~sriramsrao] for bringing up the great idea and [~kkaranasos]/[~curino]'s explanations. Definitely we need such mechanisms to have low-latency container launching to support millisec-level-latency tasks. Some questions about this, # Since the LocalRMs will be totally distributed, does it still possible to enforce capacity between queues? # Will such opportunistical containers come to view of the central RM (used to schedule CONSERVATIVE containers)? ## If yes, will the central RM can decide if a opportunistical container is valid or not (saying #containers excesses the app's limitation)? And will the preemption still works for opportunistical containers ## If no, should we have someone to coordinate such containers? # Will central scheduler state (maybe not completely, but important info like queue used resource, etc.) broadcast to distributed LocalRMs? I think it might be usaful for LocalRMs to decide which opportunistical container should go first. Thanks in advance! Wangda > Extend YARN to support distributed scheduling > --------------------------------------------- > > Key: YARN-2877 > URL: https://issues.apache.org/jira/browse/YARN-2877 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager > Reporter: Sriram Rao > > This is an umbrella JIRA that proposes to extend YARN to support distributed > scheduling. Briefly, some of the motivations for distributed scheduling are > the following: > 1. Improve cluster utilization by opportunistically executing tasks otherwise > idle resources on individual machines. > 2. Reduce allocation latency. Tasks where the scheduling time dominates > (i.e., task execution time is much less compared to the time required for > obtaining a container from the RM). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)