Wangda Tan commented on YARN-2877:

Thanks [~sriramsrao] for bringing up the great idea and 
[~kkaranasos]/[~curino]'s explanations. Definitely we need such mechanisms to 
have low-latency container launching to support millisec-level-latency tasks.

Some questions about this,
# Since the LocalRMs will be totally distributed, does it still possible to 
enforce capacity between queues?
# Will such opportunistical containers come to view of the central RM (used to 
schedule CONSERVATIVE containers)? 
## If yes, will the central RM can decide if a opportunistical container is 
valid or not (saying #containers excesses the app's limitation)? And will the 
preemption still works for opportunistical containers
## If no, should we have someone to coordinate such containers?
# Will central scheduler state (maybe not completely, but important info like 
queue used resource, etc.) broadcast to distributed LocalRMs? I think it might 
be usaful for LocalRMs to decide which opportunistical container should go 

Thanks in advance!


> Extend YARN to support distributed scheduling
> ---------------------------------------------
>                 Key: YARN-2877
>                 URL: https://issues.apache.org/jira/browse/YARN-2877
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Sriram Rao
> This is an umbrella JIRA that proposes to extend YARN to support distributed 
> scheduling.  Briefly, some of the motivations for distributed scheduling are 
> the following:
> 1. Improve cluster utilization by opportunistically executing tasks otherwise 
> idle resources on individual machines.
> 2. Reduce allocation latency.  Tasks where the scheduling time dominates 
> (i.e., task execution time is much less compared to the time required for 
> obtaining a container from the RM).

This message was sent by Atlassian JIRA

Reply via email to