Jian Fang commented on YARN-1039:

Thanks Steve for your clarification. Seems the long lived concept makes sense 
now if this flag is associated with policy switch in YARN. 

I think the above is only one part of the story. Cluster infrastructure itself 
probably is another part that we need to consider. Just like the spot instance 
feature in EC2 as mentioned in this JIRA. 

The long lived concept should have more impacts on hadoop clusters in a cloud 
environment. For example instance type could affect the container scheduling. 
We should also take this concept into consideration for some elastic features 
such as graceful expansion and shrink of a cluster in cloud. 

On the other side, I still think YARN-796 should be used together with the long 
lived concept. For example, how would resource manager know which instance 
should run a long lived daemon/task? There should be a mapping between the long 
lived concept and the tags/labels provided by instance. Right? 

> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>                 Key: YARN-1039
>                 URL: https://issues.apache.org/jira/browse/YARN-1039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Steve Loughran
>            Assignee: Craig Welch
>         Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
> A container request could support a new parameter "long-lived". This could be 
> used by a scheduler that would know not to host the service on a transient 
> (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived 
> containers on the same node

This message was sent by Atlassian JIRA

Reply via email to