Zhijie Shen commented on YARN-1039:

bq. I see. I'd assume that the "service" flag would imply long-lived, but maybe 
they could be separated.

Just think it out loudly. Please correct me if I'm wrong or missing something.

"service" and "long-lived" overlap to some extent when describing an 
application, as we usually think a service is going to run for a long time. 
However, IMHO, "service" should not be the necessity for "long-lived". 
Theoretically, a MR job can be big enough to run for a long time as well. We 
may want to differ the application with "service" from others by some of the 
applications' native characteristics. For example, "progress" is not going to 
make sense to the applications that are labeled "service", while we still want 
it for a MR job even if it runs for days, don't we? Moreover, "service" sounds 
the application-level only property, and we won't mark a single container as a 

On the other hand, "long-lived" is used to mark an application that is supposed 
to run for long time. However, it can only indicate the application is likely 
to run for a long time, but can not guarantee it will actually. I'm wondering 
if we really need to mark an application "long-lived" when submission. Is it 
feasible to justify whether an application is "long-lived" by how much time it 
has already spent in the cluster, and the "long-lived" application is going to 
be handled properly in implicit way? For example, when we come to AM retry 
opportunities (one issue for "long-lived" application), we can choose to 
refresh the "quota" given the application is working well for a while. We don't 
need to rely on "long-lived" label. The reason that I can think of why we must 
has this label upfront is that some special treatments for the "long-lived" 
application should start from the beginning.

> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>                 Key: YARN-1039
>                 URL: https://issues.apache.org/jira/browse/YARN-1039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Steve Loughran
>            Assignee: Craig Welch
>            Priority: Minor
>         Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
> A container request could support a new parameter "long-lived". This could be 
> used by a scheduler that would know not to host the service on a transient 
> (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived 
> containers on the same node

This message was sent by Atlassian JIRA

Reply via email to