Craig Welch commented on YARN-1039:

[~ste...@apache.org] wrt the need for a container level flag / a way for the 
application master to launch long lived containers - definitely, but the idea 
was for that to come as a later step - although that may be short-sighted, as 
it may be better to come up with a common way to do this for the application 
master container and the containers it later launches now instead of ending up 
with unmatched approaches later...

This first step is to provide a way for the application master to be launched 
in a long lived container (generally, an application master for a long lived 
application will need to itself be launched in a long lived container - at 
least, it needs to be possible to do so) - which is why there needs to be some 
way to indicate the need for a long lived container during application 
submission (necessary but not sufficient overall...)

[~zjshen] I was also wondering about using the tags, but after talking with 
[~xgong] we are not thinking that is the way to go because tags don't seem to 
be about changing behavior but only about freeform way to enable 

After this discussion and some looking around it really seems that what we are 
after is a way to communicate a quality of the needed container to the resource 
manager both at application submission (for the application master container) 
and also for later container launches by the master, kind of like the 
ResourceProto, which is also already present in both cases for the same reason 
(I suggested adding it there, actually, as something "necessary for the 
container" but [~xgong] objected, thinking it is really specific to metric 
qualities (cpu, memory...).

I'm going to take a look at adding something alongside /similar to the 
ResourceProto to indicate constraints/requirements for the container, starting 
with long lived, that can be common to application submission and when the 
containers are started later by the application, not necessarily a long field 
for bit manipulation but something which is also extensible 

> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>                 Key: YARN-1039
>                 URL: https://issues.apache.org/jira/browse/YARN-1039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Steve Loughran
>            Assignee: Craig Welch
>            Priority: Minor
>         Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
> A container request could support a new parameter "long-lived". This could be 
> used by a scheduler that would know not to host the service on a transient 
> (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived 
> containers on the same node

This message was sent by Atlassian JIRA

Reply via email to