Craig Welch commented on YARN-1039:

{quote}We need to have a flag to indicate an AM is long lived.  We need to have 
another flag to indicate that the containers requested by an AM will be long 

I believe the proposal meets those needs, it is one particular way to do so...

{quote}it must be set at application creation time and all containers of the 
app will be considered long lived. This is because the RM does not keep track 
of individual container requests.{quote}

I'm not sure why it matters whether or not RM keeps track of the container 
requests - the AM will request containers with scheduling constraints like long 
lived, affinity, etc, and RM considers them when selecting nodes, after that 
completes it no longer necessarily matters.  If the AM needs a relationship 
between nodes for a request, or a particular type of selection (not on a spot 
node - long-running) it will make a request for those nodes, get nodes that 
meet it's needs, and it's good to go.  It sounds as though it would be more 
flexible / meet a wider set of usecases and therefore be preferable to allow an 
application master to obtain different types of containers for different 
purposes during it's lifetime as opposed to forcing to use only one set of 
container constraints throughout

{quote}Having a long enum of flag to indicated optional qualities of the 
requested containers has been discussed in the past (in the context of some 
JIRAs related to Llama) and it has been discarded as it would mean divergence 
on the features different schedulers support.{quote}

So, for this jira there is a desire to support selecting nodes with particular 
qualities (not placing a long running process on a temporary/spot instance), 
coming soon are other needs for other similar selection/constraint logic 
(affinity, anti-affinity, etc) - not being able to indicate qualities for the 
containers would keep us for being able to support those needs, and I believe 
there is a need to support this functionality.  It's 
filtering/constraint/selection logic and could probably be generalized in a way 
which could be used by various schedulers...


> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>                 Key: YARN-1039
>                 URL: https://issues.apache.org/jira/browse/YARN-1039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Steve Loughran
>            Assignee: Craig Welch
>         Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
> A container request could support a new parameter "long-lived". This could be 
> used by a scheduler that would know not to host the service on a transient 
> (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived 
> containers on the same node

This message was sent by Atlassian JIRA

Reply via email to