Carlo Curino commented on YARN-2882:

To help understand this notion, think of containers types as a priority. The 
*guaranteed-start* containers, have higher priority, and are never over-booked 
(i.e., when they show up in the NM they are started instantaneously). By 
contrast the *queueable* containers are sent to the NM, and will be started 
only when there is room in the node. Also if *guaranteed-start* containers show 
up in a node that was completely utilized running *queueable* containers, the 
*queueable* containers are preempted/killed, to guarantee the start of the 
higher priority containers. 

_(The rest of the comment below is covered in other sub-JIRAs of YARN-2877, 
adding here some hints to the ideas for context)_
By having an explicit notion of container types, the AM can control when to use 
one type vs the other. For example, one can use *queueable* containers for 
tasks that are not yet on the critical path, and/or for short-running tasks 
(higher chance to complete).  One important use of *queueable* containers is to 
allow us to boost utilization of the nodes (having a queue of work, minimize 
the times in which the NM is idle). 

> Introducing container types
> ---------------------------
>                 Key: YARN-2882
>                 URL: https://issues.apache.org/jira/browse/YARN-2882
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Konstantinos Karanasos
> This JIRA introduces the notion of container types.
> We propose two initial types of containers: guaranteed-start and queueable 
> containers.
> Guaranteed-start are the existing containers, which are allocated by the 
> central RM and are instantaneously started, once allocated.
> Queueable is a new type of container, which allows containers to be queued in 
> the NM, thus their execution may be arbitrarily delayed.

This message was sent by Atlassian JIRA

Reply via email to