[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065547#comment-15065547 ] Karthik Kambatla commented on YARN-2882: Synced up with [~asuresh], [~kkaranasos] and [~subru] offline on this to discuss the commonalities with YARN-1011. The notion of *opportunistic* containers is common, and is governed by the following semantics: # A trusted external agent (RM or LocalRM or NM) can initiate/approve running an opportunistic container. # Additional policies on execution - queueable or over-subscription - is determined by the node's configuration. YARN-2877 would add the queueable flag and logic. YARN-1011 would add the over-subscription flag and logic. This logic may include having to monitor the usage of the node. # Only the RM can approve the promotion of an OPPORTUNISTIC container to a GUARANTEED container. In case YARN-1011, the RM instigates this directly. Haven't looked at the patch closely enough, but high-level comments: # Rename QUEUEABLE to OPPORTUNISTIC # Since a GUARANTEED container may be preempted, how about calling it REGULAR instead? # The ExecutionType is something Yarn decides on. Don't think the client API should include it. > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, YARN-2882-yarn-2877.003.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057717#comment-15057717 ] Karthik Kambatla commented on YARN-2882: In YARN-1011, we plan to allocate resources based on utilization. We need a similar differentiation - guaranteed and opportunistic. If the guaranteed containers end up using all allocated resources, the opportunistic containers are to be preempted. For this preemption, we might want to set cgroups and OOM Killer priorities for these opportunistic containers differently. Comments/questions: # Do the opportunistic containers in YARN-1011 and here share characteristics enough to be the same? If yes, can we call them opportunistic instead of queueable. Otherwise, we are probably better off calling them queueable. # To set the cgroups and OOM priorities, we need the type in container initialization context. I am fine with adding it in this patch or later. # We are considering developing YARN-1011 also on a branch. Any ideas on how we can share code? May be, deliberate more on the container execution type and commit it to trunk itself? I would like to do a more thorough review of the patch here once we finalize on these items and I am more awake :) > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057815#comment-15057815 ] Arun Suresh commented on YARN-2882: --- Glad you could chime in [~kasha]. bq. Do the opportunistic containers in YARN-1011 and here share characteristics enough to be the same? .. Although the QUEUEABLE containers will be allocated opportunistically, its naming has more to do with the fact that it will be queued at some NM. Given the fact that the latest patch introduces a new enum type (ExecutionType), instead extending the ContianerType, my vote is to probably add SPECULATIVE / OPPERTUNISTIC as a new ExecutionType for YARN-1011 in addition to QUEUEABLE to signal different mechanisms for handling the type of Request Ask. bq. To set the cgroups and OOM priorities, we need the type in container initialization context... Agreed.. But given that this patch is addressing mostly API changes, I'd prefer adding it in a subsequent patch when required. bq. We are considering developing YARN-1011 also on a branch. Any ideas on how we can share code? I like the Idea of pushing this to trunk, if we are limiting this to addition of the ExecutionType > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058771#comment-15058771 ] Wangda Tan commented on YARN-2882: -- [~asuresh], [~kasha], Since ContainerType is not in any of the releases, theoretically we can still change ContainerType to a boolean to indicate if it is an AM container or not. And then we can reuse the ContainerType to describe if it is a speculative/opportunistic container or not. Thoughts? > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059234#comment-15059234 ] Konstantinos Karanasos commented on YARN-2882: -- Thank you for the feedback, [~kasha]. Some comments below: bq. Do the opportunistic containers in YARN-1011 and here share characteristics enough to be the same? If yes, can we call them opportunistic instead of queueable. Otherwise, we are probably better off calling them queueable. Looking into YARN-1011, I also think we should differentiate QUEUEABLE containers from OPPORTUNISTIC ones. QUEUEABLE containers denote that those containers can be queued in the NM until resources become available, whereas OPPORTUNISTIC are related to the over-commitment of resources at that NM (correct me if I'm wrong). In that case, OPPORTUNISTIC should probably become an additional execution type. bq. To set the cgroups and OOM priorities, we need the type in container initialization context. I am fine with adding it in this patch or later. Agreed. [~subru] also pointed out that having the execution type in the ContainerContext will also be needed for the Timeline Service (this is the reason the ContainerType was also included in the ContainerContext). Thus, I think we should keep it in the current patch. bq. We are considering developing YARN-1011 also on a branch. Any ideas on how we can share code? May be, deliberate more on the container execution type and commit it to trunk itself? Agreed about pushing it to trunk. I will shortly upload a new patch with some small fixes. [~leftnoteasy], I propose to keep the ExecutionType naming for the GUARANTEED/QUEUEABLE/OPPORTUNISTIC containers, since it is more related to the execution rather than the container per se. However, I agree that probably we should turn the ContainerType to a boolean, if we cannot think of any other "container type". > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059314#comment-15059314 ] Konstantinos Karanasos commented on YARN-2882: -- [~leftnoteasy], I see queueable containers as a way to keep a backlog of tasks that can start execution immediately when resources get available in the NM where they are queued. So currently there is no actual overcommitment of resources. However, given the existence of the queue, we can indeed easily support overcommitment if there is a way to monitor the actual resource consumption of the currently running containers (as the one proposed in YARN-1011). I do agree that they are similar to the opportunistic containers of YARN-1011 in that they can be killed by guaranteed containers. That said, we could use the naming "opportunistic" for both cases, but we will need an extra flag that determines whether overcommitment is enabled. Would that be a better solution? > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, YARN-2882-yarn-2877.003.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059331#comment-15059331 ] Wangda Tan commented on YARN-2882: -- bq. But what if someone wants to allow overcommitment of resources but not queueing of tasks at the NMs? I think it should be handled by some configurable/pluggable policy in NM to handle such extra containers in different behavior. > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, YARN-2882-yarn-2877.003.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059303#comment-15059303 ] Wangda Tan commented on YARN-2882: -- [~kkaranasos], I think semantic of queueable and opportunistic are very close: - Both are related to resource overcommitment. (Additional resource to application) - Container with either flag could be killed without notice when any of "guaranteed" container cannot be launched. - When distributed scheduling is disabled, "queueable" container will be directly sent to central RM, when LocalRM is enabled, we could have policy to decide if a queueable request should be sent to central RM. In another word, end user should be agnostic to if distributed scheduling is involved or not. Regarding to "execution type" and "container type", I would say "execution type" is actually affecting how we allocate container as well. Not sure if it's precise enough. > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, YARN-2882-yarn-2877.003.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059344#comment-15059344 ] Konstantinos Karanasos commented on YARN-2882: -- bq. I think it should be handled by some configurable/pluggable policy in NM to handle such extra containers in different behavior. I am worried that this way we are restricting the capabilities of the system. I can see applications that want to allow overcommitment but no queuing, and possible others that want queuing but no overcommitment. By having parameters for queuing and overcommitment at each NM while being agnostic of the application, we would restrict such scenarios. > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, YARN-2882-yarn-2877.003.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059434#comment-15059434 ] Arun Suresh commented on YARN-2882: --- I was giving this some thought.. I feel that both solutions proposed above viz. # adding both OPPORTUNISTIC / SPECULATIVE and QUEUEABLE # overloading OPPORTUNISTIC and using a boolean to specify distributed / centralized scheduling is adding extra burden on the AM to qualify a request ask. A decision, which I don't think most AMs might be equipped to make. I feel an AM must just mark its essential request asks as GUARANTEED.. and the NM via some config / policy should decided how to handle non-guaranteed asks.. which might result in couple of outcomes : # make the *Distributed Scheduling* and *Opportunistic Central scheduling* mutually exclusive (only 1 can be enabled) # allow the LocalRM to handle some of the non-guaranteed asks and route rest to the RM Scheduler etc. which brings me to wonder... why cant we just have Execution type as a boolean, maybe name it *isGuaranteedAsk* ? and like [~leftnoteasy] suggested, rename ContainerType to *isAMContainer*. Of course this would mean giving the AM a little less control on the container launch guarantees if the value is set to false. > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, YARN-2882-yarn-2877.003.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059322#comment-15059322 ] Konstantinos Karanasos commented on YARN-2882: -- bq. I would prefer the above proposal. Overcommitment/queueable are two different just approaches to handle "extra" allocated containers. Makes sense. But what if someone wants to allow overcommitment of resources but not queueing of tasks at the NMs? > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, YARN-2882-yarn-2877.003.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[
https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059584#comment-15059584
]
Hadoop QA commented on YARN-2882:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
17s {color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} yarn-2877 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} yarn-2877 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 30s
{color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
11s {color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
24s {color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s
{color} | {color:green} yarn-2877 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 21s
{color} | {color:green} yarn-2877 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 28s
{color} | {color:red} Patch generated 5 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn (total was 177, now 180). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 28s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 16s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 21s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_66.
{color} |
|
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046023#comment-15046023 ] Konstantinos Karanasos commented on YARN-2882: -- Thanks for the feedback, [~leftnoteasy]: bq. How about renaming ExecutionType to guaranteed/opportunistic We opted for queueable instead of opportunistic, since queueable is more descriptive and shows better the semantics. If you think queueable is a bad name, we could change it back to opportunistic. bq. Is it necessary to put ExecutionType to ContainerStatus? It is important to put ExecutionType in the ContainerStatus, because this way the RM can differentiate between the different container/execution types. For instance, it should not account for resources that are due to queueable containers. This way less changes will be required by the AM that will not require to keep track of which containers are guaranteed and which are queueable. bq. Why need to add ExecutionType to ContainerInitializationContext/ContainerTerminationContext? Agreed. Even if it is needed in the future, you are right that it should not be part of this patch, so I will go ahead and remove it. > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045943#comment-15045943 ] Wangda Tan commented on YARN-2882: -- [~asuresh], sorry for my late response, few comments: 1) How about renaming ExecutionType to guaranteed/opportunistic, I found in early proposal it is called "opportunistic", not sure why now it is renamed "queueable" 2) Is it necessary to put ExecutionType to ContainerStatus? We can assume AM knows containers' executionTypes. 3) Why need to add ExecutionType to ContainerInitializationContext/ContainerTerminationContext? Should we assume ExecutionType will be only used by YARN services? cc: [~curino] > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[ https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043621#comment-15043621 ] Arun Suresh commented on YARN-2882: --- [~leftnoteasy], is it possible to give this one a review soon ? Since this is essentially API changes which blocks most of the other subtask. Thanks > Add ExecutionType to denote if a container execution is GUARANTEED or > QUEUEABLE > --- > > Key: YARN-2882 > URL: https://issues.apache.org/jira/browse/YARN-2882 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Attachments: YARN-2882-yarn-2877.001.patch, > YARN-2882-yarn-2877.002.patch, yarn-2882.patch > > > This JIRA introduces the notion of container types. > We propose two initial types of containers: guaranteed-start and queueable > containers. > Guaranteed-start are the existing containers, which are allocated by the > central RM and are instantaneously started, once allocated. > Queueable is a new type of container, which allows containers to be queued in > the NM, thus their execution may be arbitrarily delayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2882) Add ExecutionType to denote if a container execution is GUARANTEED or QUEUEABLE
[
https://issues.apache.org/jira/browse/YARN-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029338#comment-15029338
]
Hadoop QA commented on YARN-2882:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
46s {color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s
{color} | {color:green} yarn-2877 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s
{color} | {color:green} yarn-2877 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 37s
{color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
11s {color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
40s {color} | {color:green} yarn-2877 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s
{color} | {color:green} yarn-2877 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 33s
{color} | {color:green} yarn-2877 passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 24s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.8.0_66 with JDK
v1.8.0_66 generated 1 new issues (was 14, now 14). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 41s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn-jdk1.7.0_85 with JDK
v1.7.0_85 generated 1 new issues (was 15, now 15). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s
{color} | {color:red} Patch generated 17 new checkstyle issues in
hadoop-yarn-project/hadoop-yarn (total was 177, now 192). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 17 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 43s
{color} | {color:red} hadoop-yarn-api in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 49m 58s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 0s
{color} | {color:green} hadoop-yarn-common
