[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020350#comment-14020350
]
Bikas Saha commented on YARN-2091:
--
How about naming it hasDefaultExitCode() and directly using
ContainerExitStatus.INVALID instead of creating a new member variable.
{code}+
+ private boolean isDefaultExitCode() {
+return (this.exitCode == DEFAULT_EXIT_CODE);
+ }{code}
Can we rename this to getContainerExitStatus() so that its clear that we are
getting a ContainerExitStatus value.
{code}+ public int getReason() {
+return this.reason;
+ }{code}
I would suggest the following names
KILL_AM_STOP_CONTAINER - KILLED_BY_APPMASTER - Container was terminated by
stop request by the app master
KILL_BY_RESOURCEMANAGER - KILLED_BY_RESOURCEMANAGER - Container was terminated
by the resource manager.
KILL_FINISHED_APPMASTER - KILLED_AFTER_APP_COMPLETION - Container was
terminated after the application finished
KILL_EXCEEDED_PMEM - KILLED_EXCEEDED_PMEM - Container terminated because of
exceeding allocated physical memory
KILL_EXCEEDED_VMEM - KILLED_EXCEEDED_VMEM - Container terminated because of
exceeding allocated virtual memory
Spurious edit?
{code}- //if the current state is NEW it means the CONTAINER_INIT was
never
+ //if the current state is NEW it means the CONTAINER_INIT was never{code}
I am +1 after these comments. [~vinodkv] Do you have any further comments?
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch, YARN-2091.7.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020517#comment-14020517
]
Hadoop QA commented on YARN-2091:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12648748/YARN-2091.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3931//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3931//console
This message is automatically generated.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch, YARN-2091.7.patch,
YARN-2091.8.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020562#comment-14020562 ] Tsuyoshi OZAWA commented on YARN-2091: -- [~vinodkv], please let me know if you have further comments. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch, YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch, YARN-2091.7.patch, YARN-2091.8.patch Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018336#comment-14018336
]
Hadoop QA commented on YARN-2091:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12648400/YARN-2091.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/3911//artifact/trunk/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3911//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3911//console
This message is automatically generated.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch, YARN-2091.7.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018394#comment-14018394
]
Tsuyoshi OZAWA commented on YARN-2091:
--
* Added isDefaultExitCode() method to {{ContainerImpl}} and updated to use it.
ContainerImpl#exitCode is initialized as ContainerExitStatus.INVALID already,
so I did not change about that.
* Updated docs/comments like this:
{code}
+ /**
+ * Containers killed by AppMaster's issuing RPC(
+ * {@link org.apache.hadoop.yarn.api.ContainerManagementProtocol#
+ * stopContainers(org.apache.hadoop.yarn.api.protocolrecords.
+ * StopContainersRequest)}) explicitly.
+ */
+ public static final int KILL_AM_STOP_CONTAINER = -105;
+
+ /**
+ * Containers killed by ResourceManager's request or resync between
+ * ResourceManager and NodeManager.
+ */
+ public static final int KILL_BY_RESOURCEMANAGER = -106;
+
+ /**
+ * Containers killed by ResourceManager, due to accomplish of applications.
+ */
+ public static final int KILL_FINISHED_APPMASTER = -107;
{code}
[~bikassaha], please review it.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch, YARN-2091.7.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015611#comment-14015611
]
Bikas Saha commented on YARN-2091:
--
Can this miss a case when the exitCode has not been set (eg when the container
crashes on its own)? Should we check if the exitCode has already been set (eg.
via a kill event) and if its not set then set it from exitEvent? How can we
check if the exitCode has not been set? Maybe have some uninitialized/invalid
default value.
{code}@@ -829,7 +829,6 @@ public void transition(ContainerImpl container,
ContainerEvent event) {
@Override
public void transition(ContainerImpl container, ContainerEvent event) {
ContainerExitEvent exitEvent = (ContainerExitEvent) event;
- container.exitCode = exitEvent.getExitCode();{code}
The new exit status code need better comments/docs. E.g. what is the difference
between to 2 new appmaster related exit status. Is kill_by_resourcemanager a
generic value that can be replaced later on by a more specific reason like
preempted?
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016019#comment-14016019
]
Tsuyoshi OZAWA commented on YARN-2091:
--
[~bikassaha], thank you for the comments.
{quote}
Should we check if the exitCode has already been set (eg. via a kill event)
and if its not set then set it from exitEvent? How can we check if the exitCode
has not been set? Maybe have some uninitialized/invalid default value.
{quote}
IIUC, we can distinguish its set value from default value by checking whether
exitCode is ContainerExitStatus.INVALID because default value of {{exitCode}}
is ContainerExitStatus.INVALID. Do you have any comment about this?
{code}
if (container.exitCode == ContainerExitStatus.INVALID) {
container.exitCode = exitEvent.getExitCode();
}
{code}
About the new exit status, I'll update comments in the next patch.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016081#comment-14016081 ] Bikas Saha commented on YARN-2091: -- If we are sure that the default value is set in the code to ContainerExitStatus.INVALID then sounds good. Given that ContainerExitStatus.INVALID == 5000 we have to explicitly initialize with that value since Java will default to 0. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch, YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014793#comment-14014793
]
Hadoop QA commented on YARN-2091:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647781/YARN-2091.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3877//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3877//console
This message is automatically generated.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch, YARN-2091.5.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014870#comment-14014870
]
Hadoop QA commented on YARN-2091:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647799/YARN-2091.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3880//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3880//console
This message is automatically generated.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014874#comment-14014874
]
Tsuyoshi OZAWA commented on YARN-2091:
--
Updated points:
* Introduced new exit code - KILL_EXCEEDED_VMEM, KILL_EXCEEDED_PME,
KILL_BY_APPMASTER, KILL_BY_RESOURCEMANAGER, KILL_FINISHED_APPMASTER. Naming and
type of error code are discussion points.
* Changed ContainerKillEvent to take ContainerExitStatus as one of
constructor's arguments.
* Removed isAMAware().
* Removed to call {{ContainerExitEvent.getExitCode()}} from
ContainerKilledTransition.
[~bikassaha], could you take a look?
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch, YARN-2091.5.patch, YARN-2091.6.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013876#comment-14013876
]
Tsuyoshi OZAWA commented on YARN-2091:
--
Thank you for the suggestion, Sandy.
{quote}
ContainerExitStatus should stay an int. While ContainerStatus.getExitStatus is
technically marked Unstable, I'm sure changing this would break some
applications.
{quote}
I agree with this. In the latest patch, ContainerExitStatus stays an int.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013890#comment-14013890
]
Tsuyoshi OZAWA commented on YARN-2091:
--
Changelogs in v4:
* Added KILL_EXCEEDED_PMEM, KILL_EXCEEDED_VMEM to ContainerExitStatus.
* Updated ContainersMonitorImpl for dispatching
KILL_EXCEEDED_VMEM/KILL_EXCEEDED_PMEM.
* If the exit reason is AM-aware({{ContainerExitStatus#isAMAware()}}), pass it
to app masters. Otherwise, the exit reason is converted into
ExitCode.TERMINATED.getExitCode() for backward compatibility. AM-Aware events
are DISKS_FAILED, KILL_EXCEEDED_PMEM, KILL_EXCEEDED_VMEM currently.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013926#comment-14013926 ] Tsuyoshi OZAWA commented on YARN-2091: -- Jenkins passed last night. It's ready for review. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch, YARN-2091.4.patch Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014079#comment-14014079 ] Bikas Saha commented on YARN-2091: -- Why is isAMAware needed. All values in ContainerExitStatus are public and hence user code should already be aware of them. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch, YARN-2091.4.patch Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014245#comment-14014245 ] Tsuyoshi OZAWA commented on YARN-2091: -- Make sense. I found some test cases that check the exit code as ExitCode.TERMINATION.getCode() and I thought we need to preserve the semantics. These should be fixed, right? Thanks for the clarification. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch, YARN-2091.4.patch Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014284#comment-14014284 ] Bikas Saha commented on YARN-2091: -- We can check all cases of ContainerKillEvent and add new ExitStatus values where it makes sense or use some good default value. If a test needs to change to account for a new value then we should change the test. There may be other cases of exit status being set or tested which are unrelated to container kill event. Those can stay out of the scope of this jira. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch, YARN-2091.4.patch Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012761#comment-14012761
]
Hadoop QA commented on YARN-2091:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647416/YARN-2091.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3858//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3858//console
This message is automatically generated.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012980#comment-14012980
]
Bikas Saha commented on YARN-2091:
--
Instead of having the following if-else code everywhere, can we simply always
use the reason from the kill event. That way if we add a different reason
tomorrow (exceeded disk quota) then we dont have to find all these special
cases.
{code}+ if (killEvent.getReason() ==
ContainerExitStatus.KILL_EXCEEDED_MEMORY) {
+container.exitCode = killEvent.getReason();
+ } else {
+container.exitCode = ExitCode.TERMINATED.getExitCode();
+ }{code}
As an exercise, we can find other cases of ContainerKillEvent and see if new
enums (like exceeded_memory) can be added. Or if a suitable default value can
be found. Clearly, if we are killing then there should be a reason.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013052#comment-14013052 ] Vinod Kumar Vavilapalli commented on YARN-2091: --- Should we differentiate killing due to exceeding pmem and vmem limits separately? Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Attachments: YARN-2091.1.patch, YARN-2091.2.patch Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013071#comment-14013071 ] Bikas Saha commented on YARN-2091: -- That would make sense if YARN would allow specifying pmem and vmem separately in resource request. Without that the information is not actionable. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Attachments: YARN-2091.1.patch, YARN-2091.2.patch Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013093#comment-14013093
]
Tsuyoshi OZAWA commented on YARN-2091:
--
Thanks for the suggestions, Bikas and Vinod. It would make sense too. I'll plan
to separate pmem and vmem events and use kill reasons directly.
I have one discussion point: should we update ContainerExitStatus as Enum
because EnumSets allow us to judge which events should be handled in AM easily?
One concern is backward compatibility. This may be overkill. What do you think?
{code}
public enum ContainerExitStatus {
...
KILL_EXCEEDED_VMEM(-103),
KILL_EXCEEDED_PMEM(-104);
public static EnumSetContainerExitStatus AM_EVENTS = EnumSet.of(
DISKS_FAILED,
PREEMPTED,
KILL_EXCEEDED_VMEM,
KILL_EXCEEDED_PMEM
);
}
{code}
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013102#comment-14013102
]
Tsuyoshi OZAWA commented on YARN-2091:
--
Found that it's simple enough adding methods like isAMAware() to
ContainerExitStatus:
{code}
public class ContainerExitStatus {
public static boolean isAMAware(int exitReason) {
return (exitReason == DISKS_FAILED)
|| (exitReason == KILL_EXCEEDED_PMEM)
|| (exitReason == KILL_EXCEEDED_VMEM);
}
}
{code}
I try to leave ContainerExitStatus just a class not enum for now. Please let me
know if you have suggestions about this.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013119#comment-14013119 ] Sandy Ryza commented on YARN-2091: -- ContainerExitStatus should stay an int. While ContainerStatus.getExitStatus is technically marked Unstable, I'm sure changing this would break some applications. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013150#comment-14013150
]
Hadoop QA commented on YARN-2091:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647485/YARN-2091.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.TestContainer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3859//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3859//console
This message is automatically generated.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013216#comment-14013216
]
Hadoop QA commented on YARN-2091:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647505/YARN-2091.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3862//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3862//console
This message is automatically generated.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch, YARN-2091.2.patch, YARN-2091.3.patch,
YARN-2091.4.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011918#comment-14011918
]
Hadoop QA commented on YARN-2091:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12647261/YARN-2091.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3853//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3853//console
This message is automatically generated.
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Attachments: YARN-2091.1.patch
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010580#comment-14010580 ] Tsuyoshi OZAWA commented on YARN-2091: -- ContainerManagerImpl cannot distinguish the exit reason because ContainersMonitorImpl dispatches ContainerKillEvent without the exit reason currently. I plan to add exit reason to ContainerKillEvent. Please let me know if you have better idea. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010585#comment-14010585 ] Bikas Saha commented on YARN-2091: -- Thats the missing pieces AFAIK. That exit reason needs to be passed along internally through the NM and then on to the RM and AM. Maybe simply directly use ContainerExitStatus instead of a new reason object inside ContainerKillEvent. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[
https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010614#comment-14010614
]
Tsuyoshi OZAWA commented on YARN-2091:
--
Hi Bikas, let me clarify what simply directly use means. I meant to pass exit
reason via ContainerKillEvent like {{ContainerKillEvent(containerId,
ContainerExitStatus.KILL_EXCEEDED_MEMORY, msg)}}. Is this out of way?
Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
---
Key: YARN-2091
URL: https://issues.apache.org/jira/browse/YARN-2091
Project: Hadoop YARN
Issue Type: Task
Reporter: Bikas Saha
Assignee: Tsuyoshi OZAWA
Currently, the AM cannot programmatically determine if the task was killed
due to using excessive memory. The NM kills it without passing this
information in the container status back to the RM. So the AM cannot take any
action here. The jira tracks adding this exit status and passing it from the
NM to the RM and then the AM. In general, there may be other such actions
taken by YARN that are currently opaque to the AM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2091) Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters
[ https://issues.apache.org/jira/browse/YARN-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010630#comment-14010630 ] Bikas Saha commented on YARN-2091: -- We are on the same page. The kill reason is directly a ContainerExitStatus. Add ContainerExitStatus.KILL_EXCEEDED_MEMORY and pass it to app masters --- Key: YARN-2091 URL: https://issues.apache.org/jira/browse/YARN-2091 Project: Hadoop YARN Issue Type: Task Reporter: Bikas Saha Assignee: Tsuyoshi OZAWA Currently, the AM cannot programmatically determine if the task was killed due to using excessive memory. The NM kills it without passing this information in the container status back to the RM. So the AM cannot take any action here. The jira tracks adding this exit status and passing it from the NM to the RM and then the AM. In general, there may be other such actions taken by YARN that are currently opaque to the AM. -- This message was sent by Atlassian JIRA (v6.2#6252)
