[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[ https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309516#comment-14309516 ] Wangda Tan commented on YARN-1449: -- Canceled patch. > Protocol changes and implementations in NM side to support change container > resource > > > Key: YARN-1449 > URL: https://issues.apache.org/jira/browse/YARN-1449 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Wangda Tan (No longer used) >Assignee: Wangda Tan > Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch, > yarn-1449.5.patch > > > As described in YARN-1197, we need add API/implementation changes, > 1) Add a "changeContainersResources" method in ContainerManagementProtocol > 2) Can get succeed/failed increased/decreased containers in response of > "changeContainersResources" > 3) Add a "new decreased containers" field in NodeStatus which can help NM > notify RM such changes > 4) Added changeContainersResources implementation in ContainerManagerImpl > 5) Added changes in ContainersMonitorImpl to support change resource limit of > containers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308799#comment-14308799
]
Hadoop QA commented on YARN-1449:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618222/yarn-1449.5.patch
against trunk revision 18b2507.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6534//console
This message is automatically generated.
> Protocol changes and implementations in NM side to support change container
> resource
>
>
> Key: YARN-1449
> URL: https://issues.apache.org/jira/browse/YARN-1449
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Wangda Tan (No longer used)
>Assignee: Wangda Tan (No longer used)
> Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch,
> yarn-1449.5.patch
>
>
> As described in YARN-1197, we need add API/implementation changes,
> 1) Add a "changeContainersResources" method in ContainerManagementProtocol
> 2) Can get succeed/failed increased/decreased containers in response of
> "changeContainersResources"
> 3) Add a "new decreased containers" field in NodeStatus which can help NM
> notify RM such changes
> 4) Added changeContainersResources implementation in ContainerManagerImpl
> 5) Added changes in ContainersMonitorImpl to support change resource limit of
> containers
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[ https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308786#comment-14308786 ] Arun C Murthy commented on YARN-1449: - Cleaning up stale PA patches. > Protocol changes and implementations in NM side to support change container > resource > > > Key: YARN-1449 > URL: https://issues.apache.org/jira/browse/YARN-1449 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.2.0 >Reporter: Wangda Tan (No longer used) >Assignee: Wangda Tan (No longer used) > Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch, > yarn-1449.5.patch > > > As described in YARN-1197, we need add API/implementation changes, > 1) Add a "changeContainersResources" method in ContainerManagementProtocol > 2) Can get succeed/failed increased/decreased containers in response of > "changeContainersResources" > 3) Add a "new decreased containers" field in NodeStatus which can help NM > notify RM such changes > 4) Added changeContainersResources implementation in ContainerManagerImpl > 5) Added changes in ContainersMonitorImpl to support change resource limit of > containers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845361#comment-13845361
]
Hadoop QA commented on YARN-1449:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618222/yarn-1449.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2645//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2645//console
This message is automatically generated.
> Protocol changes and implementations in NM side to support change container
> resource
>
>
> Key: YARN-1449
> URL: https://issues.apache.org/jira/browse/YARN-1449
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch,
> yarn-1449.5.patch
>
>
> As described in YARN-1197, we need add API/implementation changes,
> 1) Add a "changeContainersResources" method in ContainerManagementProtocol
> 2) Can get succeed/failed increased/decreased containers in response of
> "changeContainersResources"
> 3) Add a "new decreased containers" field in NodeStatus which can help NM
> notify RM such changes
> 4) Added changeContainersResources implementation in ContainerManagerImpl
> 5) Added changes in ContainersMonitorImpl to support change resource limit of
> containers
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845281#comment-13845281
]
Hadoop QA commented on YARN-1449:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618208/yarn-1449.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2644//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2644//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2644//console
This message is automatically generated.
> Protocol changes and implementations in NM side to support change container
> resource
>
>
> Key: YARN-1449
> URL: https://issues.apache.org/jira/browse/YARN-1449
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch
>
>
> As described in YARN-1197, we need add API/implementation changes,
> 1) Add a "changeContainersResources" method in ContainerManagementProtocol
> 2) Can get succeed/failed increased/decreased containers in response of
> "changeContainersResources"
> 3) Add a "new decreased containers" field in NodeStatus which can help NM
> notify RM such changes
> 4) Added changeContainersResources implementation in ContainerManagerImpl
> 5) Added changes in ContainersMonitorImpl to support change resource limit of
> containers
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[ https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843808#comment-13843808 ] Wangda Tan commented on YARN-1449: -- Hi Sandy, Thanks so much to give me such comments, that's really helpful, I will update this later. > Protocol changes and implementations in NM side to support change container > resource > > > Key: YARN-1449 > URL: https://issues.apache.org/jira/browse/YARN-1449 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.2.0 >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: yarn-1449.1.patch, yarn-1449.3.patch > > > As described in YARN-1197, we need add API/implementation changes, > 1) Add a "changeContainersResources" method in ContainerManagementProtocol > 2) Can get succeed/failed increased/decreased containers in response of > "changeContainersResources" > 3) Add a "new decreased containers" field in NodeStatus which can help NM > notify RM such changes > 4) Added changeContainersResources implementation in ContainerManagerImpl > 5) Added changes in ContainersMonitorImpl to support change resource limit of > containers -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843393#comment-13843393
]
Hadoop QA commented on YARN-1449:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617532/yarn-1449.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2629//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2629//console
This message is automatically generated.
> Protocol changes and implementations in NM side to support change container
> resource
>
>
> Key: YARN-1449
> URL: https://issues.apache.org/jira/browse/YARN-1449
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: yarn-1449.1.patch, yarn-1449.3.patch
>
>
> As described in YARN-1197, we need add API/implementation changes,
> 1) Add a "changeContainersResources" method in ContainerManagementProtocol
> 2) Can get succeed/failed increased/decreased containers in response of
> "changeContainersResources"
> 3) Add a "new decreased containers" field in NodeStatus which can help NM
> notify RM such changes
> 4) Added changeContainersResources implementation in ContainerManagerImpl
> 5) Added changes in ContainersMonitorImpl to support change resource limit of
> containers
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843375#comment-13843375
]
Sandy Ryza commented on YARN-1449:
--
Can you do annotations in the same way that was done for YARN-1448.
{code}
+
+ @Override
+public ChangeContainersResourceResponse changeContainersResource(
+ChangeContainersResourceRequest request) throws YarnException,
+IOException {
+ return null;
+}
{code}
Remove tabs
{code}
+ Assert.assertTrue(request.getContainersToDecrease().get(i)
+ .equals(containerToDecrease.get(i)));
{code}
Use assertEquals?
{code}
+ Assert.assertTrue(response.getFailedChangedContainers().get(i)
+ .equals(failChanged.get(i)));
{code}
{code}
+
+ public abstract void setNewDecreasedContainers(
+ List decreasedContainers);
+
{code}
I think better to omit new and just call it DecreasedContainers
{code}
+return new Iterator() {
+
+ Iterator iter = newDecreasedContainers
+ .iterator();
{code}
Iterator indented too far
{code}
-}
\ No newline at end of file
+}
{code}
False change
{code}
+// first add all decreased containers to failed, we will support it in the
+// future
{code}
What is it that we will support in the future?
{code}
+ // check container's existance
{code}
existence
{code}
+LOG.error("we can only increase size of container in"
++ " RUNNING state, containerId=" + containerId.toString()
++ " current state is:" + currentState.name());
{code}
I think this would be better as WARN or even an INFO, as it can happen in
normal operation if a container completes while a request is in flight. Also,
capitalize "We".
{code}
+
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerState
currentState =
+ context.getContainers().get(containerId).getContainerState();
+ if (currentState !=
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerState.RUNNING)
{
{code}
This looks way over 80 characters
{code}
+ // we can consider this should successfully changed
{code}
"We can consider the change successful"
{code}
+public class ContainerChangeMonitoringEvent extends ContainersMonitorEvent {
+ private final long vmemLimit;
+ private final long pmemLimit;
{code}
No need to calculate / include the vmem here. The containers monitor should
know about it.
{code}
- private static class ProcessTreeInfo {
+ static class ProcessTreeInfo {
{code}
Unnecessary change
{code}
+public ContainerResourceToBeChanged(ContainerId containerId,
+long vmemLimit, long pmemLimit) {
{code}
vmem limit not needed here either?
{code}
+ LOG.warn("note container not found in trackingContainers, "
+ + "it maybe already completed, please check, containerId="
+ + c.containerId.toString());
{code}
"Container " + c.containerId.toString() + " that was to be resized is no longer
running.". Also, change this to info.
{code}
+
"org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.MockResourceCalculatorProcessTree");
{code}
Way over 80 characters. Better to use
MockResourceCalculatorProcessTree.class.getName(), because it's resilient to
refactorings.
{code}
+ public void testResourceChange() throws Exception {
{code}
It looks like this test relies on sleeps to wait for events to be handled.
This both makes the test run longer and can cause flaky failures. I think there
is a way to actually wait for the events to be completed. I don't know it off
the top of my head but can look it up for you if that would be helpful.
{code}
+ // add a stub here, TODO
+ @Override
+ public ChangeContainersResourceResponse changeContainersResource(
{code}
What do you mean by add a stub here?
> Protocol changes and implementations in NM side to support change container
> resource
>
>
> Key: YARN-1449
> URL: https://issues.apache.org/jira/browse/YARN-1449
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.2.0
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: yarn-1449.1.patch, yarn-1449.3.patch
>
>
> As described in YARN-1197, we need add API/implementation changes,
> 1) Add a "changeContainersResources" method in ContainerManagementProtocol
> 2) Can get succeed/failed increased/decreased containers in response of
> "changeContainersResources"
> 3) Add a "new decreased containers" field in NodeStatus which can help NM
> notify RM such changes
> 4) Added changeContainersResources implementation in ContainerManagerImpl
> 5) Added changes in ContainersMonitorImpl to support c
