[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[ https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309516#comment-14309516 ] Wangda Tan commented on YARN-1449: -- Canceled patch. Protocol changes and implementations in NM side to support change container resource Key: YARN-1449 URL: https://issues.apache.org/jira/browse/YARN-1449 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Wangda Tan (No longer used) Assignee: Wangda Tan Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch, yarn-1449.5.patch As described in YARN-1197, we need add API/implementation changes, 1) Add a changeContainersResources method in ContainerManagementProtocol 2) Can get succeed/failed increased/decreased containers in response of changeContainersResources 3) Add a new decreased containers field in NodeStatus which can help NM notify RM such changes 4) Added changeContainersResources implementation in ContainerManagerImpl 5) Added changes in ContainersMonitorImpl to support change resource limit of containers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308799#comment-14308799
]
Hadoop QA commented on YARN-1449:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618222/yarn-1449.5.patch
against trunk revision 18b2507.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6534//console
This message is automatically generated.
Protocol changes and implementations in NM side to support change container
resource
Key: YARN-1449
URL: https://issues.apache.org/jira/browse/YARN-1449
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.2.0
Reporter: Wangda Tan (No longer used)
Assignee: Wangda Tan (No longer used)
Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch,
yarn-1449.5.patch
As described in YARN-1197, we need add API/implementation changes,
1) Add a changeContainersResources method in ContainerManagementProtocol
2) Can get succeed/failed increased/decreased containers in response of
changeContainersResources
3) Add a new decreased containers field in NodeStatus which can help NM
notify RM such changes
4) Added changeContainersResources implementation in ContainerManagerImpl
5) Added changes in ContainersMonitorImpl to support change resource limit of
containers
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[ https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308786#comment-14308786 ] Arun C Murthy commented on YARN-1449: - Cleaning up stale PA patches. Protocol changes and implementations in NM side to support change container resource Key: YARN-1449 URL: https://issues.apache.org/jira/browse/YARN-1449 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.2.0 Reporter: Wangda Tan (No longer used) Assignee: Wangda Tan (No longer used) Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch, yarn-1449.5.patch As described in YARN-1197, we need add API/implementation changes, 1) Add a changeContainersResources method in ContainerManagementProtocol 2) Can get succeed/failed increased/decreased containers in response of changeContainersResources 3) Add a new decreased containers field in NodeStatus which can help NM notify RM such changes 4) Added changeContainersResources implementation in ContainerManagerImpl 5) Added changes in ContainersMonitorImpl to support change resource limit of containers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845281#comment-13845281
]
Hadoop QA commented on YARN-1449:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618208/yarn-1449.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2644//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/2644//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2644//console
This message is automatically generated.
Protocol changes and implementations in NM side to support change container
resource
Key: YARN-1449
URL: https://issues.apache.org/jira/browse/YARN-1449
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.2.0
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch
As described in YARN-1197, we need add API/implementation changes,
1) Add a changeContainersResources method in ContainerManagementProtocol
2) Can get succeed/failed increased/decreased containers in response of
changeContainersResources
3) Add a new decreased containers field in NodeStatus which can help NM
notify RM such changes
4) Added changeContainersResources implementation in ContainerManagerImpl
5) Added changes in ContainersMonitorImpl to support change resource limit of
containers
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845361#comment-13845361
]
Hadoop QA commented on YARN-1449:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618222/yarn-1449.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2645//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2645//console
This message is automatically generated.
Protocol changes and implementations in NM side to support change container
resource
Key: YARN-1449
URL: https://issues.apache.org/jira/browse/YARN-1449
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.2.0
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: yarn-1449.1.patch, yarn-1449.3.patch, yarn-1449.4.patch,
yarn-1449.5.patch
As described in YARN-1197, we need add API/implementation changes,
1) Add a changeContainersResources method in ContainerManagementProtocol
2) Can get succeed/failed increased/decreased containers in response of
changeContainersResources
3) Add a new decreased containers field in NodeStatus which can help NM
notify RM such changes
4) Added changeContainersResources implementation in ContainerManagerImpl
5) Added changes in ContainersMonitorImpl to support change resource limit of
containers
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843375#comment-13843375
]
Sandy Ryza commented on YARN-1449:
--
Can you do annotations in the same way that was done for YARN-1448.
{code}
+
+ @Override
+public ChangeContainersResourceResponse changeContainersResource(
+ChangeContainersResourceRequest request) throws YarnException,
+IOException {
+ return null;
+}
{code}
Remove tabs
{code}
+ Assert.assertTrue(request.getContainersToDecrease().get(i)
+ .equals(containerToDecrease.get(i)));
{code}
Use assertEquals?
{code}
+ Assert.assertTrue(response.getFailedChangedContainers().get(i)
+ .equals(failChanged.get(i)));
{code}
{code}
+
+ public abstract void setNewDecreasedContainers(
+ ListContainerResourceDecrease decreasedContainers);
+
{code}
I think better to omit new and just call it DecreasedContainers
{code}
+return new IteratorContainerResourceDecreaseProto() {
+
+ IteratorContainerResourceDecrease iter = newDecreasedContainers
+ .iterator();
{code}
Iterator indented too far
{code}
-}
\ No newline at end of file
+}
{code}
False change
{code}
+// first add all decreased containers to failed, we will support it in the
+// future
{code}
What is it that we will support in the future?
{code}
+ // check container's existance
{code}
existence
{code}
+LOG.error(we can only increase size of container in
++ RUNNING state, containerId= + containerId.toString()
++ current state is: + currentState.name());
{code}
I think this would be better as WARN or even an INFO, as it can happen in
normal operation if a container completes while a request is in flight. Also,
capitalize We.
{code}
+
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerState
currentState =
+ context.getContainers().get(containerId).getContainerState();
+ if (currentState !=
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerState.RUNNING)
{
{code}
This looks way over 80 characters
{code}
+ // we can consider this should successfully changed
{code}
We can consider the change successful
{code}
+public class ContainerChangeMonitoringEvent extends ContainersMonitorEvent {
+ private final long vmemLimit;
+ private final long pmemLimit;
{code}
No need to calculate / include the vmem here. The containers monitor should
know about it.
{code}
- private static class ProcessTreeInfo {
+ static class ProcessTreeInfo {
{code}
Unnecessary change
{code}
+public ContainerResourceToBeChanged(ContainerId containerId,
+long vmemLimit, long pmemLimit) {
{code}
vmem limit not needed here either?
{code}
+ LOG.warn(note container not found in trackingContainers,
+ + it maybe already completed, please check, containerId=
+ + c.containerId.toString());
{code}
Container + c.containerId.toString() + that was to be resized is no longer
running.. Also, change this to info.
{code}
+
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.MockResourceCalculatorProcessTree);
{code}
Way over 80 characters. Better to use
MockResourceCalculatorProcessTree.class.getName(), because it's resilient to
refactorings.
{code}
+ public void testResourceChange() throws Exception {
{code}
It looks like this test relies on sleeps to wait for events to be handled.
This both makes the test run longer and can cause flaky failures. I think there
is a way to actually wait for the events to be completed. I don't know it off
the top of my head but can look it up for you if that would be helpful.
{code}
+ // add a stub here, TODO
+ @Override
+ public ChangeContainersResourceResponse changeContainersResource(
{code}
What do you mean by add a stub here?
Protocol changes and implementations in NM side to support change container
resource
Key: YARN-1449
URL: https://issues.apache.org/jira/browse/YARN-1449
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.2.0
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: yarn-1449.1.patch, yarn-1449.3.patch
As described in YARN-1197, we need add API/implementation changes,
1) Add a changeContainersResources method in ContainerManagementProtocol
2) Can get succeed/failed increased/decreased containers in response of
changeContainersResources
3) Add a new decreased containers field in NodeStatus which can help NM
notify RM such changes
4) Added changeContainersResources implementation in ContainerManagerImpl
5) Added changes in
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[
https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843393#comment-13843393
]
Hadoop QA commented on YARN-1449:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617532/yarn-1449.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2629//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2629//console
This message is automatically generated.
Protocol changes and implementations in NM side to support change container
resource
Key: YARN-1449
URL: https://issues.apache.org/jira/browse/YARN-1449
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.2.0
Reporter: Wangda Tan
Assignee: Wangda Tan
Attachments: yarn-1449.1.patch, yarn-1449.3.patch
As described in YARN-1197, we need add API/implementation changes,
1) Add a changeContainersResources method in ContainerManagementProtocol
2) Can get succeed/failed increased/decreased containers in response of
changeContainersResources
3) Add a new decreased containers field in NodeStatus which can help NM
notify RM such changes
4) Added changeContainersResources implementation in ContainerManagerImpl
5) Added changes in ContainersMonitorImpl to support change resource limit of
containers
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1449) Protocol changes and implementations in NM side to support change container resource
[ https://issues.apache.org/jira/browse/YARN-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843808#comment-13843808 ] Wangda Tan commented on YARN-1449: -- Hi Sandy, Thanks so much to give me such comments, that's really helpful, I will update this later. Protocol changes and implementations in NM side to support change container resource Key: YARN-1449 URL: https://issues.apache.org/jira/browse/YARN-1449 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.2.0 Reporter: Wangda Tan Assignee: Wangda Tan Attachments: yarn-1449.1.patch, yarn-1449.3.patch As described in YARN-1197, we need add API/implementation changes, 1) Add a changeContainersResources method in ContainerManagementProtocol 2) Can get succeed/failed increased/decreased containers in response of changeContainersResources 3) Add a new decreased containers field in NodeStatus which can help NM notify RM such changes 4) Added changeContainersResources implementation in ContainerManagerImpl 5) Added changes in ContainersMonitorImpl to support change resource limit of containers -- This message was sent by Atlassian JIRA (v6.1.4#6159)
