[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295190#comment-14295190 ] Hudson commented on YARN-3086: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2019 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2019/]) YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. Contributed by Robert Metzger. (ozawa: rev f56da3ce040b16582ce8153df0d7cea00becd843) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Affects Versions: 2.6.0 Reporter: Robert Metzger Assignee: Robert Metzger Priority: Minor Fix For: 2.7.0 Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295177#comment-14295177 ] Hudson commented on YARN-3086: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #84 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/84/]) YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. Contributed by Robert Metzger. (ozawa: rev f56da3ce040b16582ce8153df0d7cea00becd843) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Affects Versions: 2.6.0 Reporter: Robert Metzger Assignee: Robert Metzger Priority: Minor Fix For: 2.7.0 Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295033#comment-14295033 ] Hudson commented on YARN-3086: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #87 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/87/]) YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. Contributed by Robert Metzger. (ozawa: rev f56da3ce040b16582ce8153df0d7cea00becd843) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Affects Versions: 2.6.0 Reporter: Robert Metzger Assignee: Robert Metzger Priority: Minor Fix For: 2.7.0 Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295051#comment-14295051 ] Hudson commented on YARN-3086: -- FAILURE: Integrated in Hadoop-Yarn-trunk #821 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/821/]) YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. Contributed by Robert Metzger. (ozawa: rev f56da3ce040b16582ce8153df0d7cea00becd843) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Affects Versions: 2.6.0 Reporter: Robert Metzger Assignee: Robert Metzger Priority: Minor Fix For: 2.7.0 Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295241#comment-14295241 ] Hudson commented on YARN-3086: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #88 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/88/]) YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. Contributed by Robert Metzger. (ozawa: rev f56da3ce040b16582ce8153df0d7cea00becd843) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/CHANGES.txt Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Affects Versions: 2.6.0 Reporter: Robert Metzger Assignee: Robert Metzger Priority: Minor Fix For: 2.7.0 Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293666#comment-14293666 ] Tsuyoshi OZAWA commented on YARN-3086: -- +1 Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Reporter: Robert Metzger Priority: Minor Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293672#comment-14293672 ] Tsuyoshi OZAWA commented on YARN-3086: -- Committed this to trunk and branch-2. Thanks [~rmetzger] for reporting and contribution. Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Reporter: Robert Metzger Priority: Minor Fix For: 2.7.0 Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293734#comment-14293734 ] Tsuyoshi OZAWA commented on YARN-3086: -- [~rmetzger], I added you as a YARN contributor. Looking forward your more contributions! Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Affects Versions: 2.6.0 Reporter: Robert Metzger Assignee: Robert Metzger Priority: Minor Fix For: 2.7.0 Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293679#comment-14293679 ] Hudson commented on YARN-3086: -- FAILURE: Integrated in Hadoop-trunk-Commit #6938 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6938/]) YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. Contributed by Robert Metzger. (ozawa: rev f56da3ce040b16582ce8153df0d7cea00becd843) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * hadoop-yarn-project/CHANGES.txt Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Affects Versions: 2.6.0 Reporter: Robert Metzger Priority: Minor Fix For: 2.7.0 Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293684#comment-14293684 ] Robert Metzger commented on YARN-3086: -- Thank you for merging! Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Affects Versions: 2.6.0 Reporter: Robert Metzger Priority: Minor Fix For: 2.7.0 Attachments: YARN-3086-2.patch, YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291140#comment-14291140
]
Hadoop QA commented on YARN-3086:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694425/YARN-3086-2.patch
against trunk revision 3703965.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6414//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6414//console
This message is automatically generated.
Make NodeManager memory configurable in MiniYARNCluster
---
Key: YARN-3086
URL: https://issues.apache.org/jira/browse/YARN-3086
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: Robert Metzger
Priority: Minor
Attachments: YARN-3086-2.patch, YARN-3086.patch
Apache Flink has a build-in YARN client to deploy it to YARN clusters.
Recently, we added more tests for the client, using the MiniYARNCluster.
One of the tests is requesting more containers than available. This test
works well on machines with enough memory, but on travis-ci (our test
environment), the available main memory is limited to 3 GB.
Therefore, I want to set custom amount of memory for each NodeManager.
Right now, the NodeManager memory is hardcoded to 4GB.
As discussed on the yarn-dev list, I'm going to create a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289546#comment-14289546
]
Robert Metzger commented on YARN-3086:
--
It seems that even on trunk tests are failing in the
hadoop-yarn-server-resourcemanager package. Looks like its pretty hard to
verify if my change is breaking anything.
I'm uploading an updated patch in a few hours...
{code}
Failed tests:
TestAMRestart.testRMAppAttemptFailuresValidityInterval:630 AppAttempt state
is not correct (timedout) expected:ALLOCATED but was:SCHEDULED
TestAMRestart.testShouldNotCountFailureToMaxAttemptRetry:405 AppAttempt state
is not correct (timedout) expected:ALLOCATED but was:SCHEDULED
TestClientRMTokens.testShortCircuitRenewCancelDifferentHostSamePort:316-checkShortCircuitRenewCancel:363
expected:getProxy but was:null
TestClientRMTokens.testShortCircuitRenewCancelDifferentHostDifferentPort:327-checkShortCircuitRenewCancel:363
expected:getProxy but was:null
TestClientRMTokens.testShortCircuitRenewCancelSameHostDifferentPort:305-checkShortCircuitRenewCancel:363
expected:getProxy but was:null
TestRMRestart.testQueueMetricsOnRMRestart:1812-assertQueueMetrics:1837
expected:2 but was:1
TestRMRestart.testRMRestartGetApplicationList:965
Wanted but not invoked:
rMAppManager.logApplicationSummary(
isA(org.apache.hadoop.yarn.api.records.ApplicationId)
);
- at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:965)
However, there were other interactions with this mock:
- at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1188)
- at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1188)
- at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1188)
- at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1188)
TestContainerResourceUsage.testUsageAfterAMRestartWithMultipleContainers:252-amRestartTests:393
Unexcpected MemorySeconds value expected:-1456158548889 but was:3265
Tests in error:
TestClientRMTokens.testShortCircuitRenewCancel:285-checkShortCircuitRenewCancel:353
» NullPointer
TestClientRMTokens.testShortCircuitRenewCancelWildcardAddress:294-checkShortCircuitRenewCancel:353
» NullPointer
TestAMAuthorization.testUnauthorizedAccess:273 » UnknownHost Invalid host
name...
TestAMAuthorization.testUnauthorizedAccess:273 » UnknownHost Invalid host
name...
{code}
Make NodeManager memory configurable in MiniYARNCluster
---
Key: YARN-3086
URL: https://issues.apache.org/jira/browse/YARN-3086
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: Robert Metzger
Priority: Minor
Attachments: YARN-3086.patch
Apache Flink has a build-in YARN client to deploy it to YARN clusters.
Recently, we added more tests for the client, using the MiniYARNCluster.
One of the tests is requesting more containers than available. This test
works well on machines with enough memory, but on travis-ci (our test
environment), the available main memory is limited to 3 GB.
Therefore, I want to set custom amount of memory for each NodeManager.
Right now, the NodeManager memory is hardcoded to 4GB.
As discussed on the yarn-dev list, I'm going to create a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290384#comment-14290384
]
Hadoop QA commented on YARN-3086:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694155/YARN-3086.patch
against trunk revision 8f26d5a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6404//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6404//console
This message is automatically generated.
Make NodeManager memory configurable in MiniYARNCluster
---
Key: YARN-3086
URL: https://issues.apache.org/jira/browse/YARN-3086
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: Robert Metzger
Priority: Minor
Attachments: YARN-3086.patch
Apache Flink has a build-in YARN client to deploy it to YARN clusters.
Recently, we added more tests for the client, using the MiniYARNCluster.
One of the tests is requesting more containers than available. This test
works well on machines with enough memory, but on travis-ci (our test
environment), the available main memory is limited to 3 GB.
Therefore, I want to set custom amount of memory for each NodeManager.
Right now, the NodeManager memory is hardcoded to 4GB.
As discussed on the yarn-dev list, I'm going to create a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290490#comment-14290490 ] Tsuyoshi OZAWA commented on YARN-3086: -- [~rmetzger] Please check whether your tests fail without your patch. If they fails without your patch, it can be environment-depend problem. In that case, you can throw a patch regardless of the failures. Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Reporter: Robert Metzger Priority: Minor Attachments: YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289349#comment-14289349
]
Tsuyoshi OZAWA commented on YARN-3086:
--
{code}
I'm was not able to run any the tests because the tests in trunk don't seem
to pass
{code}
The intermittent test failure can be observed currently for some reasons - port
conflict, shortage of resource, and timing issues. We should fix them, but the
problems are still there. I recommend you to run test under the related
directory - in this case, I recommend you to run tests under
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
instead of root directory.
{code}
Am I supposed to develop against a version specific branch?
{code}
The develop branch is trunk, so I recommend you to develop on trunk except the
problem is branch-specific problem.
Make NodeManager memory configurable in MiniYARNCluster
---
Key: YARN-3086
URL: https://issues.apache.org/jira/browse/YARN-3086
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: Robert Metzger
Priority: Minor
Attachments: YARN-3086.patch
Apache Flink has a build-in YARN client to deploy it to YARN clusters.
Recently, we added more tests for the client, using the MiniYARNCluster.
One of the tests is requesting more containers than available. This test
works well on machines with enough memory, but on travis-ci (our test
environment), the available main memory is limited to 3 GB.
Therefore, I want to set custom amount of memory for each NodeManager.
Right now, the NodeManager memory is hardcoded to 4GB.
As discussed on the yarn-dev list, I'm going to create a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289356#comment-14289356
]
Tsuyoshi OZAWA commented on YARN-3086:
--
{quote}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Oops, maybe
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
is better.
{quote}
This lines are typo. Please ignore them.
Make NodeManager memory configurable in MiniYARNCluster
---
Key: YARN-3086
URL: https://issues.apache.org/jira/browse/YARN-3086
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: Robert Metzger
Priority: Minor
Attachments: YARN-3086.patch
Apache Flink has a build-in YARN client to deploy it to YARN clusters.
Recently, we added more tests for the client, using the MiniYARNCluster.
One of the tests is requesting more containers than available. This test
works well on machines with enough memory, but on travis-ci (our test
environment), the available main memory is limited to 3 GB.
Therefore, I want to set custom amount of memory for each NodeManager.
Right now, the NodeManager memory is hardcoded to 4GB.
As discussed on the yarn-dev list, I'm going to create a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289355#comment-14289355
]
Tsuyoshi OZAWA commented on YARN-3086:
--
{code}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
{code}
Oops, maybe
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
is better.
{code}
cd hadoop # change directory to
mvn clean install -DskipTests # compile and install related jars into local
repository
cd
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
mvn test # launch test for hadoop-yarn-server-resourcemanager
{code}
It can take 1 hour or more.If you suffer the waiting time, you can skip to
launch the test since I'll help you to submit Jenkins CI with your patch.
Make NodeManager memory configurable in MiniYARNCluster
---
Key: YARN-3086
URL: https://issues.apache.org/jira/browse/YARN-3086
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: Robert Metzger
Priority: Minor
Attachments: YARN-3086.patch
Apache Flink has a build-in YARN client to deploy it to YARN clusters.
Recently, we added more tests for the client, using the MiniYARNCluster.
One of the tests is requesting more containers than available. This test
works well on machines with enough memory, but on travis-ci (our test
environment), the available main memory is limited to 3 GB.
Therefore, I want to set custom amount of memory for each NodeManager.
Right now, the NodeManager memory is hardcoded to 4GB.
As discussed on the yarn-dev list, I'm going to create a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[ https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289360#comment-14289360 ] Robert Metzger commented on YARN-3086: -- Thank you for all the help! I've updated the code... and I'm now trying to execute the tests with your instructions. If that works out, I'll upload an updated version of the patch. Make NodeManager memory configurable in MiniYARNCluster --- Key: YARN-3086 URL: https://issues.apache.org/jira/browse/YARN-3086 Project: Hadoop YARN Issue Type: Improvement Components: test Reporter: Robert Metzger Priority: Minor Attachments: YARN-3086.patch Apache Flink has a build-in YARN client to deploy it to YARN clusters. Recently, we added more tests for the client, using the MiniYARNCluster. One of the tests is requesting more containers than available. This test works well on machines with enough memory, but on travis-ci (our test environment), the available main memory is limited to 3 GB. Therefore, I want to set custom amount of memory for each NodeManager. Right now, the NodeManager memory is hardcoded to 4GB. As discussed on the yarn-dev list, I'm going to create a patch for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289338#comment-14289338
]
Tsuyoshi OZAWA commented on YARN-3086:
--
[~rmetzger] Great! At first, I'd like to comment about your patch:
How about making the default value 4 * 1024? As a result, we can remove if
statement and simplify the code path.
{code}
DEFAULT_YARN_MINICLUSTER_NM_PMEM_MB = -1;
{code}
Could you update the point?
Make NodeManager memory configurable in MiniYARNCluster
---
Key: YARN-3086
URL: https://issues.apache.org/jira/browse/YARN-3086
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: Robert Metzger
Priority: Minor
Attachments: YARN-3086.patch
Apache Flink has a build-in YARN client to deploy it to YARN clusters.
Recently, we added more tests for the client, using the MiniYARNCluster.
One of the tests is requesting more containers than available. This test
works well on machines with enough memory, but on travis-ci (our test
environment), the available main memory is limited to 3 GB.
Therefore, I want to set custom amount of memory for each NodeManager.
Right now, the NodeManager memory is hardcoded to 4GB.
As discussed on the yarn-dev list, I'm going to create a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288803#comment-14288803
]
Tsuyoshi OZAWA commented on YARN-3086:
--
The following is a suggestion from [~rmetzger] in yarn-dev mailing list:
{quote}
If you want to maintain the current value of 4GB as the default value, I
probably need to introduce a new configuration value, similar to:
public static final String YARN_MINICLUSTER_CONTROL_RESOURCE_MONITORING =
YARN_PREFIX + minicluster.control-resource-monitoring;
Or is there another approach you would prefer?
I'll add a patch to the JIRA once I've found time to fix it.
{quote}
Make NodeManager memory configurable in MiniYARNCluster
---
Key: YARN-3086
URL: https://issues.apache.org/jira/browse/YARN-3086
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: Robert Metzger
Priority: Minor
Apache Flink has a build-in YARN client to deploy it to YARN clusters.
Recently, we added more tests for the client, using the MiniYARNCluster.
One of the tests is requesting more containers than available. This test
works well on machines with enough memory, but on travis-ci (our test
environment), the available main memory is limited to 3 GB.
Therefore, I want to set custom amount of memory for each NodeManager.
Right now, the NodeManager memory is hardcoded to 4GB.
As discussed on the yarn-dev list, I'm going to create a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3086) Make NodeManager memory configurable in MiniYARNCluster
[
https://issues.apache.org/jira/browse/YARN-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288817#comment-14288817
]
Tsuyoshi OZAWA commented on YARN-3086:
--
[~rmetzger] Yeah, the approach you suggested looks good to me basically. We've
already had YARN_MC_PREFIX, so please use it.
{code}
public static final String YARN_MC_PREFIX = YARN_PREFIX + minicluster.;
{code}
I think new configuration name should be YARN_MINICLUSTER_NM_PMEM_MB
straightforwardly.
Make NodeManager memory configurable in MiniYARNCluster
---
Key: YARN-3086
URL: https://issues.apache.org/jira/browse/YARN-3086
Project: Hadoop YARN
Issue Type: Improvement
Components: test
Reporter: Robert Metzger
Priority: Minor
Apache Flink has a build-in YARN client to deploy it to YARN clusters.
Recently, we added more tests for the client, using the MiniYARNCluster.
One of the tests is requesting more containers than available. This test
works well on machines with enough memory, but on travis-ci (our test
environment), the available main memory is limited to 3 GB.
Therefore, I want to set custom amount of memory for each NodeManager.
Right now, the NodeManager memory is hardcoded to 4GB.
As discussed on the yarn-dev list, I'm going to create a patch for this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
