[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323840#comment-14323840 ] Hudson commented on HDFS-316: - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/]) HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu Yao via aw) (aw: rev b94c1117a28e996adee68fe0e181eb6f536289f4) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Fix For: 2.7.0 Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch, HDFS-316.4.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318061#comment-14318061 ] Hudson commented on HDFS-316: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #102 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/102/]) HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu Yao via aw) (aw: rev b94c1117a28e996adee68fe0e181eb6f536289f4) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Fix For: 3.0.0 Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch, HDFS-316.4.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318242#comment-14318242 ] Hudson commented on HDFS-316: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2034 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2034/]) HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu Yao via aw) (aw: rev b94c1117a28e996adee68fe0e181eb6f536289f4) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Fix For: 3.0.0 Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch, HDFS-316.4.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318108#comment-14318108 ] Hudson commented on HDFS-316: - SUCCESS: Integrated in Hadoop-Yarn-trunk #836 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/836/]) HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu Yao via aw) (aw: rev b94c1117a28e996adee68fe0e181eb6f536289f4) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java * hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Fix For: 3.0.0 Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch, HDFS-316.4.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318324#comment-14318324 ] Hudson commented on HDFS-316: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #103 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/103/]) HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu Yao via aw) (aw: rev b94c1117a28e996adee68fe0e181eb6f536289f4) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Fix For: 3.0.0 Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch, HDFS-316.4.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318374#comment-14318374 ] Hudson commented on HDFS-316: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2053 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2053/]) HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu Yao via aw) (aw: rev b94c1117a28e996adee68fe0e181eb6f536289f4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Fix For: 3.0.0 Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch, HDFS-316.4.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316457#comment-14316457 ] Hudson commented on HDFS-316: - FAILURE: Integrated in Hadoop-trunk-Commit #7072 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7072/]) HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu Yao via aw) (aw: rev b94c1117a28e996adee68fe0e181eb6f536289f4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Fix For: 3.0.0 Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch, HDFS-316.4.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[
https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316042#comment-14316042
]
Hadoop QA commented on HDFS-316:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697934/HDFS-316.4.patch
against trunk revision 7c6b654.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9532//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9532//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9532//console
This message is automatically generated.
Balancer should run for a configurable # of iterations
--
Key: HDFS-316
URL: https://issues.apache.org/jira/browse/HDFS-316
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.4.1
Reporter: Brian Bockelman
Assignee: Xiaoyu Yao
Priority: Minor
Labels: newbie
Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch,
HDFS-316.3.patch, HDFS-316.4.patch
The balancer currently exits if nothing has changed after 5 iterations.
Our site would like to constantly balance a stream of incoming data; we would
like to be able to set the number of iterations it does nothing for before
exiting; even better would be if we set it to a negative number and could
continuously run this as a daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[
https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315553#comment-14315553
]
Hadoop QA commented on HDFS-316:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697934/HDFS-316.4.patch
against trunk revision 7c6b654.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9525//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9525//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9525//console
This message is automatically generated.
Balancer should run for a configurable # of iterations
--
Key: HDFS-316
URL: https://issues.apache.org/jira/browse/HDFS-316
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.4.1
Reporter: Brian Bockelman
Assignee: Xiaoyu Yao
Priority: Minor
Labels: newbie
Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch,
HDFS-316.3.patch, HDFS-316.4.patch
The balancer currently exits if nothing has changed after 5 iterations.
Our site would like to constantly balance a stream of incoming data; we would
like to be able to set the number of iterations it does nothing for before
exiting; even better would be if we set it to a negative number and could
continuously run this as a daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[
https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315015#comment-14315015
]
Tsz Wo Nicholas Sze commented on HDFS-316:
--
Some minor comments:
- Should maxidleiteration be maxidleiterations? Or simply call it
idleiterations?
- The patch adds two newNameNodeConnectors method. One of them is not need
since it is only used by a newNameNodeConnectors. Another
newNameNodeConnectors is only used in TestMover. So we may reduce two
newNameNodeConnectors methods as below:
{code}
public static ListNameNodeConnector newNameNodeConnectors(
CollectionURI namenodes, String name, Path idPath, Configuration conf,
int maxNotChangedIterations) throws IOException {
final ListNameNodeConnector connectors = new ArrayListNameNodeConnector(
namenodes.size());
for (URI uri : namenodes) {
NameNodeConnector nnc = new NameNodeConnector(name, uri, idPath,
null, conf, maxNotChangedIterations);
nnc.getKeyManager().startBlockKeyUpdater();
connectors.add(nnc);
}
return connectors;
}
public static ListNameNodeConnector newNameNodeConnectors(
MapURI, ListPath namenodes, String name, Path idPath,
Configuration conf) throws IOException {
final ListNameNodeConnector connectors = new ArrayListNameNodeConnector(
namenodes.size());
for (Map.EntryURI, ListPath entry : namenodes.entrySet()) {
NameNodeConnector nnc = new NameNodeConnector(name, entry.getKey(),
idPath, entry.getValue(), conf, DEFAULT_MAX_NOT_CHANGED_ITERATIONS);
nnc.getKeyManager().startBlockKeyUpdater();
connectors.add(nnc);
}
return connectors;
}
{code}
TestMover can be fixed by changing DEFAULT_MAX_NOT_CHANGED_ITERATIONS to public
and passing it in TestMover.
Balancer should run for a configurable # of iterations
--
Key: HDFS-316
URL: https://issues.apache.org/jira/browse/HDFS-316
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.4.1
Reporter: Brian Bockelman
Assignee: Xiaoyu Yao
Priority: Minor
Labels: newbie
Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch,
HDFS-316.3.patch
The balancer currently exits if nothing has changed after 5 iterations.
Our site would like to constantly balance a stream of incoming data; we would
like to be able to set the number of iterations it does nothing for before
exiting; even better would be if we set it to a negative number and could
continuously run this as a daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315025#comment-14315025 ] Tsz Wo Nicholas Sze commented on HDFS-316: -- TestMover can be fixed by changing DEFAULT_MAX_NOT_CHANGED_ITERATIONS to public and passing it in TestMover. Indeed, TestMover should be fixed by changing namenodes from CollectionURI to MapURI, ListPath. Then, the newNameNodeConnectors with CollectionURI is only used by balancer and the one with MapURI, ListPath is only used by Mover. Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314973#comment-14314973 ] Chris Nauroth commented on HDFS-316: Hi Xiaoyu. This patch is looking good. Let's go ahead and update the documentation in HDFSCommands.apt.vm as part of this issue. I expect it will be a small change to add the new argument. Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315645#comment-14315645 ] Chris Nauroth commented on HDFS-316: +1 from me too. It looks like the Jenkins run failed for unrelated reasons, so let's try again. I triggered one more Jenkins run. It's build #9532, currently waiting for an available executor. Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch, HDFS-316.4.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315038#comment-14315038 ] Xiaoyu Yao commented on HDFS-316: - Thanks [~cnauroth] and [~szetszwo] for reviewing the patch. bq. Should maxidleiteration be maxidleiterations? Or simply call it idleiterations? Agree and I think idleiterations is a better name. bq. The patch adds two newNameNodeConnectors method. One of them is not need since it is only used by a newNameNodeConnectors. Another newNameNodeConnectors is only used in TestMover. So we may reduce two newNameNodeConnectors methods as below: Agree and I will update the patch based on that. Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312953#comment-14312953 ] Xiaoyu Yao commented on HDFS-316: - The patch v3 does not cover the documentation of the new -maxidliteration parameter. [~aw], let me know if we should include it in this JIRA or open a separate one for it. Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch, HDFS-316.3.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[
https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309897#comment-14309897
]
Hadoop QA commented on HDFS-316:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697066/HDFS-316.3.patch
against trunk revision 1425e3d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9459//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9459//console
This message is automatically generated.
Balancer should run for a configurable # of iterations
--
Key: HDFS-316
URL: https://issues.apache.org/jira/browse/HDFS-316
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.4.1
Reporter: Brian Bockelman
Assignee: Xiaoyu Yao
Priority: Minor
Labels: newbie
Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch,
HDFS-316.3.patch
The balancer currently exits if nothing has changed after 5 iterations.
Our site would like to constantly balance a stream of incoming data; we would
like to be able to set the number of iterations it does nothing for before
exiting; even better would be if we set it to a negative number and could
continuously run this as a daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[
https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305912#comment-14305912
]
Hadoop QA commented on HDFS-316:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672071/HDFS-316.2.patch
against trunk revision 40a4157.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9432//console
This message is automatically generated.
Balancer should run for a configurable # of iterations
--
Key: HDFS-316
URL: https://issues.apache.org/jira/browse/HDFS-316
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.4.1
Reporter: Brian Bockelman
Assignee: Xiaoyu Yao
Priority: Minor
Labels: newbie
Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch
The balancer currently exits if nothing has changed after 5 iterations.
Our site would like to constantly balance a stream of incoming data; we would
like to be able to set the number of iterations it does nothing for before
exiting; even better would be if we set it to a negative number and could
continuously run this as a daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305900#comment-14305900 ] Allen Wittenauer commented on HDFS-316: --- Re-fire the jenkins test Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Attachments: HDFS-316.0.patch, HDFS-316.1.patch, HDFS-316.2.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[
https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150467#comment-14150467
]
Hadoop QA commented on HDFS-316:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12671630/HDFS-316.1.patch
against trunk revision 5f16c98.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8239//console
This message is automatically generated.
Balancer should run for a configurable # of iterations
--
Key: HDFS-316
URL: https://issues.apache.org/jira/browse/HDFS-316
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer
Affects Versions: 2.4.1
Reporter: Brian Bockelman
Assignee: Xiaoyu Yao
Priority: Minor
Labels: newbie
Attachments: HDFS-316.0.patch, HDFS-316.1.patch
The balancer currently exits if nothing has changed after 5 iterations.
Our site would like to constantly balance a stream of incoming data; we would
like to be able to set the number of iterations it does nothing for before
exiting; even better would be if we set it to a negative number and could
continuously run this as a daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14145456#comment-14145456 ] Allen Wittenauer commented on HDFS-316: --- Please don't use camel case options. I know the rest of the system does, but they are extremely user unfriendly and something we should start actively avoiding. Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Attachments: HDFS-316.0.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[ https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14145706#comment-14145706 ] Xiaoyu Yao commented on HDFS-316: - Will update the patch with all lowercase option. -maxidliteration. Balancer should run for a configurable # of iterations -- Key: HDFS-316 URL: https://issues.apache.org/jira/browse/HDFS-316 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Affects Versions: 2.4.1 Reporter: Brian Bockelman Assignee: Xiaoyu Yao Priority: Minor Labels: newbie Attachments: HDFS-316.0.patch The balancer currently exits if nothing has changed after 5 iterations. Our site would like to constantly balance a stream of incoming data; we would like to be able to set the number of iterations it does nothing for before exiting; even better would be if we set it to a negative number and could continuously run this as a daemon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-316) Balancer should run for a configurable # of iterations
[
https://issues.apache.org/jira/browse/HDFS-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126495#comment-14126495
]
Hadoop QA commented on HDFS-316:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12667275/HDFS-316.0.patch
against trunk revision 7498dd7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7959//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7959//console
This message is automatically generated.
Balancer should run for a configurable # of iterations
--
Key: HDFS-316
URL: https://issues.apache.org/jira/browse/HDFS-316
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer
Affects Versions: 2.4.1
Reporter: Brian Bockelman
Assignee: Xiaoyu Yao
Priority: Minor
Labels: newbie
Attachments: HDFS-316.0.patch
The balancer currently exits if nothing has changed after 5 iterations.
Our site would like to constantly balance a stream of incoming data; we would
like to be able to set the number of iterations it does nothing for before
exiting; even better would be if we set it to a negative number and could
continuously run this as a daemon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
