[jira] [Commented] (HADOOP-9045) In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
[
https://issues.apache.org/jira/browse/HADOOP-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504058#comment-13504058
]
Jing Zhao commented on HADOOP-9045:
---
The patch looks very good to me. The only concern is that in
TestReplicationPolicyWithNodeGroup,
{noformat}
+ // Enable the checking for stale datanodes in the beginning
+ CONF.setBoolean(DFSConfigKeys.DFS_NAMENODE_CHECK_STALE_DATANODE_KEY,
true);
{noformat}
may be unnecessary?
In nodegroup-aware case, make sure nodes are avoided to place replica if some
replica are already under the same nodegroup
--
Key: HADOOP-9045
URL: https://issues.apache.org/jira/browse/HADOOP-9045
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Junping Du
Assignee: Junping Du
Attachments: HADOOP-9045.patch, HADOOP-9045-v2.patch,
HADOOP-9045-v3.patch, HADOOP-9045-v4.patch
In previous implementation for HADOOP-8468, 3rd replica is avoid to place on
the same nodegroup of 2nd replica. But it didn't provide check on nodegroup
of 1st replica, so if 2nd replica's rack is not efficient to place replica,
then it is possible to place 3rd and 1st replica within the same node group.
We need a change to remove all nodes from available nodes for placing replica
if there already replica on the same nodegroup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9045) In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
[
https://issues.apache.org/jira/browse/HADOOP-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501934#comment-13501934
]
Hadoop QA commented on HADOOP-9045:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554487/HADOOP-9045-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1786//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1786//console
This message is automatically generated.
In nodegroup-aware case, make sure nodes are avoided to place replica if some
replica are already under the same nodegroup
--
Key: HADOOP-9045
URL: https://issues.apache.org/jira/browse/HADOOP-9045
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Junping Du
Assignee: Junping Du
Attachments: HADOOP-9045.patch, HADOOP-9045-v2.patch,
HADOOP-9045-v3.patch
In previous implementation for HADOOP-8468, 3rd replica is avoid to place on
the same nodegroup of 2nd replica. But it didn't provide check on nodegroup
of 1st replica, so if 2nd replica's rack is not efficient to place replica,
then it is possible to place 3rd and 1st replica within the same node group.
We need a change to remove all nodes from available nodes for placing replica
if there already replica on the same nodegroup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9045) In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
[
https://issues.apache.org/jira/browse/HADOOP-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502034#comment-13502034
]
Hadoop QA commented on HADOOP-9045:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554497/HADOOP-9045-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1787//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1787//console
This message is automatically generated.
In nodegroup-aware case, make sure nodes are avoided to place replica if some
replica are already under the same nodegroup
--
Key: HADOOP-9045
URL: https://issues.apache.org/jira/browse/HADOOP-9045
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Junping Du
Assignee: Junping Du
Attachments: HADOOP-9045.patch, HADOOP-9045-v2.patch,
HADOOP-9045-v3.patch, HADOOP-9045-v4.patch
In previous implementation for HADOOP-8468, 3rd replica is avoid to place on
the same nodegroup of 2nd replica. But it didn't provide check on nodegroup
of 1st replica, so if 2nd replica's rack is not efficient to place replica,
then it is possible to place 3rd and 1st replica within the same node group.
We need a change to remove all nodes from available nodes for placing replica
if there already replica on the same nodegroup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9045) In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
[
https://issues.apache.org/jira/browse/HADOOP-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501152#comment-13501152
]
Hadoop QA commented on HADOOP-9045:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554332/HADOOP-9045-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1777//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1777//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1777//console
This message is automatically generated.
In nodegroup-aware case, make sure nodes are avoided to place replica if some
replica are already under the same nodegroup
--
Key: HADOOP-9045
URL: https://issues.apache.org/jira/browse/HADOOP-9045
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Junping Du
Assignee: Junping Du
Attachments: HADOOP-9045.patch, HADOOP-9045-v2.patch
In previous implementation for HADOOP-8468, 3rd replica is avoid to place on
the same nodegroup of 2nd replica. But it didn't provide check on nodegroup
of 1st replica, so if 2nd replica's rack is not efficient to place replica,
then it is possible to place 3rd and 1st replica within the same node group.
We need a change to remove all nodes from available nodes for placing replica
if there already replica on the same nodegroup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9045) In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
[
https://issues.apache.org/jira/browse/HADOOP-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498180#comment-13498180
]
Hadoop QA commented on HADOOP-9045:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553662/HADOOP-9045.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestHDFSFileSystemContract
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1751//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1751//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1751//console
This message is automatically generated.
In nodegroup-aware case, make sure nodes are avoided to place replica if some
replica are already under the same nodegroup
--
Key: HADOOP-9045
URL: https://issues.apache.org/jira/browse/HADOOP-9045
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Junping Du
Assignee: Junping Du
Attachments: HADOOP-9045.patch
In previous implementation for HADOOP-8468, 3rd replica is avoid to place on
the same nodegroup of 2nd replica. But it didn't provide check on nodegroup
of 1st replica, so if 2nd replica's rack is not efficient to place replica,
then it is possible to place 3rd and 1st replica within the same node group.
We need a change to remove all nodes from available nodes for placing replica
if there already replica on the same nodegroup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9045) In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
[ https://issues.apache.org/jira/browse/HADOOP-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498219#comment-13498219 ] Junping Du commented on HADOOP-9045: The unit test failed is unrelated but a umask error on directory creation, and I already file a jira HADOOP-9048 to fix it. The findbug problem is not a issue as local variable is used in controlling while-loop. So it looks no issue for this precommit test. In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup -- Key: HADOOP-9045 URL: https://issues.apache.org/jira/browse/HADOOP-9045 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-9045.patch In previous implementation for HADOOP-8468, 3rd replica is avoid to place on the same nodegroup of 2nd replica. But it didn't provide check on nodegroup of 1st replica, so if 2nd replica's rack is not efficient to place replica, then it is possible to place 3rd and 1st replica within the same node group. We need a change to remove all nodes from available nodes for placing replica if there already replica on the same nodegroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9045) In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
[ https://issues.apache.org/jira/browse/HADOOP-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498225#comment-13498225 ] Junping Du commented on HADOOP-9045: Ok. HADOOP-9042 is going to fix it. In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup -- Key: HADOOP-9045 URL: https://issues.apache.org/jira/browse/HADOOP-9045 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-9045.patch In previous implementation for HADOOP-8468, 3rd replica is avoid to place on the same nodegroup of 2nd replica. But it didn't provide check on nodegroup of 1st replica, so if 2nd replica's rack is not efficient to place replica, then it is possible to place 3rd and 1st replica within the same node group. We need a change to remove all nodes from available nodes for placing replica if there already replica on the same nodegroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9045) In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
[ https://issues.apache.org/jira/browse/HADOOP-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498273#comment-13498273 ] Tsz Wo (Nicholas), SZE commented on HADOOP-9045: ... The findbug problem is not a issue as local variable is used in controlling while-loop. So it looks no issue for this precommit test. Hi Junping, although the findbugs warnings are not harmful, we still have to fix it. Otherwise, it will fail every build in the future if we commit the patch. In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup -- Key: HADOOP-9045 URL: https://issues.apache.org/jira/browse/HADOOP-9045 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-9045.patch In previous implementation for HADOOP-8468, 3rd replica is avoid to place on the same nodegroup of 2nd replica. But it didn't provide check on nodegroup of 1st replica, so if 2nd replica's rack is not efficient to place replica, then it is possible to place 3rd and 1st replica within the same node group. We need a change to remove all nodes from available nodes for placing replica if there already replica on the same nodegroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9045) In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup
[ https://issues.apache.org/jira/browse/HADOOP-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498532#comment-13498532 ] Junping Du commented on HADOOP-9045: I am curious on this warning as line 464 is exactly the same case but without any warning. Nicholas, may be we just suppress warning? Any better idea for resolve this? In nodegroup-aware case, make sure nodes are avoided to place replica if some replica are already under the same nodegroup -- Key: HADOOP-9045 URL: https://issues.apache.org/jira/browse/HADOOP-9045 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.2-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-9045.patch In previous implementation for HADOOP-8468, 3rd replica is avoid to place on the same nodegroup of 2nd replica. But it didn't provide check on nodegroup of 1st replica, so if 2nd replica's rack is not efficient to place replica, then it is possible to place 3rd and 1st replica within the same node group. We need a change to remove all nodes from available nodes for placing replica if there already replica on the same nodegroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
