[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323894#comment-14323894 ] Hudson commented on HDFS-7647: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314226#comment-14314226
]
Yi Liu commented on HDFS-7647:
--
Sorry for coming late, I took a look at this patch again when committing
HDFS-7756 this morning and find few nits.
I make some cleanup for {{LocatedBlock}} in HDFS-7761 after the patch of this
JIRA, and details are as following.
# The usage of following two variables is unnecessary. We can remove them to
make code a bit brief.
{quote}
private final boolean hasStorageIDs;
private final boolean hasStorageTypes;
{quote}
# In this JIRA, no need to modify {{LocatedBlock#getStorageTypes}} and
{{LocatedBlock#getStorageIDs}}, we just need to update the cached
{{storageIDs}} and {{storageTypes}} after *sort*.
# Another thing is we'd better setSoftwareVersion when constructing
{{DatanodeInfoWithStorage}} from {{DatanodeInfo}}
DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Fix For: 2.7.0
Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch,
HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314049#comment-14314049 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in Hadoop-Yarn-trunk #834 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/834/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313519#comment-14313519 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in HBase-1.0 #724 (See [https://builds.apache.org/job/HBase-1.0/724/]) HBASE-12998 Compilation with Hdfs-2.7.0-SNAPSHOT is broken after HDFS-7647 (enis: rev ea431871db5a99eb64eb742558e2cccb867c355f) * hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312782#comment-14312782 ] Arpit Agarwal commented on HDFS-7647: - Thanks for the explanation - I missed that. The warning looks overaggressive but +1 for the v7 patch to avoid future findbugs warnings. I will commit it today. DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs -- Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312771#comment-14312771 ] Milan Desai commented on HDFS-7647: --- Hi [~arpitagarwal], I added those in because Findbugs was complaining that a subclass (DatanodeInfoWithStorage) didn't override equals() and hashCode() and recommending that I call the super and add an explanatory comment if I didn't want to override. Revision 5 is the same exact as Revision 7 but without the equals and hashCode overridden, so we can commit that one, but there will be two additional FindBugs warnings. DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs -- Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312916#comment-14312916 ] Milan Desai commented on HDFS-7647: --- Thanks [~arpitagarwal] for the help and review DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312969#comment-14312969 ] Ted Yu commented on HDFS-7647: -- DatanodeInfoWithStorage doesn't have audience annotation. Is it Private ? Can DatanodeInfoWithStorage reside in org.apache.hadoop.hdfs.protocol as DatanodeInfo does ? DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312976#comment-14312976 ] Arpit Agarwal commented on HDFS-7647: - Yes it is Private by default but we should tag it. [~ted_yu], Do you want to file a Jira to fix both points? Thanks. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312984#comment-14312984 ] Ted Yu commented on HDFS-7647: -- Sure. I am testing a patch - will log JIRA momentarily. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313041#comment-14313041 ] Hudson commented on HDFS-7647: -- SUCCESS: Integrated in Hadoop-trunk-Commit #7053 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7053/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312736#comment-14312736
]
Arpit Agarwal commented on HDFS-7647:
-
Hi [~milandesai], thank you again for your patience with the patch revisions!
The latest version looks good. Just a nitpick, and I do feel bad having you
upload another patch revision -
- I think {{DatanodeInfoWithStorage#equals}} and {{#hashCode}} are unnecessary
since they just forward to the superclass versions with no additional work. I
removed them and verified the test case still passes. Do you think we could
eliminate them?
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch,
HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313092#comment-14313092 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #100 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/100/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313137#comment-14313137 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2031 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2031/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313142#comment-14313142 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2050 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2050/]) HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs. (Contributed by Milan Desai) (arp: rev ab934e85947dcf2092050023909dd81ae274ff45) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeInfoWithStorage.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313428#comment-14313428 ] Hudson commented on HDFS-7647: -- FAILURE: Integrated in HBase-1.1 #159 (See [https://builds.apache.org/job/HBase-1.1/159/]) HBASE-12998 Compilation with Hdfs-2.7.0-SNAPSHOT is broken after HDFS-7647 (enis: rev 66ccc22117beb9908057f941c9e3ea3fbd6880c0) * hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313713#comment-14313713 ] Hudson commented on HDFS-7647: -- SUCCESS: Integrated in HBase-0.98 #842 (See [https://builds.apache.org/job/HBase-0.98/842/]) HBASE-12998 Compilation with Hdfs-2.7.0-SNAPSHOT is broken after HDFS-7647 (enis: rev 433672a67b2589bc77d80007c219d03f6a6bf656) * hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Fix For: 2.7.0 Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311875#comment-14311875 ] Milan Desai commented on HDFS-7647: --- Test failure and findbugs warning are unrelated; test passes locally on my machine. DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs -- Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311781#comment-14311781
]
Hadoop QA commented on HDFS-7647:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697285/HDFS-7647-7.patch
against trunk revision 1382ae5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9495//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9495//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9495//console
This message is automatically generated.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch,
HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647-7.patch, HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310699#comment-14310699
]
Hadoop QA commented on HDFS-7647:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697237/HDFS-7647-6.patch
against trunk revision cfb829e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.cli.TestHDFSCLI
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9488//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9488//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9488//console
This message is automatically generated.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch,
HDFS-7647-5.patch, HDFS-7647-6.patch, HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309643#comment-14309643 ] Arpit Agarwal commented on HDFS-7647: - Thanks for your patience [~milandesai], the latest patch look good. There are a few reproducible test failures flagged by Jenkins we'll need to address before committing. DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs -- Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310123#comment-14310123 ] Milan Desai commented on HDFS-7647: --- Some of the tests were failing because the .equals() and .hashCode() method in DatanodeInfoWithStorage also checked for storage equality, and tests like TestShortCircuitCache relied on the ability to identify DatanodeInfos in a map using a DatanodeInfoWithStorage object. The fix was to remove the equals() and hashCode() overrides so that they rely on the super. TestDecomission.testDecomissionWithOpenfile is failing because the test is waiting for the DatanodeInfoWithStorage object to become decommissioned, but it won't ever be decommissioned because it is not the same reference as the DatanodeInfo in the datanode map. This poses a larger problem in that the client may expect changes to the datanode state to be reflected in the output of LocatedBlock.getLocations(), but they won't be because we create a new DatanodeInfoWithStorage object during construction and return it in getLocations(). Instead of having DatanodeInfoWithStorage be a subclass of DatanodeInfo, we could make it be a wrapper for DatanodeInfo, StorageID, and StorageType. Then to make NetworkTopology.sortByDistance work, we would have to make it implement Node and/or DatanodeInfo. DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs -- Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310191#comment-14310191 ] Arpit Agarwal commented on HDFS-7647: - bq. This poses a larger problem in that the client may expect changes to the datanode state to be reflected in the output of LocatedBlock.getLocations(), but they won't be because we create a new DatanodeInfoWithStorage object during construction and return it in getLocations(). I think this is a test specific problem. Clients will never get access to the NameNode object, instead they will have a replica constructed from the wire message. The test case directly calls the FSNameSystem.getBlockLocations. The test should be fixed to not poll the cached object but query a new one periodically. DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs -- Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch, HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310561#comment-14310561
]
Hadoop QA commented on HDFS-7647:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697191/HDFS-7647-5.patch
against trunk revision 8de80ff.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
org.apache.hadoop.cli.TestHDFSCLI
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9479//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9479//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9479//console
This message is automatically generated.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch,
HDFS-7647-5.patch, HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309368#comment-14309368
]
Hadoop QA commented on HDFS-7647:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697012/HDFS-7647-4.patch
against trunk revision 644548f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
org.apache.hadoop.hdfs.TestDecommission
org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
org.apache.hadoop.hdfs.TestBlockReaderFactory
org.apache.hadoop.hdfs.TestDataTransferKeepalive
org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
The following test timeouts occurred in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.http.TestHttpCookieFlag
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9456//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9456//console
This message is automatically generated.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647-4.patch,
HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14299973#comment-14299973
]
Arpit Agarwal commented on HDFS-7647:
-
Hi [~milandesai],
Thank for incorporating the feedback. The patch looks great for the most part.
However I think it breaks the topology-based sorting. I caught it by adding the
following checks at the end of your test case:
{code}
// Ensure the local node is first.
final DatanodeInfoWithStorage[] sortedLocs = block.getLocations();
assertThat(sortedLocs[0].getIpAddr(), is(targetIp));
// Ensure the two decommissioned DNs were moved to the end.
assertThat(sortedLocs[sortedLocs.length-1].getAdminState(),
is(DatanodeInfo.AdminStates.DECOMMISSIONED));
assertThat(sortedLocs[sortedLocs.length-2].getAdminState(),
is(DatanodeInfo.AdminStates.DECOMMISSIONED));
{code}
where targetIp was used as follows:
{code}
final String targetIp = locs[4].getIpAddr();
// sort block locations
dm.sortLocatedBlocks(targetIp, blocks);
{code}
The first assert kept failing for me, so I spent some time debugging it. The
root cause is the following check in {{NetworkTopology#getWeight}}.
{code}
if (reader == node) {
weight = 0;
{code}
The check for local node is via a simple object ID comparison which worked
before but breaks now because the LHS is a DatanodeDescriptor while the RHS is
DatanodeInfoWithStorage. We'll need to fix that.
Also a nitpick:
{code}
if(!hasStorageTypes)
return null;
if(storageTypes != null)
return storageTypes;
...
if(!hasStorageIDs)
return null;
if(storageIDs != null)
return storageIDs;
{code}
Please add curly braces for the ifs.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647.patch
,m.t, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The
method is called by FSNamesystem.getBlockLocations(), so the client will not
know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14299989#comment-14299989
]
Arpit Agarwal commented on HDFS-7647:
-
We can replace _(reader == node)_ with _(reader.equals(node))_ in
{{NetworkTopology#getWeight}}. Also {{NetworkTopologyWithNodeGroup.getWeight}}.
This defaults to object reference equality unless {{Object.equals}} is
overriden. Although there are no other callers of this method so the change
looks safe.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14299665#comment-14299665
]
Hadoop QA commented on HDFS-7647:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695696/HDFS-7647-3.patch
against trunk revision 09ad9a8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
org.apache.hadoop.hdfs.TestDataTransferKeepalive
org.apache.hadoop.hdfs.TestDecommission
org.apache.hadoop.hdfs.TestBlockReaderFactory
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9387//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9387//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9387//console
This message is automatically generated.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647-3.patch, HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289725#comment-14289725
]
Arpit Agarwal commented on HDFS-7647:
-
Thanks for the patch [~milandesai], this looks like a good change.
Couple of comments:
# {{LocatedBlocks.getStorageTypes}} and {{.getStorageIDs}} should cache the
generated arrays on first invocation since existing callers expect these calls
to be cheap. Except for the sorting code the content of {{locs}} is not
modified once the object is initialized.
# The sorting code must invalidate the cached arrays from 1.
# We should add a unit test for sortLocatedBlocks specifically for the
invalidation.
# Also it would be good to add a comment to {{LocatedBlocks}} stating the
assumption that {{locs}} must not be modified by the caller, with the exception
of {{sortLocatedBlocks}}.
In a separate Jira it would be good to make {{locs}} an unmodifiable list or a
Guava {{ImmutableList}}. The source of the issue is that an external function
reaches into the LocatedBlock object and modifies its private fields. It
doesn't help that Java lacks support for C++-style const arrays.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288892#comment-14288892
]
Hadoop QA commented on HDFS-7647:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12693798/HDFS-7647-2.patch
against trunk revision 5f124ef.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling
org.apache.hadoop.hdfs.TestDataTransferKeepalive
org.apache.hadoop.hdfs.TestDecommission
org.apache.hadoop.hdfs.shortcircuit.TestShortCircuitCache
org.apache.hadoop.hdfs.TestBlockReaderFactory
org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9308//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9308//console
This message is automatically generated.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647-2.patch, HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286567#comment-14286567 ] Arpit Agarwal commented on HDFS-7647: - Hi Milan, while this solution would work I'd prefer the other approach you suggested. We should fix LocatedBlock to group the three fields in a new class. Existing code (except sortLocatedBlocks) can be shielded from the change by keeping public the interface the same. What do you think? DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs -- Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Attachments: HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[ https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286594#comment-14286594 ] Milan Desai commented on HDFS-7647: --- Hi [~arpitagarwal], sounds good - I will give it a shot. DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs -- Key: HDFS-7647 URL: https://issues.apache.org/jira/browse/HDFS-7647 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: Milan Desai Assignee: Milan Desai Attachments: HDFS-7647.patch DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside each LocatedBlock, but does not touch the array of StorageIDs and StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are mismatched. The method is called by FSNamesystem.getBlockLocations(), so the client will not know which StorageID/Type corresponds to which DatanodeInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286858#comment-14286858
]
Hadoop QA commented on HDFS-7647:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12693747/HDFS-7647.patch
against trunk revision 0742591.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9299//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9299//console
This message is automatically generated.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7647) DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
[
https://issues.apache.org/jira/browse/HDFS-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286883#comment-14286883
]
Yi Liu commented on HDFS-7647:
--
{quote}
We should fix LocatedBlock to group the three fields in a new class. Existing
code (except sortLocatedBlocks) can be shielded from the change by keeping
public the interface the same. What do you think?
{quote}
+1 for this approach.
DatanodeManager.sortLocatedBlocks() sorts DatanodeInfos but not StorageIDs
--
Key: HDFS-7647
URL: https://issues.apache.org/jira/browse/HDFS-7647
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Milan Desai
Assignee: Milan Desai
Attachments: HDFS-7647.patch
DatanodeManager.sortLocatedBlocks() sorts the array of DatanodeInfos inside
each LocatedBlock, but does not touch the array of StorageIDs and
StorageTypes. As a result, the DatanodeInfos and StorageIDs/StorageTypes are
mismatched. The method is called by FSNamesystem.getBlockLocations(), so the
client will not know which StorageID/Type corresponds to which DatanodeInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
