[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530370#comment-14530370 ] Uma Maheswara Rao G commented on HDFS-7310: --- Thank you Nicholas for helping Surendra on fix and review. Nice finding Surendra! .- Thanks Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529292#comment-14529292 ] Tsz Wo Nicholas Sze commented on HDFS-7310: --- FYI, Mover/Balancer may schedule two replicas to the same DN storage after this; see HDFS-8147 and HDFS-8204. Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227561#comment-14227561 ] Hudson commented on HDFS-7310: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #18 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/18/]) HDFS-7310. Mover can give first priority to local DN if it has target storage type available in local DN. (Vinayakumar B via umamahesh) (umamahesh: rev 058af60c56207907f2bedf76df4284e86d923e0c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java HDFS-7444. Addendum patch to resolve conflicts between HDFS-7444 and HDFS-7310. (wheat9: rev 978736d486a8775bc21d074010f58c28ae0fda41) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227570#comment-14227570 ] Hudson commented on HDFS-7310: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #756 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/756/]) HDFS-7310. Mover can give first priority to local DN if it has target storage type available in local DN. (Vinayakumar B via umamahesh) (umamahesh: rev 058af60c56207907f2bedf76df4284e86d923e0c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java HDFS-7444. Addendum patch to resolve conflicts between HDFS-7444 and HDFS-7310. (wheat9: rev 978736d486a8775bc21d074010f58c28ae0fda41) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227694#comment-14227694 ] Hudson commented on HDFS-7310: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1946 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1946/]) HDFS-7310. Mover can give first priority to local DN if it has target storage type available in local DN. (Vinayakumar B via umamahesh) (umamahesh: rev 058af60c56207907f2bedf76df4284e86d923e0c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java HDFS-7444. Addendum patch to resolve conflicts between HDFS-7444 and HDFS-7310. (wheat9: rev 978736d486a8775bc21d074010f58c28ae0fda41) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227699#comment-14227699 ] Hudson commented on HDFS-7310: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #18 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/18/]) HDFS-7310. Mover can give first priority to local DN if it has target storage type available in local DN. (Vinayakumar B via umamahesh) (umamahesh: rev 058af60c56207907f2bedf76df4284e86d923e0c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java HDFS-7444. Addendum patch to resolve conflicts between HDFS-7444 and HDFS-7310. (wheat9: rev 978736d486a8775bc21d074010f58c28ae0fda41) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227742#comment-14227742 ] Hudson commented on HDFS-7310: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1970 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1970/]) HDFS-7310. Mover can give first priority to local DN if it has target storage type available in local DN. (Vinayakumar B via umamahesh) (umamahesh: rev 058af60c56207907f2bedf76df4284e86d923e0c) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java HDFS-7444. Addendum patch to resolve conflicts between HDFS-7444 and HDFS-7310. (wheat9: rev 978736d486a8775bc21d074010f58c28ae0fda41) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227749#comment-14227749 ] Hudson commented on HDFS-7310: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #18 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/18/]) HDFS-7310. Mover can give first priority to local DN if it has target storage type available in local DN. (Vinayakumar B via umamahesh) (umamahesh: rev 058af60c56207907f2bedf76df4284e86d923e0c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java HDFS-7444. Addendum patch to resolve conflicts between HDFS-7444 and HDFS-7310. (wheat9: rev 978736d486a8775bc21d074010f58c28ae0fda41) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[
https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225981#comment-14225981
]
Hadoop QA commented on HDFS-7310:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683779/HDFS-7310-004.patch
against trunk revision 4a31611.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8844//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8844//console
This message is automatically generated.
Mover can give first priority to local DN if it has target storage type
available in local DN
-
Key: HDFS-7310
URL: https://issues.apache.org/jira/browse/HDFS-7310
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Vinayakumar B
Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch,
HDFS-7310-003.patch, HDFS-7310-004.patch
Currently Mover logic may move blocks to any DN which had target storage
type. But if the src DN has target storage type then mover can give highest
priority to local DN. If local DN does not contains target storage type, then
it can assign to any DN as the current logic does.
This is a thought, have not go through the code fully yet.
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226529#comment-14226529 ] Uma Maheswara Rao G commented on HDFS-7310: --- Thanks a lot Vinay for the update. Latest patch looks good to me. +1 Will commit the patch shortly. Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226556#comment-14226556 ] Hudson commented on HDFS-7310: -- FAILURE: Integrated in Hadoop-trunk-Commit #6610 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6610/]) HDFS-7310. Mover can give first priority to local DN if it has target storage type available in local DN. (Vinayakumar B via umamahesh) (umamahesh: rev 058af60c56207907f2bedf76df4284e86d923e0c) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/mover/Mover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226658#comment-14226658 ] Hudson commented on HDFS-7310: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6612 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6612/]) HDFS-7444. Addendum patch to resolve conflicts between HDFS-7444 and HDFS-7310. (wheat9: rev 978736d486a8775bc21d074010f58c28ae0fda41) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14226665#comment-14226665 ] Jing Zhao commented on HDFS-7310: - Oops sorry I missed this jira... Thanks Uma for the review and Vinay for the work! Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Fix For: 3.0.0, 2.7.0 Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch, HDFS-7310-004.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224178#comment-14224178 ] Vinayakumar B commented on HDFS-7310: - Hi [~jingzhao], if you could look at the latest patch would be great. Appreciate if others also want to take a look. Thanks Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224282#comment-14224282 ] Uma Maheswara Rao G commented on HDFS-7310: --- Thanks a lot Vinay, Let me take a look at it. Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[
https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224891#comment-14224891
]
Uma Maheswara Rao G commented on HDFS-7310:
---
Hi Vinay, Patch looks good to me. I have one question before my +1 on this.
Seems like now when you plan to move block across storage in a DN and if it is
not on transient storage, you are not doing checksum calculation. my point is,
we may need to compute checksum for having data integrity when moving across
storage? For some reason, that is not necessary, please let me know and also in
such case we need to change below log though
{code}
if (LOG.isDebugEnabled()) {
LOG.debug(Copied + srcMeta + to + dstMeta +
and calculated checksum);
LOG.debug(Copied + srcFile + to + dstFile);
}
{code}
Mover can give first priority to local DN if it has target storage type
available in local DN
-
Key: HDFS-7310
URL: https://issues.apache.org/jira/browse/HDFS-7310
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Vinayakumar B
Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch,
HDFS-7310-003.patch
Currently Mover logic may move blocks to any DN which had target storage
type. But if the src DN has target storage type then mover can give highest
priority to local DN. If local DN does not contains target storage type, then
it can assign to any DN as the current logic does.
This is a thought, have not go through the code fully yet.
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225783#comment-14225783 ] Vinayakumar B commented on HDFS-7310: - Thanks [~umamaheswararao] for the review. bq. Seems like now when you plan to move block across storage in a DN and if it is not on transient storage, you are not doing checksum calculation. my point is, we may need to compute checksum for having data integrity when moving across storage? For some reason, that is not necessary, please let me know and also in such case we need to change below log though On transient storage, checksum will not be available. So while moving a block from transient storage to other storages checksum needs to be calculated during lazyPersist. But in this case, moving from persistent to persistent, we can just copy the already existing checksum. Later if any checksum failures, then it will be detected during scans. I will update the log message. Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch, HDFS-7310-003.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[
https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201801#comment-14201801
]
Hadoop QA commented on HDFS-7310:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680093/HDFS-7310-003.patch
against trunk revision 61effcb.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8688//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8688//console
This message is automatically generated.
Mover can give first priority to local DN if it has target storage type
available in local DN
-
Key: HDFS-7310
URL: https://issues.apache.org/jira/browse/HDFS-7310
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Vinayakumar B
Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch,
HDFS-7310-003.patch
Currently Mover logic may move blocks to any DN which had target storage
type. But if the src DN has target storage type then mover can give highest
priority to local DN. If local DN does not contains target storage type, then
it can assign to any DN as the current logic does.
This is a thought, have not go through the code fully yet.
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[
https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200881#comment-14200881
]
Jing Zhao commented on HDFS-7310:
-
Thanks for working on this, [~vinayrpet]! The patch looks good to me. Only some
minor comments:
# Maybe we can let DatanodeStorageInfo#addBlock return an enum ?
# Replicas in ram_disk currently has its own lazy-persist mechanism. Thus for
the new moveBlockAcrossStorage method, maybe we can simply ignore the replica
(or throw exception) if it's currently on transient storage?
# The patch already has a new unit test in TestMover. Can we also have some
tests only focusing on the new {{moveBlockAcrossStorage}} method and the
{{replaceBlock}} protocol?
Mover can give first priority to local DN if it has target storage type
available in local DN
-
Key: HDFS-7310
URL: https://issues.apache.org/jira/browse/HDFS-7310
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Vinayakumar B
Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch
Currently Mover logic may move blocks to any DN which had target storage
type. But if the src DN has target storage type then mover can give highest
priority to local DN. If local DN does not contains target storage type, then
it can assign to any DN as the current logic does.
This is a thought, have not go through the code fully yet.
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[ https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14199814#comment-14199814 ] Vinayakumar B commented on HDFS-7310: - Hi [~szetszwo] and [~jingzhao], do you want to take a look at the patch ? Mover can give first priority to local DN if it has target storage type available in local DN - Key: HDFS-7310 URL: https://issues.apache.org/jira/browse/HDFS-7310 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Vinayakumar B Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch Currently Mover logic may move blocks to any DN which had target storage type. But if the src DN has target storage type then mover can give highest priority to local DN. If local DN does not contains target storage type, then it can assign to any DN as the current logic does. This is a thought, have not go through the code fully yet. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[
https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192980#comment-14192980
]
Hadoop QA commented on HDFS-7310:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678657/HDFS-7310-002.patch
against trunk revision 260ab6d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8621//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8621//console
This message is automatically generated.
Mover can give first priority to local DN if it has target storage type
available in local DN
-
Key: HDFS-7310
URL: https://issues.apache.org/jira/browse/HDFS-7310
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Vinayakumar B
Attachments: HDFS-7310-001.patch, HDFS-7310-002.patch
Currently Mover logic may move blocks to any DN which had target storage
type. But if the src DN has target storage type then mover can give highest
priority to local DN. If local DN does not contains target storage type, then
it can assign to any DN as the current logic does.
This is a thought, have not go through the code fully yet.
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7310) Mover can give first priority to local DN if it has target storage type available in local DN
[
https://issues.apache.org/jira/browse/HDFS-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14191881#comment-14191881
]
Hadoop QA commented on HDFS-7310:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12678458/HDFS-7310-001.patch
against trunk revision d1828d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.mover.TestStorageMover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8616//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8616//console
This message is automatically generated.
Mover can give first priority to local DN if it has target storage type
available in local DN
-
Key: HDFS-7310
URL: https://issues.apache.org/jira/browse/HDFS-7310
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Vinayakumar B
Attachments: HDFS-7310-001.patch
Currently Mover logic may move blocks to any DN which had target storage
type. But if the src DN has target storage type then mover can give highest
priority to local DN. If local DN does not contains target storage type, then
it can assign to any DN as the current logic does.
This is a thought, have not go through the code fully yet.
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
