[jira] [Commented] (HDFS-6440) Support more than 2 NameNodes
[ https://issues.apache.org/jira/browse/HDFS-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281537#comment-14281537 ] Jesse Yates commented on HDFS-6440: --- Some follow up after actually looking at the code: bq. Is it possible that doWork throws IOException other than RemoteException? Yup. In fact, the implemention of doWork at EditLogTailer#ln291 can throw an IOException if the call to the proxy for rollEditLog throws an IOException. Sure, this is a bit brittle - a remoteException could be thrown by that call (or any other) as an IOException, but that really can't be helped because we have no other way of differentiating right now. bq. 6. needCheckpoint == true implies sendRequests == true thus when call doCheckpiont(), sendRequest is always true. Yup, that was a slight logic bug. I think setting send request should look like: {code:title=StandbyCheckpointer.java} // on all nodes, we build the checkpoint. However, we only ship the checkpoint if have a // rollback request, are the checkpointer, are outside the quiet period. boolean sendRequest = needCheckpoint (isPrimaryCheckPointer || secsSinceLast = checkpointConf.getQuietPeriod()); {code} to actually not send the request every time - it wasn't going to break anything before, but now it should actually conserve bandwidth :) bq. 7. Could you break this line My IDE has that at 99 chars long - isn't 100 chars the standard line width? However, I moved the IOE from the rest of the signature up to the second half of the method declaration. bq. 11. Finally, could you reduce the changes in `MiniDFSCluster.java`, as many of them are not changed, e.g. `MiniDFSCluster.java:911-986`. I think I'm at the minimal number of changes there. Git thinks there are line add and removes frequently when things move around a bit, as this patch necessitates. Fortunately, they should be easy to ignore... but let me know if I'm missing what you are getting at. Support more than 2 NameNodes - Key: HDFS-6440 URL: https://issues.apache.org/jira/browse/HDFS-6440 Project: Hadoop HDFS Issue Type: New Feature Components: auto-failover, ha, namenode Affects Versions: 2.4.0 Reporter: Jesse Yates Assignee: Jesse Yates Attachments: Multiple-Standby-NameNodes_V1.pdf, hdfs-6440-cdh-4.5-full.patch, hdfs-6440-trunk-v1.patch, hdfs-multiple-snn-trunk-v0.patch Most of the work is already done to support more than 2 NameNodes (one active, one standby). This would be the last bit to support running multiple _standby_ NameNodes; one of the standbys should be available for fail-over. Mostly, this is a matter of updating how we parse configurations, some complexity around managing the checkpointing, and updating a whole lot of tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7573) Consolidate the implementation of delete() into a single class
[ https://issues.apache.org/jira/browse/HDFS-7573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281550#comment-14281550 ] Hudson commented on HDFS-7573: -- FAILURE: Integrated in Hadoop-trunk-Commit #6884 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6884/]) HDFS-7573. Consolidate the implementation of delete() into a single class. Contributed by Haohui Mai. (wheat9: rev 24315e7d374a1ddd4329b64350cf96fc9ab6f59c) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirDeleteOp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Consolidate the implementation of delete() into a single class -- Key: HDFS-7573 URL: https://issues.apache.org/jira/browse/HDFS-7573 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.7.0 Attachments: HDFS-7573.000.patch, HDFS-7573.001.patch, HDFS-7573.002.patch This jira proposes to consolidate the implementation of delete() in {{FSNamesystem}} and {{FSDirectory}} into a single class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6440) Support more than 2 NameNodes
[ https://issues.apache.org/jira/browse/HDFS-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HDFS-6440: -- Attachment: hdfs-6440-trunk-v1.patch Attaching patch addressing round 2 of comments. Thanks for the feedback - its getting better every round! Support more than 2 NameNodes - Key: HDFS-6440 URL: https://issues.apache.org/jira/browse/HDFS-6440 Project: Hadoop HDFS Issue Type: New Feature Components: auto-failover, ha, namenode Affects Versions: 2.4.0 Reporter: Jesse Yates Assignee: Jesse Yates Attachments: Multiple-Standby-NameNodes_V1.pdf, hdfs-6440-cdh-4.5-full.patch, hdfs-6440-trunk-v1.patch, hdfs-6440-trunk-v1.patch, hdfs-multiple-snn-trunk-v0.patch Most of the work is already done to support more than 2 NameNodes (one active, one standby). This would be the last bit to support running multiple _standby_ NameNodes; one of the standbys should be available for fail-over. Mostly, this is a matter of updating how we parse configurations, some complexity around managing the checkpointing, and updating a whole lot of tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7573) Consolidate the implementation of delete() into a single class
[ https://issues.apache.org/jira/browse/HDFS-7573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-7573: - Resolution: Fixed Fix Version/s: 2.7.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed the patch to trunk and branch-2. Thanks Charles and Jing for the reviews. Consolidate the implementation of delete() into a single class -- Key: HDFS-7573 URL: https://issues.apache.org/jira/browse/HDFS-7573 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 2.7.0 Attachments: HDFS-7573.000.patch, HDFS-7573.001.patch, HDFS-7573.002.patch This jira proposes to consolidate the implementation of delete() in {{FSNamesystem}} and {{FSDirectory}} into a single class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7635) Remove TestCorruptFilesJsp from branch-2.
[ https://issues.apache.org/jira/browse/HDFS-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281306#comment-14281306 ] Hudson commented on HDFS-7635: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #76 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/76/]) HDFS-7635. Remove TestCorruptFilesJsp from branch-2. Contributed by Chris Nauroth. (cnauroth: rev 9c94015e3678d68f4533f94108b7924193f3e509) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Remove TestCorruptFilesJsp from branch-2. - Key: HDFS-7635 URL: https://issues.apache.org/jira/browse/HDFS-7635 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7635-branch-2.001.patch HDFS-6252 removed corrupt_files.jsp, but there is still a test suite named {{TestCorruptFilesJsp}} in branch-2. The tests attempt to call corrupt_files.jsp and fail on an HTTP 404 error response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7632) MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode.
[ https://issues.apache.org/jira/browse/HDFS-7632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281305#comment-14281305 ] Hudson commented on HDFS-7632: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #76 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/76/]) HDFS-7632. MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. Contributed by Chris Nauroth. (cnauroth: rev ec4389cf7270cff4cc96313b4190422ea7c70ced) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitLocalRead.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestCachingStrategy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMissingBlocksAlert.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. - Key: HDFS-7632 URL: https://issues.apache.org/jira/browse/HDFS-7632 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.7.0 Attachments: HDFS-7632.001.patch {{MiniDFSCluster}} allows the caller to set the number of storage locations per DataNode. If this number is set higher than 2, and if the cluster is configured with more than 1 DataNode, then the calculation of the storage directory paths will be incorrect. Multiple DataNodes will attempt to use the same storage directory, and one of them will fail while trying to acquire the file lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7606) Missing null check in INodeFile#getBlocks()
[ https://issues.apache.org/jira/browse/HDFS-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281307#comment-14281307 ] Hudson commented on HDFS-7606: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #76 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/76/]) HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). Contributed by Byron Wong. (shv: rev 60cbcff2f7363e5cc386284981cac67abc965ee7) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Missing null check in INodeFile#getBlocks() --- Key: HDFS-7606 URL: https://issues.apache.org/jira/browse/HDFS-7606 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ted Yu Assignee: Byron Wong Priority: Minor Fix For: 3.0.0 Attachments: HDFS-7606-1.patch, HDFS-7606.patch, HDFS-7606.patch {code} BlockInfo[] snapshotBlocks = diff == null ? getBlocks() : diff.getBlocks(); if(snapshotBlocks != null) return snapshotBlocks; // Blocks are not in the current snapshot // Find next snapshot with blocks present or return current file blocks snapshotBlocks = getDiffs().findLaterSnapshotBlocks(diff.getSnapshotId()); {code} If diff is null and snapshotBlocks is null, NullPointerException would result from the call to diff.getSnapshotId(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7606) Missing null check in INodeFile#getBlocks()
[ https://issues.apache.org/jira/browse/HDFS-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281313#comment-14281313 ] Hudson commented on HDFS-7606: -- FAILURE: Integrated in Hadoop-Yarn-trunk #810 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/810/]) HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). Contributed by Byron Wong. (shv: rev 60cbcff2f7363e5cc386284981cac67abc965ee7) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java Missing null check in INodeFile#getBlocks() --- Key: HDFS-7606 URL: https://issues.apache.org/jira/browse/HDFS-7606 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ted Yu Assignee: Byron Wong Priority: Minor Fix For: 3.0.0 Attachments: HDFS-7606-1.patch, HDFS-7606.patch, HDFS-7606.patch {code} BlockInfo[] snapshotBlocks = diff == null ? getBlocks() : diff.getBlocks(); if(snapshotBlocks != null) return snapshotBlocks; // Blocks are not in the current snapshot // Find next snapshot with blocks present or return current file blocks snapshotBlocks = getDiffs().findLaterSnapshotBlocks(diff.getSnapshotId()); {code} If diff is null and snapshotBlocks is null, NullPointerException would result from the call to diff.getSnapshotId(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7632) MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode.
[ https://issues.apache.org/jira/browse/HDFS-7632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281311#comment-14281311 ] Hudson commented on HDFS-7632: -- FAILURE: Integrated in Hadoop-Yarn-trunk #810 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/810/]) HDFS-7632. MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. Contributed by Chris Nauroth. (cnauroth: rev ec4389cf7270cff4cc96313b4190422ea7c70ced) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMissingBlocksAlert.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestCachingStrategy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitLocalRead.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. - Key: HDFS-7632 URL: https://issues.apache.org/jira/browse/HDFS-7632 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.7.0 Attachments: HDFS-7632.001.patch {{MiniDFSCluster}} allows the caller to set the number of storage locations per DataNode. If this number is set higher than 2, and if the cluster is configured with more than 1 DataNode, then the calculation of the storage directory paths will be incorrect. Multiple DataNodes will attempt to use the same storage directory, and one of them will fail while trying to acquire the file lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7635) Remove TestCorruptFilesJsp from branch-2.
[ https://issues.apache.org/jira/browse/HDFS-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281312#comment-14281312 ] Hudson commented on HDFS-7635: -- FAILURE: Integrated in Hadoop-Yarn-trunk #810 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/810/]) HDFS-7635. Remove TestCorruptFilesJsp from branch-2. Contributed by Chris Nauroth. (cnauroth: rev 9c94015e3678d68f4533f94108b7924193f3e509) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Remove TestCorruptFilesJsp from branch-2. - Key: HDFS-7635 URL: https://issues.apache.org/jira/browse/HDFS-7635 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7635-branch-2.001.patch HDFS-6252 removed corrupt_files.jsp, but there is still a test suite named {{TestCorruptFilesJsp}} in branch-2. The tests attempt to call corrupt_files.jsp and fail on an HTTP 404 error response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
[ https://issues.apache.org/jira/browse/HDFS-7624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281363#comment-14281363 ] sam liu commented on HDFS-7624: --- The failure of TestRetryCacheWithHA should not be caused by the patch, because TestRetryCacheWithHA does not invoke any API of current test TestFileAppendRestart. TestFileAppendRestart hardcode block size without considering native OS --- Key: HDFS-7624 URL: https://issues.apache.org/jira/browse/HDFS-7624 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7624.001.patch, HDFS-7624.002.patch TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7635) Remove TestCorruptFilesJsp from branch-2.
[ https://issues.apache.org/jira/browse/HDFS-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281371#comment-14281371 ] Hudson commented on HDFS-7635: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #73 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/73/]) HDFS-7635. Remove TestCorruptFilesJsp from branch-2. Contributed by Chris Nauroth. (cnauroth: rev 9c94015e3678d68f4533f94108b7924193f3e509) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Remove TestCorruptFilesJsp from branch-2. - Key: HDFS-7635 URL: https://issues.apache.org/jira/browse/HDFS-7635 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7635-branch-2.001.patch HDFS-6252 removed corrupt_files.jsp, but there is still a test suite named {{TestCorruptFilesJsp}} in branch-2. The tests attempt to call corrupt_files.jsp and fail on an HTTP 404 error response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7635) Remove TestCorruptFilesJsp from branch-2.
[ https://issues.apache.org/jira/browse/HDFS-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281376#comment-14281376 ] Hudson commented on HDFS-7635: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2008 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2008/]) HDFS-7635. Remove TestCorruptFilesJsp from branch-2. Contributed by Chris Nauroth. (cnauroth: rev 9c94015e3678d68f4533f94108b7924193f3e509) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Remove TestCorruptFilesJsp from branch-2. - Key: HDFS-7635 URL: https://issues.apache.org/jira/browse/HDFS-7635 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7635-branch-2.001.patch HDFS-6252 removed corrupt_files.jsp, but there is still a test suite named {{TestCorruptFilesJsp}} in branch-2. The tests attempt to call corrupt_files.jsp and fail on an HTTP 404 error response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7606) Missing null check in INodeFile#getBlocks()
[ https://issues.apache.org/jira/browse/HDFS-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281377#comment-14281377 ] Hudson commented on HDFS-7606: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2008 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2008/]) HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). Contributed by Byron Wong. (shv: rev 60cbcff2f7363e5cc386284981cac67abc965ee7) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java Missing null check in INodeFile#getBlocks() --- Key: HDFS-7606 URL: https://issues.apache.org/jira/browse/HDFS-7606 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ted Yu Assignee: Byron Wong Priority: Minor Fix For: 3.0.0 Attachments: HDFS-7606-1.patch, HDFS-7606.patch, HDFS-7606.patch {code} BlockInfo[] snapshotBlocks = diff == null ? getBlocks() : diff.getBlocks(); if(snapshotBlocks != null) return snapshotBlocks; // Blocks are not in the current snapshot // Find next snapshot with blocks present or return current file blocks snapshotBlocks = getDiffs().findLaterSnapshotBlocks(diff.getSnapshotId()); {code} If diff is null and snapshotBlocks is null, NullPointerException would result from the call to diff.getSnapshotId(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7632) MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode.
[ https://issues.apache.org/jira/browse/HDFS-7632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281375#comment-14281375 ] Hudson commented on HDFS-7632: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2008 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2008/]) HDFS-7632. MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. Contributed by Chris Nauroth. (cnauroth: rev ec4389cf7270cff4cc96313b4190422ea7c70ced) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitLocalRead.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMissingBlocksAlert.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestCachingStrategy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. - Key: HDFS-7632 URL: https://issues.apache.org/jira/browse/HDFS-7632 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.7.0 Attachments: HDFS-7632.001.patch {{MiniDFSCluster}} allows the caller to set the number of storage locations per DataNode. If this number is set higher than 2, and if the cluster is configured with more than 1 DataNode, then the calculation of the storage directory paths will be incorrect. Multiple DataNodes will attempt to use the same storage directory, and one of them will fail while trying to acquire the file lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7606) Missing null check in INodeFile#getBlocks()
[ https://issues.apache.org/jira/browse/HDFS-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281372#comment-14281372 ] Hudson commented on HDFS-7606: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #73 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/73/]) HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). Contributed by Byron Wong. (shv: rev 60cbcff2f7363e5cc386284981cac67abc965ee7) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java Missing null check in INodeFile#getBlocks() --- Key: HDFS-7606 URL: https://issues.apache.org/jira/browse/HDFS-7606 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ted Yu Assignee: Byron Wong Priority: Minor Fix For: 3.0.0 Attachments: HDFS-7606-1.patch, HDFS-7606.patch, HDFS-7606.patch {code} BlockInfo[] snapshotBlocks = diff == null ? getBlocks() : diff.getBlocks(); if(snapshotBlocks != null) return snapshotBlocks; // Blocks are not in the current snapshot // Find next snapshot with blocks present or return current file blocks snapshotBlocks = getDiffs().findLaterSnapshotBlocks(diff.getSnapshotId()); {code} If diff is null and snapshotBlocks is null, NullPointerException would result from the call to diff.getSnapshotId(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7632) MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode.
[ https://issues.apache.org/jira/browse/HDFS-7632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281370#comment-14281370 ] Hudson commented on HDFS-7632: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #73 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/73/]) HDFS-7632. MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. Contributed by Chris Nauroth. (cnauroth: rev ec4389cf7270cff4cc96313b4190422ea7c70ced) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestCachingStrategy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitLocalRead.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMissingBlocksAlert.java MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. - Key: HDFS-7632 URL: https://issues.apache.org/jira/browse/HDFS-7632 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.7.0 Attachments: HDFS-7632.001.patch {{MiniDFSCluster}} allows the caller to set the number of storage locations per DataNode. If this number is set higher than 2, and if the cluster is configured with more than 1 DataNode, then the calculation of the storage directory paths will be incorrect. Multiple DataNodes will attempt to use the same storage directory, and one of them will fail while trying to acquire the file lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7632) MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode.
[ https://issues.apache.org/jira/browse/HDFS-7632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281403#comment-14281403 ] Hudson commented on HDFS-7632: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #77 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/77/]) HDFS-7632. MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. Contributed by Chris Nauroth. (cnauroth: rev ec4389cf7270cff4cc96313b4190422ea7c70ced) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMissingBlocksAlert.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestCachingStrategy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitLocalRead.java MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. - Key: HDFS-7632 URL: https://issues.apache.org/jira/browse/HDFS-7632 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.7.0 Attachments: HDFS-7632.001.patch {{MiniDFSCluster}} allows the caller to set the number of storage locations per DataNode. If this number is set higher than 2, and if the cluster is configured with more than 1 DataNode, then the calculation of the storage directory paths will be incorrect. Multiple DataNodes will attempt to use the same storage directory, and one of them will fail while trying to acquire the file lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7635) Remove TestCorruptFilesJsp from branch-2.
[ https://issues.apache.org/jira/browse/HDFS-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281404#comment-14281404 ] Hudson commented on HDFS-7635: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #77 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/77/]) HDFS-7635. Remove TestCorruptFilesJsp from branch-2. Contributed by Chris Nauroth. (cnauroth: rev 9c94015e3678d68f4533f94108b7924193f3e509) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Remove TestCorruptFilesJsp from branch-2. - Key: HDFS-7635 URL: https://issues.apache.org/jira/browse/HDFS-7635 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7635-branch-2.001.patch HDFS-6252 removed corrupt_files.jsp, but there is still a test suite named {{TestCorruptFilesJsp}} in branch-2. The tests attempt to call corrupt_files.jsp and fail on an HTTP 404 error response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7606) Missing null check in INodeFile#getBlocks()
[ https://issues.apache.org/jira/browse/HDFS-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281405#comment-14281405 ] Hudson commented on HDFS-7606: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #77 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/77/]) HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). Contributed by Byron Wong. (shv: rev 60cbcff2f7363e5cc386284981cac67abc965ee7) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Missing null check in INodeFile#getBlocks() --- Key: HDFS-7606 URL: https://issues.apache.org/jira/browse/HDFS-7606 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ted Yu Assignee: Byron Wong Priority: Minor Fix For: 3.0.0 Attachments: HDFS-7606-1.patch, HDFS-7606.patch, HDFS-7606.patch {code} BlockInfo[] snapshotBlocks = diff == null ? getBlocks() : diff.getBlocks(); if(snapshotBlocks != null) return snapshotBlocks; // Blocks are not in the current snapshot // Find next snapshot with blocks present or return current file blocks snapshotBlocks = getDiffs().findLaterSnapshotBlocks(diff.getSnapshotId()); {code} If diff is null and snapshotBlocks is null, NullPointerException would result from the call to diff.getSnapshotId(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7606) Missing null check in INodeFile#getBlocks()
[ https://issues.apache.org/jira/browse/HDFS-7606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281420#comment-14281420 ] Hudson commented on HDFS-7606: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2027 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2027/]) HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). Contributed by Byron Wong. (shv: rev 60cbcff2f7363e5cc386284981cac67abc965ee7) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Missing null check in INodeFile#getBlocks() --- Key: HDFS-7606 URL: https://issues.apache.org/jira/browse/HDFS-7606 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ted Yu Assignee: Byron Wong Priority: Minor Fix For: 3.0.0 Attachments: HDFS-7606-1.patch, HDFS-7606.patch, HDFS-7606.patch {code} BlockInfo[] snapshotBlocks = diff == null ? getBlocks() : diff.getBlocks(); if(snapshotBlocks != null) return snapshotBlocks; // Blocks are not in the current snapshot // Find next snapshot with blocks present or return current file blocks snapshotBlocks = getDiffs().findLaterSnapshotBlocks(diff.getSnapshotId()); {code} If diff is null and snapshotBlocks is null, NullPointerException would result from the call to diff.getSnapshotId(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7632) MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode.
[ https://issues.apache.org/jira/browse/HDFS-7632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281418#comment-14281418 ] Hudson commented on HDFS-7632: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2027 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2027/]) HDFS-7632. MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. Contributed by Chris Nauroth. (cnauroth: rev ec4389cf7270cff4cc96313b4190422ea7c70ced) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPendingCorruptDnMessages.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeBlockScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/shortcircuit/TestShortCircuitLocalRead.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReplication.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMissingBlocksAlert.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestProcessCorruptBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestCachingStrategy.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocal.java MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode. - Key: HDFS-7632 URL: https://issues.apache.org/jira/browse/HDFS-7632 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.7.0 Attachments: HDFS-7632.001.patch {{MiniDFSCluster}} allows the caller to set the number of storage locations per DataNode. If this number is set higher than 2, and if the cluster is configured with more than 1 DataNode, then the calculation of the storage directory paths will be incorrect. Multiple DataNodes will attempt to use the same storage directory, and one of them will fail while trying to acquire the file lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7635) Remove TestCorruptFilesJsp from branch-2.
[ https://issues.apache.org/jira/browse/HDFS-7635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281419#comment-14281419 ] Hudson commented on HDFS-7635: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2027 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2027/]) HDFS-7635. Remove TestCorruptFilesJsp from branch-2. Contributed by Chris Nauroth. (cnauroth: rev 9c94015e3678d68f4533f94108b7924193f3e509) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Remove TestCorruptFilesJsp from branch-2. - Key: HDFS-7635 URL: https://issues.apache.org/jira/browse/HDFS-7635 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Chris Nauroth Assignee: Chris Nauroth Priority: Minor Fix For: 2.7.0 Attachments: HDFS-7635-branch-2.001.patch HDFS-6252 removed corrupt_files.jsp, but there is still a test suite named {{TestCorruptFilesJsp}} in branch-2. The tests attempt to call corrupt_files.jsp and fail on an HTTP 404 error response. -- This message was sent by Atlassian JIRA (v6.3.4#6332)