[jira] [Updated] (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-385: - Fix Version/s: 1-win 1.2.0 Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0, 1.2.0, 1-win Attachments: BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt, BlockPlacementPluggable.txt, blockplacementpolicy2-branch-1.patch, blockplacementpolicy2-branch-1-win.patch, blockplacementpolicy3-branch-1.patch, blockplacementpolicy3-branch-1-win.patch, blockplacementpolicy-branch-1.patch, blockplacementpolicy-branch-1-win.patch, rat094.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumadhur Reddy Bolli updated HDFS-385: -- Attachment: blockplacementpolicy3-branch-1-win.patch blockplacementpolicy3-branch-1.patch Thanks for the feedback Nicholas. Addressed all the comments in the new patches. Please review. Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt, blockplacementpolicy-branch-1-win.patch, blockplacementpolicy-branch-1.patch, blockplacementpolicy2-branch-1-win.patch, blockplacementpolicy2-branch-1.patch, blockplacementpolicy3-branch-1-win.patch, blockplacementpolicy3-branch-1.patch, rat094.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumadhur Reddy Bolli updated HDFS-385: -- Attachment: blockplacementpolicy2-branch-1-win.patch blockplacementpolicy2-branch-1.patch Sorry. I missed newly added files in the previous patches. Submitted new patches with all files. Please review. Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt, blockplacementpolicy-branch-1-win.patch, blockplacementpolicy-branch-1.patch, blockplacementpolicy2-branch-1-win.patch, blockplacementpolicy2-branch-1.patch, rat094.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HDFS-385: --- Attachment: rat094.txt Patch for 0.94 (Build #359 failed because of this) Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt, blockplacementpolicy-branch-1-win.patch, blockplacementpolicy-branch-1.patch, rat094.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumadhur Reddy Bolli updated HDFS-385: -- Attachment: blockplacementpolicy-branch-1-win.patch blockplacementpolicy-branch-1.patch Patches to port the pluggable interface to branch-1 and branch-1-win. Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt, blockplacementpolicy-branch-1-win.patch, blockplacementpolicy-branch-1.patch The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Chansler updated HDFS-385: - Release Note: New experimental API BlockPlacementPolicy allows investigating alternate rules for locating block replicas. (was: An experimental API that allows an module external to HDFS to specify the policy that HDFS should use to allocate new blocks replicas of a file.) Editorial pass over all release notes prior to publication of 0.21. Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-385: -- Tags: (was: fb) Resolution: Fixed Release Note: An experimental API that allows an module external to HDFS to specify the policy that HDFS should use to allocate new blocks replicas of a file. Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-385: -- Status: Open (was: Patch Available) dunno why hadoopQA tests did not trigger. Cancelling to trigger HudsonQA Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-385: -- Status: Patch Available (was: Open) Trigger HudsonQA Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-385: -- Tags: fb Status: Open (was: Patch Available) Fix javadoc warnings. Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-385: -- Status: Patch Available (was: Open) Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, BlockPlacementPluggable7.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-385: -- Status: Open (was: Patch Available) Incorporate review comments. Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
