[
https://issues.apache.org/jira/browse/HDFS-7621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544545#comment-14544545
]
Zhe Zhang commented on HDFS-7621:
---------------------------------
Thanks for the update Walter.
Structural:
# {{convertToBlockWithLocations}} looks good to me know. So I'm +1 on the
{{BlockManager}} changes
# One concern on the new {{DBlock}} code is that each {{DBlock}} should
represent a single block with locations. Its super class,
{{MovedBlocks#Locations}}, is also clearly designed for a single block.
Therefore {{nonCollocatedBlock}} looks strange because each striped {{DBlock}}
actually has multiple peer blocks that it has to avoid.
# bq. Balancer handles block and doesn't know file. Balancer gets blocks from
Node.
Thanks for clarifying!
Nits:
# A few lines are too long. You might already know that we prefer each line to
be under 80 characters.
{code}
+ updateDBlockLocations(nonCollocatedBlock,
nonCollocatedBlockWithLocations);
+ DBlock newDBlock(Block block, List<MLocation> locations, DBlock
stripedBlock) {
+ final ExtendedBlock reportedBlock = new
ExtendedBlock(lsb.getBlock());
+ long numBytes =
getInternalBlockLength(lsb.getBlock().getNumBytes(),
+ final List<MLocation> reportedBlockLocation = new
ArrayList<>(1);
+ public static void verifyLocatedStripedBlocks(LocatedBlocks lbs, int
groupSize) {
+ client = NameNodeProxies.createProxy(conf,
cluster.getFileSystem(0).getUri(),
+ LocatedBlocks locatedBlocks = client.getBlockLocations(fileName, 0,
fileLen);
+ final DBlock db = mover.newDBlock(lb.getBlock().getLocalBlock(),
locations, null);
+ ClientProtocol client = NameNodeProxies.createProxy(conf,
cluster.getFileSystem(0).getUri(),
+ client.setStoragePolicy(barDir,
HdfsServerConstants.HOT_STORAGE_POLICY_NAME);
+ LocatedBlocks locatedBlocks = client.getBlockLocations(fooFile, 0,
fileLen);
+ DFSTestUtil.verifyLocatedStripedBlocks(locatedBlocks, dataBlocks +
parityBlocks);
+ DFSTestUtil.verifyLocatedStripedBlocks(locatedBlocks, dataBlocks +
parityBlocks);
{code}
I usually use a simple script to check for long lines. In case you need:
{code}
#! /usr/local/bin/python3
import sys
with open(sys.argv[1], 'r') as inf:
for line in inf:
if line.startswith('+ ') and len(line) > 81:
print(line)
{code}
# Looks like the indentation is wrong:
{code}
+ private void updateDBlockLocations(DBlock block,
+ BlockWithLocations blockWithLocations){
{code}
# {{nonCollocatedBlock}} literally means "currently not collocated". Maybe just
{{toAvoid}}? The name doesn't need to have "block" because the type already
implies it.
> Erasure Coding: update the Balancer/Mover data migration logic
> --------------------------------------------------------------
>
> Key: HDFS-7621
> URL: https://issues.apache.org/jira/browse/HDFS-7621
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Jing Zhao
> Assignee: Walter Su
> Labels: HDFS-7285
> Attachments: HDFS-7621.001.patch, HDFS-7621.002.patch,
> HDFS-7621.003.patch, HDFS-7621.004.patch
>
>
> Currently the Balancer/Mover only considers the distribution of replicas of
> the same block during data migration: the migration cannot decrease the
> number of racks. With EC the Balancer and Mover should also take into account
> the distribution of blocks belonging to the same block group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
