[
https://issues.apache.org/jira/browse/HDFS-7621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14562357#comment-14562357
]
Zhe Zhang commented on HDFS-7621:
---------------------------------
Sorry for coming to this review late.
Again, the patch looks reasonable overall. My biggest concern is that the patch
changes {{Dispatcher}} pretty heavily in place. It'd be better if we leave the
current logic mostly unchanged and add striping logic incrementally. This will
make the merge review much cleaner and easier.
{{DBlock}} actually just maps a {{Block}} to locations. The added
{{DBlockStriped}} does 2 things: 1) maps {{Block}} to locations (inherited from
{{DBlock}}); and 2) when given a {{StorageGroup}}, calculates the index of the
internal block. How about creating a structure only for purpose #2? Something
like the below:
{code}
private static class GlobalBlockGroupMap {
private final Map<Block, StripedBlockWithLocations> map = new HashMap<>();
private int getInternalBlkIdx(Block b, StorageGroup storage) {
...
}
}
{code}
Then the change to existing {{Dispatcher}} code can be simple. Right before
sending a move command to DN, just check the new map to see if we need to
translate the block ID. It seems we can add the check in {{executePendingMove}}
for this purpose, but I'm not 100% sure.
> Erasure Coding: update the Balancer/Mover data migration logic
> --------------------------------------------------------------
>
> Key: HDFS-7621
> URL: https://issues.apache.org/jira/browse/HDFS-7621
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Jing Zhao
> Assignee: Walter Su
> Labels: HDFS-7285
> Attachments: HDFS-7621.001.patch, HDFS-7621.002.patch,
> HDFS-7621.003.patch, HDFS-7621.004.patch, HDFS-7621.005.patch,
> HDFS-7621.006.patch
>
>
> Currently the Balancer/Mover only considers the distribution of replicas of
> the same block during data migration: the migration cannot decrease the
> number of racks. With EC the Balancer and Mover should also take into account
> the distribution of blocks belonging to the same block group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
