[
https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587106#comment-14587106
]
Jing Zhao commented on HDFS-8543:
---------------------------------
Thanks for working on this, Walter! So some comments other than Zhe's comments:
# One minor is that we can optimize the following code:
{code}
+ // for each duplicated index, delete some replicas until only one left
+ for (int targetIndex = duplicated.nextSetBit(0); targetIndex >= 0;
+ targetIndex = duplicated.nextSetBit(targetIndex + 1)) {
+ List<DatanodeStorageInfo> candidates = new ArrayList<>();
+ for (DatanodeStorageInfo storage : nonExcess) {
+ int index = sblk.getStorageBlockIndex(storage);
+ if (index == targetIndex) {
+ candidates.add(storage);
+ }
+ }
{code}
{{getStorageBlockIndex}} has to go through the whole object triplet array.
Looks like we can avoid making this call in the inner loop by caching the
mapping of storage and index into a temp map.
# The following code in {{chooseExcessReplicasStriped}} works currently but
looks like the 2nd and 3rd parameters are not semantically consistent with the
parameter {{candidates}}? {{candidates}} is for the internal block while the
former two parameters are about the whole block group. Maybe we can use a new
block object with the internal block's ID and replication factor 1 here.
{code}
DatanodeStorageInfo target = placementPolicy.chooseReplicaToDelete(bc,
storedBlock, groupSize, candidates, empty, excessTypes);
{code}
> Erasure Coding: processOverReplicatedBlock() handles striped block
> ------------------------------------------------------------------
>
> Key: HDFS-8543
> URL: https://issues.apache.org/jira/browse/HDFS-8543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Walter Su
> Assignee: Walter Su
> Attachments: HDFS-8543-HDFS-7285.01.patch,
> HDFS-8543-HDFS-7285.02.patch, HDFS-8543-HDFS-7285.03.patch,
> HDFS-8543-HDFS-7285.04.patch, HDFS-8543-HDFS-7285.05.patch,
> HDFS-8543-HDFS-7285.06.patch
>
>
> striped block group could be over replicated when: 1.dead DN comes back. 2.
> Balancer/Mover copies block before deletes it.
> This jira add logic for processOverReplicatedBlock() handling striped block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
