[
https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184373#comment-15184373
]
Rakesh R commented on HDFS-8786:
--------------------------------
Thanks [~jingzhao] for the reviews and good comments.
Could you please give few clarifications on first comment. I will take care
other comments while preparing next patch. Thanks!
comment-2=> agreed, will update in next patch
comment-3=> agreed, will raise a follow-on jira and work on.
comment-4=> agreed, will update in next patch
bq. comment-1=> 1. Not all the decommissioning nodes can be later used as
source nodes, since we still need to consider DataNode's current load etc. Thus
I'm not sure the calculation is correct here. In the meanwhile, I do not think
we should adjust additionalReplRequired: there is no need to leave
decommissioning nodes to the next round. Thus looks like we do not need this
change.
{{getAdditionalReplRequired()}} count/value is used while choosing the required
number of target nodes logic as shown below and these target nodes will be
given to reconstruction or replication tasks. The new calculation in my patch
is adjusting the additional replication required by ignoring the
decommissioning count, so that the target nodes for these will not be chosen
later. Here the idea is, will schedule decommissioning replication task only if
there are no other cases like, under replcias or needed a rack etc. Again
ErasureCodingWork will schedule replication task for decommissioning only if
the block group has {{#hasAllInternalBlocks}}. IIUC correctly your [previous
comment|https://issues.apache.org/jira/browse/HDFS-8786?focusedCommentId=15174447&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15174447]
also pointing to this ignore the decommssioning if any other tasks exists. I
think the existing {{BlockManager#chooseSourceDatanodes}} is sufficient for
source selection, am I missing anything?
{code}
case-1) 9 live replicas on 5 racks and 1 decommissioning replica
Choose 1 target for rack replication task and ignore decommissioning
replication task
case-2) 7 live replicas, 1 under replica and 1 decommissioning replica
Choose 1 target for reconstruction task and ignore decommissioning replication
task
case-3) 6 live replicas, 2 under replica and 1 decommissioning replica
Choose 2 target for reconstruction task and ignore decommissioning replication
task
case-4) 8 live replicas and 1 decommissioning replica
Choose 1 target for decommissioning replication task
case-5) 7 live replicas and 2 decommissioning replica
Choose 2 target for decommissioning replication task
{code}
{code}
ErasureCodingWork.java:-
void chooseTargets(BlockPlacementPolicy blockplacement,
BlockStoragePolicySuite storagePolicySuite,
Set<Node> excludedNodes) {
//....
DatanodeStorageInfo[] chosenTargets = blockplacement.chooseTarget(
getBc().getName(), getAdditionalReplRequired(), getSrcNodes()[0],
getLiveReplicaStorages(), false, excludedNodes,
getBlock().getNumBytes(),
storagePolicySuite.getPolicy(getBc().getStoragePolicyID()));
ReplicationWork.java:-
void chooseTargets(BlockPlacementPolicy blockplacement,
BlockStoragePolicySuite storagePolicySuite,
Set<Node> excludedNodes) {
//....
DatanodeStorageInfo[] chosenTargets = blockplacement.chooseTarget(
getBc().getName(), getAdditionalReplRequired(), getSrcNodes()[0],
getLiveReplicaStorages(), false, excludedNodes,
getBlock().getNumBytes(),
storagePolicySuite.getPolicy(getBc().getStoragePolicyID()));
{code}
> Erasure coding: DataNode should transfer striped blocks before being
> decommissioned
> -----------------------------------------------------------------------------------
>
> Key: HDFS-8786
> URL: https://issues.apache.org/jira/browse/HDFS-8786
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: Rakesh R
> Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch,
> HDFS-8786-003.patch, HDFS-8786-004.patch, HDFS-8786-draft.patch
>
>
> Per [discussion |
> https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
> under HDFS-8697, it's too expensive to reconstruct block groups for decomm
> purpose.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
