[
https://issues.apache.org/jira/browse/HDFS-7877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864638#comment-15864638
]
Manoj Govindassamy commented on HDFS-7877:
------------------------------------------
Thanks [~mingma]. Got it, when you club this with Upgrade Domain, the impact is
not that severe.
I will make the following change for the Maintenance Min Replication range
validation check.
{noformat}
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
@@ -484,12 +484,12 @@ public BlockManager(final Namesystem namesystem, boolean
haEnabled,
+ DFSConfigKeys.DFS_NAMENODE_MAINTENANCE_REPLICATION_MIN_KEY
+ " = " + minMaintenanceR + " < 0");
}
- if (minMaintenanceR > minR) {
+ if (minMaintenanceR > defaultReplication) {
throw new IOException("Unexpected configuration parameters: "
+ DFSConfigKeys.DFS_NAMENODE_MAINTENANCE_REPLICATION_MIN_KEY
+ " = " + minMaintenanceR + " > "
- + DFSConfigKeys.DFS_NAMENODE_REPLICATION_MIN_KEY
- + " = " + minR);
+ + DFSConfigKeys.DFS_REPLICATION_DEFAULT
+ + " = " + defaultReplication);
}
{noformat}
bq. the transition policy from ENTERING_MAINTENANCE to IN_MAINTENANCE will
become the # of live replicas >= min(dfs.namenode.maintenance.replication.min,
replication factor).
But, the transition from ENTERIN_MM to IN_MM that is happening
{{DecommissionManager#Monitor#check}} which in-turn calls
{{DecommissionManager#isSufficient}} looks ok to me. Because, we allow files to
be created with custom block replication count say 1, which can be lesser than
the default dfs.replication=3. And, since we should not be counting in the
Maintenance Replicas, the formula is, as it exists currently
{noformat}
expectedRedundancy = file_block_replication_count=1 or the
default_replication_cont=3
Math.max(
expectedRedundancy - numberReplicas.maintenanceReplicas(),
getMinMaintenanceStorageNum(block));
{noformat}
Let me know if I am missing something. Thanks.
--- related code snippets ----
{noformat}
/**
* Checks whether a block is sufficiently replicated/stored for
* decommissioning. For replicated blocks or striped blocks, full-strength
* replication or storage is not always necessary, hence "sufficient".
* @return true if sufficient, else false.
*/
private boolean isSufficient(BlockInfo block, BlockCollection bc,
NumberReplicas numberReplicas, boolean isDecommission) {
if (blockManager.hasEnoughEffectiveReplicas(block, numberReplicas, 0)) {
// Block has enough replica, skip
LOG.trace("Block {} does not need replication.", block);
return true;
}
..
..
..
// Check if the number of live + pending replicas satisfies
// the expected redundancy.
boolean hasEnoughEffectiveReplicas(BlockInfo block,
NumberReplicas numReplicas, int pendingReplicaNum) {
int required = getExpectedLiveRedundancyNum(block, numReplicas);
int numEffectiveReplicas = numReplicas.liveReplicas() + pendingReplicaNum;
return (numEffectiveReplicas >= required) &&
(pendingReplicaNum > 0 || isPlacementPolicySatisfied(block));
}
// Exclude maintenance, but make sure it has minimal live replicas
// to satisfy the maintenance requirement.
public short getExpectedLiveRedundancyNum(BlockInfo block,
NumberReplicas numberReplicas) {
final short expectedRedundancy = getExpectedRedundancyNum(block);
return (short) Math.max(expectedRedundancy -
numberReplicas.maintenanceReplicas(),
getMinMaintenanceStorageNum(block));
}
{noformat}
> Support maintenance state for datanodes
> ---------------------------------------
>
> Key: HDFS-7877
> URL: https://issues.apache.org/jira/browse/HDFS-7877
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Reporter: Ming Ma
> Assignee: Ming Ma
> Attachments: HDFS-7877-2.patch, HDFS-7877.patch,
> Supportmaintenancestatefordatanodes-2.pdf,
> Supportmaintenancestatefordatanodes.pdf
>
>
> This requirement came up during the design for HDFS-7541. Given this feature
> is mostly independent of upgrade domain feature, it is better to track it
> under a separate jira. The design and draft patch will be available soon.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
