[
https://issues.apache.org/jira/browse/HDFS-16277?focusedWorklogId=667017&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-667017
]
ASF GitHub Bot logged work on HDFS-16277:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Oct/21 14:09
Start Date: 19/Oct/21 14:09
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #3559:
URL: https://github.com/apache/hadoop/pull/3559#discussion_r731901083
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -1195,13 +1195,24 @@
"dfs.namenode.available-space-block-placement-policy.balanced-space-preference-fraction";
public static final float
DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_DEFAULT
=
0.6f;
+ public static final String
DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_KEY
=
+
"dfs.namenode.available-space-block-placement-policy.balanced-space-tolerate";
+ public static final int
DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_DEFAULT
=
+ 5;
public static final String
DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_KEY
=
"dfs.namenode.available-space-rack-fault-tolerant-block-placement-policy"
+ ".balanced-space-preference-fraction";
public static final float
DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_RACK_FAULT_TOLERANT_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_DEFAULT
=
0.6f;
+ public static final String
+
DFS_NAMENODE_AVAILABLE_SPACE_RACK_FAULT_TOLERANT_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_KEY
=
+
"dfs.namenode.available-space-rack-fault-tolerant-block-placement-policy"
+ + ".balanced-space-tolerate";
Review comment:
Can you change tolerate to tolerance everywhere in the config names
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
##########
@@ -37,6 +34,8 @@
import org.apache.hadoop.net.NetworkTopology;
import org.apache.hadoop.net.Node;
+import static org.apache.hadoop.hdfs.DFSConfigKeys.*;
+
Review comment:
Don't club imports. Keep all imports separate and the existing ones
untouched
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsBlockPlacementPolicies.md
##########
@@ -119,6 +119,24 @@ The AvailableSpaceBlockPlacementPolicy is a space balanced
block placement polic
</description>
</property>
+<property>
+<name>dfs.namenode.available-space-block-placement-policy.balanced-space-tolerate</name>
+<value>5</value>
+<description>
+ Special value between 0 and 100, noninclusive. Increases tolerance of
+ placing blocks on Datanodes with similar disk space used.
+</description>
+</property>
+
+<property>
+
<name>dfs.namenode.available-space-block-placement-policy.datanode-usgae-tolerace-fraction</name>
+ <value>0.6</value>
+ <description>
+ Special value between 1 and 100, noninclusive. Increases chance of
+ placing blocks on Datanodes with less disk space used.
+</description>
+</property>
Review comment:
Didn't catch what is this? Some copy-paste error?
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
##########
@@ -62,6 +63,11 @@ public void initialize(Configuration conf, FSClusterStats
stats,
+
DFSConfigKeys.DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_PREFERENCE_FRACTION_KEY
+ " = " + balancedPreferencePercent);
+ balancedTolerate =
+ conf.getInt(
+
DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_KEY,
+
DFS_NAMENODE_AVAILABLE_SPACE_BLOCK_PLACEMENT_POLICY_BALANCED_SPACE_TOLERATE_DEFAULT);
+
Review comment:
Rename this variable, current name doesn't make much sense.
I think we should have some validation on this value, and an upper limit as
well, May be at max 20,(document that as well)
If the value is not in the valid range(0-20) add a warn log and use the
default value.
Same for the other class as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 667017)
Time Spent: 40m (was: 0.5h)
> improve decision in AvailableSpaceBlockPlacementPolicy
> ------------------------------------------------------
>
> Key: HDFS-16277
> URL: https://issues.apache.org/jira/browse/HDFS-16277
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: block placement
> Affects Versions: 3.3.1
> Reporter: guo
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Hi
> In product environment,we may meet two or more datanode usage reaches nealy
> 100%,for exampe 99.99%,98%,97%.
> if we configure `AvailableSpaceBlockPlacementPolicy` , we also have the
> chance to choose the 99.99%(assume it is the highest usage),for we treat the
> two choosen datanode as the same usage if their storage usage are different
> within 5%.
> but this is not what we want, so i suggest we can improve the decision.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
