[
https://issues.apache.org/jira/browse/HDFS-17037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729172#comment-17729172
]
ASF GitHub Bot commented on HDFS-17037:
---------------------------------------
Hexiaoqiao commented on code in PR #5715:
URL: https://github.com/apache/hadoop/pull/5715#discussion_r1217483230
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java:
##########
@@ -104,21 +104,21 @@ void accumulateSpaces(DatanodeStorageReport r) {
for(StorageReport s : r.getStorageReports()) {
final StorageType t = s.getStorage().getStorageType();
totalCapacities.add(t, s.getCapacity());
- totalUsedSpaces.add(t, s.getDfsUsed());
+ totalUsedSpaces.add(t, (s.getCapacity() - s.getRemaining()));
}
}
@Override
Double getUtilization(DatanodeStorageReport r, final StorageType t) {
long capacity = 0L;
- long dfsUsed = 0L;
+ long totalUsed = 0L;
for(StorageReport s : r.getStorageReports()) {
if (s.getStorage().getStorageType() == t) {
capacity += s.getCapacity();
- dfsUsed += s.getDfsUsed();
+ totalUsed += (s.getCapacity() - s.getRemaining());
}
}
- return capacity == 0L? null: dfsUsed*100.0/capacity;
+ return capacity == 0L? null: totalUsed*100.0/capacity;
Review Comment:
should fix the codestyle.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java:
##########
@@ -104,21 +104,21 @@ void accumulateSpaces(DatanodeStorageReport r) {
for(StorageReport s : r.getStorageReports()) {
Review Comment:
IMO, the BalancingPolicy.Pool also have the same issue.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java:
##########
@@ -104,21 +104,21 @@ void accumulateSpaces(DatanodeStorageReport r) {
for(StorageReport s : r.getStorageReports()) {
final StorageType t = s.getStorage().getStorageType();
totalCapacities.add(t, s.getCapacity());
- totalUsedSpaces.add(t, s.getDfsUsed());
+ totalUsedSpaces.add(t, (s.getCapacity() - s.getRemaining()));
Review Comment:
It could not be true when using `s.getCapacity() - s.getRemaining()`
directly. For instance, it will be negative if one storage with total capacity
10MB，but configured Capacity 1MB for hdfs, and it remain 8MB after run for a
while, then `s.getCapacity() - s.getRemaining()` will be -7*1024*1024.
> Consider nonDfsUsed when running balancer
> -----------------------------------------
>
> Key: HDFS-17037
> URL: https://issues.apache.org/jira/browse/HDFS-17037
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Shuyan Zhang
> Assignee: Shuyan Zhang
> Priority: Major
> Labels: pull-request-available
>
> When we run balancer with `BalancingPolicy.Node` policy, our goal is to make
> each datanode storage balanced. But in the current implementation, the
> balancer doesn't account for storage used by non-dfs on the datanodes, which
> can make the situation worse for datanodes that are already strained on
> storage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
