[
https://issues.apache.org/jira/browse/HDFS-15904?focusedWorklogId=568785&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-568785
]
ASF GitHub Bot logged work on HDFS-15904:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Mar/21 05:09
Start Date: 19/Mar/21 05:09
Worklog Time Spent: 10m
Work Description: liuml07 commented on a change in pull request #2785:
URL: https://github.com/apache/hadoop/pull/2785#discussion_r597410126
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
##########
@@ -2297,7 +2297,9 @@ public void testBalancerWithSortTopNodes() throws
Exception {
maxUsage = Math.max(maxUsage, datanodeReport[i].getDfsUsed());
}
- assertEquals(200, balancerResult.bytesAlreadyMoved);
+ // Either 2 blocks of 100+100 bytes or 3 blocks of 100+100+50 bytes
Review comment:
Could add some explanation why this would happen.
The 95% usage DN will have 9 blocks of 100 bytes and 1 block of 50 byte -
all for the same file. The HDFS balancer will choose a block to move from this
node randomly. More likely it will be 100B block. Since that is greater than
`DFS_BALANCER_MAX_SIZE_TO_MOVE_KEY` which is 99L (see above settings), it will
stop here. Total bytes moved from this 95% DN will be 1 block and hence 100B.
However, chances are the first block to move from this 95% DN is the 50B
block. After this block being moved, the total moved size 50B is smaller than
`DFS_BALANCER_MAX_SIZE_TO_MOVE_KEY` , it will try to move another block. The
second block will always be 100 bytes. So total bytes moved from this 95% DN
will be 2 blocks and hence 150B (100B + 50B).
Please reword or rephrase this as comment before this assertion so readers
can have more context without thinking too much again.
Thanks,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 568785)
Time Spent: 50m (was: 40m)
> Flaky test TestBalancer#testBalancerWithSortTopNodes()
> ------------------------------------------------------
>
> Key: HDFS-15904
> URL: https://issues.apache.org/jira/browse/HDFS-15904
> Project: Hadoop HDFS
> Issue Type: Test
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TestBalancer#testBalancerWithSortTopNodes shows some flakes in around ~10
> runs or so. It's reproducible locally also. Basically, balancing either moves
> 2 blocks of size 100+100 bytes or it moves 3 blocks of size 100+100+50 bytes
> (2nd case causes flakies).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
