[ 
https://issues.apache.org/jira/browse/HDDS-17847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated HDDS-17847:
-----------------------------------
        Parent: HDDS-5626
    Issue Type: Sub-task  (was: Bug)

> Intermittent failure in balancer acceptance test: recommissioned datanode 
> keeps 2 replicas instead of 3
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17847
>                 URL: https://issues.apache.org/jira/browse/HDDS-17847
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: Balancer
>    Affects Versions: 2.3.0
>            Reporter: Sergey Soldatov
>            Assignee: Sergey Soldatov
>            Priority: Major
>
> Verify Container Balancer for RATIS/EC containers in 
> ozone-balancer/test-ratis.sh fails sometimes:
> {noformat}
> '209719296 > 104857600 * 3' should be true.
> {noformat}
> The test empties datanode1, writes 3 keys of 100 MB (9 replicas on 4 
> datanodes), puts datanode1 back in service and runs:
> {noformat}
> ozone admin containerbalancer start -t 0.1 -d 100 -i 3
> {noformat}
> It then expects datanode1 to hold exactly 3 replicas (LOWER_LIMIT:3, 
> UPPER_LIMIT:3.5).
> The threshold is a percentage, so -t 0.1 allows only about 2 MB of difference 
> on a 2 GB datanode. With 9 replicas on 4 datanodes the cluster can never be 
> balanced: a datanode with 3 replicas is always over-utilized. The balancer 
> keeps moving containers until it runs out of iterations or valid moves.
> In the failed run, iteration 1 moved all 3 containers to datanode1. In 
> iteration 2 datanode1 counted as over-utilized and the balancer moved one 
> container back. All moves succeeded, so the balancer did what it was asked. 
> Where the third replica ends up depends on how fresh the datanode usage in 
> SCM is (node report every 20 s, iteration interval 25 s).
> Fix: in test-ratis.sh use LOWER_LIMIT:2 (or 1.9). The test then checks that 
> datanode1 received data and ended with 2 or 3 replicas. The earlier step in 
> the test already checks that data moved.
> Seen in:
> * 2026-11-16, attempt 1 (passed on re-run): 
> https://github.com/ss77892/ozone/actions/runs/43892876037/job/133394786063



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to