[
https://issues.apache.org/jira/browse/HDDS-17523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17523:
------------------------------
Affects Version/s: 2.2.1
2.2.0
> Container Balancer is not resumed after SCM restart if datanodes are still
> registering at safe mode exit, and shouldRun stays true
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17523
> URL: https://issues.apache.org/jira/browse/HDDS-17523
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
>
> h3. Mechanism
> After an SCM restart or a leader change,
> {{ContainerBalancer.notifyStatusChanged}} calls {{ContainerBalancer.start}}
> when the persisted {{shouldRun}} is true. {{start}} runs the same
> {{validateConfiguration}} as a start requested by the user, and that now
> includes checks against the current cluster state: include and exclude
> datanodes must be registered (HDDS-14662), included containers must exist,
> and the number of healthy eligible datanodes times
> {{hdds.container.balancer.datanodes.involved.max.percentage.per.iteration}}
> must allow at least 2 datanodes (both HDDS-15535). If a check fails,
> {{notifyStatusChanged}} only logs "Could not start ContainerBalancer on
> raft/safe-mode status change." The persisted {{shouldRun}} stays true, no
> stop reason is recorded, and nothing retries: the notification is sent only
> on safe mode exit, leader ready and step down.
> The safe mode exit notification comes as soon as the exit rules pass, which
> can be before every datanode has registered (at least
> {{hdds.scm.safemode.min.datanode}} datanodes, default 3, and 99% of Ratis
> containers with at least one reported replica, among other rules). The
> balancing task already waits {{hdds.scm.wait.time.after.safemode.exit}}
> (default 5m) before its first iteration and evaluates the datanodes again in
> every iteration, but the check in {{start}} runs before that wait.
> h3. Trigger
> # On a cluster of 10 datanodes, run {{ozone admin containerbalancer start}}
> with default settings.
> # Restart SCM. Safe mode exits while 9 of the 10 datanodes have registered.
> With 3 replica containers, every container already has a reported replica
> once 8 of 10 datanodes have registered, and the other safe mode rules also
> pass with 9 of 10.
> # {{start}} throws "max-datanodes-percentage-to-involve-per-iteration=20
> allows at most 1 datanode(s) per iteration with 9 eligible datanode(s)". The
> last datanode registers moments later, but the balancer stays stopped.
> With the default 20% and no exclusions, a resume is rejected whenever fewer
> than 10 eligible datanodes have registered at the notification: one late
> datanode on a 10 datanode cluster, two on 11, three on 12. Larger clusters
> need more late datanodes, which gets less likely as the cluster grows.
> Exclusions and a non default percentage change which cluster sizes are
> exposed. An excluded or included datanode that has not registered yet also
> skips the resume, on any cluster size and also in 2.2.x. Two variants fail on
> every restart until the user acts: such a datanode that is down and does not
> register again, and an included container that was deleted in the meantime.
> h3. Impact
> A balancing run started by the user silently does not resume after a restart.
> {{ozone admin containerbalancer status}} reports it as not running with no
> stop reason, while the persisted state still says it should run, so the next
> restart or leader change tries again. Nothing is lost and the user can start
> it again. A leader change between running SCMs is less exposed, because
> followers also receive datanode heartbeats. I did not run a mini cluster
> restart; the scenarios were run on the mock SCM of {{TestContainerBalancer}}.
> h3. Reproduction
> PASS on unmodified source, deterministic.
> [^TestBugCR2BalancerNotResumedAfterRestart.java] ({{hadoop-hdds/server-scm}})
> uses the mock SCM of {{TestContainerBalancer}} with the persisted balancer
> state in memory. It starts the balancer with {{startBalancer}}, stops it as
> SCM shutdown does, then creates a second {{ContainerBalancer}} over the same
> persisted state and calls {{notifyStatusChanged}}. Three tests: 9 of 10
> datanodes healthy at the notification with default settings; an excluded
> datanode that has not registered; an included container that no longer exists
> (two restarts). Each asserts the warning, status STOPPED and {{shouldRun}}
> true. A passing test means the defect is present.
> h3. Patch
> [^CR-2-resume-balancer-while-datanodes-register.patch], against
> 9118d5a85a37db8c64404d56703b7bacb040e063. It also applies to master at
> f59f8d3d123.
> The cluster state checks move from {{validateConfiguration}} into a new
> {{validateClusterState}}, which only {{startBalancer}} calls, right after
> {{validateConfiguration}}. A start requested by the user is validated exactly
> as before, in the same order and with the same messages, so the failures
> HDDS-15535 reports to the CLI are unchanged. A resume still runs the size and
> timeout checks, and leaves the cluster state to the task, which looks at it
> after the post safe mode wait. If the cluster is still too small or nothing
> can be selected then, the task stops and persists {{shouldRun=false}}, with
> stop reason {{CAN_NOT_BALANCE_ANY_MORE}} (pool too small, no included
> container left) or {{INITIALIZATION_FAILED}} (included datanode not
> registered), as it does today when the cluster changes during a run and as
> resumes did before HDDS-14662.
> Covered by {{testNotifyStartsWhileDatanodesAreStillRegistering}} in the
> existing {{TestContainerBalancer}}: after the restart only 5 of 10 datanodes
> are healthy and the excluded datanode has not registered, {{startBalancer}}
> is still rejected, and {{notifyStatusChanged}} resumes the balancer with the
> persisted configuration. Without the change it fails with "expected:
> <RUNNING> but was: <STOPPED>".
> {{testNotifyResumeWithMissingIncludedContainerStopsWithReason}}, on the 20
> datanode mock cluster of the balancer task tests, resumes a run whose
> included container no longer exists: the task stops with
> {{CAN_NOT_BALANCE_ANY_MORE}}, schedules no move and persists
> {{shouldRun=false}}. Without the change it fails because {{shouldRun}} stays
> true.
> Found by code review of the SCM container balancer start, stop and failover
> resume paths, as part of the TLA+ verification effort under HDDS-15926, on
> commit 9118d5a85a37db8c64404d56703b7bacb040e063. Checked against HDDS issues
> and apache/ozone pull requests for duplicates before filing. The attached
> patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
