Siyao Meng created HDDS-17523:
---------------------------------
Summary: Container Balancer is not resumed after SCM restart if
datanodes are still registering at safe mode exit, and shouldRun stays true
Key: HDDS-17523
URL: https://issues.apache.org/jira/browse/HDDS-17523
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Mechanism
After an SCM restart or a leader change,
{{ContainerBalancer.notifyStatusChanged}} calls {{ContainerBalancer.start}}
when the persisted {{shouldRun}} is true. {{start}} runs the same
{{validateConfiguration}} as a start requested by the user, and that now
includes checks against the current cluster state: include and exclude
datanodes must be registered (HDDS-14662), included containers must exist, and
the number of healthy eligible datanodes times
{{hdds.container.balancer.datanodes.involved.max.percentage.per.iteration}}
must allow at least 2 datanodes (both HDDS-15535). If a check fails,
{{notifyStatusChanged}} only logs "Could not start ContainerBalancer on
raft/safe-mode status change." The persisted {{shouldRun}} stays true, no stop
reason is recorded, and nothing retries: the notification is sent only on safe
mode exit, leader ready and step down.
The safe mode exit notification comes as soon as the exit rules pass, which can
be before every datanode has registered (at least
{{hdds.scm.safemode.min.datanode}} datanodes, default 3, and 99% of Ratis
containers with at least one reported replica, among other rules). The
balancing task already waits {{hdds.scm.wait.time.after.safemode.exit}}
(default 5m) before its first iteration and evaluates the datanodes again in
every iteration, but the check in {{start}} runs before that wait.
h3. Trigger
# On a cluster of 10 datanodes, run {{ozone admin containerbalancer start}}
with default settings.
# Restart SCM. Safe mode exits while 9 of the 10 datanodes have registered.
With 3 replica containers, every container already has a reported replica once
8 of 10 datanodes have registered, and the other safe mode rules also pass with
9 of 10.
# {{start}} throws "max-datanodes-percentage-to-involve-per-iteration=20 allows
at most 1 datanode(s) per iteration with 9 eligible datanode(s)". The last
datanode registers moments later, but the balancer stays stopped.
With the default 20% and no exclusions, a resume is rejected whenever fewer
than 10 eligible datanodes have registered at the notification: one late
datanode on a 10 datanode cluster, two on 11, three on 12. Larger clusters need
more late datanodes, which gets less likely as the cluster grows. Exclusions
and a non default percentage change which cluster sizes are exposed. An
excluded or included datanode that has not registered yet also skips the
resume, on any cluster size and also in 2.2.x. Two variants fail on every
restart until the user acts: such a datanode that is down and does not register
again, and an included container that was deleted in the meantime.
h3. Impact
A balancing run started by the user silently does not resume after a restart.
{{ozone admin containerbalancer status}} reports it as not running with no stop
reason, while the persisted state still says it should run, so the next restart
or leader change tries again. Nothing is lost and the user can start it again.
A leader change between running SCMs is less exposed, because followers also
receive datanode heartbeats. I did not run a mini cluster restart; the
scenarios were run on the mock SCM of {{TestContainerBalancer}}.
h3. Reproduction
PASS on unmodified source, deterministic.
[^TestBugCR2BalancerNotResumedAfterRestart.java] ({{hadoop-hdds/server-scm}})
uses the mock SCM of {{TestContainerBalancer}} with the persisted balancer
state in memory. It starts the balancer with {{startBalancer}}, stops it as SCM
shutdown does, then creates a second {{ContainerBalancer}} over the same
persisted state and calls {{notifyStatusChanged}}. Three tests: 9 of 10
datanodes healthy at the notification with default settings; an excluded
datanode that has not registered; an included container that no longer exists
(two restarts). Each asserts the warning, status STOPPED and {{shouldRun}}
true. A passing test means the defect is present.
h3. Patch
[^CR-2-resume-balancer-while-datanodes-register.patch], against
9118d5a85a37db8c64404d56703b7bacb040e063. It also applies to master at
f59f8d3d123.
The cluster state checks move from {{validateConfiguration}} into a new
{{validateClusterState}}, which only {{startBalancer}} calls, right after
{{validateConfiguration}}. A start requested by the user is validated exactly
as before, in the same order and with the same messages, so the failures
HDDS-15535 reports to the CLI are unchanged. A resume still runs the size and
timeout checks, and leaves the cluster state to the task, which looks at it
after the post safe mode wait. If the cluster is still too small or nothing can
be selected then, the task stops and persists {{shouldRun=false}}, with stop
reason {{CAN_NOT_BALANCE_ANY_MORE}} (pool too small, no included container
left) or {{INITIALIZATION_FAILED}} (included datanode not registered), as it
does today when the cluster changes during a run and as resumes did before
HDDS-14662.
Covered by {{testNotifyStartsWhileDatanodesAreStillRegistering}} in the
existing {{TestContainerBalancer}}: after the restart only 5 of 10 datanodes
are healthy and the excluded datanode has not registered, {{startBalancer}} is
still rejected, and {{notifyStatusChanged}} resumes the balancer with the
persisted configuration. Without the change it fails with "expected: <RUNNING>
but was: <STOPPED>".
{{testNotifyResumeWithMissingIncludedContainerStopsWithReason}}, on the 20
datanode mock cluster of the balancer task tests, resumes a run whose included
container no longer exists: the task stops with {{CAN_NOT_BALANCE_ANY_MORE}},
schedules no move and persists {{shouldRun=false}}. Without the change it fails
because {{shouldRun}} stays true.
Found by code review of the SCM container balancer start, stop and failover
resume paths, as part of the TLA+ verification effort under HDDS-15926, on
commit 9118d5a85a37db8c64404d56703b7bacb040e063. Checked against HDDS issues
and apache/ozone pull requests for duplicates before filing. The attached patch
is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
