Siyao Meng created HDDS-17522:
---------------------------------
Summary: Stopping Container Balancer during its delayed start logs
a false ERROR "Container Balancer is stopped abnormally"
Key: HDDS-17522
URL: https://issues.apache.org/jira/browse/HDDS-17522
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-7-delayed-start-interrupt-is-normal-stop.patch,
TestBugCR7DelayedStartInterruptLoggedAsError.java
h3. Mechanism
After a leader change, safe mode exit or SCM restart with {{shouldRun=true}}
persisted, {{ContainerBalancer.start}} starts the task with a delayed start,
and {{ContainerBalancerTask.run}} sleeps for
{{hdds.scm.wait.time.after.safemode.exit}} (default 5 minutes) before
balancing. That sleep is inside the {{try}} whose generic {{catch (Exception
e)}} logs ERROR "Container Balancer is stopped abnormally" with the exception.
Both stop paths ({{ContainerBalancer.stopBalancer}} and
{{ContainerBalancer.stop}}) interrupt the task thread, so a stop during the
delay always ends in that catch with {{java.lang.InterruptedException: sleep
interrupted}}. The same interrupt during the sleeps inside
{{ContainerBalancerTask.balance}} is handled as a normal stop: logged at INFO,
with the interrupt flag restored.
h3. Trigger
# The balancer is running, so {{shouldRun=true}} is persisted.
# SCM restarts, fails over or leaves safe mode; the new task sleeps for the
delayed start.
# Within that delay the operator runs {{ozone admin containerbalancer stop}},
or the SCM steps down, enters safe mode or shuts down.
h3. Impact
A requested stop is reported in the SCM log as an ERROR with an exception,
which can mislead operators and trigger log based alerting. Nothing else is
wrong: the stop reason shown by {{ozone admin containerbalancer status}} is
correct (USER_REQUESTED or SCM_STATE_CHANGE), because both stop paths record
their reason before interrupting and the first recorded reason is kept, and the
thread exits right after the catch, so the lost interrupt flag has no effect.
(The stop reason display comes from HDDS-15667, which is not released yet, so
released versions show only the false ERROR log.)
h3. Reproduction
PASS on unmodified source, deterministic.
[^TestBugCR7DelayedStartInterruptLoggedAsError.java]
({{hadoop-hdds/server-scm}}) uses the mock SCM of {{TestContainerBalancer}}
with the persisted state in memory and a 300 second delay. It persists
{{shouldRun=true}}, calls {{start}} and waits for the task to sleep, then stops
it with {{stopBalancer}} in one test and with {{stop}} in another; both assert
the ERROR line and the {{InterruptedException}} in the task log and the correct
stop reason. A control test stops during the sleep in {{balance}} and asserts
that no ERROR is logged. A passing test means the defect is present.
h3. Patch
[^CR-7-delayed-start-interrupt-is-normal-stop.patch], against
9118d5a85a37db8c64404d56703b7bacb040e063. It also applies to master at
f59f8d3d123.
{{ContainerBalancerTask.run}} catches {{InterruptedException}} before the
generic catch, logs "Container Balancer was interrupted during the delayed
start." at INFO and restores the interrupt flag, as {{balance}} does.
{{balance}} declares no checked exception, so only the delayed start sleep
reaches the new catch; other exceptions are still logged as ERROR and recorded
with the ERROR stop reason.
{{TestContainerBalancer#testDelayedStartOnSCMStatusChange}}, which already
stops the balancer during the delay, now also asserts that the task log has no
"stopped abnormally" and that the stop reason is USER_REQUESTED. Without the
change it fails on the ERROR line.
Found by code review of the SCM container balancer start, stop and failover
resume paths, as part of the TLA+ verification effort under HDDS-15926, on
commit 9118d5a85a37db8c64404d56703b7bacb040e063. Checked against HDDS issues
and apache/ozone pull requests for duplicates before filing. The attached patch
is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
