[
https://issues.apache.org/jira/browse/HDDS-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17522:
----------------------------------
Labels: pull-request-available (was: )
> Stopping Container Balancer during its delayed start logs a false ERROR
> "Container Balancer is stopped abnormally"
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17522
> URL: https://issues.apache.org/jira/browse/HDDS-17522
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Assignee: Meng-Shuan Tsai
> Priority: Minor
> Labels: pull-request-available
> Attachments: CR-7-delayed-start-interrupt-is-normal-stop.patch,
> TestBugCR7DelayedStartInterruptLoggedAsError.java
>
>
> h3. Mechanism
> After a leader change, safe mode exit or SCM restart with {{shouldRun=true}}
> persisted, {{ContainerBalancer.start}} starts the task with a delayed start,
> and {{ContainerBalancerTask.run}} sleeps for
> {{hdds.scm.wait.time.after.safemode.exit}} (default 5 minutes) before
> balancing. That sleep is inside the {{try}} whose generic {{catch (Exception
> e)}} logs ERROR "Container Balancer is stopped abnormally" with the
> exception. Both stop paths ({{ContainerBalancer.stopBalancer}} and
> {{ContainerBalancer.stop}}) interrupt the task thread, so a stop during the
> delay always ends in that catch with {{java.lang.InterruptedException: sleep
> interrupted}}. The same interrupt during the sleeps inside
> {{ContainerBalancerTask.balance}} is handled as a normal stop: logged at
> INFO, with the interrupt flag restored.
> h3. Trigger
> # The balancer is running, so {{shouldRun=true}} is persisted.
> # SCM restarts, fails over or leaves safe mode; the new task sleeps for the
> delayed start.
> # Within that delay the operator runs {{ozone admin containerbalancer stop}},
> or the SCM steps down, enters safe mode or shuts down.
> h3. Impact
> A requested stop is reported in the SCM log as an ERROR with an exception,
> which can mislead operators and trigger log based alerting. Nothing else is
> wrong: the stop reason shown by {{ozone admin containerbalancer status}} is
> correct (USER_REQUESTED or SCM_STATE_CHANGE), because both stop paths record
> their reason before interrupting and the first recorded reason is kept, and
> the thread exits right after the catch, so the lost interrupt flag has no
> effect. (The stop reason display comes from HDDS-15667, which is not released
> yet, so released versions show only the false ERROR log.)
> h3. Reproduction
> PASS on unmodified source, deterministic.
> [^TestBugCR7DelayedStartInterruptLoggedAsError.java]
> ({{hadoop-hdds/server-scm}}) uses the mock SCM of {{TestContainerBalancer}}
> with the persisted state in memory and a 300 second delay. It persists
> {{shouldRun=true}}, calls {{start}} and waits for the task to sleep, then
> stops it with {{stopBalancer}} in one test and with {{stop}} in another; both
> assert the ERROR line and the {{InterruptedException}} in the task log and
> the correct stop reason. A control test stops during the sleep in {{balance}}
> and asserts that no ERROR is logged. A passing test means the defect is
> present.
> h3. Patch
> [^CR-7-delayed-start-interrupt-is-normal-stop.patch], against
> 9118d5a85a37db8c64404d56703b7bacb040e063. It also applies to master at
> f59f8d3d123.
> {{ContainerBalancerTask.run}} catches {{InterruptedException}} before the
> generic catch, logs "Container Balancer was interrupted during the delayed
> start." at INFO and restores the interrupt flag, as {{balance}} does.
> {{balance}} declares no checked exception, so only the delayed start sleep
> reaches the new catch; other exceptions are still logged as ERROR and
> recorded with the ERROR stop reason.
> {{TestContainerBalancer#testDelayedStartOnSCMStatusChange}}, which already
> stops the balancer during the delay, now also asserts that the task log has
> no "stopped abnormally" and that the stop reason is USER_REQUESTED. Without
> the change it fails on the ERROR line.
> Found by code review of the SCM container balancer start, stop and failover
> resume paths, as part of the TLA+ verification effort under HDDS-15926, on
> commit 9118d5a85a37db8c64404d56703b7bacb040e063. Checked against HDDS issues
> and apache/ozone pull requests for duplicates before filing. The attached
> patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
