[
https://issues.apache.org/jira/browse/HDDS-16571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133525#comment-18133525
]
Siyao Meng commented on HDDS-16571:
-----------------------------------
Verification run complete. The Specula pipeline ran all phases against the
pinned commit: code analysis, TLA+ specification generation, harness and trace
collection, trace validation plus TLC model checking, bug confirmation with the
adversarial Challenger debate, and severity classification.
h3. Run environment
{noformat}
Ozone commit: 9118d5a85a37db8c64404d56703b7bacb040e063
Specula: v1.2.0 plus 35 commits (f5285a2b), with local changes
Agent/model: claude-code, Claude Opus 5.5 (1M context)
Effort: medium
TLC limits: 28G memory, 8 workers
Total: 1h 57m agent time, 52.6M tokens, about $40 (estimate)
{noformat}
h3. Result
10 candidate findings came out of model checking (3) and code review (7), and
all 10 were investigated individually.
||Disposition||Count||
|Reproduced|4|
|Masked (defect present, harm prevented by another component today)|2|
|False positive|4|
Three of the reproduced findings come from the same weak spot: the persisted
{{shouldRun}} flag and what the balancer actually does can drift apart. They
need separate fixes, so they are separate issues. Each issue was then validated
a second time by hand against the pinned commit: a regression test was added to
{{TestContainerBalancer}} and shown to fail without the fix and pass with it,
the bug was confirmed still present on apache master (checked on 2026-10-28),
and the fix patch, reproduction and report were reviewed independently before
filing.
h3. Filed
||Issue||Priority||Summary||
|HDDS-17523|Major|Container Balancer is not resumed after SCM restart if
datanodes are still registering at safe mode exit, and shouldRun stays true|
|HDDS-17524|Minor|Container Balancer stop can be overwritten by the balancing
task's progress save, so the stopped balancer resumes after SCM failover or
restart|
|HDDS-17525|Minor|Container balancer that stops on its own keeps persisted
shouldRun=true if the save fails, so a later leader change starts a new run|
|HDDS-17522|Minor|Stopping Container Balancer during its delayed start logs a
false ERROR "Container Balancer is stopped abnormally"|
Priorities follow the manual validation, not the pipeline's severity classes.
Each issue has a proposed fix patch and a standalone reproduction test attached.
h3. Not filed
* Masked: a leader ready notification that arrives while the previous balancer
task is still stopping is dropped, so a ready leader can keep
{{shouldRun=true}} with no balancer running. It was reached only with an added
source delay (0 of 300 attempts on unmodified code), because the step down stop
waits for the task and re-election takes seconds. A non blocking {{stop()}}
(HDDS-16029) would remove that mask.
* Masked: a late {{notifyLeaderReady}} after {{notifyNotLeader}} can leave a
follower with {{isLeaderReady=true}}. For the balancer, the {{isLeader}} check
on the notify path, the NotLeaderException on the replicated save and the step
down stop prevent any effect. Other SCM services that check only
{{isLeaderReady()}} were not examined.
* False positive: a stop whose save times out but later commits leaves the task
running with {{shouldRun=false}} persisted. Every outcome is one an admin can
get from a timed out stop, and the save first order is intended so that a stop
is never lost.
* False positive: an interrupted move wait could let the saved iteration index
skip unfinished work. Every path that interrupts the task marks it stopping
first, so the RUNNING re-check always fires.
* False positive: {{ContainerBalancer.saveConfiguration}} writes the config
field from the task thread without a lock. The task only writes back the same
config object, so no thread sees an older config.
* False positive: a new task's {{MoveManager.resetState()}} drops pending move
futures without completing them. No thread can still be waiting on them at that
point.
h3. Reproduce
{code:none}
specula run --agent=claude-code --effort=medium --model='claude-opus-5-5[1m]' \
--max-parallel=2 --enable-reviews --confirm-debate \
--tlc-memory-limit=28G --tlc-worker-limit=8 \
"scm-container-balancer|apache/ozone|Java|Use the target-specific guidance"
{code}
h3. Caveats
* All reproductions run in the {{TestContainerBalancer}} mock SCM harness, not
on a real Ratis cluster. Failover, restart and safe mode exit are driven
through the real {{notifyStatusChanged}} path. With the fixes of HDDS-17523,
HDDS-17524 and HDDS-17522 applied, {{TestContainerBalancerOperations}} passed
on a mini cluster, and {{TestFailoverWithSCMHA}} also passed with the
HDDS-17523 fix. The HDDS-17525 fix was not run on a mini cluster.
* HDDS-17523 with the default configuration needs a cluster of 10 or more
datanodes where enough of them are still unregistered at safe mode exit (1 of
10, 2 of 11, 3 of 12). The safe mode arithmetic is calculated, not observed on
a real restart. In 2.2.x it is reached through an include or exclude datanode
that is not registered yet.
* HDDS-17524 needs the stop to land within one Ratis round trip of an iteration
end, so with the default 3 minute interval it is rare per stop (argued from the
code, not measured). It was hit without timing help only with a balancing
interval of 0.
* HDDS-17525 needs the Ratis write of the final save to be rejected while the
SCM stays leader, for example by the pending request limit or a leadership
transfer that fails. The tests inject that failure.
* The patches were built and tested on the pinned commit and apply to master at
f59f8d3d123. HDDS-17524 changes the same part of {{TestContainerBalancer}} as
HDDS-17523 and HDDS-17525, so whichever lands second needs a trivial rebase of
that test file. All other pairs apply on top of each other.
* The patches are proposals for review.
Generated with Specula (Claude Opus 5.5).
> [Specula] Verify scm-container-balancer (Apache Ozone master @9118d5a)
> ----------------------------------------------------------------------
>
> Key: HDDS-16571
> URL: https://issues.apache.org/jira/browse/HDDS-16571
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Minor
> Labels: specula
> Attachments: scm-container-balancer.guidance.md
>
>
> This is a runnable Apache Ozone formal-verification target for the Specula
> pipeline (TLA+ model checking plus trace validation against the real code).
> It has not been run yet. Anyone can pick it up: run Specula against the
> pinned commit, then report findings on this issue. Part of umbrella
> HDDS-15926.
> h3. Target
> {{scm-container-balancer}}. Scope and code entry points are in the attached
> guidance file [^scm-container-balancer.guidance.md].
> h3. Run environment
> {noformat}
> Apache Ozone commit: 9118d5a85a37db8c64404d56703b7bacb040e063
> Specula: https://github.com/specula-org/Specula (v1.1.0 or later)
> Toolchain: JDK 21, ~28 GB RAM for TLC, an LLM coding agent
> (claude-code or codex adapter)
> Runtime: hours per target; needs an agent API budget
> {noformat}
> h3. How to run
> # Clone Specula and install per its README; clone apache/ozone and check out
> the pinned commit above.
> # Place the attached guidance file as the target's .prompt-extra.md (see the
> Specula README for target layout).
> # Run:
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "scm-container-balancer|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> # Deliverables land under the run's {{.specula-output/}}:
> {{confirmed-bugs.md}}, {{bug-severity.md}}, {{summary.md}}, and
> {{confirmation/<id>/investigation.md}}.
> h3. Reporting results
> Comment here with the outcome (0 findings is still useful: it records the
> target as covered at this commit). For each confirmed bug, file a separate
> Ozone Bug and link it to this issue with the "Testing" link type ("Testing
> discovered").
> h3. Notes
> * The attached guidance file names an earlier commit ({{20e6a1c}}); use the
> commit above. Code entry-point line numbers in the guidance were captured at
> an earlier commit and may have shifted on this head; treat them as
> approximate. Specula re-analyzes the source, so the run does not depend on
> them.
> * Recon-building targets need a local Specula patch to skip Maven {{target/}}
> out-of-tree symlinks in {{snapshotlib._validate_source_tree}}, else the run
> crashes at finalize.
> * effort=medium is the tested default for large targets; effort=high can hit
> per-turn timeouts on the biggest ones.
> * Findings are proposals, to be confirmed by human review and testing before
> any fix is merged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
