[
https://issues.apache.org/jira/browse/HDDS-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127223#comment-18127223
]
Siyao Meng commented on HDDS-16546:
-----------------------------------
Verification run complete. The Specula pipeline ran all phases against the
pinned commit: code analysis, TLA+ specification generation, harness and trace
collection, trace validation plus TLC model checking, bug confirmation with the
adversarial Challenger debate, and severity classification.
h3. Run environment
{noformat}
Ozone commit: cce1ca28944886218132f01c0e271c1fd0700078
Specula: v1.2.0-27-g0263f82a (with local changes)
Agent/model: claude-code, Claude Opus 5.5 (1M context)
Effort: medium
TLC limits: 28G memory, 8 workers
Total: 4h 19m agent time, 190.8M tokens, about $124 (estimate)
{noformat}
h3. Result
28 candidate findings came out of model checking (12) and code review (16), and
all 28 were investigated individually.
||Disposition||Count||
|Reproduced|19|
|Masked (defect present, harm prevented by another component today)|3|
|Limited by the test environment|1|
|False positive|5|
Each reproduced finding was then validated a second time by hand against the
pinned commit: a regression test was added to an existing suite and shown to
fail without the fix and pass with it, the bug was confirmed still present on
apache master (checked on 2026-10-14), and the fix patch, reproduction and
report were reviewed independently before filing. The finding limited by the
test environment was validated the same way and is filed with a reproduction
and no patch. One more defect found during that review was filed as well.
h3. Filed
||Issue||Priority||Summary||
|HDDS-17017|Major|Per disk RocksDB of a failed datanode volume is reopened by a
racing write or compaction and never closed until restart|
|HDDS-17018|Major|Container imported or created while its datanode volume fails
stays registered on the failed volume and is reported to SCM as live|
|HDDS-17019|Major|Failure of a datanode DB volume is not propagated to the data
volumes whose RocksDB it holds, so they stay in service|
|HDDS-17020|Major|Datanode volume failure closes the failed volume's RocksDB
while holding the volume set write lock, stalling container creates and node
reports|
|HDDS-17016|Major|Datanode with more failed volumes than tolerated keeps
serving after restart, then stops on a later volume check that finds nothing
wrong|
|HDDS-17022|Major|RECOVERING container (EC reconstruction target) keeps its
space reservation after it is closed, deleted or marked unhealthy|
|HDDS-17013|Major|Tarball and untar directory of a container import interrupted
by a datanode crash are never removed from the volume|
|HDDS-17012|Minor|Container data scan cut short by a closed RocksDB is accepted
as complete, so a partial checksum is reported and the replica stays CLOSED|
|HDDS-17009|Minor|Container create retried on another volume after an
IOException leaves the full container size reserved on the abandoned volume|
|HDDS-17026|Minor|Datanode container create that fails with an unchecked
exception leaks the reserved container size on the volume until restart|
|HDDS-17023|Minor|Failed container push releases the receiver's space
reservation twice when the sender's half close arrives after the error|
|HDDS-17024|Minor|Failed .container write during mark for close rolls the
container back to OPEN without its space reservation|
|HDDS-17025|Minor|Container push with a negative size reserves a negative
amount on the receiving volume that is never released|
|HDDS-17008|Minor|Duplicate container resolved by another volume's
ContainerReader at datanode start can leave a reservation for the deleted copy|
|HDDS-17014|Minor|Volume check during SchemaV3 finalization can fail a healthy
data volume because createDbStore sets dbLoaded before dbParentDir|
|HDDS-17015|Minor|ConcurrentModificationException in DbVolume.closeAllDbStore
leaves a failed DB volume active and stops the periodic disk check|
|HDDS-17010|Minor|Datanode volume check waits the full disk check timeout on an
empty volume set, delaying the check of the next volume set|
|HDDS-17011|Minor|Datanode automatic small SST file compaction leaks a native
CompactRangeOptions on every range compaction|
|HDDS-17021|Minor|Failed mark for close of a RECOVERING container drops it from
stale recovering tracking, so it is never marked UNHEALTHY|
Priorities follow the manual validation, not the pipeline's severity classes.
Each issue has a proposed fix patch and a standalone reproduction test
attached, except HDDS-17020, which has a reproduction and a suggested fix.
h3. Not filed
* Reproduced, already reported: a per disk RocksDB closed by RocksDB on ENOSPC
stays cached and its volume stays healthy. This is HDDS-13994.
* Reproduced, root cause already reported: a volume whose per disk DB did not
load at startup stays in service, which also lets the periodic compaction open
a RocksDB in the process working directory. This is HDDS-11485. The reservation
leak it causes on every container create is a separate defect and is filed
above.
* Masked: SchemaV3 finalization stops at the first volume that cannot create
its DB without failing that volume. The periodic volume check fails such a
volume and the next finalize retry succeeds, except for a permission fault on
the cluster ID directory alone. The proposed fix would fail a healthy data
volume when a dedicated DB volume is at fault, so it is not proposed.
* Masked: the per disk DB cache is shut down before background services stop,
so a late task can reopen a DB. Production shutdown always ends in process exit.
* Masked: a failed container ID table write leaves a half registered container.
The DB failure that causes it lasts for the rest of the process, and startup
rebuilds the table.
* False positives (5): the reported unsynchronized field is already volatile
and its NPE is handled like an IOException; no read only store parameter
exists; DB volumes are still checked (only a log line and a metric differ); the
only IOException in the reported check loop is a shutdown interrupt; the raw
string key and the changing storage ID harm no consumer.
h3. Reproduce
{code:none}
specula run --agent=claude-code --effort=medium --model='claude-opus-5-5[1m]' \
--max-parallel=2 --enable-reviews --confirm-debate \
--tlc-memory-limit=28G --tlc-worker-limit=8 \
"dn-volume-db-lifecycle|apache/ozone|Java|Use the target-specific guidance"
{code}
h3. Caveats
* Several reproductions need controlled timing: HDDS-17014 and HDDS-17008 hold
one thread with a latch or a slowed log appender, HDDS-17015 simulates the
concurrent DB registration with a mocked cache, and the create path of
HDDS-17018 holds the create with a spy. Without that help the windows were not
hit (for example 0 of 400 duplicate pairs for HDDS-17008).
* HDDS-17020 measured a bounded stall (2 s with a held iterator, about 51 s by
arithmetic for a full block at the default scanner rate). A hang that lasts as
long as a hung disk is argued from the code, not run.
* HDDS-17019 and HDDS-17016 change behaviour on purpose: data volumes whose DB
volume failed are now failed, and a datanode with more failed volumes than a
finite tolerance now stops at startup. HDDS-17016 has a paragraph meant for a
release note.
* The reproductions of HDDS-17012 and HDDS-17013 use file permissions and need
a non root user.
* Failures injected by mocks or set directly (a negative push size in
HDDS-17025, an unchecked create failure in the regression test of HDDS-17026)
were used only where the issue says so.
* The patches were built and tested on the pinned commit and apply to master at
2536fe9cbc3, alone and together where they touch the same code (HDDS-17017 with
HDDS-17018 and HDDS-17019; HDDS-17009 with HDDS-17026; HDDS-17022 with
HDDS-17024).
* The patches are proposals for review.
Generated with Specula (Claude Opus 5.5).
> [Specula] Verify dn-volume-db-lifecycle (Apache Ozone master @cce1ca2)
> ----------------------------------------------------------------------
>
> Key: HDDS-16546
> URL: https://issues.apache.org/jira/browse/HDDS-16546
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Minor
> Labels: help-wanted, specula
> Attachments: dn-volume-db-lifecycle.guidance.md
>
>
> This is a runnable Apache Ozone formal-verification target for the Specula
> pipeline (TLA+ model checking plus trace validation against the real code).
> It has not been run yet. Anyone can pick it up: run Specula against the
> pinned commit, then report findings on this issue. Part of umbrella
> HDDS-15926.
> h3. Target
> {{dn-volume-db-lifecycle}}. Scope and code entry points are in the attached
> guidance file [^dn-volume-db-lifecycle.guidance.md].
> h3. Run environment
> {noformat}
> Apache Ozone commit: cce1ca28944886218132f01c0e271c1fd0700078
> Specula: https://github.com/specula-org/Specula (v1.1.0 or later)
> Toolchain: JDK 21, ~28 GB RAM for TLC, an LLM coding agent
> (claude-code or codex adapter)
> Runtime: hours per target; needs an agent API budget
> {noformat}
> h3. How to run
> # Clone Specula and install per its README; clone apache/ozone and check out
> the pinned commit above.
> # Place the attached guidance file as the target's .prompt-extra.md (see the
> Specula README for target layout).
> # Run:
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-volume-db-lifecycle|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> # Deliverables land under the run's {{.specula-output/}}:
> {{confirmed-bugs.md}}, {{bug-severity.md}}, {{summary.md}}, and
> {{confirmation/<id>/investigation.md}}.
> h3. Reporting results
> Comment here with the outcome (0 findings is still useful: it records the
> target as covered at this commit). For each confirmed bug, file a separate
> Ozone Bug and link it to this issue with the "Testing" link type ("Testing
> discovered").
> h3. Notes
> * The attached guidance file names an earlier commit ({{20e6a1c}}); use the
> commit above. Code entry-point line numbers in the guidance were captured at
> an earlier commit and may have shifted on this head; treat them as
> approximate. Specula re-analyzes the source, so the run does not depend on
> them.
> * Recon-building targets need a local Specula patch to skip Maven {{target/}}
> out-of-tree symlinks in {{snapshotlib._validate_source_tree}}, else the run
> crashes at finalize.
> * effort=medium is the tested default for large targets; effort=high can hit
> per-turn timeouts on the biggest ones.
> * Findings are proposals, to be confirmed by human review and testing before
> any fix is merged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
