[
https://issues.apache.org/jira/browse/HDDS-16474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125412#comment-18125412
]
Siyao Meng commented on HDDS-16474:
-----------------------------------
Verification run complete. The Specula pipeline ran all phases against the
pinned commit: code analysis, TLA+ specification generation, harness and trace
collection, trace validation plus TLC model checking, bug confirmation with the
adversarial Challenger debate, and severity classification.
h3. Run environment
{noformat}
Ozone commit: 1e528b97aed3eef1be6846f1f0767c1ef98a883a
Specula: v1.2.0 plus 24 commits (6190c183), with local changes
Agent/model: claude-code, Claude Opus 5.5 (1M context)
Effort: medium
TLC limits: 28G memory, 8 workers
Total: 5h 31m agent time, 137.2M tokens, about $106 (estimate)
{noformat}
h3. Result
12 candidate findings came out of model checking (8) and code review (4), and
all 12 were investigated individually. No spec repair requests were raised by
confirmation.
||Disposition||Count||
|Reproduced|9|
|Masked (defect present, harm prevented by another component today)|2|
|False positive|1|
Two reproduced findings share one fix, so they are one issue. Each issue was
then validated a second time by hand against the pinned commit: a regression
test was added to an existing suite and shown to fail without the fix and pass
with it, the bug was confirmed still present on apache master (checked on
2026-10-09), and the fix patch, reproduction and report were reviewed
independently before filing. The reviews changed several of them: one fix was
replaced by a different design, one was found to be incomplete and is filed
with that limit stated, and two reports were corrected in what they claim. Two
more issues (HDDS-16814 and HDDS-16815) were found during those reviews and
went through the same validation and review. They are not pipeline findings and
are not in the counts above.
h3. Filed
||Issue||Priority||Summary||
|HDDS-16809|Major|Empty QUASI_CLOSED container is deleted although no known
replica has its sequence ID, and the returning data replica is force deleted|
|HDDS-16810|Major|Reconciliation raises the container BCSID to a block BCSID
taken from a peer whose own container BCSID is lower|
|HDDS-16811|Major|Reconciliation can restore a deleted block, or commit it
without its file when block deletion runs during the repair|
|HDDS-16807|Major|DiskBalancer can move a container that is being reconciled,
the repairs are lost and SCM gets a report from the retired replica|
|HDDS-16804|Minor|ReplicationSupervisor loses a command of the new SCM leader
when an equal task of the previous leader is still queued|
|HDDS-16805|Minor|Container reconciliation does not verify chunks pulled from a
peer when ozone.client.verify.checksum is false on the datanode|
|HDDS-16806|Minor|Container scanner builds an empty merkle tree for containers
written with checksum type NONE, which breaks reconciliation of those
containers|
|HDDS-16808|Minor|Data checksum reported to SCM and stored in RocksDB is not
refreshed when block deletion changes the container checksum tree|
|HDDS-16815|Major|Container data scanner thread ends when a chunk's stored
bytes per checksum is invalid, and a large stored value sizes the scan's read
buffer|
|HDDS-16814|Minor|DiskBalancer move drops the in memory data checksum of the
moved container replica, so the datanode reports data checksum 0 for it|
Priorities follow the manual validation, not the pipeline's severity classes.
Each issue has a proposed fix patch and a standalone reproduction test attached.
h3. Not filed
* Masked: reconciliation can still write a block row and the container BCSID
through a container object that a concurrent delete has already removed. The
two windows are a few statements wide, were hit only with a pause added to the
source, and the leftover rows are cleared when the affected replica is deleted
again. HDDS-10929 is the existing issue for coordinating reconciliation with
container deletion.
* Masked: the merkle tree, the RocksDB checksum key and the reported data
checksum can diverge after a scanner and reconciliation race, a failed put
block or a crash between the two writes. The race is HDDS-14908. The other
cases need an injected fault or a crash, affect only the reported value, and
are realigned by the next data scan or reconciliation.
* False positive: a reconcile command carries no deadline and reports DONE when
no peer could be used. Late execution was judged intended and nothing consumes
that status.
h3. Reproduce
{code:none}
specula run --agent=claude-code --effort=medium --model='claude-opus-5-5[1m]' \
--max-parallel=2 --enable-reviews --confirm-debate \
--tlc-memory-limit=28G --tlc-worker-limit=8 \
"container-reconciliation|apache/ozone|Java|Use the target-specific guidance"
{code}
h3. Caveats
* The issues from the pipeline findings need either an administrator's {{ozone
admin container reconcile}}, a running DiskBalancer, a non default setting or
node failures in a particular order. HDDS-16814 needs a DiskBalancer move.
* The reproduction of HDDS-16815 writes the stored chunk checksum value into
the container DB directly. Its patch covers the container data scan only.
* HDDS-16809 needs three datanode outages on one pipeline in a fixed order, one
of them permanent. Its patch is a guard in {{EmptyContainerHandler}} only. The
report describes a second part in the container report handler that the patch
does not fix and asks for opinions on it.
* HDDS-16810 is on master only (the logic came with HDDS-15542). Its patch adds
an optional field to a datanode to datanode response. In one schedule described
in the report the patched code leaves a container QUASI_CLOSED where the
unpatched code closes it through the defective round.
* HDDS-16811 changes what block deletion writes to the container merkle tree,
which changes the data checksum of ordinary containers and differs between
datanode versions during a rolling upgrade. The report states this and names a
layout feature gate as the stricter alternative.
* HDDS-16806 and HDDS-16805 need a non default client checksum setting.
* The mini cluster suite {{TestContainerCommandReconciliation}} did not start
in the test environment (its secure SCM HA cluster timed out during setup), so
none of the datanode patches was run against it. The reports say which suites
were run. It should pass in CI before any of these patches goes in.
* The patches were built and tested on the pinned commit and apply to master at
8ca44914771. The HDDS-16811 patch applies on top of the HDDS-16808 patch.
HDDS-16811 also touches the same test class as HDDS-16810 and HDDS-16806
({{TestContainerReconciliationWithMockDatanodes}}), so whichever of those lands
later needs a trivial rebase of its test hunks. The HDDS-16815 patch and the
HDDS-16806 patch change the same lines of {{KeyValueContainerCheck.scanBlock}},
so the later one needs a rebase. HDDS-16814 should go in after or together with
HDDS-16807, as its report explains. The other patches are independent of each
other.
* The patches are proposals for review.
Generated with Specula (Claude Opus 5.5).
> [Specula] Verify container-reconciliation (Apache Ozone master @1e528b9)
> ------------------------------------------------------------------------
>
> Key: HDDS-16474
> URL: https://issues.apache.org/jira/browse/HDDS-16474
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Minor
> Labels: help-wanted, specula
> Attachments: container-reconciliation.guidance.md
>
>
> This is a runnable Apache Ozone formal-verification target for the Specula
> pipeline (TLA+ model checking plus trace validation against the real code).
> It has not been run yet. Anyone can pick it up: run Specula against the
> pinned commit, then report findings on this issue. Part of umbrella
> HDDS-15926.
> h3. Target
> {{container-reconciliation}}. Scope and code entry points are in the attached
> guidance file [^container-reconciliation.guidance.md].
> h3. Run environment
> {noformat}
> Apache Ozone commit: 1e528b97aed3eef1be6846f1f0767c1ef98a883a
> Specula: https://github.com/specula-org/Specula (v1.1.0 or later)
> Toolchain: JDK 21, ~28 GB RAM for TLC, an LLM coding agent
> (claude-code or codex adapter)
> Runtime: hours per target; needs an agent API budget
> {noformat}
> h3. How to run
> # Clone Specula and install per its README; clone apache/ozone and check out
> the pinned commit above.
> # Place the attached guidance file as the target's .prompt-extra.md (see the
> Specula README for target layout).
> # Run:
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "container-reconciliation|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> # Deliverables land under the run's {{.specula-output/}}:
> {{confirmed-bugs.md}}, {{bug-severity.md}}, {{summary.md}}, and
> {{confirmation/<id>/investigation.md}}.
> h3. Reporting results
> Comment here with the outcome (0 findings is still useful: it records the
> target as covered at this commit). For each confirmed bug, file a separate
> Ozone Bug and link it to this issue with the "Testing" link type ("Testing
> discovered").
> h3. Notes
> * The attached guidance file names an earlier commit ({{20e6a1c}}); use the
> commit above. Code entry-point line numbers in the guidance were captured at
> an earlier commit and may have shifted on this head; treat them as
> approximate. Specula re-analyzes the source, so the run does not depend on
> them.
> * Recon-building targets need a local Specula patch to skip Maven {{target/}}
> out-of-tree symlinks in {{snapshotlib._validate_source_tree}}, else the run
> crashes at finalize.
> * effort=medium is the tested default for large targets; effort=high can hit
> per-turn timeouts on the biggest ones.
> * Findings are proposals, to be confirmed by human review and testing before
> any fix is merged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
