Ritesh Shukla created HDDS-16011:
------------------------------------
Summary: Formalize BCSID semantics and heuristics for container
reconciliation
Key: HDDS-16011
URL: https://issues.apache.org/jira/browse/HDDS-16011
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Datanode
Reporter: Ritesh Shukla
h3. Problem
Container reconciliation is the only writer of block and container BCSIDs for
non-open containers, yet the BCSID contract is undocumented: the design doc
(hadoop-hdds/docs/content/design/container-reconciliation.md, HDDS-10239) never
mentions BCSID. The implicit contract has produced three independent bug lanes:
* HDDS-15542: a trailing peer chunk filtered from the diff as unhealthy
({{reportChunkIfHealthy}}) leaves {{allChunksSuccessful}} true, advancing the
BCSID past data never ingested; the in-loop unhealthy skip is a second lane to
the same state.
* HDDS-12986: a stale peer merkle tree combined with a fresh peer BCSID
advances the BCSID past chunks the stale tree never listed.
All violate the same invariant through different mechanisms.
h3. Invariant
A replica advertising BCSID N asserts it durably holds every chunk committed
through N.
* The merkle tree / data checksum is the *data-plane* convergence oracle:
repair is best effort, partial repair is committed and kept, and unrecoverable
damage is an accepted steady state with an honest cross-replica mismatch.
* The BCSID is an *attestation* consumed without verification by read admission
({{BlockUtils.verifyBCSId}} container gate, {{BlockManagerImpl.getBlock}} block
gate), SCM under-replication source selection (max-sequenceId filter),
over-replication victim selection, and quasi-closed force-close winner
selection.
* Under-claiming is safe and self-correcting (replica declines reads, is
excluded as a source, converges on a later pass). Over-claiming is permanent
(container BCSID is a monotonic max), invisible to the scanner (the replica is
internally consistent), and defeats read failover and SCM safety logic.
h3. Scope / deliverables
# Design doc: add a BCSID semantics section to {{container-reconciliation.md}}
and correct drift (implemented repair is single-peer and prefix-contiguous, not
the documented cross-replica chunk union; unhealthy peer chunks are excluded
from the diff; repair paths accept QUASI_CLOSED and UNHEALTHY replicas, not
only CLOSED; Phase I trigger is the admin CLI, not automatic).
# Adoption rule (shipped in HDDS-15542,
https://github.com/apache/ozone/pull/10496): adopt the peer's block BCSID only
when the local block covers the peer's committed {{BlockData}} chunk list. The
getBlock snapshot carries chunk list and BCSID atomically, which also closes
the HDDS-12986 interleaving. Document why the diff cannot serve as the
completeness oracle.
# Convergence heuristic (deferred from PR review
https://github.com/apache/ozone/pull/7474#discussion_r1965940647): after
reconciling with a peer, when the post-repair merkle trees are identical, raise
the container BCSID to the max of the two. Verified content equality is exactly
what a BCSID claim asserts; this closes the residual availability gap where a
content-identical replica under-claims indefinitely (an empty diff runs no
repair and therefore never adopts).
# Quasi-closed analysis: BCSID adoption runs on QUASI_CLOSED/UNHEALTHY replicas
and feeds quasi-closed-stuck force-close selection by sequenceId; document and,
if needed, constrain.
# Coordinate HDDS-12986: the correctness portion is closed by the
snapshot-consistent gate; remaining scope is the optional optimization of
dropping getBlock, which would require an alternative completeness oracle.
h3. References
HDDS-10239 (umbrella), HDDS-15542 (https://github.com/apache/ozone/pull/10496),
HDDS-12986 (PR #8903, auto-closed unmerged).
Generated-by: Claude Code (Claude Fable 5)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
