[
https://issues.apache.org/jira/browse/HDDS-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16446:
------------------------------
Description:
h3. Finding
For quasi-closed-stuck containers an UNHEALTHY IN_SERVICE replica is counted as
a valid copy, so an origin with only 1 healthy copy is declared adequately
replicated (no repair) and over-delete can drop a healthy copy while keeping
the UNHEALTHY one; a persistent durability reduction with no downstream handler
to recover it, reachable from ordinary replica reports.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding CR-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-ratis-replication
Guidance:
campaigns/ozone-9fbf9ee/targets/012-scm-ratis-replication/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-ratis-replication|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
Precondition: a quasi-closed-stuck container (multiple origins). One "other"
origin has 1 healthy QUASI_CLOSED IN_SERVICE replica plus 1 UNHEALTHY
IN_SERVICE replica (a data-scan or corruption failure on an in-service,
node-healthy datanode). The target for an "other" origin is 2 copies.
Action: ReplicationManager runs the quasi-closed-stuck replication check, which
builds QuasiClosedStuckReplicaCount from the reported replicas.
Observed wrong outcome: getInService counts the UNHEALTHY replica as a valid
copy, so the origin reports 2 in-service copies, equal to its target, and
isUnderReplicated() returns false. No under-replication result is enqueued and
no repair is scheduled, even though the origin has only 1 genuinely healthy
copy. The lingering UNHEALTHY replica is not removed while a healthy copy
coexists (the unhealthy-replication handler returns early when healthy replicas
exist), so nothing downstream recovers the missing healthy copy: a persistent
durability reduction. The same over-count can make an origin with 3 healthy
plus 1 UNHEALTHY in-service copies look over-replicated (4 > target 3), so the
over-replication handler deletes a copy while the UNHEALTHY one may survive.
h3. Root cause
In QuasiClosedStuckReplicaCount
(hadoop-hdds/server-scm/.../QuasiClosedStuckReplicaCount.java), the constructor
adds every IN_SERVICE replica to inServiceReplicasByOrigin regardless of the
replica's health state, so getInService counts UNHEALTHY replicas as valid
copies. This is inconsistent with the rest of the class, which treats UNHEALTHY
replicas as unreliable: hasHealthyReplicas and computeBestOrigins both exclude
UNHEALTHY replicas, and the class is documented as ranking origins by their
highest healthy BCSID. getInService feeds both getUnderReplicatedReplicas and
getOverReplicatedOrigins, so the miscount corrupts both adequacy and
over-replication decisions.
h3. Suggested fix
The attached patch [^HDDS-16446.001.patch] excludes UNHEALTHY replicas when
populating inServiceReplicasByOrigin, so getInService counts only genuinely
healthy in-service copies, consistent with computeBestOrigins. An origin with 1
healthy plus 1 UNHEALTHY copy is now correctly under-replicated, and an
UNHEALTHY copy no longer inflates an origin into a false over-replication. Two
tests are added to TestQuasiClosedStuckReplicaCount: one asserts the
under-replication case (1 healthy plus 1 UNHEALTHY on an "other" origin), and
one asserts an UNHEALTHY in-service copy does not push a healthy-at-target
origin into over-replication. The existing QuasiClosedStuck replica-count,
over-replication, and under-replication handler suites still pass. Victim
selection in the over-replication handler (health/BCSID-blind ordering) is left
as separate hardening.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
For quasi-closed-stuck containers an UNHEALTHY IN_SERVICE replica is counted as
a valid copy, so an origin with only 1 healthy copy is declared adequately
replicated (no repair) and over-delete can drop a healthy copy while keeping
the UNHEALTHY one; a persistent durability reduction with no downstream handler
to recover it, reachable from ordinary replica reports.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding CR-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-ratis-replication
Guidance:
campaigns/ozone-9fbf9ee/targets/012-scm-ratis-replication/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-ratis-replication|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> SCM counts an UNHEALTHY replica as valid for quasi closed stuck containers,
> risking loss of the healthy copy
> ------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16446
> URL: https://issues.apache.org/jira/browse/HDDS-16446
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16446.001.patch
>
>
> h3. Finding
> For quasi-closed-stuck containers an UNHEALTHY IN_SERVICE replica is counted
> as a valid copy, so an origin with only 1 healthy copy is declared adequately
> replicated (no repair) and over-delete can drop a healthy copy while keeping
> the UNHEALTHY one; a persistent durability reduction with no downstream
> handler to recover it, reachable from ordinary replica reports.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding CR-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: scm-ratis-replication
> Guidance:
> campaigns/ozone-9fbf9ee/targets/012-scm-ratis-replication/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "scm-ratis-replication|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition: a quasi-closed-stuck container (multiple origins). One "other"
> origin has 1 healthy QUASI_CLOSED IN_SERVICE replica plus 1 UNHEALTHY
> IN_SERVICE replica (a data-scan or corruption failure on an in-service,
> node-healthy datanode). The target for an "other" origin is 2 copies.
> Action: ReplicationManager runs the quasi-closed-stuck replication check,
> which builds QuasiClosedStuckReplicaCount from the reported replicas.
> Observed wrong outcome: getInService counts the UNHEALTHY replica as a valid
> copy, so the origin reports 2 in-service copies, equal to its target, and
> isUnderReplicated() returns false. No under-replication result is enqueued
> and no repair is scheduled, even though the origin has only 1 genuinely
> healthy copy. The lingering UNHEALTHY replica is not removed while a healthy
> copy coexists (the unhealthy-replication handler returns early when healthy
> replicas exist), so nothing downstream recovers the missing healthy copy: a
> persistent durability reduction. The same over-count can make an origin with
> 3 healthy plus 1 UNHEALTHY in-service copies look over-replicated (4 > target
> 3), so the over-replication handler deletes a copy while the UNHEALTHY one
> may survive.
> h3. Root cause
> In QuasiClosedStuckReplicaCount
> (hadoop-hdds/server-scm/.../QuasiClosedStuckReplicaCount.java), the
> constructor adds every IN_SERVICE replica to inServiceReplicasByOrigin
> regardless of the replica's health state, so getInService counts UNHEALTHY
> replicas as valid copies. This is inconsistent with the rest of the class,
> which treats UNHEALTHY replicas as unreliable: hasHealthyReplicas and
> computeBestOrigins both exclude UNHEALTHY replicas, and the class is
> documented as ranking origins by their highest healthy BCSID. getInService
> feeds both getUnderReplicatedReplicas and getOverReplicatedOrigins, so the
> miscount corrupts both adequacy and over-replication decisions.
> h3. Suggested fix
> The attached patch [^HDDS-16446.001.patch] excludes UNHEALTHY replicas when
> populating inServiceReplicasByOrigin, so getInService counts only genuinely
> healthy in-service copies, consistent with computeBestOrigins. An origin with
> 1 healthy plus 1 UNHEALTHY copy is now correctly under-replicated, and an
> UNHEALTHY copy no longer inflates an origin into a false over-replication.
> Two tests are added to TestQuasiClosedStuckReplicaCount: one asserts the
> under-replication case (1 healthy plus 1 UNHEALTHY on an "other" origin), and
> one asserts an UNHEALTHY in-service copy does not push a healthy-at-target
> origin into over-replication. The existing QuasiClosedStuck replica-count,
> over-replication, and under-replication handler suites still pass. Victim
> selection in the over-replication handler (health/BCSID-blind ordering) is
> left as separate hardening.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
