Siyao Meng created HDDS-16144:
---------------------------------
Summary: SCM may purge a delete transaction before a newly copied
replica has acknowledged it
Key: HDDS-16144
URL: https://issues.apache.org/jira/browse/HDDS-16144
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
{{SCMDeletedBlockTransactionStatusManager.commitTransactions}} decides whether
a delete transaction can be purged from the deleted-block log using a replica
set snapshot taken earlier in the same call, and then removes the transaction
from the durable log without re-checking membership. A replica that is copied
to a new datanode after the snapshot but before the purge, and that never
received or acknowledged the deletion, can therefore keep the blocks forever:
SCM has already purged the transaction and never re-issues it.
h3. Steps to reproduce
# A delete transaction for a container is outstanding and has been sent to the
current replicas.
# {{commitTransactions}} captures the replica set via
{{containerManager.getContainerReplicas(containerId)}} and evaluates
{{canPurge}} from that snapshot (all snapshot replicas have committed).
# Around the same time, replication manager copies the container to a new
datanode. The copy carries the container RocksDB, so the new replica inherits
whatever delete state the source had; if the source had not yet physically
deleted the blocks, the new replica holds them with no pending SCM delete
transaction.
# {{commitTransactions}} proceeds to purge:
{{transactionToDNsCommitMap.remove(txID)}} and
{{removeTransactions(txIDsToBeDeleted)}} delete the row from the durable
deleted-block log. There is no re-validation of the current replica set at
removal time.
The transaction is gone from the SCM log and is never recreated. SCM does not
reconcile a replica reported {{deleteTransactionId}} against the container and
does not re-issue delete transactions from it (the field is effectively unused
in server-scm), so the lagging replica is never told to delete.
h3. Root cause
The purge decision and the durable removal are not guarded by a consistent,
current replica set. {{commitTransactions}} evaluates {{canPurge}} against a
replica snapshot and then {{removeTransactions}} removes the row with no
membership re-check. SCM has no catch-up mechanism that compares a replica's
{{deleteTransactionId}} to the container's and re-issues missed deletions; once
the transaction is purged it is never re-delivered.
h3. Impact
A container replica that becomes current in the snapshot-to-purge window
without having applied the deletion permanently retains the deleted blocks, an
undeleted-block space leak on that datanode with no automatic recovery at the
SCM level. The leak is confined to datanode capacity; there is no client
visible data corruption. The SCM-level premature purge reproduces
deterministically; demonstrating the physical on-disk leak requires a
mini-cluster with adversarial copy-versus-delete timing, so this is filed as an
environment-limited (not fully cluster-reproduced) finding.
h3. Relation to HDDS-11498
HDDS-11498 improved deletion efficiency and the "send to all replicas or not at
all" handling in {{commitTransactions}}, but does not address a replica that
becomes current after the ACK-set snapshot is taken and before the durable
removal. No filed issue reports this stale-snapshot premature purge.
h3. Suggested fix
Re-validate the current replica set immediately before removing a transaction
from the durable log (recompute {{canPurge}} against a fresh
{{getContainerReplicas}} under the same critical section), or reconcile a newly
added replica's {{deleteTransactionId}} against the container and re-issue any
missed delete transactions before allowing the purge, so a replica that has not
acknowledged the deletion cannot be orphaned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
