Siyao Meng created HDDS-16142:
---------------------------------
Summary: SCM deleted block transaction summary can be double
counted after a Ratis submit timeout
Key: HDDS-16142
URL: https://issues.apache.org/jira/browse/HDDS-16142
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
The volatile deleted-block transaction summary in
{{SCMDeletedBlockTransactionStatusManager}} (the {{totalTxCount}},
{{totalBlockCount}}, {{totalBlockSize}}, {{totalReplicatedBlockSize}} counters
surfaced by {{getDeletedBlockSummary}}) can be decremented twice for a single
transaction, driving the summary below the set of transactions it actually
accounts for. The undercount is persisted durably and reloaded on leader
transfer, so it is permanent.
h3. Steps to reproduce
# Add a delete transaction so the summary is incremented and the transaction is
queued for the durable deleted-block log.
# The Ratis submit for the add times out. {{SCMRatisServerImpl.submitRequest}}
does {{submitClientRequestAsync(req).get(timeout)}}, which abandons the future
without cancelling the request, so {{addTransactions}} takes the
{{IOException}} path and rolls the summary back down, assuming the durable add
never landed.
# The abandoned request still commits and applies afterwards, so the
transaction is durably present in the deleted-block log even though the summary
was reverted.
# The container later becomes terminal. {{getTransactions}} re-caches the
transaction size and {{removeTransactions}} decrements the summary a second
time.
After this sequence the volatile summary is at least 1 below the number of
transactions accounted for. {{removeTransactionsFromDB(txIDs, getSummary())}}
then persists the undercount durably, so it survives leader transfer and
restart.
h3. Root cause
{{SCMDeletedBlockTransactionStatusManager.addTransactions}} increments the
summary, calls {{deletedBlockLogStateManager.addTransactionsToDB(txList,
getSummary())}}, and on {{IOException}} reverts the increment in its catch
block on the assumption that the durable add did not happen. A Ratis submit
timeout breaks that assumption: {{submitRequest}} abandons the client request
future after the timeout without cancelling it, so the request can still be
committed and applied, persisting the transaction. The now durable transaction
is later removed normally and {{removeTransactions}} decrements the summary
again for the same transaction, a double decrement for a single transaction.
h3. Impact
The deleted-block transaction summary is observability only. It is surfaced by
the SCM client protocol {{getDeletedBlockSummary}}, {{ozone admin scm}}, Recon,
and JMX metrics, and does not affect deletion correctness or
{{getNumOfValidTransactions()}}. The consequence is a persistent, operator
visible undercount of pending deleted-block work that survives leader transfer
and restart. There is no data loss and no premature deletion.
h3. Relation to HDDS-13184, HDDS-15726, HDDS-14572
HDDS-13184 introduced the deleted-block summary accounting. HDDS-15726 (the
most recent change to this file) added the {{addTransactions}} and
{{removeTransactions}} rollback-on-DB-failure catch blocks that this bug
exploits; it does not account for the submit-timeout-but-still-applied case.
HDDS-14572 added the leader-transfer summary reconstruction that makes the
undercount survive failover. None of these reports or fixes the double
decrement.
h3. Suggested fix
Do not treat an {{IOException}} from {{addTransactionsToDB}} as proof that the
durable mutation did not land. Prefer reconciling the summary from the
authoritative durable state (the deleted-block log and the persisted summary)
on the failure path and on leader transfer, rather than maintaining it purely
through incremental increment and rollback. Alternatively, make the submit path
settle or cancel the request so a timed-out submit cannot later commit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
