[
https://issues.apache.org/jira/browse/HDDS-16145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyesh Karatha reassigned HDDS-16145:
--------------------------------------
Assignee: Priyesh Karatha
> Checkpoint flush during delete transaction removal can double count the SCM
> deleted block summary
> -------------------------------------------------------------------------------------------------
>
> Key: HDDS-16145
> URL: https://issues.apache.org/jira/browse/HDDS-16145
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Priyesh Karatha
> Priority: Major
>
> A checkpoint or leader-transfer flush that interposes in the middle of
> {{DeletedBlockLogStateManagerImpl.removeTransactionsFromDB}} can re-expose a
> delete-transaction row that is still durable, causing the block-deletion
> scanner to select it again and decrement the volatile summary a second time,
> to a permanent and impossible negative value.
> h3. Steps to reproduce
> # The block-deletion scanner drives terminal cleanup, which calls
> {{removeTransactionsFromDB(txIDs, summary)}}. That method first marks the ids
> in {{deletingTxIDs}} (the scanner-hide set), then removes each row from the
> transaction buffer, then buffers the updated summary, as separate steps.
> # A checkpoint download ({{InterSCMGrpcService}}) or leader-transfer flush
> runs {{transactionBuffer.flush()}} between the mark and the buffered removal.
> {{onFlush}} then resets {{deletingTxIDs}} to a fresh empty set, clearing the
> hide marker while the row is still durable (its {{removeFromBuffer}} had not
> been flushed).
> # The next scan no longer sees the row as hidden, re-selects it, and terminal
> cleanup decrements the summary again for the same transaction.
> The summary is driven to a negative value (for a single transaction it
> reaches -1), which is persisted durably and reloaded verbatim on restart and
> leader transfer.
> h3. Root cause
> {{removeTransactionsFromDB}} performs the hide marking
> ({{deletingTxIDs.addAll}}), the row removal ({{removeFromBuffer}}), and the
> summary update as separate, non-atomic steps.
> {{SCMHADBTransactionBufferImpl.flush()}} takes only the write lock and does
> not consult the in-progress apply/removal state, and {{onFlush}} resets
> {{deletingTxIDs}}. A flush interposed mid-removal therefore clears the hide
> marker for a row that is still durable, so the scanner re-selects it and
> double-decrements the summary. A negative deleted-block count is an
> impossible undercount.
> h3. Impact
> The deleted-block transaction summary is observability only. It is surfaced
> by the SCM client protocol {{getDeletedBlockSummary}}, {{ozone admin scm}},
> Recon {{PendingDeletionEndpoint}}, and JMX metrics, and does not affect
> deletion correctness. The consequence is a persistent, operator visible
> accounting corruption (a negative pending-deletion count) that survives
> restart and leader transfer. There is no data loss and no premature deletion.
> h3. Relation to HDDS-15665
> HDDS-15665 stabilizes an already-open deleted-block iterator against
> concurrent modification, but does not make {{removeTransactionsFromDB}}
> atomic with respect to {{flush()}}, nor does it protect a new iterator opened
> after a mid-removal flush clears the hide marker. No filed issue reports this
> checkpoint-flush re-exposure and summary double decrement.
> h3. Suggested fix
> Make the hide marking, row removal, and summary update in
> {{removeTransactionsFromDB}} atomic with respect to flush, or have
> {{flush()}}/{{onFlush}} preserve the {{deletingTxIDs}} hide markers for rows
> whose removal has not yet been flushed, so a flush cannot re-expose a
> still-durable row that is being removed. Recomputing the summary from the
> durable rows on reload would also prevent a negative value from being
> persisted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
