Siyao Meng created HDDS-16143:
---------------------------------
Summary: SCM leader transfer flush can split a delete transaction
apply and persist a summary undercount
Key: HDDS-16143
URL: https://issues.apache.org/jira/browse/HDDS-16143
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
When SCM leadership changes, {{SCMStateMachine.notifyLeaderChanged}} calls
{{transactionBuffer.flush()}} unconditionally. If that flush interposes in the
middle of a delete-transaction apply, it can persist the transaction rows
without the matching summary update, and the subsequent leader-side
reconstruction reads the resulting durable summary undercount into the
in-memory summary, baking in a permanent undercount that survives restart.
h3. Steps to reproduce
# A delete-transaction apply is in progress.
{{DeletedBlockLogStateManagerImpl.addTransactionsToDB}} buffers the transaction
rows first and then buffers the updated summary as a separate {{addToBuffer}}
call.
# An SCM leader transfer fires and {{SCMStateMachine.notifyLeaderChanged}}
calls {{transactionBuffer.flush()}} between the row buffering and the summary
buffering.
# The flush persists the rows with the stale (pre-update) summary, so the
durable summary is now lower than the durable row count.
# {{onBecomeLeader}} then runs {{initDataDistributionData}}, which reloads the
in-memory summary from that undercounted durable value rather than recomputing
it from the rows.
The reconstructed in-memory summary is permanently short; the next durable
write re-persists the undercount, so it survives further leader transfers and
restart.
h3. Root cause
{{addTransactionsToDB}} performs the row mutation and the summary mutation as
two separate buffer operations, so they are not atomic with respect to a flush.
The leader-transfer path uses the unguarded {{transactionBuffer.flush()}} in
{{notifyLeaderChanged}} (not the {{flushIfNeeded}} variant that skips when an
apply is in progress), so a flush can split an in-progress apply.
{{onBecomeLeader}} then reconstructs the in-memory summary from the split
durable value via {{initDataDistributionData}}, converting a transient durable
inconsistency into a permanent in-memory one.
h3. Impact
The deleted-block transaction summary is observability only. It is surfaced by
the SCM client protocol {{getDeletedBlockSummary}}, {{ozone admin scm}}, Recon,
and JMX metrics, and does not affect deletion correctness. The consequence is a
persistent, operator visible summary undercount that survives leader transfer
and restart. There is no data loss and no premature deletion.
h3. Relation to HDDS-14572, HDDS-15065
HDDS-14572 added the leader-transfer summary reconstruction ({{onBecomeLeader}}
to {{initDataDistributionData}}) that this bug relies on to make the undercount
permanent. HDDS-15065 added the guarded {{flushIfNeeded}} path (which skips
flushing while an apply is in progress) but did not guard the {{flush()}} in
{{notifyLeaderChanged}}. Neither reports or fixes this split-then-reconstruct
undercount.
h3. Suggested fix
Make the row-and-summary durable mutation atomic with respect to flush (buffer
both under the same guarded section), or guard the leader-transfer {{flush()}}
the same way {{flushIfNeeded}} is guarded so it cannot split an in-progress
apply, or have {{initDataDistributionData}} recompute the summary from the
durable rows rather than trusting a possibly-split persisted summary value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
