Siyao Meng created HDDS-16688:
---------------------------------
Summary: OM checkpoint transfers snapshot SST files before taking
snapshotCache.lock()
Key: HDDS-16688
URL: https://issues.apache.org/jira/browse/HDDS-16688
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: HDDS-16688-MC-3-checkpoint-prelock-sst.patch
h3. Mechanism
In {{OMDBCheckpointServletInodeBasedXfer.collectDbDataToTransfer}} the snapshot
directories are enumerated at {{:259}} and their SST files transferred by the
loop at {{:277-284}}, but {{snapshotCache.lock()}} is only taken at
{{:315-316}}. The block comment at {{:303-313}} describes precisely the race
the loop above it is exposed to.
The loop uses the 5 argument {{collectFilesFromDir}} overload ({{:501-510}}),
which delegates with {{ignoreNoSuchFileException = false}} ({{:512-516}}),
while the sibling sst backup call at {{:337-340}} passes {{true}} for the same
class of hazard.
h3. Impact
A snapshot purge committing between {{Files.list(dbDir)}} and the per file read
makes the unprotected SST transfer throw {{NoSuchFileException}}, aborting the
whole checkpoint. The {{/dbCheckpoint}} request fails with HTTP 500, which
affects follower bootstrap and admin tarball download. Externally observable,
but per request: a later attempt can succeed.
h3. Reproduction
{{repro/test_bugMC-3_checkpoint_prelock_sst.sh}} with
{{repro/java/SpeculaMc3CheckpointPrelockTest.java}}. Evidence in
{{repro/logs/ev-mc3.log}}.
h3. Patch
{{HDDS-XXXXX-MC-3-checkpoint-prelock-sst.patch}} is attached. It applies to
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535 and carries the fix plus a regression
test that was verified to fail without the fix and pass with it.
{code:none}
mvn test -pl hadoop-ozone/ozone-manager
-Dtest=TestOMDBCheckpointServletInodeBasedXferNonLeader
{code}
Found by TLA+ model checking and code review of the OM snapshot cache under
HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with
Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
