Siyao Meng created HDDS-16920:
---------------------------------
Summary: SCM creates the DB inconsistent marker only after moving
scm.db away, so a stop in between restarts on an empty DB
Key: HDDS-16920
URL: https://issues.apache.org/jira/browse/HDDS-16920
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-2-create-db-marker-before-moving-scm-db.patch,
TestBugMC2CrashAfterDBMoveOpensEmptyDB.java
h3. Mechanism
{{HAUtils.replaceDBWithCheckpoint}} renames {{scm.db}} to
{{scm.db.backup.<index>_<time>}} and only then creates
{{dbInconsistentMarker}}. The marker is meant to stop SCM from starting on a
half installed DB ("this marker file will fail it from starting up"). If the
process stops between the rename and the marker, the directory has neither
{{scm.db}} nor the marker. On the next start {{SCMMetadataStoreImpl.start}}
checks only the marker, and {{DBStoreBuilder.createDBStore}} creates a new
empty DB because {{createIfMissing}} is true. The old DB stays in the backup
directory and nothing restores or deletes it.
h3. Trigger
# A follower SCM installs a checkpoint from the leader
({{SCMHAManagerImpl.installCheckpoint}}).
# The SCM process stops (kill, OOM) right after {{Files.move}} of {{scm.db}}
and before {{Files.createFile}} of the marker. The window is two consecutive
file system calls; in the reproduction the stop is injected.
h3. Impact
* The SCM starts on an empty {{scm.db}} without any error, against the marker
contract. Its old DB is left orphaned in {{scm.db.backup.*}}.
* Committed state is not lost cluster wide (the other SCMs have it). The window
is entered only after Ratis acknowledged the install, so with the current code
the follower cannot catch up afterwards anyway, whether it restarts on the
empty DB or on the old one. As long as it stays a follower it does not start
its datanode server, so no datanode report is processed against the empty DB
(argued, not run).
h3. Reproduction
PASS on the unmodified source. [^TestBugMC2CrashAfterDBMoveOpensEmptyDB.java]
({{hadoop-hdds/server-scm}}) writes a real SCM DB with a container and a
transaction info, runs {{HAUtils.replaceDBWithCheckpoint}} with a static
{{Files.move}} spy that does the real move and then throws an {{Error}} (the
stop), and opens a new {{SCMMetadataStoreImpl}} on the same directory. Output:
{{after the crash: scm.db exists=false, marker exists=false, backups=1}}, then
{{restart accepted, scm.db exists=true, transactionInfo=null, container
#1=null}}. A passing test means the defect is present.
h3. Patch
[^MC-2-create-db-marker-before-moving-scm-db.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad.
The patch creates the marker before the move and deletes it again if the move
fails, so a failed backup leaves the directory unchanged. The rest of the
method is unchanged. A stop anywhere between the move and the end of the copy
now leaves the marker, and the existing check refuses to start ("Cannot load
SCM DB as it is in an inconsistent state"). With the patch the reproduction
fails because the marker exists after the stop.
Covered by {{testReplaceDBWithCheckpointCreatesMarkerBeforeMovingDB}} and
{{testReplaceDBWithCheckpointRemovesMarkerIfMoveFails}} in a new
{{TestHAUtils}} (there is no existing unit test class for {{HAUtils}}). Without
the change the first fails with "Expecting AtomicBoolean(false) to have value:
true". With it {{TestHAUtils}}, {{TestRDBSnapshotProvider}} and
{{TestSCMHAUtils}} (5 tests) and the mini cluster suites
{{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} (5 tests) pass,
and checkstyle is clean. The patch does not add an fsync of the marker or the
directory, so it covers a process stop; after a power loss the file system may
still persist the rename without the marker.
{{OzoneManager.replaceOMDBWithCheckpoint}} still moves the {{om.db}} items to
the backup before {{moveCheckpointFiles}} creates the marker (by code reading);
HDDS-16026 changed its failure handling, not this order, and the OM side is a
separate follow up that this patch does not change.
Found by TLA+ model checking and code review of the SCM Ratis snapshot and
checkpoint install paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
