Siyao Meng created HDDS-16921:
---------------------------------
Summary: SCM checkpoint install rollback restores the old DB
without closing the open checkpoint DB, so it reloads the rejected checkpoint
Key: HDDS-16921
URL: https://issues.apache.org/jira/browse/HDDS-16921
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-6-close-scm-db-before-rollback-restore.patch,
TestBugMC6RollbackKeepsCheckpointStore.java
h3. Mechanism
{{SCMHAManagerImpl.installCheckpoint}} closes the metadata store, swaps the
checkpoint into {{scm.db}} and calls {{reloadSCMState}}. If that fails, the
rollback calls {{HAUtils.replaceDBWithCheckpoint}} with the backup of the old
DB and then {{reloadSCMState}} again, before {{exitSystem}}. When the first
reload failed after {{SCMMetadataStoreImpl.start}} had opened the checkpoint (a
later service rejected it), that store is still open during the rollback:
* the directory of the open RocksDB is renamed to a second {{scm.db.backup.*}}
and the old DB is copied back to {{scm.db}};
* {{SCMMetadataStoreImpl.start}} does nothing while its store is set, so the
rollback's {{reloadSCMState}} rebinds every service to the rejected checkpoint
again instead of the restored old DB. With a deterministic failure it fails
again with the same error.
h3. Trigger
# A follower SCM installs a checkpoint that RocksDB opens but a service rejects
in {{startServices}}. Examples: a damaged data block in a table that a service
reads by key, a failed precondition in a service's {{reinitialize}}, or (as in
the reproduction) an apparent layout version this software does not know.
# The rollback runs.
h3. Impact
* In production {{exitSystem(1)}} in the {{finally}} block terminates the
process right after the rollback, so nothing serves the wrong state, and on
restart {{scm.db}} holds the old DB. What remains is that the rollback reload,
which is meant to check the old DB before terminating, checks the wrong DB, and
that the open DB's directory is renamed under it: new RocksDB files of the open
store would be created in the restored {{scm.db}} before the exit (argued, not
observed).
* Only with a test exit manager, which returns instead of terminating, does the
SCM go on serving the rejected checkpoint, whose directory has been renamed,
while {{scm.db}} on disk holds the old DB.
h3. Reproduction
PASS on the unmodified source. [^TestBugMC6RollbackKeepsCheckpointStore.java]
({{hadoop-ozone/integration-test}}) runs a one SCM mini cluster, takes a
checkpoint of its DB, sets the checkpoint's apparent layout version one above
the software version and its transaction info above the applied index,
allocates a container and writes a marker key that only the current DB has, and
installs the checkpoint through {{SCMStateMachine.pause}},
{{setInstallingSnapshotData}} and {{reinitialize}}, with a counting exit
manager. Output: {{exitSystem(1, Failed to reload SCM state and instantiate
services.)}}; after the rollback {{the live store reads apparent version 101
(checkpoint value 101), marker null, ContainerManager has #1: false}}, while
{{restored DB on disk at scm.db has marker old and #1: true}}. A passing test
means the defect is present.
h3. Patch
[^MC-6-close-scm-db-before-rollback-restore.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad.
The patch calls {{stopServices()}} (close the metadata store) in the rollback
before the old DB is restored, so the rollback's reload opens the restored old
DB. It is a no op when the failed reload never opened the store. The process
still exits as before. With the patch the reproduction fails because after the
rollback the live store and ContainerManager read the restored old DB (marker
present, container #1 present).
Covered by {{testRollbackAfterReloadFailureRestoresOldDB}} in the existing
{{TestSCMInstallSnapshotWithHA}}. Without the change the rollback's own
{{reloadSCMState}} throws "Initialization failed. Disk contains unknown
apparent version 101 for software version ZDU (100)", because it reloaded the
rejected checkpoint. With it {{TestSCMInstallSnapshotWithHA}} and
{{TestSCMInstallSnapshot}} (6 tests), {{TestSCMHAManagerImpl}} and
{{TestSCMStateMachine}} (18 tests) pass and checkstyle is clean.
Found by TLA+ model checking and code review of the SCM Ratis snapshot and
checkpoint install paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
