[
https://issues.apache.org/jira/browse/HDDS-16921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16921:
------------------------------
Attachment: MC-6-close-scm-db-before-rollback-restore.patch
> SCM checkpoint install rollback restores the old DB without closing the open
> checkpoint DB, so it reloads the rejected checkpoint
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16921
> URL: https://issues.apache.org/jira/browse/HDDS-16921
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-6-close-scm-db-before-rollback-restore.patch,
> TestBugMC6RollbackKeepsCheckpointStore.java
>
>
> h3. Mechanism
> {{SCMHAManagerImpl.installCheckpoint}} closes the metadata store, swaps the
> checkpoint into {{scm.db}} and calls {{reloadSCMState}}. If that fails, the
> rollback calls {{HAUtils.replaceDBWithCheckpoint}} with the backup of the old
> DB and then {{reloadSCMState}} again, before {{exitSystem}}. When the first
> reload failed after {{SCMMetadataStoreImpl.start}} had opened the checkpoint
> (a later service rejected it), that store is still open during the rollback:
> * the directory of the open RocksDB is renamed to a second
> {{scm.db.backup.*}} and the old DB is copied back to {{scm.db}};
> * {{SCMMetadataStoreImpl.start}} does nothing while its store is set, so the
> rollback's {{reloadSCMState}} rebinds every service to the rejected
> checkpoint again instead of the restored old DB. With a deterministic failure
> it fails again with the same error.
> h3. Trigger
> # A follower SCM installs a checkpoint that RocksDB opens but a service
> rejects in {{startServices}}. Examples: a damaged data block in a table that
> a service reads by key, a failed precondition in a service's
> {{reinitialize}}, or (as in the reproduction) an apparent layout version this
> software does not know.
> # The rollback runs.
> h3. Impact
> * In production {{exitSystem(1)}} in the {{finally}} block terminates the
> process right after the rollback, so nothing serves the wrong state, and on
> restart {{scm.db}} holds the old DB. What remains is that the rollback
> reload, which is meant to check the old DB before terminating, checks the
> wrong DB, and that the open DB's directory is renamed under it: new RocksDB
> files of the open store would be created in the restored {{scm.db}} before
> the exit (argued, not observed).
> * Only with a test exit manager, which returns instead of terminating, does
> the SCM go on serving the rejected checkpoint, whose directory has been
> renamed, while {{scm.db}} on disk holds the old DB.
> h3. Reproduction
> PASS on the unmodified source. [^TestBugMC6RollbackKeepsCheckpointStore.java]
> ({{hadoop-ozone/integration-test}}) runs a one SCM mini cluster, takes a
> checkpoint of its DB, sets the checkpoint's apparent layout version one above
> the software version and its transaction info above the applied index,
> allocates a container and writes a marker key that only the current DB has,
> and installs the checkpoint through {{SCMStateMachine.pause}},
> {{setInstallingSnapshotData}} and {{reinitialize}}, with a counting exit
> manager. Output: {{exitSystem(1, Failed to reload SCM state and instantiate
> services.)}}; after the rollback {{the live store reads apparent version 101
> (checkpoint value 101), marker null, ContainerManager has #1: false}}, while
> {{restored DB on disk at scm.db has marker old and #1: true}}. A passing test
> means the defect is present.
> h3. Patch
> [^MC-6-close-scm-db-before-rollback-restore.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> The patch calls {{stopServices()}} (close the metadata store) in the rollback
> before the old DB is restored, so the rollback's reload opens the restored
> old DB. It is a no op when the failed reload never opened the store. The
> process still exits as before. With the patch the reproduction fails because
> after the rollback the live store and ContainerManager read the restored old
> DB (marker present, container #1 present).
> Covered by {{testRollbackAfterReloadFailureRestoresOldDB}} in the existing
> {{TestSCMInstallSnapshotWithHA}}. Without the change the rollback's own
> {{reloadSCMState}} throws "Initialization failed. Disk contains unknown
> apparent version 101 for software version ZDU (100)", because it reloaded the
> rejected checkpoint. With it {{TestSCMInstallSnapshotWithHA}} and
> {{TestSCMInstallSnapshot}} (6 tests), {{TestSCMHAManagerImpl}} and
> {{TestSCMStateMachine}} (18 tests) pass and checkstyle is clean.
> Found by TLA+ model checking and code review of the SCM Ratis snapshot and
> checkpoint install paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
