[
https://issues.apache.org/jira/browse/HDDS-16927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16927:
------------------------------
Attachment: TestBugMC8CheckpointDirLeak.java
> SCM never deletes the checkpoint it downloads for a Ratis snapshot install,
> leaving a full DB copy in its snapshot directory per install
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16927
> URL: https://issues.apache.org/jira/browse/HDDS-16927
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-8-delete-downloaded-scm-checkpoint-after-install.patch,
> TestBugMC8CheckpointDirLeak.java
>
>
> h3. Mechanism
> {{SCMSnapshotProvider.getSCMDBSnapshot}} downloads the leader's checkpoint
> into the SCM Ratis snapshot directory ({{ozone.scm.ha.ratis.snapshot.dir}}),
> untars it to {{scm.db-<leader>-<time>}} and deletes only the tarball.
> {{HAUtils.replaceDBWithCheckpoint}} copies that directory into {{scm.db}}
> with {{copyDirectory}} and documents that the checkpoint is not deleted
> there. Its caller {{SCMHAManagerImpl.installCheckpoint}} deletes only the
> backup of the old DB, and {{SCMStateMachine}} deletes nothing. The failure
> paths do not delete it either: a failed secret key fetch or a
> {{verifyCheckpointFromLeader}} that catches an exception and returns null in
> {{notifyInstallSnapshotFromLeader}}, and an install that is rejected as too
> old or fails in {{installCheckpoint}}. Only the index check in
> {{HAUtils.verifyTransactionInfo}} deletes a downloaded checkpoint. SCM
> startup creates the snapshot directory but never cleans it.
> h3. Trigger
> Any checkpoint install on an SCM. With default settings this happens on every
> SCM added with {{--bootstrap}}: Ratis (3.3.1) sends a snapshot notification
> to a bootstrapping follower once. With
> {{ozone.scm.ha.ratis.log.purge.enabled}} it also happens each time a follower
> falls behind the purged log.
> h3. Impact
> * Each install leaves one full copy of the SCM DB on the metadata volume, and
> nothing reclaims it (run for one install per SCM; repeated installs on the
> same SCM follow from the same code, not run).
> * Repeated installs need free space for the next untar and copy on the same
> volume, so the leak can eventually make an install fail for lack of space
> (argued, not run).
> * No effect on correctness.
> h3. Reproduction
> PASS on the unmodified source, default configuration.
> [^TestBugMC8CheckpointDirLeak.java] ({{hadoop-ozone/integration-test}})
> starts three SCMs with one held back and starts it, so it bootstraps and
> installs the leader's checkpoint. Afterwards its snapshot directory holds one
> {{scm.db-<leader>-<time>}} directory (70,399 bytes in the run), and the
> follower that the mini cluster bootstraps at startup holds one too. A passing
> test means the defect is present. With the patch all snapshot directories are
> empty after the install and the test fails at its {{hasSize(1)}} assertion.
> h3. Patch
> [^MC-8-delete-downloaded-scm-checkpoint-after-install.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> {{SCMStateMachine}} deletes the downloaded checkpoint with
> {{DBCheckpoint.cleanupCheckpoint()}} once it is not needed any more: in
> {{reinitialize}} after {{installCheckpoint}} returns or throws, and in
> {{notifyInstallSnapshotFromLeader}} when the secret key fetch fails or the
> verification returns null. A failed delete is logged and does not affect the
> install. This matches the cleanup the OM does in
> {{OzoneManager.installSnapshotFromLeader}}.
> Not covered by the patch: the tar and partial directory left by a failed
> untar, the DB copies left by a rollback after a failed reload (SCM exits
> there), a checkpoint stored by {{setInstallingSnapshotData}} for which Ratis
> never calls {{reinitialize}} before the SCM stops, for example when the
> leader changes before its next install notification (code reading), and
> directories left by earlier versions or crashes, which can be deleted by hand
> while no install is running.
> Covered by assertions added to
> {{TestSCMInstallSnapshot.testInstallCheckPoint}}: the checkpoint directory is
> gone after the in process install, and a checkpoint older than the applied
> index is rejected and deleted as well. Without the change the first assertion
> fails because the directory still exists. With it {{TestSCMInstallSnapshot}},
> {{TestSCMInstallSnapshotWithHA}}, {{TestSCMSnapshot}},
> {{TestSecretKeySnapshot}}, {{TestScmHAFinalization}} and the server-scm
> {{TestSCMStateMachine}} pass, and checkstyle is clean. This patch and
> HDDS-16924 both append to {{TestSCMInstallSnapshot.testInstallCheckPoint}},
> so whichever lands second needs a trivial rebase.
> Found by TLA+ model checking and code review of the SCM Ratis snapshot and
> checkpoint install paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
