Siyao Meng created HDDS-16927:
---------------------------------
Summary: SCM never deletes the checkpoint it downloads for a Ratis
snapshot install, leaving a full DB copy in its snapshot directory per install
Key: HDDS-16927
URL: https://issues.apache.org/jira/browse/HDDS-16927
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-8-delete-downloaded-scm-checkpoint-after-install.patch,
TestBugMC8CheckpointDirLeak.java
h3. Mechanism
{{SCMSnapshotProvider.getSCMDBSnapshot}} downloads the leader's checkpoint into
the SCM Ratis snapshot directory ({{ozone.scm.ha.ratis.snapshot.dir}}), untars
it to {{scm.db-<leader>-<time>}} and deletes only the tarball.
{{HAUtils.replaceDBWithCheckpoint}} copies that directory into {{scm.db}} with
{{copyDirectory}} and documents that the checkpoint is not deleted there. Its
caller {{SCMHAManagerImpl.installCheckpoint}} deletes only the backup of the
old DB, and {{SCMStateMachine}} deletes nothing. The failure paths do not
delete it either: a failed secret key fetch or a {{verifyCheckpointFromLeader}}
that catches an exception and returns null in
{{notifyInstallSnapshotFromLeader}}, and an install that is rejected as too old
or fails in {{installCheckpoint}}. Only the index check in
{{HAUtils.verifyTransactionInfo}} deletes a downloaded checkpoint. SCM startup
creates the snapshot directory but never cleans it.
h3. Trigger
Any checkpoint install on an SCM. With default settings this happens on every
SCM added with {{--bootstrap}}: Ratis (3.3.1) sends a snapshot notification to
a bootstrapping follower once. With {{ozone.scm.ha.ratis.log.purge.enabled}} it
also happens each time a follower falls behind the purged log.
h3. Impact
* Each install leaves one full copy of the SCM DB on the metadata volume, and
nothing reclaims it (run for one install per SCM; repeated installs on the same
SCM follow from the same code, not run).
* Repeated installs need free space for the next untar and copy on the same
volume, so the leak can eventually make an install fail for lack of space
(argued, not run).
* No effect on correctness.
h3. Reproduction
PASS on the unmodified source, default configuration.
[^TestBugMC8CheckpointDirLeak.java] ({{hadoop-ozone/integration-test}}) starts
three SCMs with one held back and starts it, so it bootstraps and installs the
leader's checkpoint. Afterwards its snapshot directory holds one
{{scm.db-<leader>-<time>}} directory (70,399 bytes in the run), and the
follower that the mini cluster bootstraps at startup holds one too. A passing
test means the defect is present. With the patch all snapshot directories are
empty after the install and the test fails at its {{hasSize(1)}} assertion.
h3. Patch
[^MC-8-delete-downloaded-scm-checkpoint-after-install.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad.
{{SCMStateMachine}} deletes the downloaded checkpoint with
{{DBCheckpoint.cleanupCheckpoint()}} once it is not needed any more: in
{{reinitialize}} after {{installCheckpoint}} returns or throws, and in
{{notifyInstallSnapshotFromLeader}} when the secret key fetch fails or the
verification returns null. A failed delete is logged and does not affect the
install. This matches the cleanup the OM does in
{{OzoneManager.installSnapshotFromLeader}}.
Not covered by the patch: the tar and partial directory left by a failed untar,
the DB copies left by a rollback after a failed reload (SCM exits there), a
checkpoint stored by {{setInstallingSnapshotData}} for which Ratis never calls
{{reinitialize}} before the SCM stops, for example when the leader changes
before its next install notification (code reading), and directories left by
earlier versions or crashes, which can be deleted by hand while no install is
running.
Covered by assertions added to
{{TestSCMInstallSnapshot.testInstallCheckPoint}}: the checkpoint directory is
gone after the in process install, and a checkpoint older than the applied
index is rejected and deleted as well. Without the change the first assertion
fails because the directory still exists. With it {{TestSCMInstallSnapshot}},
{{TestSCMInstallSnapshotWithHA}}, {{TestSCMSnapshot}},
{{TestSecretKeySnapshot}}, {{TestScmHAFinalization}} and the server-scm
{{TestSCMStateMachine}} pass, and checkstyle is clean. This patch and
HDDS-16924 both append to {{TestSCMInstallSnapshot.testInstallCheckPoint}}, so
whichever lands second needs a trivial rebase.
Found by TLA+ model checking and code review of the SCM Ratis snapshot and
checkpoint install paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
