[
https://issues.apache.org/jira/browse/HDDS-17297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17297:
------------------------------
Attachment:
CR-5-serialize-sst-filtering-reconfig-with-checkpoint-install.patch
> Reconfiguring the SST filtering interval while an OM installs a checkpoint
> starts an SstFilteringService on the replaced KeyManager
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17297
> URL: https://issues.apache.org/jira/browse/HDDS-17297
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-5-serialize-sst-filtering-reconfig-with-checkpoint-install.patch,
> TestBugCR5SstFilteringReconfigDuringInstall.java
>
>
> h3. Mechanism
> {{OzoneManager.reconfOzoneSnapshotSSTFilteringServiceInterval}} reads the
> {{keyManager}} field several times and calls
> {{KeyManagerImpl.startSnapshotSstFilteringService}} on it. It runs on the
> reconfiguration task thread and nothing makes it wait for
> {{OzoneManager.installCheckpoint}}, which stops the old {{KeyManager}},
> replaces the DB and then, in {{reloadOMState}}, creates a new metadata
> manager (with a new lock) and a new {{KeyManagerImpl}}.
> {{installSnapshotFromLeader}} holds the OM monitor for the whole install, but
> the reconfiguration function does not take it. A reconfiguration that runs
> after the old key manager was stopped and before the new one is assigned
> starts a service on the old key manager, which nothing references afterwards.
> {{reconfOzoneKeyDeletingLimitPerTask}} has the same race: it calls
> {{getKeyManager().getDeletingService()}} without the monitor.
> That service took its bootstrap lock from the replaced metadata manager when
> it was created, while {{OMDBCheckpointServlet}} resolves the current lock on
> every call. Its task looks up the current metadata manager and snapshot
> manager on every run, so it works on the live snapshots.
> h3. Trigger
> # SST filtering is disabled ({{ozone.snapshot.filtering.service.interval}}
> set to 0) and the admin enables it again with {{ozone admin reconfig
> --service=OM start}}.
> # At the same time the OM installs a checkpoint from the leader (any follower
> that lags behind the purged Raft log), and the reconfiguration task reaches
> this property inside the install, between the stop of the old key manager and
> the creation of the new one.
> The window is the DB replacement and reload, so the race is narrow. Only an
> admin can start the reconfiguration task.
> h3. Impact
> * An {{SstFilteringService}} keeps running on the replaced key manager.
> Setting the interval back to 0 stops only the service of the current key
> manager, so filtering continues although the admin disabled it. Shown in the
> reproduction.
> * The orphan service is not excluded by the checkpoint servlet's bootstrap
> write lock. Shown in the reproduction. That it actually deletes SST files of
> a snapshot while a checkpoint is being taken is argued, not run.
> * Argued from code, not run: with filtering enabled (the default), the same
> race makes the {{Objects.requireNonNull}} check in the reconfiguration
> function fail with {{NullPointerException}}, because the stopped key manager
> no longer has a service. The reconfiguration of the property is then reported
> as failed and no orphan is left.
> * A reconfiguration of {{ozone.key.deleting.limit.per.task}} inside the same
> window fails with {{NullPointerException}}, because the stopped key manager
> has no {{KeyDeletingService}}, and is reported as failed. Shown in the
> regression test. Argued from code, not run: one that runs just before the
> window sets the limit on the service that the install then replaces, and the
> new service can read the old value from the configuration, so the change can
> be lost silently.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugCR5SstFilteringReconfigDuringInstall.java]
> ({{hadoop-ozone/ozone-manager}}) starts one OM through {{OmTestManagers}},
> disables filtering, and installs a checkpoint of its own DB with
> {{installCheckpoint}} (presented with a newer transaction index so that it
> proceeds). The existing {{checkpointBackupInjector}} hook holds the install
> inside the window while the test calls
> {{ReconfigurationHandler.reconfigurePropertyImpl}}, the call the
> reconfiguration task makes for each changed property, on its own thread. It
> shows that the reconfiguration does not wait for the install, that the old
> key manager has a running service, that this service survives disabling
> filtering again, and that it gets its bootstrap read lock while
> {{OMDBCheckpointServlet.Lock}} holds the bootstrap write lock. A passing test
> means the defect is present.
> h3. Patch
> [^CR-5-serialize-sst-filtering-reconfig-with-checkpoint-install.patch],
> against e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master
> at 669912118cf.
> {{reconfOzoneSnapshotSSTFilteringServiceInterval}},
> {{reconfOzoneKeyDeletingLimitPerTask}} and {{installCheckpoint}} are now
> {{synchronized}}. The install path from Ratis already holds the OM monitor in
> {{installSnapshotFromLeader}}, so these reconfigurations now wait for a
> running install and then apply to the services of the new key manager.
> Marking {{installCheckpoint}} as well keeps that true for its direct callers;
> the lock is reentrant, so the Ratis path is unchanged.
> {{installSnapshotFromLeader}} holds the monitor for the whole install,
> including the checkpoint download, so on a follower that is downloading a
> checkpoint the reconfiguration task, and every property after this one in the
> same task, waits for the download. It waits in the property loop, outside
> {{reconfigLock}}, so {{ozone admin reconfig status}} keeps answering that the
> task is running.
> Covered by {{testServiceReconfigurationWaitsForInstall}} in the existing
> {{TestOMRatisSnapshots}}: on the active follower of the suite's HA cluster it
> holds an install inside the window with the suite's existing fault injector,
> runs both reconfigurations on their own threads, and checks that the replaced
> key manager has no filtering service, the new one has a filtering service
> with the new interval, and its key deleting service has the new limit.
> Without the change the test fails at the limit reconfiguration with the
> {{NullPointerException}}; with only the filtering part of the test it fails
> because the replaced key manager has a filtering service. With it
> {{TestOMRatisSnapshots}} and {{TestOzoneIntegrationNonHA}} (including the OM
> reconfiguration and reconfig shell tests) pass, and checkstyle is clean. With
> the patch the reproduction fails at the assertion that the old key manager
> got a service.
> Found by code review of the Ozone Manager lock ordering paths, as part of the
> TLA+ verification effort under HDDS-15926, on commit
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-13464 (made this interval reconfigurable), HDDS-15103 (a
> race between OM start and snapshot install), HDDS-17120 and HDDS-16878
> (checkpoint servlets that keep a DB store an install replaced, the same
> pattern for another object). The attached patch is a proposal for review.
> Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
