[
https://issues.apache.org/jira/browse/HDDS-17461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17461:
------------------------------
Attachment: TestBugCR9StateReloadStaleMetricsAndCacheThreads.java
> After a checkpoint install, OM volume, bucket and snapshot directory metrics
> keep reading the replaced metadata manager
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17461
> URL: https://issues.apache.org/jira/browse/HDDS-17461
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-9a-recreate-metadata-metrics-after-checkpoint-install.patch,
> TestBugCR9StateReloadStaleMetricsAndCacheThreads.java
>
>
> h3. Mechanism
> {{VolumeUtilizationMetrics}} and {{BucketUtilizationMetrics}} are created
> once in the {{OzoneManager}} constructor and keep a final reference to the
> startup {{OmMetadataManagerImpl}}; every collection iterates that manager's
> volume or bucket cache. {{OMSnapshotDirectoryMetrics}} likewise keeps the
> manager it was created with.
> {{OzoneManager.reloadOMState}}, run by {{installCheckpoint}}, builds a new
> metadata manager and re-wires the managers and services that hold it, but not
> these three sources. So after an install they keep reading the stopped
> manager: the utilization metrics report the volumes and buckets as they were
> before the install, and the snapshot directory metrics report zeros because
> the old manager's store is closed.
> {{OMMetrics.startSnapshotDirectoryMetrics}} only creates the source when none
> exists, so the stop and start pair in {{OzoneManager.restart}} (used by the
> mini cluster) does not switch it to the new manager either, and {{restart}}
> never registers the two utilization sources again after {{stop}} unregistered
> them.
> h3. Trigger
> A follower OM installs a checkpoint from the leader, which is the normal
> catch up for an OM that was down while the leader purged the log it needs (or
> a newly bootstrapped OM). No fault or operator action is needed.
> h3. Impact
> On that OM, the per volume and per bucket utilization metrics and the
> snapshot directory metrics are wrong until the OM process restarts: volumes
> and buckets created after the install are missing, deleted ones remain, and
> usage values are frozen. Requests and data are not affected. Volume
> utilization metrics exist only on master; the bucket metrics are released
> since 2.0.0 and the snapshot directory metrics since 2.2.0.
> h3. Reproduction
> [^TestBugCR9StateReloadStaleMetricsAndCacheThreads.java]
> ({{hadoop-ozone/integration-test}}, target path and Maven command in the
> class comment) runs three OMs without datanodes with one OM not started,
> creates 120 volumes and a bucket past a purge gap of 50, then starts that OM
> and waits until Ratis has installed a checkpoint on it and it caught up with
> the leader. Its DB then has every volume and the bucket, but its
> {{VolumeUtilizationMetrics}} report only {{s3v}}, its
> {{BucketUtilizationMetrics}} report no bucket, and its snapshot directory
> metrics do not read its current metadata manager. The test also checks that
> the replaced table caches keep their cleanup executors running, which
> HDDS-17460 covers. A passing test means the defects are present; with this
> patch the three metrics checks fail.
> h3. Patch
> [^CR-9a-recreate-metadata-metrics-after-checkpoint-install.patch], against
> a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master at
> b6d0181075e.
> A new private {{OzoneManager.restartMetadataManagerMetrics}}, called from
> both {{reloadOMState}} and {{restart}}, unregisters the two utilization
> sources and creates them again for the current metadata manager, and stops
> and starts the snapshot directory metrics (replacing the pair {{restart}}
> already had). {{OMMetrics.stopSnapshotDirectoryMetrics}} now unregisters the
> source and drops it, so the next start creates one for the manager it is
> given. This is the pattern HDDS-17038 used for the double buffer metrics on
> the same install path. Every source is unregistered before it is registered
> again; unregistering a source that {{stop}} already removed does nothing.
> Covered by new checks at the end of {{testInstallSnapshotWithClientWrite}} in
> the existing {{TestOMRatisSnapshots}}: after the install, the follower's
> volume and bucket utilization sources must report the test volume and bucket,
> and its snapshot directory metrics must read its current metadata manager.
> Without the change the volume check fails ({{["s3v"]}} does not contain the
> test volume), and without the {{OMMetrics}} part the snapshot directory check
> fails. A new check in {{testRestartOMWithVolumeOperation}} of the existing
> {{TestOzoneManagerRestart}} asserts that the volume utilization source is
> registered after {{restart}}; it fails without the change. With the change
> {{TestOMRatisSnapshots}}, {{TestOMInstallSnapshotDuringBootstrapping}},
> {{TestOzoneManagerRestart}}, {{TestOzoneManagerHAWithStoppedNodes}} (which
> restarts OMs), {{TestOmMetrics}} and the unit tests of the three metrics
> classes and {{OMMetrics}} pass, and checkstyle is clean.
> {{TestOMRatisSnapshotTransfer}} timed out here downloading the checkpoint,
> with and without the patch, before any reload, so it was not seen passing.
> Found by code review of the OM table cache, double buffer flush and startup
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing (nearest: HDDS-17038,
> the same defect for the double buffer metrics, fixed with the same pattern;
> HDDS-10476, HDDS-14041 and HDDS-15333, which added these metrics). The
> attached patch is a proposal for review. Generated with Specula (Claude Opus
> 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
