[
https://issues.apache.org/jira/browse/HDDS-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeja Chintalapati reassigned HDDS-17460:
------------------------------------------
Assignee: Sreeja Chintalapati
> Table cache cleanup executors are never shut down, so OM checkpoint installs
> and Recon OM DB syncs leak threads and keep old caches in memory
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17460
> URL: https://issues.apache.org/jira/browse/HDDS-17460
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 0.5.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.2.1, 1.4.0, 2.0.0,
> 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Assignee: Sreeja Chintalapati
> Priority: Minor
> Attachments:
> CR-9b-shut-down-table-cache-executors-on-metadata-manager-stop.patch,
> TestBugCR9StateReloadStaleMetricsAndCacheThreads.java
>
>
> h3. Mechanism
> Each {{TypedTable}} with a cache creates a {{FullTableCache}} or a
> {{PartialTableCache}}, and each of those owns a single thread executor. The
> {{FullTableCache}} one is a scheduled executor that runs {{cleanupTask}}
> every second from the moment the cache is created; the {{PartialTableCache}}
> one starts its thread on the first {{cleanup}}. Neither cache, {{TypedTable}}
> nor {{OmMetadataManagerImpl}} ever shuts these executors down, so a metadata
> manager that is stopped, or whose tables are replaced, leaves its cleanup
> threads running. The scheduled FULL cache task also keeps a reference to its
> cache, so the cache cannot be collected.
> h3. Trigger
> * OM: {{OzoneManager.reloadOMState}}, run when a follower OM installs a
> checkpoint from the leader, stops the old {{OmMetadataManagerImpl}} and
> creates a new one. This is the normal catch up for an OM that was down while
> the leader purged the log it needs, or a newly bootstrapped OM. In tests,
> every stopped OM leaves its threads in the test JVM.
> * Recon: {{ReconOmMetadataManagerImpl.updateOmDB}}, run on every full OM DB
> snapshot sync, initializes new FULL cache tables over the old ones and closes
> the old store, but not the old caches.
> h3. Impact
> Each install on an OM leaves one thread per FULL cache table (volume and
> bucket, plus the revoked STS token table on master) and one per PARTIAL table
> that had been flushed into, plus the replaced FULL caches, which hold a full
> in memory copy of those tables, until the OM process restarts. Each full
> snapshot sync in Recon leaves the same FULL cleanup threads and replaced
> caches behind (by code reading, since 2.0.0, when the FULL cache got its
> scheduled thread; the new Recon test shows the threads). The FULL threads
> wake every second. The threads are daemon threads, so shutdown is not
> blocked, and requests and data are not affected. The growth is bounded by the
> number of installs or syncs.
> h3. Reproduction
> [^TestBugCR9StateReloadStaleMetricsAndCacheThreads.java]
> ({{hadoop-ozone/integration-test}}, target path and Maven command in the
> class comment) runs three OMs without datanodes with one OM not started,
> creates 120 volumes and a bucket past a purge gap of 50, then starts that OM
> and waits until Ratis has installed a checkpoint on it and it caught up with
> the leader. The replaced metadata manager's volume table cache executor is
> then not shut down, and none of the FULL cache cleanup threads that were
> alive before the install has ended. With this patch, the three FULL cleanup
> threads of the replaced manager ({{volumeTable}}, {{bucketTable}},
> {{s3RevokedStsTokenTable}}) end. The OM in this test never ran before the
> install, so its PARTIAL caches never started a thread and are not checked;
> the unit test below covers them. The test also checks the OM metrics sources
> that keep the replaced metadata manager, which a separate issue covers. A
> passing test means the defects are present; with this patch the two cache
> checks fail.
> h3. Patch
> [^CR-9b-shut-down-table-cache-executors-on-metadata-manager-stop.patch],
> against a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master
> at b6d0181075e.
> A new {{TableCache.close}}, a no op by default, shuts down the executor in
> {{FullTableCache}} and {{PartialTableCache}}. A new {{Table.closeCache}},
> also a no op by default, calls it from {{TypedTable}}.
> {{OmMetadataManagerImpl.stop}} calls {{closeCache}} on all its tables, and
> the table initializer of {{OmMetadataManagerImpl}} calls it on a table it
> replaces, which is the Recon sync case (OM initializes each table once per
> manager). On install the double buffer is flushed and stopped before the old
> manager is stopped, and on OM stop the Ratis server stops first, so no
> cleanup reaches a closed cache; Recon never calls {{cleanupCache}}. Starting
> a manager again after stop builds new tables and caches as before.
> Covered by {{testStopShutsDownTableCacheThreads}} in the existing
> {{TestOmMetadataManager}}: it creates a metadata manager, makes sure both a
> FULL and a PARTIAL cleanup thread run, stops the manager and waits for those
> threads to end. And by a new check in {{testUpdateOmDB}} of the existing
> {{TestReconOmMetadataManagerImpl}}: after the second {{updateOmDB}}, the FULL
> cleanup threads of the replaced tables must end. Without the change both
> waits time out. With the change the table and cache suites in
> {{hadoop-hdds/framework}} ({{TestTableCache}}, {{TestTypedTable}},
> {{TestRDBTableStore}}, {{TestTypedRDBTableStore}}),
> {{TestOmMetadataManager}}, {{TestSnapshotCache}}, {{TestOmSnapshotManager}},
> the {{TestOzoneManagerDoubleBuffer}} suites, {{TestOMRatisSnapshots}},
> {{TestOMInstallSnapshotDuringBootstrapping}}, {{TestOzoneManagerRestart}} and
> the Recon {{TestReconOmMetadataManagerImpl}}, {{TestOMDBUpdatesHandler}} and
> {{TestOzoneManagerServiceProviderImpl}} pass, and checkstyle is clean. Not
> run: a heap check that the old caches become collectable, and a Recon sync
> against a running OM.
> Found by code review of the OM table cache, double buffer flush and startup
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing (nearest: HDDS-11201,
> which added the scheduled FULL cache cleanup, and HDDS-11403). The attached
> patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
