Siyao Meng created HDDS-17460:
---------------------------------
Summary: Table cache cleanup executors are never shut down, so OM
checkpoint installs and Recon OM DB syncs leak threads and keep old caches in
memory
Key: HDDS-17460
URL: https://issues.apache.org/jira/browse/HDDS-17460
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-9b-shut-down-table-cache-executors-on-metadata-manager-stop.patch,
TestBugCR9StateReloadStaleMetricsAndCacheThreads.java
h3. Mechanism
Each {{TypedTable}} with a cache creates a {{FullTableCache}} or a
{{PartialTableCache}}, and each of those owns a single thread executor. The
{{FullTableCache}} one is a scheduled executor that runs {{cleanupTask}} every
second from the moment the cache is created; the {{PartialTableCache}} one
starts its thread on the first {{cleanup}}. Neither cache, {{TypedTable}} nor
{{OmMetadataManagerImpl}} ever shuts these executors down, so a metadata
manager that is stopped, or whose tables are replaced, leaves its cleanup
threads running. The scheduled FULL cache task also keeps a reference to its
cache, so the cache cannot be collected.
h3. Trigger
* OM: {{OzoneManager.reloadOMState}}, run when a follower OM installs a
checkpoint from the leader, stops the old {{OmMetadataManagerImpl}} and creates
a new one. This is the normal catch up for an OM that was down while the leader
purged the log it needs, or a newly bootstrapped OM. In tests, every stopped OM
leaves its threads in the test JVM.
* Recon: {{ReconOmMetadataManagerImpl.updateOmDB}}, run on every full OM DB
snapshot sync, initializes new FULL cache tables over the old ones and closes
the old store, but not the old caches.
h3. Impact
Each install on an OM leaves one thread per FULL cache table (volume and
bucket, plus the revoked STS token table on master) and one per PARTIAL table
that had been flushed into, plus the replaced FULL caches, which hold a full in
memory copy of those tables, until the OM process restarts. Each full snapshot
sync in Recon leaves the same FULL cleanup threads and replaced caches behind
(by code reading, since 2.0.0, when the FULL cache got its scheduled thread;
the new Recon test shows the threads). The FULL threads wake every second. The
threads are daemon threads, so shutdown is not blocked, and requests and data
are not affected. The growth is bounded by the number of installs or syncs.
h3. Reproduction
[^TestBugCR9StateReloadStaleMetricsAndCacheThreads.java]
({{hadoop-ozone/integration-test}}, target path and Maven command in the class
comment) runs three OMs without datanodes with one OM not started, creates 120
volumes and a bucket past a purge gap of 50, then starts that OM and waits
until Ratis has installed a checkpoint on it and it caught up with the leader.
The replaced metadata manager's volume table cache executor is then not shut
down, and none of the FULL cache cleanup threads that were alive before the
install has ended. With this patch, the three FULL cleanup threads of the
replaced manager ({{volumeTable}}, {{bucketTable}}, {{s3RevokedStsTokenTable}})
end. The OM in this test never ran before the install, so its PARTIAL caches
never started a thread and are not checked; the unit test below covers them.
The test also checks the OM metrics sources that keep the replaced metadata
manager, which a separate issue covers. A passing test means the defects are
present; with this patch the two cache checks fail.
h3. Patch
[^CR-9b-shut-down-table-cache-executors-on-metadata-manager-stop.patch],
against a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master at
b6d0181075e.
A new {{TableCache.close}}, a no op by default, shuts down the executor in
{{FullTableCache}} and {{PartialTableCache}}. A new {{Table.closeCache}}, also
a no op by default, calls it from {{TypedTable}}.
{{OmMetadataManagerImpl.stop}} calls {{closeCache}} on all its tables, and the
table initializer of {{OmMetadataManagerImpl}} calls it on a table it replaces,
which is the Recon sync case (OM initializes each table once per manager). On
install the double buffer is flushed and stopped before the old manager is
stopped, and on OM stop the Ratis server stops first, so no cleanup reaches a
closed cache; Recon never calls {{cleanupCache}}. Starting a manager again
after stop builds new tables and caches as before.
Covered by {{testStopShutsDownTableCacheThreads}} in the existing
{{TestOmMetadataManager}}: it creates a metadata manager, makes sure both a
FULL and a PARTIAL cleanup thread run, stops the manager and waits for those
threads to end. And by a new check in {{testUpdateOmDB}} of the existing
{{TestReconOmMetadataManagerImpl}}: after the second {{updateOmDB}}, the FULL
cleanup threads of the replaced tables must end. Without the change both waits
time out. With the change the table and cache suites in
{{hadoop-hdds/framework}} ({{TestTableCache}}, {{TestTypedTable}},
{{TestRDBTableStore}}, {{TestTypedRDBTableStore}}), {{TestOmMetadataManager}},
{{TestSnapshotCache}}, {{TestOmSnapshotManager}}, the
{{TestOzoneManagerDoubleBuffer}} suites, {{TestOMRatisSnapshots}},
{{TestOMInstallSnapshotDuringBootstrapping}}, {{TestOzoneManagerRestart}} and
the Recon {{TestReconOmMetadataManagerImpl}}, {{TestOMDBUpdatesHandler}} and
{{TestOzoneManagerServiceProviderImpl}} pass, and checkstyle is clean. Not run:
a heap check that the old caches become collectable, and a Recon sync against a
running OM.
Found by code review of the OM table cache, double buffer flush and startup
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing (nearest: HDDS-11201,
which added the scheduled FULL cache cleanup, and HDDS-11403). The attached
patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
