LuciferYang opened a new issue, #12123: URL: https://github.com/apache/gravitino/issues/12123
### Version main branch ### Describe what's wrong When a catalog that used the MySQL Connector/J driver is dropped or reloaded, its isolated ClassLoader is not garbage collected, so Metaspace grows on every reload and eventually hits `OutOfMemoryError: Metaspace`. Two references keep the ClassLoader alive after cleanup runs: 1. MySQL's `AbandonedConnectionCleanupThread` (`mysql-cj-abandoned-connection-cleanup`) holds the driver's ClassLoader as its context ClassLoader. A plain thread interrupt does not release it; only `uncheckedShutdown()` clears the tracked-connection map so the thread and ClassLoader can be collected. 2. `ClassLoaderResourceCleanerUtils.clearThreadLocalMap` only sweeps `Gravitino-webserver-*` threads. ThreadLocals that reference the dropped ClassLoader can also sit on other threads (Caffeine ForkJoinPool workers, catalog-cleaner, Hadoop daemons), and those are never cleared. `closeClassLoaderResource` is the shared routine that releases a catalog ClassLoader's resources, but it releases neither of the above references, so a MySQL-backed catalog leaks its ClassLoader whenever cleanup runs. ### Error message and/or stacktrace `java.lang.OutOfMemoryError: Metaspace` after repeatedly creating and dropping MySQL-backed catalogs (Metaspace committed size grows monotonically with reload count). ### How to reproduce 1. Gravitino version: main branch 2. Create a catalog that loads the MySQL driver (a `jdbc-mysql` catalog, or an Iceberg JDBC catalog on MySQL). 3. Drop the catalog, then repeat create/drop in a loop. 4. Observe with `jcmd <pid> VM.metaspace` (or a heap dump) that the isolated ClassLoader and the `mysql-cj-abandoned-connection-cleanup` thread survive each drop; Metaspace committed size keeps climbing. ### Additional context The fix is scoped to `ClassLoaderResourceCleanerUtils`: shut down the MySQL `AbandonedConnectionCleanupThread` (guarded so it only acts when the target ClassLoader owns the driver class), and broaden the ThreadLocal sweep beyond webserver threads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
