binhuiliu opened a new issue, #12246:
URL: https://github.com/apache/gravitino/issues/12246
### Version
Apache Gravitino 1.3.0 (`apache/gravitino:1.3.0`)
This version already contains the fix from #10844 / #10854 that closes
`HiveClientFactory` when a `HiveClientPool` is closed.
### Describe what's wrong
`gravitino-server` still reaches `java.lang.OutOfMemoryError: Metaspace`
after long-running periodic access to a Hive catalog.
The workload does not use a custom cloud IAM authenticator or token-derived
client-pool cache keys. A Hive catalog is accessed periodically, with an
interval longer than the default five-minute Hive client-pool cache eviction
interval. This allows the cached client pool to expire between otherwise normal
Hive catalog requests.
Although the #10854 close path is present in 1.3.0, a heap dump shows that
closed `HiveClientClassLoader` instances remain strongly reachable through the
global Log4j/SLF4J logger registry.
The relevant retention path reported by Eclipse Memory Analyzer is:
```text
jdk.internal.loader.ClassLoaders$AppClassLoader
-> org.slf4j.LoggerFactory
-> org.apache.logging.slf4j.SLF4JServiceProvider
-> org.apache.logging.slf4j.Log4jLoggerFactory
-> registry
-> java.util.concurrent.ConcurrentHashMap
-> logger/configuration objects associated with HiveClientClassLoader
-> org.apache.gravitino.hive.client.HiveClientClassLoader
```
This means that calling `HiveClientFactory.close()` and
`URLClassLoader.close()` does not make the Hive client classloader collectible.
The classloader remains reachable and its classes continue consuming Metaspace.
### Expected behavior
After a Hive client pool is evicted and closed, its `HiveClientFactory` and
`HiveClientClassLoader` should become unreachable and eligible for garbage
collection, unless they are intentionally reused.
Periodic Hive catalog access should not cause the number of retained Hive
client classloaders or Metaspace usage to grow over time.
### Actual behavior
After approximately 12 days of uptime:
```text
MaxMetaspaceSize: 512.00 MB
Metaspace used: 509.23 MB
HiveClientClassLoader instances: 109
Classes loaded by those classloaders: 52,702
Full GC count: 5,616
OutOfMemoryError: Metaspace occurrences: approximately 100
```
Eclipse Memory Analyzer reported:
```text
109 HiveClientClassLoader instances
retained heap: approximately 62 MB
737 org.apache.logging.log4j.core.config.properties.PropertiesConfiguration
instances
1,473 org.apache.logging.log4j.core.appender.RollingFileAppender instances
566 java.util.zip.ZipFile$Source instances
```
108 of the 109 `HiveClientClassLoader` instances were reachable through
`Log4jLoggerFactory.registry`.
The Java heap itself was not exhausted. At the time of diagnosis, heap
occupancy was approximately 182 MB out of a 1 GB heap, while Metaspace was
approximately 509 MB out of its 512 MB limit.
### Error message and stacktrace
Hive catalog schema/table listing fails with:
```text
org.apache.gravitino.exceptions.GravitinoRuntimeException: Failed to connect
to Hive Metastore
at
org.apache.gravitino.hive.HiveClientPool.newClient(HiveClientPool.java:56)
...
Caused by: org.apache.gravitino.exceptions.GravitinoRuntimeException:
Metaspace
at
org.apache.gravitino.hive.client.HiveExceptionConverter.convertException(HiveExceptionConverter.java:199)
at
org.apache.gravitino.hive.client.HiveExceptionConverter.toGravitinoException(HiveExceptionConverter.java:118)
at
HIVE3//org.apache.gravitino.hive.client.HiveShimV3.createMetaStoreClient(HiveShimV3.java:181)
...
Caused by: java.lang.OutOfMemoryError: Metaspace
```
The Hive Metastore container remains healthy and accepts connections. The
failure occurs while Gravitino creates or initializes the Hive client.
### How to reproduce
The following periodic Hive catalog access pattern led to the issue:
1. Start Gravitino 1.3.0 with a Hive catalog and the default Hive
client-pool settings.
2. Start the JVM with a bounded Metaspace, for example:
```text
-Xms1024m -Xmx1024m -XX:MaxMetaspaceSize=512m
```
3. Query the Hive catalog schemas and tables periodically through any
supported client path.
4. Keep the interval between accesses longer than the default client-pool
cache eviction interval (five minutes), so that a client pool is evicted and
recreated between access cycles. In the observed environment, the interval was
approximately 30 minutes.
5. Monitor the process over time:
```bash
jcmd <pid> VM.metaspace basic scale=MB
jcmd <pid> VM.classloader_stats
```
6. Observe that retained `HiveClientClassLoader` instances and Metaspace
usage increase. Eventually Hive catalog operations fail with `OutOfMemoryError:
Metaspace`.
An accelerated reproducer may be possible by configuring a shorter
`client.pool-cache.eviction-interval-ms`, repeatedly querying the Hive catalog
after each eviction, and forcing GC between iterations. This accelerated form
has not yet been independently verified.
### Relationship to #10844 / #10854
This appears to be a follow-up to #10844 rather than the same missing-close
bug.
#10854 added the following lifecycle:
```text
HiveClientPool.close()
-> super.close()
-> closeClientFactory()
-> HiveClientFactory.close()
-> HiveClientClassLoader.close()
```
That close path is present in the affected 1.3.0 build. However, the heap
dump shows that the closed classloaders remain strongly reachable through
`Log4jLoggerFactory.registry`.
Therefore, #10854 ensures that `close()` is invoked, but does not ensure
that the Hive client classloader becomes collectible after close.
### Additional context
- Java: OpenJDK 17.0.16
- Operating system: Ubuntu 22.04
- In the observed deployment, the periodic access originated from an
OpenMetadata workflow querying the Hive catalog through Trino. This is
deployment context rather than a required reproduction condition.
- Gravitino container remained `running` and `healthy`.
- The Gravitino HTTP version/health endpoint continued returning HTTP 200
while Hive catalog operations failed.
- The host had approximately 15 GB of available physical memory, so this was
not a host-level memory exhaustion event.
- Full heap dumps may contain deployment configuration or application data
and are therefore not attached publicly. Sanitized MAT statistics and
additional object-retention details can be provided if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
