chenwyi2 opened a new issue, #12032:
URL: https://github.com/apache/gravitino/issues/12032
### Version
main branch
### Describe what's wrong
Describe the bug
When Kerberos authentication is enabled for the Iceberg REST catalog, the
process accumulates a large number of idle daemon threads named
check-Iceberg-Hive-tgt-0 (or similar ticket-refresh threads). These threads
never exit and grow roughly in sync with catalog-wrapper cache recreation
(default eviction interval: 1 hour).
In a long-running production instance (~30 days), a thread dump showed ~1000
threads, most of which were stuck in:
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take
Each thread waited on a different Condition object, which indicates many
separate ScheduledThreadPoolExecutor instances were created and never shut down.
Root cause
IcebergCatalogWrapperManager.createCatalogWrapper() does:
new CatalogWrapperForREST(...) → creates catalog A + KerberosClient A + TGT
refresh thread A
new KerberosAwareIcebergCatalogProxy(rest).getProxy(catalogName, config) →
CGLIB Enhancer.create(constructorArgs) re-invokes the constructor → creates
catalog B + KerberosClient B + TGT refresh thread B
The cache only retains the proxy. On eviction, proxy.close() is intercepted
and delegated to target.close(), which closes A only. B is never closed, so its
scheduler thread leaks forever.
Relevant code:
iceberg/.../KerberosAwareIcebergCatalogProxy.java — e.create(argClass, new
Object[]{...}) re-runs constructors
iceberg/.../authentication/kerberos/KerberosClient.java — creates
ScheduledThreadPoolExecutor named check-Iceberg-Hive-tgt-%d
iceberg-rest-server/.../IcebergCatalogWrapperManager.java — expireAfterWrite
cache recreates wrappers periodically
Note: migrating to the shared catalogs/hadoop-auth KerberosClient improves
per-client lifecycle (e.g. shutdown before re-login), but does not fix this
leak, because the orphaned client from CGLIB construction is still never closed.
Expected behavior
Each Kerberos-enabled catalog wrapper lifecycle should own exactly one
KerberosClient / TGT refresh executor. When the wrapper is closed (including
cache eviction), that executor must be shut down and the thread must exit.
Recreating the wrapper must not leave orphaned refresh threads behind.
Suggested fix
Instantiate the CGLIB proxy without invoking IcebergCatalogWrapper /
CatalogWrapperForREST constructors (e.g. via Objenesis + Factory.setCallback),
and keep all method calls delegated to the already-constructed target.
Avoid “fix only in close()” as the primary solution: the proxy should never
own a second catalog/Kerberos client in the first place.
### Error message and/or stacktrace
"check-Iceberg-Hive-tgt-0" #17526 daemon prio=5 os_prio=0 cpu=140.41ms
elapsed=2117176.03s tid=0x00007fe07ad9dab0 nid=0xf2d10 waiting on condition
[0x00007fdce77f6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000800b5ab4d18> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:252)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:1672)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
at
java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1062)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1122)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
at java.lang.Thread.run([email protected]/Thread.java:833)
### How to reproduce
1.1.0
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
