beyondhj opened a new issue, #11301:
URL: https://github.com/apache/gravitino/issues/11301
### Version
main branch
### Describe what's wrong
The ClassLoaderResourceCleanerUtils.closeStatsDataClearerInFileSystem()
method incorrectly shuts down the static scheduler field in
org.apache.hadoop.metrics2.lib.MutableQuantiles. This scheduler is a
JVM-level global static resource shared by all components that use Hadoop
metrics, not scoped to any specific ClassLoader.
Once this scheduler is shut down, any subsequent usage of MutableQuantiles
in the JVM will fail because the scheduler cannot be restarted.
Steps to reproduce
1. Create an Iceberg catalog (or any catalog that triggers
IcebergCatalogOperations.close())
2. Trigger catalog close (e.g., via cache eviction, alterCatalog,
dropCatalog, etc.)
3. The closeClassLoaderResource() method is called
4. MutableQuantiles.scheduler is shut down via scheduler.shutdownNow()
5. Any subsequent operation that uses Hadoop metrics2 MutableQuantiles
will fail
Expected behavior
The MutableQuantiles.scheduler should NOT be shut down when closing a
catalog's ClassLoader, because:
1. It is a static field shared globally across the JVM
2. It is not tied to any specific ClassLoader
3. Shutting it down permanently breaks metrics functionality for all
components
Actual behavior
The scheduler is shut down, causing permanent failure of
MutableQuantiles-based metrics in the entire JVM process.
Root cause analysis
The problematic code is in ClassLoaderResourceCleanerUtils.java:
private static void closeStatsDataClearerInFileSystem(ClassLoader
targetClassLoader)
throws Exception {
// ...
Class<?> mutableQuantilesClass =
Class.forName("org.apache.hadoop.metrics2.lib.MutableQuantiles",
true, targetClassLoader);
ScheduledExecutorService scheduler =
(ScheduledExecutorService)
FieldUtils.readStaticField(mutableQuantilesClass, "scheduler",
true);
scheduler.shutdownNow(); // <-- BUG: This shuts down a JVM-global
static resource
// ...
}
The assumption that MutableQuantiles.scheduler belongs to the catalog's
ClassLoader is incorrect. The class may be loaded by a parent ClassLoader (or
the system ClassLoader), and the static field is shared across the entire JVM.
### Error message and/or stacktrace
2026-05-29` 22:43:53.745 ERROR [Gravitino-webserver-73]
[org.apache.gravitino.server.web.rest.ExceptionHandlers$BaseExceptionHandler.handle(ExceptionHandlers.java:1051)]
- Failed to operate object operation [LIST] under
[iceberg_prod.test_db_1.test_table_1], reason [Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7801783c[Not
completed, task =
java.util.concurrent.Executors$RunnableAdapter@6d6df57b[Wrapped task =
org.apache.hadoop.metrics2.lib.MutableQuantiles$RolloverSample@2837ffe2]]
rejected from
java.util.concurrent.ScheduledThreadPoolExecutor@79c0cae2[Terminated, pool size
= 0, active threads = 0, queued tasks = 0, completed tasks = 23664]]
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7801783c[Not
completed, task =
java.util.concurrent.Executors$RunnableAdapter@6d6df57b[Wrapped task =
org.apache.hadoop.metrics2.lib.MutableQuantiles$RolloverSample@2837ffe2]]
rejected from
java.util.concurrent.ScheduledThreadPoolExecutor@79c0cae2[Terminated, pool size
= 0, active threads = 0, queued tasks = 0, completed tasks = 23664]
at
java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
~[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
~[?:?]
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
~[?:?]
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:680)
~[?:?]
at
org.apache.hadoop.metrics2.lib.MutableQuantiles.<init>(MutableQuantiles.java:110)
~[gravitino-aws-bundle-1.2.1.jar:?]
at
org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:224)
~[gravitino-aws-bundle-1.2.1.jar:?]
at
org.apache.hadoop.fs.s3a.S3AInstrumentation.quantiles(S3AInstrumentation.java:332)
~[gravitino-aws-bundle-1.2.1.jar:?]
at
org.apache.hadoop.fs.s3a.S3AInstrumentation.<init>(S3AInstrumentation.java:226)
~[gravitino-aws-bundle-1.2.1.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:400)
~[gravitino-aws-bundle-1.2.1.jar:?]
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
~[gravitino-aws-bundle-1.2.1.jar:?]
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
~[gravitino-aws-bundle-1.2.1.jar:?]
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
~[gravitino-aws-bundle-1.2.1.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
~[gravitino-aws-bundle-1.2.1.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
~[gravitino-aws-bundle-1.2.1.jar:?]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
~[gravitino-aws-bundle-1.2.1.jar:?]
at org.apache.iceberg.hadoop.Util.getFs(Util.java:55)
~[iceberg-core-1.10.1.jar:?]
at
org.apache.iceberg.hadoop.HadoopInputFile.fromLocation(HadoopInputFile.java:56)
~[iceberg-core-1.10.1.jar:?]
at
org.apache.iceberg.hadoop.HadoopFileIO.newInputFile(HadoopFileIO.java:87)
~[iceberg-core-1.10.1.jar:?]
at
org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:294)
~[iceberg-core-1.10.1.jar:?]
### How to reproduce
- Gravitino version: 1.2.1
- Hadoop version: 3.x
Prerequisites:
- A running Gravitino server
- An Iceberg catalog with default Hadoop FileSystem configuration
Steps:
1. Start Gravitino server
2. Create an Iceberg catalog
3. Create a schema and a table in the Iceberg catalog
4. Then wait for the cache to expire (e.g., 1 hour) without accessing the
catalog
5. Try to load the table again
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
