zhangstar333 opened a new issue, #8548: URL: https://github.com/apache/paimon/issues/8548
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version - Paimon: 1.1.1 in the affected deployment - Hadoop FileSystem: Aliyun OSS Hadoop FileSystem - Access pattern: many concurrent Paimon scan splits - Hadoop FS cache: disabled to isolate credentials ### Compute Engine java ### Minimal reproduce step We observed unbounded JVM thread growth when Paimon reads object storage through `HadoopFileIO` in an embedded compute-engine JVM. The issue is reproducible when all of the following are true: 1. The engine creates/deserializes a Paimon `Table` (and thus a `HadoopFileIO`) for many scan splits. 2. Hadoop FileSystem cache is disabled, for example: ```text fs.oss.impl.disable.cache=true This configuration is commonly required to avoid credential reuse between different AK/SK or STS tokens. 3. The underlying Hadoop FileSystem implementation creates executor threads. In our case it is: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem ## Observed behavior HadoopFileIO maintains a per-instance map: Map<Pair<String, String>, FileSystem> fsMap; When the Hadoop global FileSystem cache is disabled, each HadoopFileIO instance creates its own FileSystem instance through path.getFileSystem(conf). However, HadoopFileIO does not override close(). Therefore, calling: table.fileIO().close(); does not close the FileSystem instances because FileIO.close() is currently a no-op by default. For OSS, each unclosed AliyunOSSFileSystem may retain transfer/copy executor threads. With many scan splits, the JVM thread count keeps increasing and can eventually reach thousands or more. We checked Paimon 1.4.2, the release-1.4 branch, and current master; HadoopFileIO still has fsMap but does not close its FileSystem instances. ### What doesn't meet your expectations? When a HadoopFileIO instance is no longer used, its owned non-shared FileSystem instances should be released so that connector-specific resources, including executor threads, are shut down. ## Question / proposal Would the Paimon community consider implementing resource cleanup in HadoopFileIO.close()? Conceptually, it would close and clear the FileSystem instances stored in the instance-local fsMap. There are two design questions we would appreciate guidance on: 1. Is HadoopFileIO intended to own and close the FileSystem instances in its fsMap? 2. How should cleanup behave when Hadoop global FileSystem cache is enabled, so that closing one HadoopFileIO does not unexpectedly affect another user of a shared cached FileSystem? A Paimon-level fix would help all compute engines using HadoopFileIO with disabled Hadoop FS cache, not only our embedded-engine scenario. ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
