sundapeng opened a new issue, #10522: URL: https://github.com/apache/paimon/issues/10522
### Search before asking - [x] I searched in the issues and found nothing similar. ### Paimon version Master at `b24751731a22fa9788205897b6554f247a3fe300`. ### Compute Engine Engine independent: any reader or writer of a REST catalog table with `data-token.enabled`. ### Minimal reproduce step 1. Access an OSS table through a REST catalog with `data-token.enabled=true`. `RESTTokenFileIO` creates one delegate per data token with `file-io.allow-cache=false`, so the delegate owns its file systems instead of sharing cached ones. 2. Keep the job running across token refreshes. Each refresh creates a new delegate, and the previous one stays in `RESTTokenFileIO`'s `FILE_IO_CACHE` until it is evicted (10 hours without access, or more than `setFileIOCacheMaximumSize` entries, 1,000 by default). 3. On eviction the removal listener calls `close()` on the delegate. The delegate returned by `FileIO.get(oss://...)` is `OSSLoader.OSSPluginFileIO`, and `PluginFileIO` does not override `FileIO.close()`, whose default does nothing. ### What doesn't meet your expectations? `OSSFileIO.close()`, which closes its file systems when `allow-cache` is false, is never reached. The `AliyunOSSFileSystem` instances of every evicted delegate are left open, together with their OSS clients and thread pools, and a long-running job accumulates one set per refreshed token. The same applies to the other loaders built on `PluginFileIO` (S3, Azure, GS, OBS, COSN). ### Anything else? Forwarding `close()` from `PluginFileIO` to the plugin `FileIO` is not enough on its own: an evicted delegate can still back streams that are open, such as a long read or a multipart upload, and closing its file systems on eviction would break them. Closing after the last stream would need `RESTTokenFileIO` to track the streams opened through it, including those opened by callers that take the delegate from `fileIO()` directly. This came up in the review of #10178 and was left out of that PR. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
