JingsongLi commented on PR #8962: URL: https://github.com/apache/paimon/pull/8962#issuecomment-5467406603
Thanks for the investigation and for splitting the changes. After review, we do not plan to support `fs.oss.impl.disable.cache=true` in Paimon. `HadoopFileIO` and the `FileIO` lifecycle were designed around Hadoop-managed/shared `FileSystem` instances. Paimon does not provide an end-to-end ownership or lease contract for a per-`FileIO` filesystem, especially for streams and iterators that can outlive the wrapper or cache entry. The additional changes needed in `FileIO.get`, `PluginFileIO`, `ResolvingFileIO`, and `RESTTokenFileIO` show that making `HadoopFileIO.close()` destructive is not a local bug fix. It changes the resource-lifetime contract across the stack and can close a filesystem while it is still in use. We do not want to take on that contract for a configuration Paimon was not designed to support. Therefore we do not plan to resolve #8548, and this PR should not be merged. Deployments using Paimon should keep the Hadoop filesystem cache enabled. Please close this PR. Thanks again for the work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
