gianm opened a new pull request, #19539: URL: https://github.com/apache/druid/pull/19539
This patch adds the setting `backgroundFetchExternalFiles`. When set, cloud storage files referenced by `ExternalSegment` (`EXTERN`) are fetched asynchronously into the task's storage locations. The setting defaults to true. To support this, new infrastructure is added: 1) `VirtualStorageManager`, a layer on top of `StorageLocation` that provides a simple "fetch and cache a file" API. 2) `StorageLoadingThreadPool`, an extraction of the thread pool from `SegmentLocalCacheManager` so it can be shared with `VirtualStorageManager`. 3) `AsyncResource`, a Future-like utility that provides better tools for managing Closeable resources. It is used by `VirtualStorageManager` to provide the asynchronously-fetched file handles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
