adriangb opened a new pull request, #17758: URL: https://github.com/apache/datafusion/pull/17758
Currently we prefetch 1 file. In bandwith limited systems (e.g. object storage) having wider IO than CPU tends to make sense. This makes the prefetch configurable so that we can open >2 files at a time. My main concern is memory use. For now users will have to tune this manually. Long term I think we should track how much prefetched data is 'waiting' using a memory pool + the newly added memory tracking in arrow -> you can set a pretty large prefetch value and it will be throttled by memory. This means that scans with very selective filters (that produce few rows per file) will benefit heavily from wide IO prefetching while scans that produce large amounts of data will do little prefetching if they hit memory limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
