tschwarzinger commented on issue #26682: URL: https://github.com/apache/datafusion/issues/26682#issuecomment-6575114525
One take on prefetching and `progressive_io` trade-off from someone not too familiar with the Parquet scanner: Even when `progressive_io` is turned on, turning on prefetching could make sense if it fetches across the row group boundaries: | Time | Decoder | Fetcher | |--------|--------|--------| | 1 | - | filter 1 columns, RG 1 | | 2 |filter 1, RG 1 | filter 1 columns, RG 2 | | 3 | filter 1, RG 2 | filter 2 columns, RG 1, using selection | | 4 | filter 2, RG 1 | filter 2 columns, RG 2, using selection | | 5 | ... | ... | Each later-stage fetch still uses the earlier filters’ results, so I/O stays as small as with `progressive_io`, while the CPU keeps busy by working on other row groups. The I/O buffers for all row groups except the "current" one would be limited by the read-ahead byte budget. I am not sure how this plays into the current implementation of the morselizer though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
