alamb commented on PR #11165: URL: https://github.com/apache/arrow-rs/pull/11165#issuecomment-5800174450
> The ranges the scan may need, regardless of whether the decoder or an external cache already has those bytes; or > The additional ranges the decoder is expected to request, after accounting for its current buffers? I did preferr this approach as I thought it was easier to explain. I also thought it had the invariant that if the preview API returns a range, then some future decode attempt will return NeedsData for that range. However, the invariant will not be true when pushdown filtering is enabled (and preview can return ranges that would not actually be fetched) > This keeps Parquet responsible for determining what the scan may need, while leaving availability, prefetching, request coalescing, and memory-budget decisions to the layers that own them. It also allows the same planning API to support different cache and I/O implementations without needing to understand each one. Well, technically speaking I think both approaches keep the responsibility of prefetching, etc to layers other than Parquet. If the decoder already has cached data, then that data came from outside the decoder from one of those other layers. But that is likely me being pedantic So in conclusion I think either approach is fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
