Dandandan commented on PR #23492: URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5297628109
> I have been thinking about this PR and how it could play into our "turn pushdown on by default" story > > We have put a lot of work in upstream to make applying filters during parquet decoding faster (thanks to @hhhizzz @haohuaijin and many others). However, enabling filter pushdown today **also** changes the I/O patterns (so that the all the needed columns are not fetched up front, but instead are fetched sequentially) > > I think one thing we could do with a nice I/O pre-fetching abstraction would be to start a request to fetch bytes for all columns required in a single request, so the I/O pattern is the same -- and then just enable the pushdown filter application once we already had the data 🤔 I think that's "tricky" because the depends on the IO backend, no? * For local SSD it can often be better to serialize the filters, so the first filter can reduce IO for the second, second for the third, etc. * For object store backends it can be better to reduce the number of calls (for maybe slightly higher bandwidth usage). But bandwidth still important to consider (e.g. for AWS, S3 GET is only ~100MB/s, a standard high core instance might have only a couple of GB/s network bandwidth...) I think for ClickBench it might not matter (no offset index?) - so perhaps we should special case this (just request the data). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
