alamb commented on PR #23492: URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5301788924
> > * For local SSD it can often be better to serialize the filters, so the first filter can reduce IO for the second, second for the third, etc. > * For object store backends it can be better to reduce the number of calls (for maybe slightly higher bandwidth usage). But bandwidth still important to consider (e.g. for AWS, S3 GET is only ~100MB/s, a standard high core instance might have only a couple of GB/s network bandwidth..., and other limits) Yes for sure -- this is the key difference Where I was heading is that previously, in my mind we can't turn on predicate pushdown because it caused slowdowns. The slowdowns were caused by two reasons: 1. It changed the I/O patterns (which slows down local SSD, but likely improves for object store) 2. It was computationally slower in some cases (figuring out bitmasks, rows that passed, etc) I think after the great work from @hhhizzz @haohuaijin and others upstream in arrow-rs/parquet we are close (if not already) at the point where it is computationally faster. All that remains is the different I/O. So if we can have a mode where we can turn on predicate pushdown and the I/O pattern is the same then we could (finally) turn on predicate pushdown by default. > I think for ClickBench it might not matter (no offset index?) - so perhaps we should special case this (just request the columns). This is a great point -- maybe that could be the trigger (if we are running with predicate pushdown but there is no page index, just fetch the column page data for all columns in one go) 🤔 > Should we make a ClickBench dataset with a page index? Or an option on the CB benchmark if possible? We could -- though I think we should figure out what exactly we are trying to measure / accomplish with such a benchmark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
