alamb opened a new issue, #24393: URL: https://github.com/apache/datafusion/issues/24393
### Is your feature request related to a problem or challenge? - Related to https://github.com/apache/datafusion/issues/3463 - This is an idea for https://github.com/apache/datafusion/issues/20324 Namely, the idea is to avoid regressions when we turn on filter pushdown We haven't been able to turn on predicate pushdown because it caused slowdowns. The slowdowns were caused by two reasons: 1. predicate pushdown changes the I/O patterns 2. Evaluating predicates is computationally slower in some cases (figuring out bitmasks, rows that passed, etc) I think after the great work from @hhhizzz @haohuaijin and others upstream in arrow-rs/parquet we are close (if not already) at the point where it is computationally the same or faster All that remains is the different I/O The I/O pattern today is 1. Default: read all column data needed for both filtering and projections when reading a row group 2. Predicate Pushdown (`pushdown_filters`): sequentially fetch the columns required to evaluate filter expressions, and then after filtering is complete, fetches columns needed to evaluate the projections @Dandandan / @adriangb also noted on https://github.com/apache/datafusion/pull/23492#issuecomment-5297913079 that we can't prune I/O for files without a page index (notably the clickbench dataset) anyways , so other than peak buffer needs, there is no benefit to changing the I/O to read columns sequentially in that case ### Describe the solution you'd like I would like is some way to turn on filter pushdown but **keep the I/O pattern the same as today** (aka read all required data pages for a Row Gorup) up front in one request That way, once we sort out any lingering CPU slowdowns, we can turn on predicate pushdown by default and leave the I/O pattern as a policy choice (as the I/O policy will likely different for different I/O systems -- e.g. NVMe vs ObjectStore. ### Describe alternatives you've considered So I think we should separate out the notions of "filter evaluation mechanics" from the I/O pattern. One idea is to add a parquet configuration setting like 'progressive_io' (defaults to false, the behavior today). When true it would fetch data progressively, as happens today when filter_pushdown is enabled So we would have a feature matrix like | filter_pushdown | progressive_io | description | |--------|--------|--------| | false | false | Same as todays behavior | | true | false | filter pushdown, but still keep same I/O (my proposed new default) | | false | true | N/A (just a decode, so no progressive filtering) | true | true | progressive filter evaluation and I/O (what happens today when `filter_pushdown`) is enabled ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
