xudong963 opened a new pull request, #27119: URL: https://github.com/apache/datafusion/pull/27119
## Which issue does this PR close? Closes #27114. ## Rationale for this change Each aggregate with `FILTER` currently filters every input column before evaluating its arguments. Queries with several filtered aggregates repeatedly copy columns that each aggregate does not read. This change filters only the columns needed by that aggregate, while preserving evaluation after filtering so rejected rows cannot cause errors such as division by zero. ## What changes are included in this PR? - Record the input columns used by each aggregate's arguments, including its `ORDER BY` expressions. Unread columns use data-free `NullArray` placeholders to keep column indices valid. - Apply this to ungrouped aggregation, compacting grouped hash aggregation, and skip-partial state conversion. Skip-partial conversion shares one selected batch across all arguments. - Add SQL regressions for multiple aggregates, ordering, NULL and empty selections, fallible arguments, multiple partitions, and multi-argument skip-partial conversion. Physical-plan tests verify that unused columns are not copied and that skip-partial execution is exercised. For the issue's 20M-row reproducer with 15 filtered aggregates, local medians of 21 warm runs after three warm-ups were: | Query | main (`70f2ed01b`) | This change | Speedup | |---|---:|---:|---:| | No `GROUP BY` | 68 ms | 37 ms | 1.84x | | `GROUP BY quantity % 8` | 95 ms | 56 ms | 1.70x | Both builds used `release-nonlto`; all results matched. The corresponding `CASE WHEN` controls measured 46/47 ms and 80/86 ms. Other local builds were running concurrently, so these timings include scheduling noise. ## What is the testing strategy for this PR? - `cargo fmt --all` and `cargo clippy --profile ci --all-targets --all-features -- -D warnings` passed. - The extended workspace test configuration completed across 61 Rust test binaries: 13,493 passed and 8 ignored. Disk exhaustion interrupted the original command; after freeing build artifacts, the remaining compiled test binaries were run from their crate directories. The complete SQLLogicTest suite was rerun separately: all 548 files passed. - License-header validation and spelling checks on changed files passed. - Ran 10 representative cases from the existing `aggregate_filter` Criterion benchmark against main, covering column/literal arguments, selection rates, group cardinality, and unfiltered controls. An initial 5.8% slowdown in an unfiltered control did not consistently reproduce in three alternating-order repetitions (fix/main ratios 0.998, 0.943, and 1.019). ## Are there any user-facing changes? Lower copying overhead for aggregate `FILTER` queries. SQL behavior and public APIs are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
