galaxytemple opened a new pull request, #27450:
URL: https://github.com/apache/datafusion/pull/27450
## Which issue does this PR close?
- Closes #26809.
## Rationale for this change
Ungrouped `MIN(col)` / `MAX(col)` queries with a filter (e.g., `SELECT
MAX(ts) FROM t WHERE val > 0`) and TopK queries over a **`FilterExec`**
currently scan Parquet files and row groups in their original listing order
rather than statistics order:
1. **`FilterExec` did not implement `try_pushdown_sort`**: Even though
`FilterExec` preserves input ordering, it did not forward sort pushdown
requests to its input, blocking sort pushdown whenever a `FilterExec` remained
above the scan.
2. **`PushdownSort` only matched `SortExec`**: Ungrouped `MIN`/`MAX`
`AggregateExec` nodes already emit a dynamic filter (`AggrDynFilter`,
introduced in #18644), but without a scan-order hint from `PushdownSort`, files
and row groups were scanned in listing order rather than statistics order,
preventing the dynamic filter bound from tightening early.
3. **`AggregateExec::init_dynamic_filter` only allowed
`AggregateMode::Partial`**: In single-partition plans,
`CombinePartialFinalAggregate` merges `Partial + Final` into
`AggregateMode::Single` before `FilterPushdown(Post)` runs, which dropped
`dynamic_filter` to `None` even though a single-partition scan can still read
multiple files or row groups.
Another possible approach was rewriting `SELECT MAX(col)` into `ORDER BY col
DESC LIMIT 1` in the logical optimizer. However, `LIMIT 1` returns 0 rows on
empty input (while `MAX` returns a `[NULL]` row). And using the existing
`AggregateExec` dynamic filter and `try_pushdown_sort` felt much simpler.
## What changes are included in this PR?
- Implement `FilterExec::try_pushdown_sort` in
`datafusion/physical-plan/src/filter.rs`:
- Forward the sort request to the input and rebuild `FilterExec` via
`FilterExecBuilder::from(self).with_input(new_input)`.
- Remap sort columns through `self.projection` using
`FilterRemapper::with_column_mapping` (same as
`FilterExec::handle_child_pushdown_result`).
- Return `Unsupported` if `self.fetch.is_some()`.
- Allow `AggregateMode::Single` in `AggregateExec::init_dynamic_filter`.
- In `PushdownSort`, push `[col DESC NULLS LAST]` for ungrouped `MAX(col)`
and `[col ASC NULLS LAST]` for `MIN(col)` down to the aggregate input when
dynamic filter pushdown is enabled and the input has no `fetch`.
## What is the testing strategy for this PR?
- Added unit tests in `filter.rs` checking that
`FilterExec::try_pushdown_sort` forwards the sort order, remaps column indices
when a projection is present, and returns `Unsupported` when `fetch` is set.
- Added snapshot tests in `pushdown_sort.rs` for sort pushdown through
`FilterExec` and for ungrouped `MIN`/`MAX` aggregates.
- Tested locally by generating dummy 50 Parquet files (10M rows total, 200k
rows per file in ascending `ts` order) and running `EXPLAIN ANALYZE`:
- `SELECT MAX(ts) FROM t WHERE val > 0`: **2.78s → 29ms** (reads
`file_050` first and skips the remaining 49 files via dynamic filter).
- `SELECT ts FROM t WHERE val > 0 ORDER BY ts DESC LIMIT 1`: **2.75s →
34ms**.
## Are there any user-facing changes?
No API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]