NGA-TRAN opened a new pull request, #24501:
URL: https://github.com/apache/datafusion/pull/24501
## Which issue does this PR close?
- Closes #23569
- Depends on #24500 (test coverage that pinned today's hash-shuffle plan)
## Rationale for this change
A table that is range-partitioned on `timestamp` (for example by hour) and
grouped by `date_bin` / `date_trunc` of that timestamp is already
partition-disjoint when bins do not straddle split points. DataFusion still
planned Partial → hash `RepartitionExec` → Final.
After this change, `Range([timestamp])` subset-satisfies
`KeyPartitioned([key, f(timestamp)])` when `f` is monotonic (`date_bin`,
`date_trunc`) and evaluating `f` at each split vs. its predecessor shows the
bins are disjoint. Aggregation then runs as one streaming `SinglePartitioned`
step with `ordering_mode=Sorted`.
Bins that do straddle the split (for example `date_trunc('day')` on
hour-partitioned data) still hash-repartition.
#24500's expected plan changes from:
```text
AggregateExec FinalPartitioned
RepartitionExec Hash([key, date_bin(...)])
AggregateExec Partial, ordering_mode=Sorted
DataSourceExec Range([timestamp]), 2 file groups
```
to:
```text
AggregateExec SinglePartitioned, ordering_mode=Sorted
DataSourceExec Range([timestamp]), 2 file groups
```
## What changes are included in this PR?
- Project `Range` through monotonic grouping expressions when bins stay
disjoint.
- Treat that case as subset satisfaction so `SanityCheckPlan` accepts
`SinglePartitioned` over `Range([timestamp])`.
- Update `range_sorted_time_bin_agg.slt` for `date_bin(60s)` and
`date_trunc('hour')` (no shuffle) and `date_trunc('day')` (shuffle remains).
## Are these changes tested?
- Unit tests in `partitioning.rs` for aligned `date_bin` /
`date_trunc('hour')` vs unaligned split and `date_trunc('day')`.
- `cargo test --test sqllogictests -- range_sorted_time_bin_agg.slt`
## Are there any user-facing changes?
Queries that group by `date_bin` / `date_trunc` of a range-partitioned
timestamp may skip a hash shuffle when bins do not cross file-group boundaries.
Results are unchanged.
## Test plan
- [x] `cargo test -p datafusion-physical-expr --lib -- partitioning`
- [x] `cargo test --test sqllogictests -- range_sorted_time_bin_agg.slt`
- [ ] CI
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
