NGA-TRAN opened a new pull request, #24440: URL: https://github.com/apache/datafusion/pull/24440
## Which issue does this PR close? - Closes #24438. ## Rationale for this change When data is range-partitioned on group-by keys, merging many data partitions into fewer DataFusion partitions (to keep `target_partitions` near the CPU count) can lose the global `(key, time)` sort order. `AggregateExec` then falls back to blocking hash aggregation even though group-by keys still do not overlap across the merged partitions. This PR lets callers declare that guarantee so aggregation can stream without requiring a global sort. ## What changes are included in this PR? - Add a `partition_disjoint_exprs` property on `EquivalenceProperties`, with setter/clear/satisfies helpers. - Let `MemoryExec` / `TestMemoryExec` attach the property via `try_with_partition_disjoint_keys`. - When `InputOrderMode` is `Linear` but group-by keys are partition-disjoint, `AggregateExec` uses incremental emission and a `GroupOrdering::PartitionDisjoint` path that emits completed groups when each partition finishes. - Clear the property on `CoalescePartitionsExec` and `RepartitionExec`, where the disjointness guarantee is lost. - Treat the property like ordering when deciding whether preserving distribution enables streaming. ## Are these changes tested? - Unit test `partition_disjoint_partial_aggregate_planning` checks that the property enables streaming planning, and that the same unsorted input without the property stays blocking. - Fuzz test `partition_disjoint_streaming_aggregate_test` compares partition-disjoint incremental aggregation against a concatenated baseline. ## Are there any user-facing changes? Yes: public APIs on `EquivalenceProperties` and `MemoryExec` / `TestMemoryExec` to declare partition-disjoint group keys. No SQL syntax changes. Downstream engines that merge range-partitioned files can set this property so `AggregateExec` streams. Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
