andygrove opened a new pull request, #2806: URL: https://github.com/apache/datafusion-ballista/pull/2806
# Which issue does this PR close? No issue. Split out of #2792. # Rationale for this change Until AQE decides an undecided join (`DynamicJoinSelectionExec`), the join reports the output partitioning of the join it was planned as. A `CollectLeft` hash join passes its right input's partitioning through. So a join that was planned with a one-partition table on the right, such as `nation`, reports one output partition, even though the join it resolves into may build that small table and output the other side's partitions. DataFusion's `CombinePartialFinalAggregate` runs during each replan. It sees an aggregate over what claims to be one partition and merges the `Partial` and `Final` pair into a single `Single` aggregate, which later replans cannot split back apart. Once the join resolves, the aggregate's input is coalesced and exchanged below it, and every task of the consuming stage reads and aggregates the whole input. In TPC-H q11 this hits the scalar `sum` subquery. At SF1000 with 256 partitions, each of 16 tasks read all 32M input rows, and the query took 14.7 s. With this change it takes 5.4 s (Spark + Comet on the same cluster: 43.7 s). The behavior is on `main` today. # What changes are included in this PR? - `DynamicJoinSelectionExec` reports `UnknownPartitioning` with the larger of its inputs' partition counts when the planned join reports fewer, both when it is built (`from_hash_join`, `from_sort_join`) and in `with_new_children`. Partial and final aggregates above it therefore stay split. Once the join is decided, the real join's properties take over. - Test `undecided_join_reports_at_least_its_widest_input`: a `CollectLeft` join with a one-partition right input must report the wider left input's partition count, including after `replace_children`. # Are there any user-facing changes? No. Only plans with an aggregate above an undecided join over a one-partition input change: the aggregate stays partial and final instead of becoming single. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
