zhuqi-lucas commented on code in PR #21976:
URL: https://github.com/apache/datafusion/pull/21976#discussion_r3214871632
##########
datafusion/physical-optimizer/src/optimizer.rs:
##########
@@ -170,18 +169,12 @@ impl PhysicalOptimizer {
// those are handled by the later `FilterPushdown` rule.
// See `FilterPushdownPhase` for more details.
Arc::new(FilterPushdown::new()),
- // The EnforceDistribution rule is for adding essential
repartitioning to satisfy distribution
- // requirements. Please make sure that the whole plan tree is
determined before this rule.
- // This rule increases parallelism if doing so is beneficial to
the physical plan; i.e. at
- // least one of the operators in the plan benefits from increased
parallelism.
- Arc::new(EnforceDistribution::new()),
- // The CombinePartialFinalAggregate rule should be applied after
the EnforceDistribution rule
+ // EnsureRequirements: merged EnforceDistribution + EnforceSorting
into a
+ // single idempotent rule with distribution-aware pushdown_sorts.
+ // See https://github.com/apache/datafusion/issues/21973
Review Comment:
Thanks @2010YOUY01 for review and good suggestion!
Will apply the suggestion in the next push (per alamb's plan I'll land a
test-extraction PR first and rebase this on top).
On "distribution-aware pushdown_sorts": the old pushdown_sorts didn't
propagate the parent's required_input_distribution, so a SortExec could get
pushed below something requiring SinglePartition (e.g. under GlobalLimitExec) —
giving an invalid plan because the sort then runs on multiple partitions. The
fix propagates the requirement and inserts a SortPreservingMergeExec where
needed. Will link to #21973 in the comment block when I update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
