zhuqi-lucas opened a new pull request, #25688:
URL: https://github.com/apache/datafusion/pull/25688
## Which issue does this PR close?
- Prerequisite for #25572 (the phase split, without the convergence loop
yet).
- Related to #25355.
## Rationale for this change
The physical optimizer runs a single hand-ordered rule list. Some of those
passes are not optimizations, they are *enforcers* of invariants:
`EnsureRequirements` inserts the repartitioning and sorting needed to satisfy
the distribution and ordering requirements every operator declares. The logical
layer already separates these two kinds of passes (`AnalyzerRule` makes a plan
valid, `OptimizerRule` makes it faster); the physical layer does not.
Giving enforcement its own phase is the prerequisite for #25572: once the
optimizer rules no longer include enforcement, they can be re-run to a fixed
point without re-running the expensive enforcement passes each time.
## What changes are included in this PR?
- A new `PhysicalAnalyzerRule` trait (mirrors the logical `AnalyzerRule`)
and a `PhysicalAnalyzer` rule list, plumbed through `Session` / `SessionState`
/ `SessionStateBuilder` symmetrically with the physical optimizer rules.
- `EnsureRequirements` implements `PhysicalAnalyzerRule` and is the sole
default analyzer. It is **also** kept as a `PhysicalOptimizerRule`, so any
custom rule list that still registers it keeps working.
- The planner runs the analyzer phase and then the optimizer phase.
### A note on ordering (why not a literal "all analyzers, then all
optimizers")
The sketch in #25585 ran all analyzers before all optimizers. Implemented
literally that breaks the default pipeline: `EnsureRequirements` would run
before `JoinSelection`, which decides broadcast (single-partition) vs
partitioned joins and therefore changes the required input distribution.
Enforcing first repartitions the inputs of a join that `JoinSelection` then
turns into a single-partition broadcast join, and `SanityCheckPlan` rejects the
result (`does not satisfy distribution requirements: SinglePartition`). This
matches the existing per-rule comments in `optimizer.rs` ("JoinSelection ...
should run before EnsureRequirements").
So the analyzer phase runs at `EnsureRequirements`' historical position:
after the optimizer passes that establish requirements and before those that
assume them (right before `CombinePartialFinalAggregate` in the default list).
The default pipeline order, and every resulting plan, is therefore unchanged.
Rearranging the list into cleaner phases is left to the follow-up in #25572.
## Are these changes tested?
Yes:
- Unit tests that the default analyzer list is `[EnsureRequirements]` and
that it is no longer in the default optimizer list.
- A planner test that a custom `PhysicalAnalyzerRule` registered via the
builder runs during planning, and that enforcement still produces a valid plan.
- The full `sqllogictest` suite passes with no plan changes (zero golden
churn), confirming the default order is preserved.
## Are there any user-facing changes?
- New public API: `PhysicalAnalyzerRule`, `PhysicalAnalyzer`,
`Session::physical_analyzers`, `SessionState::physical_analyzers` /
`add_physical_analyzer_rule`, and
`SessionStateBuilder::with_physical_analyzer_rule(s)`.
- `DefaultPhysicalPlanner::optimize_physical_plan`'s observer callback now
receives the rule name (`&str`) instead of `&dyn PhysicalOptimizerRule`, so
analyzer passes can be surfaced too (for example in `EXPLAIN VERBOSE`).
- Migration note: a custom `Session` that overrides `physical_optimizers()`
should also override `physical_analyzers()` (for example by delegating to
`SessionState`), otherwise enforcement no longer runs for that session.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
