zhuqi-lucas opened a new issue, #25360:
URL: https://github.com/apache/datafusion/issues/25360
### Describe the bug
Inside one `EnsureRequirements::optimize` call, the distribution phase and
the sorting phase frequently both rewrite the plan and arrive back where they
started. The call does two full tree rewrites and returns an equivalent plan.
I instrumented every phase of `EnsureRequirements::optimize`
(`physical-optimizer/src/ensure_requirements/mod.rs`) to report whether it
changed the rendered plan, and ran a real 34-node query plan through a chain
that enforces requirements six times. Over 31 calls:
```
16x net=unchanged phases that fired: (2a ensure_distribution, 2b
ensure_sorting)
7x net=unchanged phases that fired: ()
4x net=CHANGED phases that fired: (2a, 2b, 3b
replace_with_order_preserving_variants)
4x net=CHANGED phases that fired: (2a, 2b)
```
So of 23 calls that changed nothing overall, **16 got there by having 2a
rewrite the plan and 2b rewrite it back**. Only 7 were genuinely quiet.
A second, independent measurement agrees. After making the rule report
transformation accurately (filed separately), comparing each call's input
against its output:
```
8x content=diff pointer=diff real work
16x content=SAME pointer=diff changed nothing, still rebuilt the tree
7x content=SAME pointer=SAME genuinely untouched
```
The 16 and the 7 line up exactly with the phase data.
### To Reproduce
Instrument the phase boundaries in `EnsureRequirements::optimize` with a
rendered-plan comparison and plan a query whose scan has an output ordering
that conflicts with a window function's required sort, through a chain with
several enforcement passes. I have not managed to reduce this to the built-in
chain over built-in sources; it is reliable on the chain that surfaced it.
### Expected behavior
A call that changes nothing should not rewrite the plan twice to get there.
Beyond the wasted work, this is what makes the rule's output a fresh object
every time, which costs every caller that wants to know whether anything
happened.
Relatedly, the rule does not reach its fixpoint in one pass: two consecutive
applications both changed the plan in these measurements, the second moving a
`RepartitionExec` below a `SortExec` and switching that sort to per-partition.
A chain that enforces requirements once after its own rewrites therefore may
not be getting a settled plan.
### Additional context
Found while measuring #25355 / #25356. Roughly 60ms of a 209ms physical
optimization phase in that plan is enforcement passes that change nothing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
