zhuqi-lucas opened a new issue, #25362:
URL: https://github.com/apache/datafusion/issues/25362
### Describe the bug
`EnsureRequirements` does not reach its fixpoint in one application.
Applying it to a plan it has just produced can still change that plan, so a
chain that enforces requirements once after its own rewrites may be left with a
plan the rule itself would improve.
Observed on a real 34-node plan, logging each call's input against its
output:
```
ER#7 input P6 -> output P7 changed
ER#8 input P7 -> output P8 changed <- applied to its own output,
changed again
ER#9 input P8 -> output P8 unchanged <- fixpoint reached on the third
application
```
The surviving difference between P7 and P8 is a `RepartitionExec` moving
from above a `SortExec` to below it, with the sort switching to
`preserve_partitioning=true`:
```text
P7: RepartitionExec: Hash([ticker], 12) P8: ProjectionExec
ProjectionExec FilterExec
FilterExec BoundedWindowAggExec
BoundedWindowAggExec SortExec
preserve_partitioning=true
SortExec preserve_partitioning=false
RepartitionExec: Hash([ticker], 12)
<scan> <scan>
```
P8 is the better plan: the sort runs per partition instead of after a merge.
### To Reproduce
Plan a query whose scan has an output ordering that conflicts with a window
function's required sort, through a chain with several enforcement passes, and
compare consecutive `EnsureRequirements` outputs. I have not managed to reduce
this to the built-in chain over built-in sources; it is reliable on the chain
that surfaced it, and I am happy to help narrow it down.
### Expected behavior
Either the rule converges in one application, or it iterates internally
until it does. As it stands, whether a plan gets the better arrangement depends
on how many times the surrounding chain happens to call the rule, which is not
something a chain author can reason about.
This also means the number of enforcement passes a chain needs is not one
per plan-mutating rule, as the built-in ordering comments imply, but that plus
however many the rule needs to settle.
### Additional context
Found while measuring #25355 / #25356. The wasted-work side of the same area
is #25360; the transformation-reporting side is #25361.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
