dwsmith1983 opened a new pull request, #7555:
URL: https://github.com/apache/datafusion-comet/pull/7555
## Which issue does this PR close?
Closes #7553.
#6785 should merge after this one: it restores the whole reported order the
same way for windows that run natively, and its Spark fallback relies on this
rule.
## Rationale for this change
When a window stays in Spark over an input whose order Comet may report
falsely, `sortDoubtedWindowInputs` sorts that input again, but only by the
window's own keys. Spark had already planned the operators above the window
around the longer order the input reported, so a second window ordered by more
keys, or a `sortWithinPartitions` Spark removed as redundant, sees rows out of
order. That happens when the input really is out of order past the window's
keys (a native nested loop join with more than one build row), and also when it
is in order but the inserted sort spills and the merge reorders ties.
## What changes are included in this PR?
- The inserted sort uses the whole order the window's input reports when
that order covers the window's keys, and the window's own keys otherwise.
- The contributor guide's note on this rule says the same.
## How are these changes tested?
Two tests in `CometInMemoryCacheSuite` run a Spark window ordered by `v`
under one ordered by `(v, w)` over a sorted cache joined through a native
nested loop join, and check the rows against Spark, the order of a
`sortWithinPartitions("v", "w")` above the lower window, and that the only sort
in the plan is the one this rule adds, by `(v, w)`. One uses two build rows,
the other one build row with a forced spill. Both fail without the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
