dwsmith1983 opened a new issue, #7553:
URL: https://github.com/apache/datafusion-comet/issues/7553
### Describe the bug
Found in #6785. `CometExecRule.sortDoubtedWindowInputs` (from #7256) puts a
`SortExec` under a window that stays in Spark when Comet may report a false
order for its input. The sort uses only the window's own required ordering.
Spark planned the operators above that window around the longer order the input
reported, so they lose it:
- A second Spark window ordered by `(v, w)` above a window ordered by `v` is
not sorted again, because `mayReportFalseInputOrder` now stops at the new sort.
- A `sortWithinPartitions(v, w)` above was already removed as redundant.
Two ways this returns wrong rows on main (891008e3c):
1. The input really is out of order past the window's keys. A native
broadcast nested loop join with two build rows emits its rows one build row at
a time (#6898). Sorting by `v` alone leaves `w` out of order, and `row_number()
OVER (ORDER BY v, w)` above it is wrong.
2. The input really is in order. With one build row the join keeps it, but
the inserted sort by `v` spills, and merging the spill files reorders rows that
tie on `v`. Without the spill the same query matches Spark.
### Steps to reproduce
AQE off, `spark.sql.autoBroadcastJoinThreshold=-1`, one input partition:
```scala
val cached = spark.range(0, 4000, 1, 1)
.selectExpr("id % 4 AS v", "id AS w")
.sortWithinPartitions("v", "w")
.cache()
val build = spark.range(2).selectExpr("id AS z")
cached.join(broadcast(build), expr("w % 2 <= z"))
.select("v", "w")
.selectExpr("v", "w",
"sum(w) OVER (ORDER BY v) AS s",
"row_number() OVER (ORDER BY v, w) AS r")
```
Both windows stay in Spark. The plan has `Sort [v ASC NULLS FIRST]` under
the lower window and no sort under the upper one, and `r` differs from Spark:
within `v = 0`, `w = 0` comes after `w = 3996`. For the second case, use
`spark.range(1)` as the build side, 200000 rows, and
`spark.shuffle.spill.numElementsForceSpillThreshold=10000`.
### Expected behavior
The rows match Spark. The inserted sort should restore the whole order the
input reported (the part that covers the window's required ordering), not only
the window's keys, so the operators Spark planned above it still see that order.
### Additional context
#6785 sorts a native window's input the same way and has the same gap; its
fix restores the full reported order natively. This issue covers the rule for
windows that run in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
