comphead commented on PR #7256: URL: https://github.com/apache/datafusion-comet/pull/7256#issuecomment-6612993769
Thanks @andygrove. 3ff09398ec fixes the unpartitioned case instead of documenting it, so no issue is needed to track it. 1. The collation fallback no longer depends on trust. A window with a collated key always falls back to Spark. When `mayReportFalseOrder` doubts the order a window's input reports, `CometExecRule` sorts the input of a window that stays in Spark, so Spark's window sees the rows in order and applies the collation. The comments on the guard and on the new pass say so. The trust check now decides only what the native window can rely on. A window with `ORDER BY` but no `PARTITION BY` over a doubted order falls back too, since the native window checks no order there. 2. `compatibility/operators.md` now lists collated keys among the window's fallbacks. It said so only for `WindowGroupLimitExec` before, and with this change it holds for the window in every case. 3. `a window that stays in Spark over a nested loop join sorts the join's rows again` runs sunchao's unpartitioned fixture, his partitioned one from the review before, and the same unpartitioned shape with an integer key. It compares the rows with Spark, and it fails without the new pass. 4. On `rust-test`: since #7311 the upstream run skips it for a pull request from a fork, and the fork run behind `CI (fork)` runs it. It passed there on `1bb942a33a`. The nested loop join's own order (#6898) is unchanged. A partitioned native window above it still fails with the ordering error, and #6785 fixes that by keeping such a join in Spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
