comphead commented on PR #7256: URL: https://github.com/apache/datafusion-comet/pull/7256#issuecomment-6664147045
Thanks @andygrove for the approval and the AQE point, and thanks @sunchao and @rich7420 for the earlier rounds. 0f730d5b5c runs `a window that stays in Spark over a nested loop join sorts the join's rows again` with AQE off and on. With AQE on it checks that the plan is adaptive, strips it with `stripAQEPlan`, and checks that every window reads from a sort. Under AQE, `CometExecRule` runs again for the stage, so the added sort over the native join can become a `CometSortExec`, and the integer window above it can then run natively. That is still correct, so the check accepts a Spark or Comet window over a Spark or Comet sort. I have not run it locally, so CI gives the first verdict. The same commit widens the new pass from `WindowExec` to `WindowExecBase`. `WindowInPandasExec` (Spark 3.4 to 4.0) and `ArrowWindowPythonExec` (4.1) take their required ordering from the same trait, and Comet never converts them, so a Python window over a doubted order now gets the same sort. No test covers the Python case, since it needs a Python worker. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
