comphead opened a new pull request, #7140: URL: https://github.com/apache/datafusion-comet/pull/7140
## Which issue does this PR close? No issue filed. Related: #7139 adds window benchmarks, from the same investigation. ## Rationale for this change `CometWindowExec` declares only `dataSize` and `numPartitions`, and nothing sets either, so the time spent in the native window never reaches the Spark UI or the event log. In a stage that ran two windows between two sorts, the reported operator time covered about a quarter of the task time, and the windows turned out to be most of the rest. ## What changes are included in this PR? - `CometWindowExec` reports `CometMetricNode.baselineMetrics`, the output rows and the time spent in the operator, which `BoundedWindowAggExec` and `WindowAggExec` both record. - When a window function returns a type Spark does not expect, such as `row_number`, the planner wraps the window in a projection that casts it. The window is now an additional native plan of that operator, so its time is reported together with the projection's instead of being left out. The output rows still come from the projection. ## How are these changes tested? A new test in `CometWindowExecSuite` runs one window that needs the cast (`row_number`) and one that does not (`max`), and checks that both report 1000 output rows and a nonzero time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
