JkSelf commented on code in PR #10667:
URL:
https://github.com/apache/incubator-gluten/pull/10667#discussion_r2339298412
##########
backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala:
##########
@@ -524,6 +524,74 @@ class MiscOperatorSuite extends
VeloxWholeStageTransformerSuite with AdaptiveSpa
}
}
+ test("optimize sort window to streaming window") {
+ withSQLConf(VeloxConfig.COLUMNAR_VELOX_WINDOW_TYPE.key -> "sort") {
Review Comment:
@zjuwangg Just curious why you choose to use sort window not straeming
window in you env? Because the sort operator already before the window
opeartor, and then we can directly use the StreamigWindow in velox side.
I plan to remove the SortWindow type and switch to using the StreamingWindow
directly, since Spark already inserts a sort operator before the window
operator. This means we no longer need to apply the `EliminateLocalSort `rule
to remove the sort, and then use `EnsureRequirements ` to re-insert it when the
window type is streaming.
Initially, we provided both window types because we believed the SortWindow
could support spilling and help fix some OOM scenarios. However, in Velox,
spilling in SortWindow is only supported during the addInput stage, not during
getOutput, so it doesn't effectively address most OOM cases. As a result, we
developed `PartitionStreamingWindow `and `RowsStreamingWindow `to better handle
these situations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
