pavan51 commented on code in PR #57346:
URL: https://github.com/apache/spark/pull/57346#discussion_r3626368097
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4465,6 +4465,15 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val WINDOW_MONOTONIC_DEQUE_ENABLED =
+ buildConf("spark.sql.window.monotonicDeque.enabled")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .doc("Use monotonic deques for sliding min/max aggregate window frames.
" +
+ "This provides O(N) complexity instead of O(N * W) or O(N log W).")
+ .version("5.0.0")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
Unlike the segment tree, the monotonic deque has O(1) amortized cost and
does not suffer from tree-building overhead on small windows. However, keeping
it disabled by default for its initial release as a safer approach to mitigate
risk. I've updated the default to false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
