0lai0 commented on issue #5022: URL: https://github.com/apache/datafusion-comet/issues/5022#issuecomment-5079484025
I looked into how Spark computes the RANGE frame bound, and I think it explains the divergence. Spark builds the bound expression based on the ORDER BY column's data type, in [`WindowEvaluatorFactoryBase.createBoundOrdering`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactoryBase.scala): 1. DATE → [`DateAdd`](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala), which is a plain `Int + Int` (no `addExact`, no ANSI check), so it silently wraps on overflow. 2. DECIMAL → [`DecimalAddNoOverflowCheck`](https://github.com/apache/spark/blob/cce435599ed332cd5b785d49b26d4e7dde631d8c/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/decimalExpressions.scala#L204), which deliberately skips the overflow check, so it neither returns NULL nor throws. So Spark's bound compute is intentionally overflow-tolerant. The bound value is only used as a comparison key (not stored as a result), so wrapping / skipping the check is safe and keeps the frame boundary consistent with the actual row ordering. DataFusion computes the frame bound in [`window_state.rs`](https://github.com/apache/datafusion/blob/f1ab86dad406a189e43ac19d24965b3fbf9dbba9/datafusion/expr/src/window_state.rs#L390-L443) using `add_checked` / `sub_checked`. On overflow these return `Err`, and the bound collapses to the partition edge, that's where the result diverges from Spark. So making the DATE and DECIMAL cases actually compatible would mean changing this overflow behavior on the DataFusion side. Note the collapse is a deliberate choice, so this would likely need to be an opt-in behavior rather than changing the default for all DataFusion users. Is this the right direction? If so, I'm happy to do the upstream DataFusion work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
