andygrove opened a new issue, #5022:
URL: https://github.com/apache/datafusion-comet/issues/5022
## Summary
Native `RANGE` window frames with an explicit `PRECEDING`/`FOLLOWING` offset
diverge from Spark when the boundary arithmetic `current +/- offset` overflows.
This affects both `DATE` ORDER BY (#4974) and `DECIMAL` ORDER BY (#4987).
## Behavior
On boundary overflow:
- **Spark** computes the boundary through normal expression evaluation and
produces a concrete value.
- `DATE`: via `DateAdd`, which is plain `Int + Int` with silent wraparound
(not `Math.addExact`, not ANSI-gated). It never throws and yields a wrapped
date.
- `DECIMAL`: via normal decimal arithmetic, returning `NULL` under
non-ANSI and throwing under ANSI.
- **Comet/DataFusion** computes the boundary via `ScalarValue::add`/`sub`.
For both `Date32 + IntervalDayTime` and `Decimal128` near max precision, the
arithmetic returns `Err` on overflow, and the window frame collapses the bound
to the partition edge.
So the two engines produce different frames (and under ANSI, Spark throws
where Comet does not).
## Practical likelihood
- `DECIMAL`: rare but plausible, needs values near `Decimal128` max
precision.
- `DATE`: extremely remote, overflowing `Date32` needs an offset of roughly
2.1 billion days (~5.88 million years) from a realistic date.
## Options
1. Make the native boundary overflow match Spark (best): wrap for `DATE`,
`NULL`/throw for `DECIMAL` depending on ANSI. This likely means upstream
DataFusion work.
2. Mark the affected cases `Incompatible(Some(...))` and gate the native
path behind `spark.comet.operator.<name>.allowIncompatible`, falling back to
Spark by default. Follows the `CometDataWritingCommand` precedent.
Whichever route, a test at the type boundary should pin the chosen behavior
rather than avoiding the edge.
## Related
- #4974 (DATE, currently runs native by default, undocumented)
- #4987 (DECIMAL, currently runs native by default, documented as a known
difference)
- Both are part of #4834.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
