andygrove opened a new issue, #5065:
URL: https://github.com/apache/datafusion-comet/issues/5065
### Describe the bug
Under ANSI mode, `IntegralDivide` (the `div` operator) silently returns a
wrong value for `Long.MinValue div -1` instead of throwing
`ARITHMETIC_OVERFLOW`.
Spark 4.1 (`arithmetic.scala`) sets `checkDivideOverflow = true` for
`LongType` when `failOnError` is set, and throws
`QueryExecutionErrors.overflowInIntegralDivideError` ("Overflow in integral
divide. Use `try_divide` to tolerate divisor being 0 and return NULL instead.")
when `input1 == Long.MinValue && input2 == -1`.
Comet's serde
(`spark/src/main/scala/org/apache/comet/serde/arithmetic.scala`,
`CometIntegralDivide`) casts both operands to `DECIMAL(19,0)` and performs a
decimal divide. The result `9223372036854775808` fits in precision 19, so the
`CheckOverflow` wrapper passes. The final cast back to `LongType` is hard-coded
to legacy eval mode:
```scala
// arithmetic.scala:348
CometCast.castToProto(expr, None, LongType, childExpr.get,
CometEvalMode.LEGACY)
```
The legacy decimal-to-int64 path truncates
(`native/spark-expr/src/conversion_funcs/numeric.rs`,
`cast_decimal_to_int32_up!` non-ANSI arm), producing `-9223372036854775808`
with no error.
`SparkError::IntegralDivideOverflow` already exists in
`native/common/src/error.rs` and is wired through `ShimSparkErrorConverter`,
but no native kernel ever constructs it.
### Steps to reproduce
```sql
SET spark.sql.ansi.enabled=true;
SELECT CAST(-9223372036854775808 AS LONG) div CAST(-1 AS LONG);
```
Spark: throws `SparkArithmeticException` with error class
`ARITHMETIC_OVERFLOW` ("Overflow in integral divide").
Comet: returns `-9223372036854775808`.
### Expected behavior
Comet should throw the same `ARITHMETIC_OVERFLOW` error as Spark under ANSI
mode (and return the wrapped value under legacy mode, which it already does).
### Additional context
Found by an ANSI-mode audit of all native expressions against Spark 4.1.1.
Divide-by-zero handling in `IntegralDivide` is correct; only the `Long.MinValue
div -1` overflow case is affected. There is no existing test for this input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
