andygrove opened a new issue, #24145: URL: https://github.com/apache/datafusion/issues/24145
### Describe the bug Under `datafusion.execution.enable_ansi_mode = true`, `datafusion-spark`'s `abs` correctly raises on integral overflow, but the message is one DataFusion invents rather than Spark's: ``` DataFusion error: Arrow error: Compute error: Int32 overflow on abs(-2147483648) ``` Spark 4.2.0 raises: ``` [ARITHMETIC_OVERFLOW] overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22003 ``` The condition detected is the same. Only the text differs. DataFusion does not model Spark's error classes or SQLSTATE values, so an exact reproduction is not possible, but the message body is expressible. This one matters more than a typical message mismatch because `abs.rs` is the in-repo precedent that new `datafusion-spark` ANSI implementations are pointed at. Its wording gets copied. ### Cross-version note The message text is version dependent, and 4.2.0 is the version that changed it: | Spark | Message for `abs(-2147483648)` | | ----- | --------------------------------------------- | | 3.5.8 | `[ARITHMETIC_OVERFLOW] integer overflow. ...` | | 4.0.4 | `[ARITHMETIC_OVERFLOW] integer overflow. ...` | | 4.1.3 | `[ARITHMETIC_OVERFLOW] integer overflow. ...` | | 4.2.0 | `[ARITHMETIC_OVERFLOW] overflow. ...` | Spark 4.2.0 added a canonicalization step to `ExecutionErrors.arithmeticOverflowError` that rewrites any message matching `\w+ overflow` to plain `overflow`, so the JDK strings `integer overflow` and `long overflow` and the `MathUtils` strings `byte overflow` and `short overflow` all collapse to the same text. The reason given in the source is JIT hot-throw behavior, [JDK-8367990](https://bugs.openjdk.org/browse/JDK-8367990). Before 4.2.0 the width appeared in the message, so DataFusion's current text is closer in spirit to those versions than to 4.2.0, though equal to none of them. DataFusion has no mechanism for version-specific expectations, tracked by https://github.com/apache/datafusion/issues/23887. ### To Reproduce ```sql set datafusion.execution.enable_ansi_mode = true; select abs((-2147483648)::INT); -- DataFusion error: Arrow error: Compute error: Int32 overflow on abs(-2147483648) ``` The array path produces a second spelling, because the macro behind it stringifies the Arrow array type rather than the scalar type: ```sql select abs(a) FROM (VALUES (-2147483647::INT), ((-2147483648)::INT)) AS t(a); -- DataFusion error: Arrow error: Compute error: Int32Array overflow on abs(-2147483648) ``` Both spellings are asserted today in `datafusion/sqllogictest/test_files/spark/math/abs.slt`, four of each. Spark's message does not vary by input shape. ### Expected behavior A message matching Spark 4.2.0's `ARITHMETIC_OVERFLOW` text, and the same message on the scalar and array paths. ### Scope This is the `abs` half of the repository-wide question raised in https://github.com/apache/datafusion/issues/23897. That issue notes that changing `pmod` alone would leave the crate less consistent while `abs` keeps its own wording. The two should be decided together. The two paths in `abs` are not equally easy, and the array one is the reason this is worth its own issue rather than a drive-by fix. - **Scalar path.** The message is built by `scalar_compute_op!` in `datafusion/spark/src/function/math/abs.rs`. It is local to the spark crate and is two `format!` calls. - **Array path.** The message is built by `make_try_abs_function!` in `datafusion/functions/src/math/abs.rs`, which `datafusion-spark` imports and reuses. Core DataFusion's own `abs` raises through the same macro, so the text cannot be changed there without changing the error message core DataFusion users see. Matching Spark on the array path means `datafusion-spark` needs its own kernel rather than borrowing core's. That second point generalizes past `abs`: wherever a spark function reuses a core macro or kernel, Spark-specific error text is not available without first un-sharing the code. ### Relevant code - `datafusion/spark/src/function/math/abs.rs` - `datafusion/functions/src/math/abs.rs`, `make_try_abs_function!` - `datafusion/sqllogictest/test_files/spark/math/abs.slt` Surfaced by the audit-datafusion-spark-expression skill. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
