ShayanGho opened a new issue, #24950: URL: https://github.com/apache/datafusion/issues/24950
### Describe the bug With the integer-width fix in #24943 applied, `datafusion-spark` accepts BIGINT arguments to `factorial` by implicitly casting them to INT. That cast raises an Arrow overflow error regardless of `datafusion.execution.enable_ansi_mode`. Spark 4.2.0 raises `CAST_OVERFLOW` in ANSI mode. In non-ANSI mode, its BIGINT-to-INT cast wraps to a signed 32-bit integer, and `factorial` evaluates that wrapped value. The result can be NULL or a valid factorial. This is a follow-up to #24940 / #24943. Reproduced against PR #24943 at `ff3968e66e9cef4dee8c034b568386505a6590fb`. The PR is still open as of 2026-09-04. Without that fix, BIGINT arguments are rejected during type coercion. ### To Reproduce DataFusion with Spark functions registered and #24943 applied: ```sql SET datafusion.execution.enable_ansi_mode = false; SELECT factorial(CAST(5000000000 AS BIGINT)); -- Arrow error: Cast error: Can't cast value 5000000000 to type Int32 SELECT factorial(CAST(4294967301 AS BIGINT)); -- Arrow error: Cast error: Can't cast value 4294967301 to type Int32 ``` Running either SELECT with `datafusion.execution.enable_ansi_mode = true` also raises the corresponding Arrow cast error. For these constant expressions, the error is reported by the `simplify_expressions` optimizer rule. Spark SQL, verified with a live Spark 4.2.0 session (`pyspark==4.2.0`): ```sql SET spark.sql.ansi.enabled = false; SELECT factorial(CAST(5000000000 AS BIGINT)); -- NULL SELECT factorial(CAST(4294967301 AS BIGINT)); -- 120 SET spark.sql.ansi.enabled = true; SELECT factorial(CAST(5000000000 AS BIGINT)); -- [CAST_OVERFLOW] SELECT factorial(CAST(4294967301 AS BIGINT)); -- [CAST_OVERFLOW] ``` In non-ANSI mode, `5000000000` wraps to `705032704`, which is outside `factorial`'s supported range of 0 through 20. `4294967301` wraps to `5`, so its factorial is `120`. Returning NULL for every overflowing BIGINT would therefore still diverge from Spark. ### Expected behavior Match Spark's implicit BIGINT-to-INT cast semantics: with `datafusion.execution.enable_ansi_mode = false`, wrap to signed INT and evaluate `factorial` on that value. With ANSI mode enabled, raise an overflow error. ### Additional context `SparkFactorial::new` in `datafusion/spark/src/function/math/factorial.rs` uses `Signature::coercible` after #24943. Type coercion inserts the cast before the function runs. Reading the ANSI setting inside `factorial` alone cannot change a cast that has already failed. The implementation needs to account for Spark's cast semantics at that earlier stage. The latest #24943 diff has no out-of-range BIGINT `query error` case in `datafusion/sqllogictest/test_files/spark/math/factorial.slt`. Regression coverage should include both ANSI modes and an overflowing value that wraps into 0..20, for scalar and column inputs. Related to #24941, which covers additional accepted input types and their ANSI-dependent casts. Part of #23929. Surfaced by the audit-datafusion-spark-expression skill. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
