ShayanGho opened a new issue, #24941:
URL: https://github.com/apache/datafusion/issues/24941
### Describe the bug
Spark's `Factorial` extends `ImplicitCastInputTypes` with `inputTypes =
Seq(IntegerType)`. Under
ANSI mode the implicit cast rule (`AnsiTypeCoercion.implicitCast` ->
`Cast.canANSIStoreAssign`)
allows any numeric type to be cast to INT and any string to be cast to an
atomic type; the legacy
rule allows the same. So Spark accepts DECIMAL, FLOAT, DOUBLE and STRING
arguments. DataFusion
rejects all of them at planning time.
### To Reproduce
Spark SQL, `pyspark==4.2.0`, both ANSI settings unless noted:
```sql
SELECT factorial(CAST(5 AS DECIMAL(10,0))); -- 120
SELECT factorial(5.0); -- 120
SELECT factorial(CAST(5.7 AS DOUBLE)); -- 120 (fraction truncated)
SELECT factorial(CAST(20.9 AS FLOAT)); -- 2432902008176640000
SELECT factorial('5'); -- 120
SELECT factorial(' 5 '); -- 120
SELECT factorial('5.0'); -- ANSI: [CAST_INVALID_INPUT]
error; non-ANSI: 120
SELECT factorial('abc'); -- ANSI: [CAST_INVALID_INPUT]
error; non-ANSI: NULL
SELECT factorial(''); -- ANSI: [CAST_INVALID_INPUT]
error; non-ANSI: NULL
SELECT factorial(CAST(99999999999 AS DECIMAL(20,0))); -- ANSI:
[CAST_OVERFLOW] error; non-ANSI: NULL
```
DataFusion (`datafusion-cli --spark`): on `main` every one of these fails
with
`Failed to coerce arguments to satisfy a call to 'factorial' function:
coercion from Utf8 to the
signature Exact(Int32) failed` (and likewise for Float64 / Decimal128). With
the integer-width fix
for #24940 applied they still fail, now with
`Function 'factorial' requires Int32, but received String (DataType: Utf8)`
(and likewise
`Float64` and `Decimal(10, 0)`), so this divergence survives that fix.
### Expected behavior
Match Spark: accept the types Spark accepts and apply Spark's cast
semantics, including the
ANSI-dependent behaviour for malformed strings and overflow. (DataFusion
currently fails at the
cast in both modes, since the function does not consult
`datafusion.execution.enable_ansi_mode`.)
### Additional context
Not fixed together with the integer-width fix because it needs a decision,
not a mechanical change:
1. #23889 is the same `ImplicitCastInputTypes` gap for `next_day`; its
discussion raises the general
question of whether `datafusion-spark` should model Spark's implicit
casts, and a general
approach should probably cover both.
2. The string and overflow cases are ANSI dependent in Spark, so any
widening has to route through
a Spark-compatible cast rather than `arrow::compute::cast`, which errors
in both modes.
Relevant code: `datafusion/spark/src/function/math/factorial.rs`,
`SparkFactorial::new`.
Surfaced by the `audit-datafusion-spark-expression` skill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
