andygrove opened a new issue, #376:
URL: https://github.com/apache/datafusion-spark/issues/376

   `nanvl` gets two of its Spark behaviors only from its `simplify` rewrite to 
`CASE WHEN isnan(expr1) THEN expr2 ELSE expr1 END`. When `simplify` doesn't 
run, `invoke_with_args` handles the call. That happens with the sqllogictest 
`evaluate` mode (#316), and with a caller such as Comet that builds the 
physical expression directly. In that path:
   
   - **expr2 is evaluated for every row.** With ANSI mode on, `SELECT nanvl(c2, 
1 / 0) FROM (VALUES (0), (1)) t(c2)` raises `Divide by zero`. Spark returns `0, 
1`, because it only evaluates `1 / 0` on rows where `c2` is NaN.
   - **String arguments are rejected.** `SELECT nanvl('123.5', 1.0::double)` 
fails with `nanvl expects Float32 or Float64 arguments at this point, got 
Utf8`, where Spark returns `123.5`. The ANSI-aware cast (NULL with ANSI off, an 
error with it on) lives only in `simplify`.
   
   The string case can be fixed in `invoke_with_args` by casting a string 
argument to Float64, with `safe` set from `enable_ansi_mode`. A scalar UDF 
always receives its arguments already evaluated, so a lazy expr2 can't come 
from `invoke_with_args` alone.
   
   The seven records in `conditional/nanvl.slt` that need `simplify` are marked 
`skipif evaluate` until this is fixed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to