dwsmith1983 opened a new issue, #7671: URL: https://github.com/apache/datafusion-comet/issues/7671
### Describe the bug With ANSI off, `/`, `div` and `%` return wrong results when the divisor is a stateful nondeterministic expression, and so do `ln`, `log10` and `log2` when their argument is one. It is the same problem as #5781: Comet evaluates the operand twice, once in the guard that turns a zero divisor or a non-positive argument into NULL and once as the value, and the two copies see different rows. - `/` and `div`: `nullIfWhenPrimitive` in `spark/src/main/scala/org/apache/comet/serde/arithmetic.scala` wraps the divisor as `If(EqualTo(x, 0), null, x)`. - `%`: `null_if_zero_primitive` in `native/spark-expr/src/math_funcs/modulo_expr.rs` builds the same `IfExpr` natively. - `ln`, `log10`, `log2`: `nullIfNegative` in `spark/src/main/scala/org/apache/comet/serde/math.scala` wraps the argument as `If(LessThanOrEqual(x, 0), null, x)`. SQL `log(base, x)` goes through `CometLogarithm` and is not affected. None of these checks whether the operand is deterministic, and all of them run natively. ### Steps to reproduce ```sql SET spark.sql.ansi.enabled=false; CREATE TABLE t (id INT, v INT) USING parquet; -- one file, so monotonically_increasing_id() is the row index 0..15 INSERT INTO t SELECT CAST(id AS INT), CAST(id + 1 AS INT) FROM range(0, 16, 1, 1); SELECT id, v / IF(monotonically_increasing_id() % 2 = 0, 2, 0) FROM t; SELECT id, v div IF(monotonically_increasing_id() % 2 = 0, 2, 0) FROM t; SELECT id, v % IF(monotonically_increasing_id() % 2 = 0, 3, 0) FROM t; SELECT id, isnan(ln(IF(monotonically_increasing_id() % 2 = 0, CAST(v AS DOUBLE), CAST(-1 AS DOUBLE)))) FROM t; ``` On main (ba16e1a53), Spark 4.1: - `/`: Comet returns `Infinity` for ids 2, 6, 10 and 14, where Spark returns 1.5, 3.5, 5.5 and 7.5. - `div`: Comet returns 0 for the same ids, where Spark returns 1, 3, 5 and 7. - `%`: Comet fails with `CometNativeException: Divide by zero error`, where Spark returns NULL on the odd rows. - `ln`, `log10`, `log2`: Comet returns NaN for ids 2, 6, 10 and 14, where Spark returns a number. ### Expected behavior The same results as Spark. As on `branch-1.2` in #7670, these could fall back to Spark for a nondeterministic operand, or go through the codegen dispatcher as #5867 does for `size`, `arrays_zip` and `map_from_arrays`. ### Additional context Found while preparing #7670, the `branch-1.2` change for #5781. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
