comphead opened a new issue, #26334: URL: https://github.com/apache/datafusion/issues/26334
### Describe the bug Spark's `Pow` is `java.lang.StrictMath.pow` ([mathExpressions.scala#L1254-L1260](https://github.com/apache/spark/blob/v4.0.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L1254-L1260)). `SparkPow` uses Rust's `f64::powf` and adds a special case that returns `+Infinity` when the base is zero and the exponent is negative ([pow.rs#L100-L108](https://github.com/apache/datafusion/blob/9dec77584ea39d93aa01bba7f880b318b37d2c24/datafusion/spark/src/function/math/pow.rs#L100-L108) and [pow.rs#L126-L140](https://github.com/apache/datafusion/blob/9dec77584ea39d93aa01bba7f880b318b37d2c24/datafusion/spark/src/function/math/pow.rs#L126-L140)). The comment on the second one says "Spark: 0^negative = +Infinity (covers both 0.0 and -0.0)", which does not match Spark. Java and C99 also differ for a base of `1` with a NaN exponent and for a base of `1` or `-1` with an infinite exponent. Java returns NaN and C99 returns `1`. ### To Reproduce Spark 4.0.0: ```sql SELECT pow(CAST('-0.0' AS DOUBLE), -1.0D) -- -Infinity SELECT pow(CAST('-0.0' AS DOUBLE), -2.0D) -- Infinity SELECT pow(CAST('-0.0' AS DOUBLE), -3.0D) -- -Infinity SELECT pow(1.0D, CAST('NaN' AS DOUBLE)) -- NaN SELECT pow(1.0D, CAST('Infinity' AS DOUBLE)) -- NaN SELECT pow(-1.0D, CAST('Infinity' AS DOUBLE)) -- NaN ``` DataFusion, from the existing tests: - [pow.slt#L100](https://github.com/apache/datafusion/blob/9dec77584ea39d93aa01bba7f880b318b37d2c24/datafusion/sqllogictest/test_files/spark/math/pow.slt#L100) expects `Infinity` for `pow(-0.0, -1.0)`. - [pow.slt#L74](https://github.com/apache/datafusion/blob/9dec77584ea39d93aa01bba7f880b318b37d2c24/datafusion/sqllogictest/test_files/spark/math/pow.slt#L74) expects `1` for `pow(1.0, NaN)`. The other cases were not run on DataFusion. By reading the special case, `pow(-0.0, -3.0)` returns `+Infinity`. The infinite exponents go through `powf`, which follows C99 and returns `1`. ### Expected behavior Match `StrictMath.pow` for these special cases. A base of `-0.0` with a negative odd integer exponent gives `-Infinity`, and a base of `1` with a NaN exponent, or a base of `1` or `-1` with an infinite exponent, gives NaN. The `base == 0.0` special case looks unnecessary, because C99 already gives `+Infinity` for `0.0` and `-Infinity` for `-0.0` with an odd negative exponent (not run here). The expectations at `pow.slt#L74` and `#L100` need to change. Last-digit differences between the platform `pow` and `StrictMath.pow` for finite inputs are out of scope here. ### Additional context Found by running the queries in `datafusion/sqllogictest/test_files/spark` on a local Spark and comparing the results with the expected values in the files, which is the kind of check proposed in #17045. Spark results come from Apple's build of Spark 4.0.0 (`4.0.0.65-apple`) with a UTC session and `spark.sql.ansi.enabled=false` unless stated. The Spark source links point at the `v4.0.0` tag. DataFusion results are the sqllogictest expectations at 9dec77584, which CI enforces. Anything marked as not run was derived from reading the code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
