andygrove opened a new pull request, #333: URL: https://github.com/apache/datafusion-spark/pull/333
## Which issue does this PR close? None. This is groundwork for the math functions, and logarithms (stacked on this PR) need it to assert Spark's exact results. ## Rationale for this change DataFusion's sqllogictest engine rounds a top-level DOUBLE to 15 decimal places (12 for FLOAT) and prints `-0.0` as `0`. That hides exactly the last-digit and signed-zero differences that matter when emulating Spark's `java.lang.StrictMath`. For example, `log(3, 81)` is `4.000000000000001` in Spark and `4` with Rust's `f64::ln`, and both print as `4` today. The workaround, `SELECT f(x) = <exact>::double` returning a boolean, only says that a value is wrong, not what it is. ## What changes are included in this PR? - `crates/sqllogictest/tests/engine/mod.rs`: a small `AsyncDB` engine that prints DOUBLE and FLOAT values (including dictionary-encoded ones) as the shortest text that round-trips, with no exponent, keeping the sign of `-0.0`. Every other type is still formatted by DataFusion's `convert_batches`, and column types still come from the real schema, so nothing else changes. - `--complete` flag: `cargo test -p datafusion-spark-sqllogictest -- <filter> --complete` rewrites expected results with the actual ones. - PySpark runner: formats doubles the same way, and its unit tests and README are updated. - Six existing files updated: `atan2`, `csc`, `expm1`, `pmod`, `pow` and `sec`. Each change only adds digits (e.g. `pmod(-7.2, 3.0)` is `1.7999999999999998`, not `1.8`), apart from the signed-zero cases below. - **`atan2` fix.** The new output exposed a bug that rounding had hidden. Spark evaluates `Math.atan2(y + 0.0, x + 0.0)`, which turns `-0.0` into `0.0`. So `atan2(-0.0, -1.0)` is π, not −π, and `atan2(-0.0, 1.0)` is `0`, not `-0`. The existing test and the docs asserted the Java `Math.atan2` values. Both are fixed and a test for `atan2(0.0, -0.0)` is added. - `pow(-0.0, 1.0)` now shows `-0`, which is correct: Spark returns `-0.0`. The Spark values were confirmed by evaluating Spark 4.2.0's catalyst expressions (`Atan2`, `Pow`) directly from the `spark-catalyst_2.13-4.2.0` jar, through both the interpreted and generated-code paths. ## Are these changes tested? - `cargo fmt --all -- --check`: clean - `cargo clippy --all-targets --all-features -- -D warnings`: clean - `cargo test -p datafusion-spark`: 335 passed - `cargo test -p datafusion-spark-sqllogictest`: 249 of 249 files passed - `python3 -m unittest test_pyspark_slt_runner`: passed ## Are there any user-facing changes? `atan2` now treats `-0.0` as `0.0`, as Spark does. The rest only affects tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
