andygrove opened a new pull request, #333:
URL: https://github.com/apache/datafusion-spark/pull/333

   ## Which issue does this PR close?
   
   None. This is groundwork for the math functions, and logarithms (stacked on 
this PR) need it to assert Spark's exact results.
   
   ## Rationale for this change
   
   DataFusion's sqllogictest engine rounds a top-level DOUBLE to 15 decimal 
places (12 for FLOAT) and prints `-0.0` as `0`. That hides exactly the 
last-digit and signed-zero differences that matter when emulating Spark's 
`java.lang.StrictMath`. For example, `log(3, 81)` is `4.000000000000001` in 
Spark and `4` with Rust's `f64::ln`, and both print as `4` today. The 
workaround, `SELECT f(x) = <exact>::double` returning a boolean, only says that 
a value is wrong, not what it is.
   
   ## What changes are included in this PR?
   
   - `crates/sqllogictest/tests/engine/mod.rs`: a small `AsyncDB` engine that 
prints DOUBLE and FLOAT values (including dictionary-encoded ones) as the 
shortest text that round-trips, with no exponent, keeping the sign of `-0.0`. 
Every other type is still formatted by DataFusion's `convert_batches`, and 
column types still come from the real schema, so nothing else changes.
   - `--complete` flag: `cargo test -p datafusion-spark-sqllogictest -- 
<filter> --complete` rewrites expected results with the actual ones.
   - PySpark runner: formats doubles the same way, and its unit tests and 
README are updated.
   - Six existing files updated: `atan2`, `csc`, `expm1`, `pmod`, `pow` and 
`sec`. Each change only adds digits (e.g. `pmod(-7.2, 3.0)` is 
`1.7999999999999998`, not `1.8`), apart from the signed-zero cases below.
   - **`atan2` fix.** The new output exposed a bug that rounding had hidden. 
Spark evaluates `Math.atan2(y + 0.0, x + 0.0)`, which turns `-0.0` into `0.0`. 
So `atan2(-0.0, -1.0)` is π, not −π, and `atan2(-0.0, 1.0)` is `0`, not `-0`. 
The existing test and the docs asserted the Java `Math.atan2` values. Both are 
fixed and a test for `atan2(0.0, -0.0)` is added.
   - `pow(-0.0, 1.0)` now shows `-0`, which is correct: Spark returns `-0.0`.
   
   The Spark values were confirmed by evaluating Spark 4.2.0's catalyst 
expressions (`Atan2`, `Pow`) directly from the `spark-catalyst_2.13-4.2.0` jar, 
through both the interpreted and generated-code paths.
   
   ## Are these changes tested?
   
   - `cargo fmt --all -- --check`: clean
   - `cargo clippy --all-targets --all-features -- -D warnings`: clean
   - `cargo test -p datafusion-spark`: 335 passed
   - `cargo test -p datafusion-spark-sqllogictest`: 249 of 249 files passed
   - `python3 -m unittest test_pyspark_slt_runner`: passed
   
   ## Are there any user-facing changes?
   
   `atan2` now treats `-0.0` as `0.0`, as Spark does. The rest only affects 
tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to