mbutrovich commented on code in PR #333:
URL: https://github.com/apache/datafusion-spark/pull/333#discussion_r4602667036
##########
crates/sqllogictest/scripts/pyspark_slt_runner.py:
##########
@@ -925,20 +925,20 @@ def _shortest_f32(value: float) -> str:
return repr(value)
-def _round_float(value: float, single: bool, digits: int) -> str:
- """DataFusion's float_to_str: round, drop trailing zeros, no exponent."""
+def _plain_float(value: float, single: bool) -> str:
+ """The test engine's float text: shortest round-trip digits, no exponent.
+
+ This is Rust's `Display` for floats: a whole number has no `.0`, and `-0.0`
+ keeps its sign. NaN ignores its sign.
+ """
Review Comment:
The engine now keeps the sign of `-0.0`, but `numeric_difference` still
treats `-0` and `0` as the same number
([L1184](https://github.com/apache/datafusion-spark/blob/8e7bb9a73086393511e662b99ebf29f0be7363a4/crates/sqllogictest/scripts/pyspark_slt_runner.py#L1184)
compares `float(e) == float(a)`). It reports them as "same numbers, printed
with a different scale", which hides a real difference. Running the runner over
every file on this branch, `math/pmod.slt:231` lands in that bucket:
```
SELECT pmod(-5.0::DOUBLE, 2.5::DOUBLE) as pmod_float_6;
DataFusion (expected):
0
Spark 4.2.0:
-0
```
Could `numeric_difference` count a sign difference on zero as a real
difference? For example, at L1184:
```python
if e_float == a_float and math.copysign(1, e_float) ==
math.copysign(1, a_float):
continue
```
The `pmod` result itself is a bug on `main`. `spark_pmod` detects a negative
remainder with Arrow's `lt` ([modulus.rs
L157](https://github.com/apache/datafusion-spark/blob/125600164aa825e2fe805887dfdb384d4182f2a8/crates/spark/src/function/math/modulus.rs#L157)),
and Arrow orders floats so that `-0.0 < 0.0`. Spark's `if (remainder < 0)` is
false for `-0.0` ([arithmetic.scala
L1147-L1155](https://github.com/apache/spark/blob/32f7299601108917fb01920a54e084595b7b3bf8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L1147-L1155)),
so Spark returns `-0.0`, and `spark_pmod` adds the divisor and returns `0`. I
didn't find an issue for it. With this PR, `pmod.slt:231` asserts the wrong
sign. Could we record Spark's `-0` there as a commented-out query that links an
issue, until `pmod` is fixed?
##########
crates/sqllogictest/test_files/math/pmod.slt:
##########
@@ -429,4 +429,4 @@ SELECT pmod(10.5, 3.0) as pmod_scalar_3;
query R
SELECT pmod(-7.2, 3.0) as pmod_scalar_4;
----
-1.8
+1.7999999999999998
Review Comment:
Spark reads the untyped `-7.2` and `3.0` as `DECIMAL`, so it returns `1.8`
for this query, and the runner reports it as "results differ only in
floating-point digits". Before this PR the rounding hid that, and now the file
asserts DataFusion's DOUBLE result. Could we add a comment saying this follows
DataFusion's literal typing, with a link to #97? The `::float8` query at lines
211-214 already checks the DOUBLE result, and Spark agrees with that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]