Spenserrrr opened a new pull request, #58319:
URL: https://github.com/apache/spark/pull/58319
### What changes were proposed in this pull request?
`_fmod_func` cast both operands to double before taking the remainder,
including on the branch that is reached only when neither operand is floating.
Integral operands now take the remainder as longs, and only the result is cast
back to double. The floating branch is unchanged.
### Why are the changes needed?
A double carries 53 bits of mantissa, so an integral operand above 2**53 was
rounded before the remainder was ever taken:
```python
>>> pdf = pd.DataFrame({"x1": [9007199254740993], "x2": [2]})
>>> np.fmod(pdf.x1, pdf.x2)[0]
# pandas
1
>>> np.fmod(ps.from_pandas(pdf).x1, ps.from_pandas(pdf).x2).to_pandas()[0]
# before
0.0
```
Every epoch-nanosecond value is past that boundary, since 2**53 nanoseconds
is about 104 days. The same cast also gave an integral remainder a sign of zero
it cannot have, because an IEEE remainder carries the dividend's sign: `-64 %
2` returned `-0.0` where NumPy returns `0`. The `pandas_udf` that SPARK-58582
replaced called NumPy directly, so this restores its results.
The floating branch is faithful as it stands: NumPy also promotes a mixed
integral/floating pair to float64 and loses the same bits. A remainder above
2**53 is still rounded by the double return type, which requires a divisor
above 2**53; the result is exact for any divisor up to it.
### Does this PR introduce _any_ user-facing change?
No change relative to a released version, since SPARK-58582 is not in a
release branch. Within unreleased master, `np.fmod` on integral columns returns
the exact remainder for operands above 2**53, and an integral remainder of zero
is `0.0` rather than `-0.0`.
### How was this patch tested?
New rows in `test_np_fmod`: an integral frame above 2**53 compared exactly
rather than approximately, since a relative tolerance accepts an off-by-one at
those magnitudes, plus an explicit `np.signbit` assertion because `almost=True`
treats `-0.0` and `0.0` as equal. Both were confirmed to fail without the
change by reverting only `numpy_compat.py`.
Also ran the classic and Spark Connect parity suites for
`test_numpy_compat`, and checked parity with pandas by hand for
int8/int16/int32/int64, nullable `Int64` with nulls and zero divisors, mixed
integral/floating pairs, `-2**63 % -1`, and decimal columns, which keep taking
the double path.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
