kokhlo opened a new issue, #58307:
URL: https://github.com/apache/spark/issues/58307
### Title
[PYTHON][PANDAS API ON SPARK] pandas-on-Spark floordiv returns incorrect
results for unrepresentable quotients and int64 above 2^53
### Description
`pyspark.pandas` floor division (`//`) produces incorrect results in two
scenarios:
1. **Floating-point operands where the quotient rounds across an integer
boundary**
Example: `ps.Series([1.0]) // 0.1` returns `10.0`, but pandas and NumPy
return `9.0`.
2. **Int64 operands above 2^53**
Example: `ps.Series([-9007199254740993]) // 2` returns
`-4503599627370496`, but pandas returns `-4503599627370497` (off by 1).
Both bugs stem from the use of `F.floor(col / divisor)` in
`python/pyspark/pandas/data_type_ops/num_ops.py`.
**Affected code paths** (4 instances):
- `IntegralOps.floordiv` (~line 516)
- `FractionalOps.floordiv` (~line 390)
- `IntegralOps.rfloordiv` (~line 556)
- `FractionalOps.rfloordiv` (~line 417)
---
### Root Cause
#### Float Case
`1.0 / 0.1` in IEEE-754 double is not exactly representable and rounds
**up** to exactly `10.0`.
Taking `floor(10.0)` gives `10`, but pandas/NumPy use a remainder-based
approach (as NumPy's `npy_divmod` does) to produce the correct `9`.
#### Int Case
Spark's `/` operator always divides as `double`. For operands above 2^53,
casting to double loses precision in the low bits before the division happens.
The quotient is then off by 1 when floored.
---
### Reproducer
```python
import pyspark.pandas as ps
import pandas as pd
# Case 1: 1.0 // 0.1
ps_series = ps.Series([1.0])
pd_series = pd.Series([1.0])
print("pyspark:", (ps_series // 0.1).values[0]) # 10.0 — WRONG
print("pandas: ", (pd_series // 0.1).values[0]) # 9.0 — CORRECT
# Case 2: -9007199254740993 // 2
ps_series_int = ps.Series([-9007199254740993])
pd_series_int = pd.Series([-9007199254740993])
print("pyspark:", (ps_series_int // 2).values[0]) # -4503599627370496 —
WRONG
print("pandas: ", (pd_series_int // 2).values[0]) # -4503599627370497 —
CORRECT
```
---
### Expected Fix
1. **For fractional types**: Use a remainder-based quotient (as NumPy does)
rather than `F.floor(col / divisor)`.
Example: `quotient = (col - (col % divisor)) / divisor`
(with sign correction when remainder is non-zero and
numerator/denominator have opposite signs).
2. **For integral types**: Compute the quotient in integer space before
casting to float, so int64 values above 2^53 are not prematurely rounded.
---
### Acknowledgements
This issue was identified in draft PR #58306 (which fixes the
`numpy_compat._floor_divide_func` mapping but deliberately leaves the
`num_ops.py` paths unfixed). That PR's author noted:
> "Worth flagging for reviewers: the same two defects are live in the path
that *is* reached, `data_type_ops/num_ops.py` … That is deliberately not fixed
here: it is a user-facing behaviour change on a long-standing path and deserves
its own JIRA and a maintainer's call on whether to make it."
I'm opening this issue per that guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
