uros-b commented on code in PR #57783:
URL: https://github.com/apache/spark/pull/57783#discussion_r3719228297
##########
python/pyspark/pandas/numpy_compat.py:
##########
@@ -108,8 +108,8 @@
lambda s1, s2: np.copysign(s1, s2), DoubleType()
),
"float_power": lambda c1, c2: F.pow(c1.cast("double"), c2.cast("double")),
- "floor_divide": pandas_udf( # type: ignore[call-overload]
- lambda s1, s2: np.floor_divide(s1, s2), DoubleType()
+ "floor_divide": lambda c1, c2: F.when(c2 == 0, F.lit(0.0)).otherwise(
Review Comment:
The zero-divisor guard F.when(c2 == 0, F.lit(0.0)) fires for both integer
and float zero divisors, returning 0.0 in all cases. For float inputs, NumPy's
contract is: np.floor_divide(1.0, 0.0) = inf, np.floor_divide(-1.0, 0.0) =
-inf, np.floor_divide(0.0, 0.0) = nan. The old pandas-udf path preserved those
IEEE 754 results. The new implementation returns 0.0 instead; a silent
behavioral divergence for float-column users. The fix is to only suppress the
zero divisor for integer inputs (e.g. guard on F.typeof similar to the fmod
PR), or to use the type-dispatch pattern already present in fmod to return
float("nan") for float-type zero divisors and float("inf") when c1 > 0,
float("-inf") when c1 < 0.
##########
python/pyspark/pandas/tests/test_numpy_compat.py:
##########
@@ -196,6 +196,19 @@ def test_np_ldexp(self):
self.assert_eq(np.ldexp(psdf.x, psdf.exp), np.ldexp(pdf.x, pdf.exp),
almost=True)
+ def test_np_floor_divide(self):
Review Comment:
The added test exercises only integer inputs. No float frame is included, so
the float-zero-divisor divergence above is not caught. The sibling PR #57784
(fmod) correctly adds both integer and float frames with infinities and NaN;
this PR should do the same to provide comparable coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
