uros-b commented on code in PR #57784:
URL: https://github.com/apache/spark/pull/57784#discussion_r3719221601
##########
python/pyspark/pandas/numpy_compat.py:
##########
@@ -116,7 +116,18 @@
.otherwise(F.greatest(c1, c2))
.cast("double"),
"fmin": lambda c1, c2: F.least(c1, c2).cast("double"),
- "fmod": pandas_udf(lambda s1, s2: np.fmod(s1, s2), DoubleType()), # type:
ignore[call-overload]
+ "fmod": lambda c1, c2: F.when(
Review Comment:
When c2 is NULL and c1 is non-null, the outer when(c1.isNull() |
c2.isNull(), c1.cast("double")) returns the dividend's value rather than NULL.
Under the old pandas-udf, Spark NULL was converted to pandas NaN before calling
np.fmod, so np.fmod(x, NaN) = NaN; the null-divisor propagated as NaN. The new
code returns c1 unchanged for a null divisor. In practice, float columns in
pandas-on-Spark use NaN (not NULL) for missing values, so the
practically-affected population is nullable-integer columns with Python None
entries (e.g. pd.array([1, 2, None], dtype="Int64")); those are fully supported
and the divergence is reachable. Fix: the outer NULL clause is superfluous —
(c1 % c2) already propagates NULL through Spark's built-in null semantics in
both positions; removing the outer when clause entirely is cleaner and fixes
the regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
