Spenserrrr opened a new pull request, #58019:
URL: https://github.com/apache/spark/pull/58019
### What changes were proposed in this pull request?
This is a follow-up to the native conversions of `fmax`/`fmin` (SPARK-58553)
and `reciprocal` (SPARK-58465, SPARK-58646) in `pyspark.pandas.numpy_compat`.
Those conversions replaced the `pandas_udf` fallbacks with native Spark
expressions, but for these three functions the native expression reproduces
behavior that C/IEEE leave unspecified, so it only matches NumPy on the
platform and NumPy version where that behavior was pinned down.
This PR restricts the native path to the verified platform and otherwise
falls back to the real NumPy ufunc via `pandas_udf` (which matches NumPy on any
platform/version by construction). Two module-level predicates express the two
independent axes of divergence:
- `_native_platform = platform.system() == "Linux" and platform.machine() ==
"x86_64"`
- `_numpy_tie_returns_first = LooseVersion(np.__version__) >=
LooseVersion("2.3.0")`
`reciprocal` gates on `_native_platform` only (its divergence is
architecture-specific, not version-specific); `fmax`/`fmin` gate on
`_native_platform and _numpy_tie_returns_first`.
### Why are the changes needed?
The scheduled ARM (`ubuntu-24.04-arm`, aarch64) and macOS jobs fail on
`test_np_fmax_fmin` and `test_np_reciprocal_integer`, because NumPy's own
result for these operations is not portable:
- `reciprocal` of an integer `0` is computed through a float→int cast of
`+inf`, which is undefined behavior in C and resolved per CPU architecture:
x86-64 yields `INT64_MIN`, aarch64 yields `INT64_MAX`.
- The `fmax`/`fmin` signed-zero tie (equal operands differing only in the
sign of zero, e.g. `+0.0` vs `-0.0`) is unspecified by C99. NumPy's scalar
result returns the first operand from 2.3.0 on and the second before, and it
also differs across architectures.
The native expressions were tuned to the x86-64 Linux / NumPy >= 2.3.0
result, so they diverge on other architectures and older NumPy. The
minimum-dependencies job (older NumPy) was addressed for `fmax`/`fmin` in an
earlier SPARK-58553 follow-up, but the underlying cross-platform (architecture)
divergence remained on aarch64 and macOS, and `reciprocal` was affected as well.
### Does this PR introduce _any_ user-facing change?
No. On the verified platform the native expression is unchanged. On other
platforms/versions the result now matches the installed NumPy (via the UDF)
instead of a platform-specific value, restoring behavior-preserving parity with
NumPy. There is no API change.
### How was this patch tested?
No test changes were needed: the existing `test_np_fmax_fmin`,
`test_np_reciprocal_integer`, and the `NumPyCompatTests` compat sweep use the
installed NumPy as the reference, so the mapping now tracks NumPy on every
platform.
Verified on GitHub Actions across all affected environments — ARM
(`ubuntu-24.04-arm`, aarch64), macOS (`macos-26`), the minimum-dependencies
build (older NumPy), and the regular x86-64 Linux build — all green. Also ran
the full `NumPyCompatTests` locally on x86-64 Linux with both the native path
and the UDF fallback path exercised, both passing.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
