Spenserrrr opened a new pull request, #57978:
URL: https://github.com/apache/spark/pull/57978
### What changes were proposed in this pull request?
SPARK-58553 replaced the `pandas_udf`-based `np.fmax` / `np.fmin`
implementations in the pandas API on Spark with native `F.greatest` / `F.least`
expressions. When the two operands are equal (for example `+0.0` and `-0.0`),
the native mapping breaks the tie by returning the **first** operand.
NumPy changed this signed-zero tie-break at **2.3.0**: `>= 2.3.0` returns
the first operand, while older versions return the **second**. The native
mapping therefore matches NumPy `>= 2.3.0` but disagrees with older versions on
the sign of a `±0.0` result.
This PR selects the tie operand based on the installed NumPy version so the
result matches `np.fmax` / `np.fmin` on that NumPy: return the first operand on
`>= 2.3.0`, the second on older versions. The implementation stays fully native
(`F.greatest` / `F.least`); only which operand is returned on a signed-zero tie
differs by version.
**Why not restore the original `pandas_udf` fallback for old NumPy?** The
original UDF matched the installed NumPy automatically (it calls `np.fmax` in a
Python worker), so restoring it for `< 2.3.0` would also be correct. But that
reintroduces the per-batch JVM <-> Python round trip that SPARK-58553 removed,
losing the performance and optimizer benefits for those users. Since the
signed-zero tie is the **only** cross-version difference (verified exhaustively
over every combination of `{-inf, -2, -1, -0.0, +0.0, 1, 2, inf, nan}` from
NumPy 1.23.2 through 2.4.1 — the tie-break flips at 2.3.0 and nothing else
changes), and NumPy `< 2.3.0` is a frozen release range, selecting the matching
tie operand keeps the native fast path while producing results identical to
`np.fmax` / `np.fmin`.
**Scope:** `fmax` / `fmin` is the only affected function. The full
`test_numpy_compat.py` suite (18 tests, including the generic mapping sweeps
and every other SPARK-58532 conversion — `fmod`, `ldexp`, `heaviside`,
`reciprocal`, `float_power`, bitwise shifts, `signbit`, etc.) passes on the
minimum dependencies; the signed-zero tie in `fmax` / `fmin` is the only
version-sensitive behavior.
### Why are the changes needed?
The scheduled "Build / Python-only (Minimum dependencies of PySpark)" build
(NumPy 1.23.2) fails `pyspark.pandas.tests.test_numpy_compat
NumPyCompatTests.test_np_fmax_fmin`. The test asserts the sign bit of the
result via `np.signbit`, and on the two `±0.0` tie rows the native mapping
(first operand) disagrees with the reference computed from the installed NumPy
(second operand on 1.23.2). Regular CI runs a newer NumPy (`>= 2.3.0`), where
the native choice matches, which is why the original change passed pre-merge CI
and the failure only surfaced in the minimum-dependency build.
### Does this PR introduce _any_ user-facing change?
No. There is no change relative to any released Spark version (the released
implementation used the `pandas_udf`, which already matched the installed
NumPy). This aligns the unreleased native implementation from SPARK-58553 with
`np.fmax` / `np.fmin` on NumPy `< 2.3.0`. The numeric value is unchanged in all
cases (`+0.0` and `-0.0` are numerically equal); only the sign bit of a zero
result on a `±0.0` tie is corrected to match the installed NumPy.
### How was this patch tested?
-
`pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_fmax_fmin` and
the full `NumPyCompatTests` suite (18 tests) pass on NumPy 2.4.1 and in a
minimum-dependency environment (NumPy 1.23.2, pandas 2.2.0, pyarrow 18.0.0)
that reproduces the scheduled build.
- Confirmed the failure reproduces on NumPy 1.23.2 without this change and
is resolved with it.
- Verified across installed NumPy wheels (1.23.2 through 2.4.1) that the
signed-zero `±0.0` tie is the only `fmax` / `fmin` behavior that differs
between versions, and that the tie-break flips at exactly 2.3.0.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
