Yicong-Huang opened a new pull request, #58218:
URL: https://github.com/apache/spark/pull/58218
### What changes were proposed in this pull request?
This is a follow-up of #57856 (SPARK-58646), which replaced the scalar
pandas UDF fallback of `np.reciprocal` with native Spark expressions for
non-floating-point inputs.
That change routed every non-`float`/`double` dtype through a single integer
branch that hard-codes the int64 minimum as the divide-by-zero sentinel and
casts the quotient through `long`. This does not match the previous pandas UDF
(`np.reciprocal` applied to the pandas `Series`) for decimals, booleans, and
narrower integers:
| dtype | input | previous UDF (`np.reciprocal` -> Double) | merged native
(#57856) |
|---|---|---|---|
| int64 (bigint) | `0` | `-9.2e18` (int64 min) | `-9.2e18` (unchanged) |
| int32 (int) | `0` | `-2147483648` (int32 min) | `-9.2e18` |
| int8/int16 (tinyint/smallint) | `0` | `0` (numpy `1 // 0` does not
overflow on narrow widths) | `-9.2e18` |
| boolean | `False` | `0.0` (numpy promotes bool to int8: `True -> 1`,
`False -> 0`) | `-9.2e18` |
| decimal | `2.5` | `0.4` (numpy takes a true floating reciprocal) | `0.0`
(truncated to long) |
This PR restores parity:
- Decimal inputs now flow through the floating-point reciprocal branch
(`typeof` starts with `decimal`), since numpy computes a true reciprocal for
them. A decimal `0` (which the old UDF could not handle --
`np.reciprocal(Decimal('0'))` raises `DivisionByZero`) now maps to `inf`,
consistent with the floating-point branch.
- The integer/boolean branch now picks the divide-by-zero sentinel by column
width -- int32 minimum for `int`, int64 minimum for `bigint`, and `0` for the
narrower widths (`tinyint`, `smallint`, and `boolean` promoted to int8) -- and
casts through `long` so boolean and narrower integers can take part in the
division.
int64 columns, the only case exercised by the original PR, are unchanged.
### Why are the changes needed?
The merged native expression regressed the observable pandas-on-Spark
behavior for decimal, boolean, and narrower-integer columns relative to the
pandas UDF it replaced. This restores parity so that `np.reciprocal` produces
the same results as before across all supported dtypes.
### Does this PR introduce _any_ user-facing change?
No. #57856 is unreleased (master only), so this only fixes an unreleased
regression before it ships; there is no change relative to any released Spark
version.
### How was this patch tested?
Added `test_np_reciprocal_non_default_dtypes` in
`python/pyspark/pandas/tests/test_numpy_compat.py`, inherited by the Spark
Connect parity suite, asserting `np.reciprocal(psser)` equals
`np.reciprocal(pdf)` for int8/int16/int32, boolean, and decimal columns
(covering positive, negative, and the per-width zero-overflow sentinel).
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
