Yicong-Huang opened a new pull request, #57856: URL: https://github.com/apache/spark/pull/57856
### What changes were proposed in this pull request? This PR replaces the Python `pandas_udf` fallback used by `np.reciprocal` on integer columns in pandas-on-Spark with a native Spark SQL expression. The float/double branch of the `reciprocal` mapping in `python/pyspark/pandas/numpy_compat.py` was already native; only the integer fallback still wrapped `np.reciprocal` in a scalar pandas UDF. The new expression reproduces NumPy's integer semantics natively: integer division truncated toward zero (`1 -> 1`, `-1 -> -1`, every other magnitude `-> 0`), with `0` mapping to the int64 minimum to match NumPy's overflow behavior on integer arrays. ### Why are the changes needed? Evaluating a Python UDF per batch incurs serialization to and from the Python worker, which is far more expensive than a native Catalyst expression that runs entirely in the JVM. Removing the UDF for the integer path avoids that round trip and keeps the behavior identical to the previous implementation. ### Does this PR introduce _any_ user-facing change? No. The output values are unchanged for all integer inputs (verified against `np.reciprocal` on the equivalent pandas Series for positive, negative, zero, and int64 boundary values). ### How was this patch tested? Added `test_np_reciprocal_integer` to `python/pyspark/pandas/tests/test_numpy_compat.py` covering positive, negative, and zero integer inputs (including the int64 min/max boundaries), asserting parity with `np.reciprocal` on the pandas reference. The test is inherited by the Spark Connect parity suite (`test_parity_numpy_compat.py`). Both the classic and Connect suites pass. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
