Spenserrrr opened a new pull request, #57912: URL: https://github.com/apache/spark/pull/57912
### What changes were proposed in this pull request? This PR replaces the `pandas_udf`-based implementation of `numpy.copysign` in the pandas API on Spark with a native Spark SQL expression. `np.copysign(a, b)` returns the magnitude of `a` with the sign bit of `b`. The native expression handles the edge cases that a naive `b < 0` test would miss: - **Signed zero**: `-0.0` has its sign bit set (`np.copysign(1.0, -0.0) == -1.0`) even though `-0.0 < 0` is `False` in Spark. The sign test additionally checks `cast(b as string) == '-0.0'`, mirroring the existing `reciprocal` mapping in the same file. - **Null**: NA in either argument propagates to NULL, guarded explicitly so that a null `b` does not fall through the sign predicate (a null condition in a `CASE WHEN` is not matched, which would otherwise leak a wrong non-null result). ### Why are the changes needed? The `pandas_udf` fallback ships every row JVM -> Python worker -> JVM through Arrow serialization, runs NumPy, and ships the results back. A native expression runs in the JVM, code-generates, and is visible to the optimizer. This is part of the umbrella [SPARK-58532](https://issues.apache.org/jira/browse/SPARK-58532). ### Does this PR introduce _any_ user-facing change? No. The behavior is preserved, including the double result type of the previous `pandas_udf` implementation. ### How was this patch tested? Added `test_np_copysign` to `NumPyCompatTestsMixin`, covering integer inputs (float result), infinities and NaN in both positions, signed zero (pinned with `np.signbit` since `almost=` cannot distinguish `-0.0` from `+0.0`), and null propagation via a nullable `Int64` frame. The Connect parity suite inherits the mixin automatically. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
