This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new b0a5d5b02d68 [SPARK-58469][PS] Use native expressions for NumPy rint
b0a5d5b02d68 is described below
commit b0a5d5b02d685652cf04a597c09cca9fba3de27e
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Aug 3 16:43:42 2026 +0800
[SPARK-58469][PS] Use native expressions for NumPy rint
### What changes were proposed in this pull request?
Replace the pandas UDF mapping for NumPy rint on pandas-on-Spark objects
with the native Spark `rint` expression after casting the input to `double`.
Add coverage for halfway values, signed zero, NaN, and infinities.
### Why are the changes needed?
The operation can be evaluated by Spark without crossing the Python worker
boundary while preserving the existing double result type.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added parity coverage in `NumPyCompatTests.test_np_math_functions`.
- Passed Ruff format and lint checks for the changed Python files.
- Focused PySpark tests were not run locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57672 from zhengruifeng/pandas-native-rint-dev2.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit b24e4d7a2cabebc95f7ea664ca3d3b5f29c68b83)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 2 +-
python/pyspark/pandas/tests/test_numpy_compat.py | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index 316e2e9e3afd..baa107660185 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -68,7 +68,7 @@ unary_np_spark_mappings = {
"reciprocal": pandas_udf( # type: ignore[call-overload]
lambda s: np.reciprocal(s), DoubleType()
),
- "rint": pandas_udf(lambda s: np.rint(s), DoubleType()), # type:
ignore[call-overload]
+ "rint": lambda c: F.rint(c.cast("double")),
"sign": F.signum,
"signbit": lambda c: F.when(c < 0, True).otherwise(False),
"sin": F.sin,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index bd7ba4c3e210..219f21a6efbc 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -104,6 +104,7 @@ class NumPyCompatTestsMixin:
(np.negative, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
(np.positive, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
(np.rad2deg, [-np.inf, -64.0, -np.pi, 0.0, np.pi, 64.0, np.inf,
np.nan]),
+ (np.rint, [-np.inf, -2.5, -1.5, -0.5, -0.0, 0.0, 0.5, 1.5, 2.5,
np.inf, np.nan]),
(np.sign, [-np.inf, -64.0, -2.0, -0.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
(np.sinh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
(np.square, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]