This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 691d1567f03c [SPARK-58470][PS] Use native expressions for NumPy trunc
691d1567f03c is described below
commit 691d1567f03c1ec6ed6e831315325e2950350dd2
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Mon Aug 3 16:47:04 2026 +0800
[SPARK-58470][PS] Use native expressions for NumPy trunc
### What changes were proposed in this pull request?
Replace the pandas UDF mapping for NumPy trunc on pandas-on-Spark objects
with native Spark expressions. The implementation uses the remainder of the
absolute double value and reapplies the sign, preserving nulls, NaN,
infinities, and signed zero. Add coverage for these values and fractional
positive and negative inputs.
### Why are the changes needed?
Spark does not provide a numeric trunc function, but the operation can be
evaluated with native expressions without crossing the Python worker boundary.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added parity coverage in `NumPyCompatTests.test_np_math_functions`.
- Manually verified the native expression for signed zero, fractional
values, infinities, NaN, and large doubles.
- Passed Ruff format and lint checks for the changed Python files.
- Focused PySpark tests were not run locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57673 from zhengruifeng/pandas-native-trunc-dev2.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/numpy_compat.py | 11 ++++++++++-
python/pyspark/pandas/tests/test_numpy_compat.py | 18 ++++++++++++++++++
2 files changed, 28 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/numpy_compat.py
b/python/pyspark/pandas/numpy_compat.py
index baa107660185..1653121d3124 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -23,6 +23,7 @@ from pyspark.sql.pandas.functions import pandas_udf
from pyspark.sql.types import DoubleType, BooleanType
from pyspark.pandas.base import IndexOpsMixin
+
unary_np_spark_mappings = {
"abs": F.abs,
"absolute": F.abs,
@@ -78,7 +79,15 @@ unary_np_spark_mappings = {
"square": lambda c: c.cast("double") * c,
"tan": F.tan,
"tanh": F.tanh,
- "trunc": pandas_udf(lambda s: np.trunc(s), DoubleType()), # type:
ignore[call-overload]
+ "trunc": lambda c: F.when(
+ c.cast("double").isNull()
+ | F.isnan(c.cast("double"))
+ | c.cast("double").isin(float("-inf"), float("inf")),
+ c.cast("double"),
+ ).otherwise(
+ F.signum(c.cast("double"))
+ * (F.abs(c.cast("double")) - (F.abs(c.cast("double")) % F.lit(1.0)))
+ ),
}
binary_np_spark_mappings = {
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index 219f21a6efbc..2530d6b04a80 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -109,6 +109,24 @@ class NumPyCompatTestsMixin:
(np.sinh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
(np.square, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
(np.tanh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (
+ np.trunc,
+ [
+ -np.inf,
+ -64.0,
+ -2.0,
+ -1.5,
+ -0.5,
+ -0.0,
+ 0.0,
+ 0.5,
+ 1.5,
+ 2.0,
+ 64.0,
+ np.inf,
+ np.nan,
+ ],
+ ),
):
with self.subTest(name=np_func.__name__, values=values):
pdf = pd.DataFrame({"a": values})
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]