This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 94d47152ec05 [SPARK-58249][SPARK-58234][PS][TEST][FOLLOWUP] Test
native NumPy ufuncs with special values
94d47152ec05 is described below
commit 94d47152ec053fb2ee7bc9fdf257e6c95502940d
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri Jul 24 09:31:40 2026 +0800
[SPARK-58249][SPARK-58234][PS][TEST][FOLLOWUP] Test native NumPy ufuncs
with special values
### What changes were proposed in this pull request?
Add pandas-on-Spark parity coverage for the native NumPy ufunc mappings
added in #57397 and #57417. The tests compare pandas-on-Spark results with
pandas/NumPy for invalid mathematical inputs, `NaN`, and positive and negative
infinity. They retain the `int64` minimum-value case for `np.fabs`.
### Why are the changes needed?
The mappings now execute native Spark SQL functions instead of scalar
pandas UDFs. IEEE special values and invalid function domains are where the two
implementations can differ, so the coverage verifies that the native results
remain NumPy-compatible.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added pandas-on-Spark comparisons with pandas/NumPy for invalid inputs,
`NaN`, and infinities.
- Ran `build/sbt -Phive package`.
- Ran `SPARK_TESTING=1 SPARK_PREPEND_CLASSES=1
PYSPARK_PYTHON=$CONDA_PREFIX/bin/python
PYSPARK_DRIVER_PYTHON=$CONDA_PREFIX/bin/python bin/pyspark
pyspark.pandas.tests.test_numpy_compat` in the `spark-dev-313` conda
environment.
- Ran `SPARK_TESTING=1 SPARK_PREPEND_CLASSES=1
PYSPARK_PYTHON=$CONDA_PREFIX/bin/python
PYSPARK_DRIVER_PYTHON=$CONDA_PREFIX/bin/python python/run-tests --testnames
"pyspark.pandas.tests.test_numpy_compat
NumPyCompatTests.test_np_math_functions"` in the `spark-dev-313` conda
environment.
- Ran `git diff --check`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57458 from zhengruifeng/pandas-native-math-special-values-dev2.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/pandas/tests/test_numpy_compat.py | 35 +++++++++++++-----------
1 file changed, 19 insertions(+), 16 deletions(-)
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py
b/python/pyspark/pandas/tests/test_numpy_compat.py
index 2864f8637796..ed5e53a8bd68 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -87,27 +87,30 @@ class NumPyCompatTestsMixin:
def test_np_math_functions(self):
for np_func, values in (
- (np.arccosh, [1.0, 2.0, 4.0]),
- (np.arcsinh, [-2.0, 0.0, 2.0]),
- (np.arctanh, [-0.5, 0.0, 0.5]),
- (np.cosh, [-2.0, 0.0, 2.0]),
- (np.deg2rad, [-180.0, 0.0, 180.0]),
- (np.exp2, [-2.0, 0.0, 2.0]),
+ (np.arccosh, [-np.inf, -1.0, 0.0, 1.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.arcsinh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
+ (
+ np.arctanh,
+ [-np.inf, -64.0, -1.0, 0.0, 1.0, 64.0, np.inf, np.nan],
+ ),
+ (np.cosh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.deg2rad, [-np.inf, -64.0, -180.0, 0.0, 180.0, 64.0, np.inf,
np.nan]),
+ (np.exp2, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
(np.fabs, [np.iinfo(np.int64).min, -2, 0, 2]),
- (np.negative, [-2.0, 0.0, 2.0]),
- (np.positive, [-2.0, 0.0, 2.0]),
- (np.rad2deg, [-np.pi, 0.0, np.pi]),
- (np.sign, [-2.0, -0.0, 0.0, 2.0, np.nan]),
- (np.sinh, [-2.0, 0.0, 2.0]),
- (np.square, [-2.0, 0.0, 2.0]),
- (np.tanh, [-2.0, 0.0, 2.0]),
+ (np.fabs, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.negative, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
+ (np.positive, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
+ (np.rad2deg, [-np.inf, -64.0, -np.pi, 0.0, np.pi, 64.0, np.inf,
np.nan]),
+ (np.sign, [-np.inf, -64.0, -2.0, -0.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
+ (np.sinh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.square, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf,
np.nan]),
+ (np.tanh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
):
- with self.subTest(name=np_func.__name__):
+ with self.subTest(name=np_func.__name__, values=values):
pdf = pd.DataFrame({"a": values})
psdf = ps.from_pandas(pdf)
- result = np_func(psdf.a)
- self.assert_eq(result, np_func(pdf.a), almost=True)
+ self.assert_eq(np_func(psdf.a), np_func(pdf.a), almost=True)
def test_np_spark_compat_series(self):
from pyspark.pandas.numpy_compat import unary_np_spark_mappings,
binary_np_spark_mappings
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
