zhengruifeng opened a new pull request, #57458: URL: https://github.com/apache/spark/pull/57458
### What changes were proposed in this pull request? Add pandas-on-Spark parity coverage for the native NumPy ufunc mappings added in #57397 and #57417. The tests compare pandas-on-Spark results with pandas/NumPy for invalid mathematical inputs, `NaN`, and positive and negative infinity. They retain the `int64` minimum-value case for `np.fabs`. ### Why are the changes needed? The mappings now execute native Spark SQL functions instead of scalar pandas UDFs. IEEE special values and invalid function domains are where the two implementations can differ, so the coverage verifies that the native results remain NumPy-compatible. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Added pandas-on-Spark comparisons with pandas/NumPy for invalid inputs, `NaN`, and infinities. - Ran `build/sbt -Phive package`. - Ran `SPARK_TESTING=1 SPARK_PREPEND_CLASSES=1 PYSPARK_PYTHON=$CONDA_PREFIX/bin/python PYSPARK_DRIVER_PYTHON=$CONDA_PREFIX/bin/python bin/pyspark pyspark.pandas.tests.test_numpy_compat` in the `spark-dev-313` conda environment. - Ran `SPARK_TESTING=1 SPARK_PREPEND_CLASSES=1 PYSPARK_PYTHON=$CONDA_PREFIX/bin/python PYSPARK_DRIVER_PYTHON=$CONDA_PREFIX/bin/python python/run-tests --testnames "pyspark.pandas.tests.test_numpy_compat NumPyCompatTests.test_np_math_functions"` in the `spark-dev-313` conda environment. - Ran `git diff --check`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
