Haotian Sun created SPARK-59359:
-----------------------------------
Summary: Add tests for NumPy ufunc type coercion
Key: SPARK-59359
URL: https://issues.apache.org/jira/browse/SPARK-59359
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 5.0.0
Reporter: Haotian Sun
pandas-on-Spark gates the operand types each NumPy ufunc accepts with
_np_spark_accepted_types in python/pyspark/pandas/numpy_compat.py, a table
transcribed by hand from what NumPy accepts. Nothing verifies that
transcription, and dev/requirements.txt requires numpy>=1.23.2 with no upper
bound, so a NumPy release that moves a coercion leaves the gate over-rejecting
or under-rejecting silently.
Add a golden-file test under python/pyspark/tests/upstream/numpy/ that records,
for every ufunc pandas-on-Spark dispatches, the dtype NumPy coerces each
operand to. It takes no Spark session and acts as a drift canary, in the same
shape as the existing upstream/pyarrow golden tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
