[
https://issues.apache.org/jira/browse/SPARK-59359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng reassigned SPARK-59359:
-------------------------------------
Assignee: Haotian Sun
> Add tests for NumPy ufunc type coercion
> ---------------------------------------
>
> Key: SPARK-59359
> URL: https://issues.apache.org/jira/browse/SPARK-59359
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Haotian Sun
> Assignee: Haotian Sun
> Priority: Major
> Labels: pull-request-available
>
> pandas-on-Spark gates the operand types each NumPy ufunc accepts with
> _np_spark_accepted_types in python/pyspark/pandas/numpy_compat.py, a table
> transcribed by hand from what NumPy accepts. Nothing verifies that
> transcription, and dev/requirements.txt requires numpy>=1.23.2 with no upper
> bound, so a NumPy release that moves a coercion leaves the gate
> over-rejecting or under-rejecting silently.
> Add a golden-file test under python/pyspark/tests/upstream/numpy/ that
> records, for every ufunc pandas-on-Spark dispatches, the dtype NumPy coerces
> each operand to. It takes no Spark session and acts as a drift canary, in the
> same shape as the existing upstream/pyarrow golden tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
