zhengruifeng opened a new pull request, #57237:
URL: https://github.com/apache/spark/pull/57237
### What changes were proposed in this pull request?
Select dedicated golden files for PyArrow Array.to_pandas() according to the
installed pandas major version. The existing expectations move to pandas_2, and
pandas_3 records the string dtype behavior from pandas 3.0.3 and PyArrow 24.0.0.
### Why are the changes needed?
Pandas 3 uses its dedicated string dtype when converting Arrow string
arrays. The shared golden expected Series[object], causing the pandas 3 CI job
to fail with six mismatches for string and large_string arrays.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- pandas 2.3.3 / PyArrow 23.0.1:
conda run -n spark-dev-313 env PYTHONPATH=python python -m unittest
pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_default
- pandas 3.0.3 / PyArrow 24.0.0 / NumPy 2.5.1, matching the failed CI job:
conda run -n spark-dev-313-p3 env PYTHONPATH=/tmp/spark-pandas3-ci:python
python -m unittest
pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_default
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
