zhengruifeng opened a new pull request, #56402:
URL: https://github.com/apache/spark/pull/56402
### What changes were proposed in this pull request?
The pandas UDF coercion tests (`test_pandas_udf_input_type` and
`test_pandas_udf_return_type`) used to load a single golden file generated
under
Pandas 2 and patch it in memory to work under Pandas 3, where several
defaults
differ:
- `datetime64` ndarrays default to `[us]` instead of `[ns]`
- `Categorical` categories use `str` instead of `object`
- the same casts return microseconds instead of nanoseconds
- string lists coerce to `Decimal` where Pandas 2 errored
This PR replaces that in-memory patching with dedicated golden files
generated
under Pandas 3 (the `_base_pandas3` suffix). The running pandas version
selects
the golden file; Pandas 2 keeps using the existing `_base` files unchanged.
A second, temporary commit switches the default CI `PYSPARK_IMAGE_TO_TEST` to
`python-312-pandas-3` so this PR's GitHub Actions run exercises the new
golden
files under Pandas 3. **That commit is for testing only and will be reverted
before merge.**
### Why are the changes needed?
In-memory patching couples the golden data to hand-maintained assumptions
about
how Pandas 3 differs from Pandas 2 (column renames, ns->us scaling, cell
flips).
Generating real golden files under Pandas 3 captures the actual behavior
directly, and is easier to maintain and review.
### Does this PR introduce _any_ user-facing change?
No. Test-only change (plus a temporary CI-only commit).
### How was this patch tested?
- Regenerated the golden files with `SPARK_GENERATE_GOLDEN_FILES=1` under a
Pandas 3 environment (pandas 3.0.2, numpy 2.4.3, pyarrow 23.0.1, Python
3.13).
- Ran the two pandas UDF coercion tests against the new golden files under
Pandas 3.
- The temporary CI image commit runs the PySpark suite under Pandas 3 in this
PR's GitHub Actions.
Note: the pre-existing `test_python_udf_input_type` (`with_arrow_and_pandas`
variant) is known to fail under Pandas 3 because the legacy pandas conversion
turns `None` into `nan` for the `string_null` case. That is not addressed in
this PR.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (model: claude-opus-4-8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
