Spenserrrr opened a new pull request, #57435:
URL: https://github.com/apache/spark/pull/57435
### What changes were proposed in this pull request?
This PR adds a golden-file test that pins the behavior of PyArrow's
`Array.to_pandas(coerce_temporal_nanoseconds=True)` across Arrow data types.
- New test:
`python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_coerce_temporal.py`
- New golden files: `golden_pyarrow_arrow_to_pandas_coerce_temporal.{csv,md}`
- Registered the module in `dev/sparktestsupport/modules.py`.
The test follows the same pattern as `test_pyarrow_arrow_to_pandas_default`
(SPARK-54944). The only behavioral difference is that each source array is
converted with `arr.to_pandas(coerce_temporal_nanoseconds=True)`. The source
arrays are limited to the temporal types the argument affects (`timestamp` and
`duration` in units
s/ms/us/ns, tz-aware `timestamp`, `date`, and `time`), and an overflow case
and a few non-temporal control rows (`int64`/`float64`/`string`) to demonstrate
the argument leaves non-temporal types unaffected.
The golden files record the resulting pandas Series dtype and values. They
were generated with pandas 2.3.3, pyarrow 24.0.0, and numpy 2.4.1. Notable
behavior:
- `timestamp[s|ms|us|ns]` all coerce to `datetime64[ns]`.
- tz-aware `timestamp[us, tz=UTC]` coerces to `datetime64[ns, UTC]`.
- `duration[s|ms|us|ns]` all coerce to `timedelta64[ns]`.
- `date32`/`date64` and `time32`/`time64` remain `object` dtype
- A far-future second-resolution timestamp (year 2500) that cannot fit in
the nanosecond range raises `ArrowInvalid`.
### Why are the changes needed?
This is part of SPARK-54936 ("Monitor behaviour changes from upstream").
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New golden-file test. Generated the golden files with
`SPARK_GENERATE_GOLDEN_FILES=1` (pandas 2.3.3, pyarrow 24.0.0, numpy 2.4.1),
reviewed every cell, then confirmed the test passes in comparison mode:
python -m pytest
python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_coerce_temporal.py
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
