Spenserrrr opened a new pull request, #57861:
URL: https://github.com/apache/spark/pull/57861
### What changes were proposed in this pull request?
This is a **pure refactor** of the existing `pa.Array.to_pandas` golden
tests, done first so that `pa.ChunkedArray.to_pandas` coverage can be added
cleanly in a follow-up. It adds no new test cases and changes no behavior; it
only reorganizes the shared machinery.
- Move `_PyArrowToPandasTestBase` into
`test_pyarrow_arrow_to_pandas_default.py` and have
`PyArrowArrayToPandasDefaultTests` extend it, so the base class is the single
home for the machinery both test files share.
- Split the source-array inventory into type-family group methods on the
base -- `_numeric_sources`, `_string_binary_sources`, `_temporal_sources`,
`_nested_sources`, `_dictionary_sources` -- unioned by `_build_source_arrays`.
A concrete test can now reuse the whole inventory or one group.
- `PyArrowArrayToPandasCoerceTemporalTests` builds its rows from
`_temporal_sources()` instead of hand-rolling the temporal types, removing the
duplicated definitions. Its golden file is regenerated: **same rows and same
cell values, only the row order changes** to match the shared temporal group.
- The default test class reuses the base's `_to_pandas_cell` instead of
reimplementing the conversion inline.
- Move `_verify_zero_copy` / `_arrow_buffers` (used only by the zero-copy
tests) out of the base and into `PyArrowArrayToPandasZeroCopyTests`, so no
class inherits helpers it does not use.
- The non-default file now imports the shared base directly rather than
importing the whole default module to avoid test re-collection; a base with no
`test_*` methods is not collected.
Class hierarchy after this change:
```
GoldenFileTestMixin
<- _PyArrowToPandasTestBase (in ..._default.py;
_to_pandas_cell + 5 group methods + _build_source_arrays; no test_*)
<- PyArrowArrayToPandasDefaultTests (..._default.py)
<- PyArrowArrayToPandasCoerceTemporalTests
(..._non_default.py; uses _temporal_sources())
<- PyArrowArrayToPandasZeroCopyTests
(..._non_default.py; owns _verify_zero_copy / _arrow_buffers)
<- PyArrowArrayToPandasIntegerObjectNullsTests (..._non_default.py)
```
### Why are the changes needed?
`pa.ChunkedArray.to_pandas` shares `to_pandas`'s entire keyword surface and
returns a Series, exactly like `pa.Array.to_pandas`, so its tests belong in
these same files. Adding them requires the source inventory to be reusable by
group (the ChunkedArray rows extend the existing ones) and the shared machinery
to live in one place. Doing this reorganization on its own keeps the follow-up
focused purely on the new coverage, and keeps this diff easy to verify as
behavior-preserving.
I will add the `pa.ChunkedArray.to_pandas` tests in a follow-up PR after
this one is merged.
### Does this PR introduce _any_ user-facing change?
No. Test-only refactor.
### How was this patch tested?
- The four to_pandas golden files were regenerated. The `default`,
`zero_copy`, `zero_copy_arrow_backed`, and `integer_object_nulls` goldens are
**byte-identical**; the `coerce_temporal` golden differs **only in row order**
(same rows, same cell values, confirmed by a sorted diff).
- The full to_pandas test suite passes without
`SPARK_GENERATE_GOLDEN_FILES`, and passes across a sweep of pyarrow
18/19/20/21/22/23/24/25 x pandas 2 and 3 (16/16).
- `dev/lint-python --ruff` is clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
