This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new e36c662dbe9d [SPARK-58656][PYTHON][TEST] Centralize the to_pandas
golden test inventory into a shared base
e36c662dbe9d is described below
commit e36c662dbe9d8bdd97198d41670cac4dc283e885
Author: Spenser Sun <[email protected]>
AuthorDate: Mon Aug 10 12:08:54 2026 +0800
[SPARK-58656][PYTHON][TEST] Centralize the to_pandas golden test inventory
into a shared base
### What changes were proposed in this pull request?
This is a **pure refactor** of the existing `pa.Array.to_pandas` golden
tests, done first so that `pa.ChunkedArray.to_pandas` coverage can be added
cleanly in a follow-up. It adds no new test cases and changes no behavior; it
only reorganizes the shared machinery.
- Move `_PyArrowToPandasTestBase` into
`test_pyarrow_arrow_to_pandas_default.py` and have
`PyArrowArrayToPandasDefaultTests` extend it, so the base class is the single
home for the machinery both test files share.
- Split the source-array inventory into type-family group methods on the
base -- `_numeric_sources`, `_string_binary_sources`, `_temporal_sources`,
`_nested_sources`, `_dictionary_sources` -- unioned by `_build_source_arrays`.
A concrete test can now reuse the whole inventory or one group.
- `PyArrowArrayToPandasCoerceTemporalTests` builds its rows from
`_temporal_sources()` instead of rewriting the temporal types, removing the
duplicated definitions. Its golden file is regenerated: **same rows and same
cell values, only the row order changes** to match the shared temporal group.
- The default test class reuses the base's `_to_pandas_cell` instead of
reimplementing the conversion inline.
- Move `_verify_zero_copy` / `_arrow_buffers` (used only by the zero-copy
tests) out of the base and into `PyArrowArrayToPandasZeroCopyTests`, so no
class inherits helpers it does not use.
- The non-default file now imports the shared base directly rather than
importing the whole default module to avoid test re-collection; a base with no
`test_*` methods is not collected.
Class hierarchy after this change:
```
GoldenFileTestMixin
<- _PyArrowToPandasTestBase (in ..._default.py;
_to_pandas_cell + 5 group methods + _build_source_arrays; no test_*)
<- PyArrowArrayToPandasDefaultTests (..._default.py)
<- PyArrowArrayToPandasCoerceTemporalTests
(..._non_default.py; uses _temporal_sources())
<- PyArrowArrayToPandasZeroCopyTests
(..._non_default.py; owns _verify_zero_copy / _arrow_buffers)
<- PyArrowArrayToPandasIntegerObjectNullsTests
(..._non_default.py)
```
### Why are the changes needed?
`pa.ChunkedArray.to_pandas` shares `to_pandas`'s entire keyword surface and
returns a Series, exactly like `pa.Array.to_pandas`, so its tests belong in
these same files. Adding them requires the source inventory to be reusable by
group (the ChunkedArray rows extend the existing ones) and the shared machinery
to live in one place. Doing this reorganization on its own keeps the follow-up
focused purely on the new coverage, and keeps this diff easy to verify as
behavior-preserving.
I will add the `pa.ChunkedArray.to_pandas` tests in a follow-up PR after
this one is merged.
### Does this PR introduce _any_ user-facing change?
No. Test-only refactor.
### How was this patch tested?
- The four to_pandas golden files were regenerated. The `default`,
`zero_copy`, `zero_copy_arrow_backed`, and `integer_object_nulls` goldens are
**byte-identical**; the `coerce_temporal` golden differs **only in row order**
(same rows, same cell values, confirmed by a sorted diff).
- The full to_pandas test suite passes without
`SPARK_GENERATE_GOLDEN_FILES`, and passes across a sweep of pyarrow
18/19/20/21/22/23/24/25 x pandas 2 and 3 (16/16).
- `dev/lint-python --ruff` is clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57861 from Spenserrrr/chunkedarray-to-pandas-tests.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 661188e8fd26924a3fd44acb281b1f82066ea71b)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
...den_pyarrow_arrow_to_pandas_coerce_temporal.csv | 16 +-
...lden_pyarrow_arrow_to_pandas_coerce_temporal.md | 16 +-
.../test_pyarrow_arrow_to_pandas_default.py | 92 +++++--
.../test_pyarrow_arrow_to_pandas_non_default.py | 295 ++++++++-------------
4 files changed, 195 insertions(+), 224 deletions(-)
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
index d4a8df1a2a62..4f8439cd1d2e 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
@@ -1,4 +1,10 @@
test case pyarrow array pandas series pandas series
(date_as_object=False)
+date32:standard [2024-01-01, 2024-06-15]@date32[day]
[datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
[Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]]
+date32:nullable [2024-01-01, None]@date32[day] [datetime.date(2024, 1,
1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
+date32:empty []@date32[day] []@Series[object]
[]@Series[datetime64[ns]]
+date64:standard [2024-01-01, 2024-06-15]@date64[ms]
[datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
[Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]]
+date64:nullable [2024-01-01, None]@date64[ms] [datetime.date(2024, 1,
1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
+date64:empty []@date64[ms] []@Series[object]
[]@Series[datetime64[ns]]
timestamp[s]:standard [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[s]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
timestamp[s]:nullable [2024-01-01 12:00:00, None]@timestamp[s]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
timestamp[s]:empty []@timestamp[s] []@Series[datetime64[ns]]
[]@Series[datetime64[ns]]
@@ -14,7 +20,6 @@ timestamp[ns]:empty []@timestamp[ns]
[]@Series[datetime64[ns]] []@Series[datetim
timestamp[us,tz=UTC]:standard [2024-01-01 12:00:00+00:00, 2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]]
timestamp[us,tz=UTC]:nullable [2024-01-01 12:00:00+00:00, None]@timestamp[us,
tz=UTC] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'),
NaT]@Series[datetime64[ns, UTC]] [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), NaT]@Series[datetime64[ns, UTC]]
timestamp[us,tz=UTC]:empty []@timestamp[us, tz=UTC]
[]@Series[datetime64[ns, UTC]] []@Series[datetime64[ns, UTC]]
-timestamp[s]:overflow [2500-01-01 00:00:00]@timestamp[s]
ERR@ArrowInvalid ERR@ArrowInvalid
duration[s]:standard [1 day, 0:00:00, 2:30:00]@duration[s] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]]
duration[s]:nullable [1 day, 0:00:00, None]@duration[s] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[ns]] [Timedelta('1 days
00:00:00'), NaT]@Series[timedelta64[ns]]
duration[s]:empty []@duration[s] []@Series[timedelta64[ns]]
[]@Series[timedelta64[ns]]
@@ -27,13 +32,6 @@ duration[us]:empty []@duration[us]
[]@Series[timedelta64[ns]] []@Series[timedelt
duration[ns]:standard [1 days 00:00:00, 0 days 02:30:00]@duration[ns]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
duration[ns]:nullable [1 days 00:00:00, None]@duration[ns] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[ns]] [Timedelta('1 days
00:00:00'), NaT]@Series[timedelta64[ns]]
duration[ns]:empty []@duration[ns] []@Series[timedelta64[ns]]
[]@Series[timedelta64[ns]]
-duration[s]:overflow [109500 days, 0:00:00]@duration[s]
[Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
[Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
-date32:standard [2024-01-01, 2024-06-15]@date32[day]
[datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
[Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]]
-date32:nullable [2024-01-01, None]@date32[day] [datetime.date(2024, 1,
1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
-date32:empty []@date32[day] []@Series[object]
[]@Series[datetime64[ns]]
-date64:standard [2024-01-01, 2024-06-15]@date64[ms]
[datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
[Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]]
-date64:nullable [2024-01-01, None]@date64[ms] [datetime.date(2024, 1,
1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
-date64:empty []@date64[ms] []@Series[object]
[]@Series[datetime64[ns]]
time32[s]:standard [12:30:00, 18:45:30]@time32[s] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object]
time32[s]:nullable [12:30:00, None]@time32[s] [datetime.time(12, 30),
None]@Series[object] [datetime.time(12, 30), None]@Series[object]
time32[s]:empty []@time32[s] []@Series[object]
[]@Series[object]
@@ -46,6 +44,8 @@ time64[us]:empty []@time64[us] []@Series[object]
[]@Series[object]
time64[ns]:standard [12:30:00, 18:45:30]@time64[ns] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object]
time64[ns]:nullable [12:30:00, None]@time64[ns] [datetime.time(12, 30),
None]@Series[object] [datetime.time(12, 30), None]@Series[object]
time64[ns]:empty []@time64[ns] []@Series[object]
[]@Series[object]
+timestamp[s]:overflow [2500-01-01 00:00:00]@timestamp[s]
ERR@ArrowInvalid ERR@ArrowInvalid
+duration[s]:overflow [109500 days, 0:00:00]@duration[s]
[Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
[Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
int64:standard [0, 1, -1]@int64 [0, 1, -1]@Series[int64] [0, 1,
-1]@Series[int64]
int64:nullable [0, 1, None]@int64 [0.0, 1.0, nan]@Series[float64] [0.0,
1.0, nan]@Series[float64]
float64:standard [0.0, 1.5, -1.5]@float64 [0.0, 1.5,
-1.5]@Series[float64] [0.0, 1.5, -1.5]@Series[float64]
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
index 45eec337f059..e6b3a061c97d 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
@@ -1,5 +1,11 @@
| test case | pyarrow array
| pandas series
| pandas series (date_as_object=False)
|
|-------------------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
+| date32:standard | [2024-01-01, 2024-06-15]@date32[day]
| [datetime.date(2024, 1, 1),
datetime.date(2024, 6, 15)]@Series[object]
| [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]] |
+| date32:nullable | [2024-01-01, None]@date32[day]
| [datetime.date(2024, 1, 1),
None]@Series[object]
| [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
|
+| date32:empty | []@date32[day]
| []@Series[object]
| []@Series[datetime64[ns]]
|
+| date64:standard | [2024-01-01, 2024-06-15]@date64[ms]
| [datetime.date(2024, 1, 1),
datetime.date(2024, 6, 15)]@Series[object]
| [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]] |
+| date64:nullable | [2024-01-01, None]@date64[ms]
| [datetime.date(2024, 1, 1),
None]@Series[object]
| [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
|
+| date64:empty | []@date64[ms]
| []@Series[object]
| []@Series[datetime64[ns]]
|
| timestamp[s]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[s] | [Timestamp('2024-01-01
12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
|
| timestamp[s]:nullable | [2024-01-01 12:00:00, None]@timestamp[s]
| [Timestamp('2024-01-01 12:00:00'),
NaT]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
|
| timestamp[s]:empty | []@timestamp[s]
| []@Series[datetime64[ns]]
| []@Series[datetime64[ns]]
|
@@ -15,7 +21,6 @@
| timestamp[us,tz=UTC]:standard | [2024-01-01 12:00:00+00:00, 2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] | [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] | [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] |
| timestamp[us,tz=UTC]:nullable | [2024-01-01 12:00:00+00:00,
None]@timestamp[us, tz=UTC] | [Timestamp('2024-01-01
12:00:00+0000', tz='UTC'), NaT]@Series[datetime64[ns, UTC]]
| [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'),
NaT]@Series[datetime64[ns, UTC]] |
| timestamp[us,tz=UTC]:empty | []@timestamp[us, tz=UTC]
| []@Series[datetime64[ns, UTC]]
| []@Series[datetime64[ns, UTC]]
|
-| timestamp[s]:overflow | [2500-01-01 00:00:00]@timestamp[s]
| ERR@ArrowInvalid
| ERR@ArrowInvalid
|
| duration[s]:standard | [1 day, 0:00:00, 2:30:00]@duration[s]
| [Timedelta('1 days 00:00:00'), Timedelta('0
days 02:30:00')]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] |
| duration[s]:nullable | [1 day, 0:00:00, None]@duration[s]
| [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]]
|
| duration[s]:empty | []@duration[s]
| []@Series[timedelta64[ns]]
| []@Series[timedelta64[ns]]
|
@@ -28,13 +33,6 @@
| duration[ns]:standard | [1 days 00:00:00, 0 days
02:30:00]@duration[ns] | [Timedelta('1 days
00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] |
| duration[ns]:nullable | [1 days 00:00:00, None]@duration[ns]
| [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]]
|
| duration[ns]:empty | []@duration[ns]
| []@Series[timedelta64[ns]]
| []@Series[timedelta64[ns]]
|
-| duration[s]:overflow | [109500 days, 0:00:00]@duration[s]
| [Timedelta('-104004 days
+00:25:26.290448384')]@Series[timedelta64[ns]]
| [Timedelta('-104004 days
+00:25:26.290448384')]@Series[timedelta64[ns]]
|
-| date32:standard | [2024-01-01, 2024-06-15]@date32[day]
| [datetime.date(2024, 1, 1),
datetime.date(2024, 6, 15)]@Series[object]
| [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]] |
-| date32:nullable | [2024-01-01, None]@date32[day]
| [datetime.date(2024, 1, 1),
None]@Series[object]
| [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
|
-| date32:empty | []@date32[day]
| []@Series[object]
| []@Series[datetime64[ns]]
|
-| date64:standard | [2024-01-01, 2024-06-15]@date64[ms]
| [datetime.date(2024, 1, 1),
datetime.date(2024, 6, 15)]@Series[object]
| [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]] |
-| date64:nullable | [2024-01-01, None]@date64[ms]
| [datetime.date(2024, 1, 1),
None]@Series[object]
| [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
|
-| date64:empty | []@date64[ms]
| []@Series[object]
| []@Series[datetime64[ns]]
|
| time32[s]:standard | [12:30:00, 18:45:30]@time32[s]
| [datetime.time(12, 30), datetime.time(18, 45,
30)]@Series[object]
| [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object]
|
| time32[s]:nullable | [12:30:00, None]@time32[s]
| [datetime.time(12, 30), None]@Series[object]
| [datetime.time(12, 30), None]@Series[object]
|
| time32[s]:empty | []@time32[s]
| []@Series[object]
| []@Series[object]
|
@@ -47,6 +45,8 @@
| time64[ns]:standard | [12:30:00, 18:45:30]@time64[ns]
| [datetime.time(12, 30), datetime.time(18, 45,
30)]@Series[object]
| [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object]
|
| time64[ns]:nullable | [12:30:00, None]@time64[ns]
| [datetime.time(12, 30), None]@Series[object]
| [datetime.time(12, 30), None]@Series[object]
|
| time64[ns]:empty | []@time64[ns]
| []@Series[object]
| []@Series[object]
|
+| timestamp[s]:overflow | [2500-01-01 00:00:00]@timestamp[s]
| ERR@ArrowInvalid
| ERR@ArrowInvalid
|
+| duration[s]:overflow | [109500 days, 0:00:00]@duration[s]
| [Timedelta('-104004 days
+00:25:26.290448384')]@Series[timedelta64[ns]]
| [Timedelta('-104004 days
+00:25:26.290448384')]@Series[timedelta64[ns]]
|
| int64:standard | [0, 1, -1]@int64
| [0, 1, -1]@Series[int64]
| [0, 1, -1]@Series[int64]
|
| int64:nullable | [0, 1, None]@int64
| [0.0, 1.0, nan]@Series[float64]
| [0.0, 1.0, nan]@Series[float64]
|
| float64:standard | [0.0, 1.5, -1.5]@float64
| [0.0, 1.5, -1.5]@Series[float64]
| [0.0, 1.5, -1.5]@Series[float64]
|
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
index e3a9a0c0f5bd..a7e2b6e056d1 100644
---
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
@@ -27,6 +27,10 @@ arguments (no types_mapper, no self_destruct, etc.),
tracking:
- How null values are handled (NaN, None, NaT, etc.)
- Whether values are preserved correctly after conversion
+The shared ``_PyArrowToPandasTestBase`` holds the conversion helper and the
source-array
+inventory, split into type-family group methods that these and the non-default
tests
+reuse whole or by group.
+
## Golden File Cell Format
Each cell uses the value@type format:
@@ -71,23 +75,28 @@ if have_pyarrow:
import pyarrow as pa
[email protected](
- not have_pyarrow or not have_pandas or not have_numpy,
- pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
-)
-class PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
+class _PyArrowToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
"""
- Tests pa.Array.to_pandas() with default arguments via golden file
comparison.
+ Shared machinery for pa.Array.to_pandas() golden file tests.
- Covers all major Arrow types: integers, floats, bool, string, binary,
- decimal, date, timestamp, duration, time, null, and nested types.
- Each type is tested both without and with null values.
+ Holds the conversion helper and the source-array inventory, split into
type-family
+ group methods reusable whole (via ``_build_source_arrays``) or one at a
time.
+ Defines no ``test_*`` of its own.
"""
- def _build_source_arrays(self):
- """Build an ordered dict of named source PyArrow arrays for testing."""
- import pyarrow as pa
-
+ def _to_pandas_cell(self, arr, **to_pandas_kwargs) -> str:
+ """
+ Convert ``arr`` via ``to_pandas(**to_pandas_kwargs)`` and format the
+ result as a golden-file cell, returning ``ERR@<ExceptionClass>`` if the
+ conversion raises.
+ """
+ try:
+ return self.repr_value(arr.to_pandas(**to_pandas_kwargs),
max_len=0)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
+ def _numeric_sources(self):
+ """Integer, float, and boolean arrays."""
sources = {}
# =====================================================================
@@ -136,6 +145,12 @@ class
PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
sources["bool:nullable"] = pa.array([True, False, None], pa.bool_())
sources["bool:empty"] = pa.array([], pa.bool_())
+ return sources
+
+ def _string_binary_sources(self):
+ """String, binary, and decimal arrays."""
+ sources = {}
+
# =====================================================================
# String types
# =====================================================================
@@ -168,6 +183,12 @@ class
PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
)
sources["decimal128:empty"] = pa.array([], pa.decimal128(5, 2))
+ return sources
+
+ def _temporal_sources(self):
+ """Date, timestamp, duration, and time arrays."""
+ sources = {}
+
# =====================================================================
# Date types
# =====================================================================
@@ -226,6 +247,12 @@ class
PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
sources["time64[ns]:nullable"] = pa.array([t1, None], pa.time64("ns"))
sources["time64[ns]:empty"] = pa.array([], pa.time64("ns"))
+ return sources
+
+ def _nested_sources(self):
+ """Null, list, struct, map, and their nested combinations."""
+ sources = {}
+
# =====================================================================
# Null type
# =====================================================================
@@ -307,6 +334,12 @@ class
PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
pa.map_(pa.string(), pa.map_(pa.string(), pa.int64())),
)
+ return sources
+
+ def _dictionary_sources(self):
+ """Dictionary-encoded arrays."""
+ sources = {}
+
# =====================================================================
# Dictionary type
# =====================================================================
@@ -325,6 +358,33 @@ class
PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
return sources
+ def _build_source_arrays(self):
+ """Build an ordered dict of named source PyArrow arrays for testing."""
+ sources = {}
+ for group in [
+ self._numeric_sources(),
+ self._string_binary_sources(),
+ self._temporal_sources(),
+ self._nested_sources(),
+ self._dictionary_sources(),
+ ]:
+ sources.update(group)
+ return sources
+
+
[email protected](
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
+)
+class PyArrowArrayToPandasDefaultTests(_PyArrowToPandasTestBase):
+ """
+ Tests pa.Array.to_pandas() with default arguments via golden file
comparison.
+
+ Covers all major Arrow types: integers, floats, bool, string, binary,
+ decimal, date, timestamp, duration, time, null, and nested types.
+ Each type is tested both without and with null values.
+ """
+
def test_to_pandas_default(self):
"""Test pa.Array.to_pandas() with default arguments against golden
file."""
sources = self._build_source_arrays()
@@ -356,11 +416,7 @@ class
PyArrowArrayToPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase):
if col_name == "pyarrow array":
return self.repr_value(arr, max_len=0)
else:
- try:
- result = arr.to_pandas()
- return self.repr_value(result, max_len=0)
- except Exception as e:
- return f"ERR@{type(e).__name__}"
+ return self._to_pandas_cell(arr)
self.compare_or_generate_golden_matrix(
row_names=row_names,
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
index 7e0dc3a0823a..b619a443912b 100644
---
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
@@ -28,12 +28,10 @@ non-Arrow path. These tests record how each Arrow type
behaves under those
arguments so CI fails loudly if the behavior drifts across pandas/PyArrow/NumPy
upgrades.
-The shared ``_PyArrowToPandasTestBase`` holds the golden-file matrix driver and
-the per-cell conversion helper. Each concrete test class supplies its own
source
-arrays (the rows relevant to the argument under test) and one test method per
-argument combination (each producing its own golden file). New ``to_pandas``
-arguments (e.g. ``zero_copy_only``, ``integer_object_nulls``) can be added as
-additional classes without touching the existing ones.
+The shared ``_PyArrowToPandasTestBase`` (in
``test_pyarrow_arrow_to_pandas_default.py``)
+holds the conversion helper and the source-array inventory. Each class reuses
the
+inventory whole or by group and adds a test method per argument combination
(its own
+golden file), so a new argument is a new class rather than an edit to the
existing ones.
## Golden File Cell Format
@@ -71,11 +69,12 @@ from pyspark.testing.utils import (
pandas_requirement_message,
numpy_requirement_message,
)
-from pyspark.testing.goldenutils import GoldenFileTestMixin
-# Imported as a module, not `from ... import
PyArrowArrayToPandasDefaultTests`, so
-# that unittest does not collect and re-run the default test class from this
module.
-from pyspark.tests.upstream.pyarrow import test_pyarrow_arrow_to_pandas_default
+# Import the shared base (which defines no test_* methods), not a concrete
test class,
+# so that unittest does not collect and re-run the default file's tests here.
+from pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_default
import (
+ _PyArrowToPandasTestBase,
+)
if have_pandas:
import pandas as pd
@@ -85,107 +84,6 @@ if have_numpy:
import numpy as np
-class _PyArrowToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
- """
- Shared machinery for pa.Array.to_pandas() golden file tests.
-
- Concrete subclasses provide their own ``_build_source_arrays`` (the rows)
and
- one or more ``test_*`` methods that call
``compare_or_generate_golden_matrix``.
- This base defines no ``test_*`` methods, so it contributes no tests itself.
- """
-
- def _to_pandas_cell(self, arr, **to_pandas_kwargs) -> str:
- """
- Convert ``arr`` via ``to_pandas(**to_pandas_kwargs)`` and format the
- result as a golden-file cell, returning ``ERR@<ExceptionClass>`` if the
- conversion raises.
- """
- try:
- return self.repr_value(arr.to_pandas(**to_pandas_kwargs),
max_len=0)
- except Exception as e:
- return f"ERR@{type(e).__name__}"
-
- @staticmethod
- def _arrow_buffers(arrow_obj):
- """
- Every non-null buffer backing ``arrow_obj``.
-
- A ChunkedArray has no buffers of its own -- its data lives in its
chunks --
- so it is expanded first. ``None`` entries (e.g. an absent validity
bitmap)
- are skipped.
- """
- if isinstance(arrow_obj, pa.ChunkedArray):
- chunks = [arrow_obj.chunk(i) for i in range(arrow_obj.num_chunks)]
- else:
- chunks = [arrow_obj]
-
- buffers = []
- for chunk in chunks:
- for buffer in chunk.buffers():
- if buffer is not None:
- buffers.append(buffer)
- return buffers
-
- @classmethod
- def _verify_zero_copy(cls, arr, **to_pandas_kwargs) -> str:
- """
- Independently verify whether ``to_pandas`` reused ``arr``'s buffers,
- instead of trusting PyArrow's own ``zero_copy_only`` verdict.
-
- The check inspects whatever storage pandas returned rather than
assuming a
- numpy-backed Series, so it stays correct as pandas moves more dtypes to
- Arrow-backed storage.
-
- Returns ``"zero-copy"``, ``"partial-copy"``, ``"copied"``, or
- ``"ERR@<ExceptionClass>"``.
- """
- try:
- series = arr.to_pandas(**to_pandas_kwargs)
- except Exception as e:
- return f"ERR@{type(e).__name__}"
-
- backing_array = series.array
-
- # Arrow-backed result: compare buffer addresses, reading the stored
data
- # back through the public __arrow_array__ protocol. to_numpy() would
- # materialize a copy here and wrongly report no sharing. Keying on the
- # protocol rather than on ArrowDtype also covers dtypes that are
- # Arrow-backed without being ArrowDtype, such as pandas 3's string.
- #
- # The result has several buffers (validity, offsets, values) and can
borrow
- # some while allocating others, so count how many it borrowed rather
than
- # stopping at the first match. The result's buffers are the
denominator
- # because the question is what pandas had to allocate.
- if hasattr(backing_array, "__arrow_array__"):
- source_addresses = {buffer.address for buffer in
cls._arrow_buffers(arr)}
- stored_buffers = cls._arrow_buffers(pa.array(backing_array))
- borrowed = 0
- for buffer in stored_buffers:
- if buffer.address in source_addresses:
- borrowed += 1
- if borrowed == len(stored_buffers):
- return "zero-copy"
- return "partial-copy" if borrowed else "copied"
-
- # numpy-backed result: np.shares_memory accounts for slice offsets, so
it
- # is robust where raw address equality is not.
- pandas_values = series.to_numpy()
- for buffer in cls._arrow_buffers(arr):
- if np.shares_memory(np.frombuffer(buffer, dtype=np.uint8),
pandas_values):
- return "zero-copy"
-
- # Zero bytes cannot overlap, so an empty result is zero-copy if the
numpy
- # array still borrows the Arrow array (numpy's .base is its memory
owner).
- if len(arr) == 0:
- owner = pandas_values
- while (base := getattr(owner, "base", None)) is not None:
- if base is arr:
- return "zero-copy"
- owner = base
-
- return "copied"
-
-
@unittest.skipIf(
not have_pyarrow or not have_pandas or not have_numpy,
pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
@@ -206,41 +104,17 @@ class
PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
"""
def _build_source_arrays(self):
- """Build an ordered dict of named source PyArrow arrays for testing."""
- sources = {}
-
- # =====================================================================
- # Timestamp types (the primary target of coerce_temporal_nanoseconds)
- # =====================================================================
- dt1 = datetime.datetime(2024, 1, 1, 12, 0, 0)
- dt2 = datetime.datetime(2024, 6, 15, 18, 30, 0)
- for unit in ["s", "ms", "us", "ns"]:
- sources[f"timestamp[{unit}]:standard"] = pa.array([dt1, dt2],
pa.timestamp(unit))
- sources[f"timestamp[{unit}]:nullable"] = pa.array([dt1, None],
pa.timestamp(unit))
- sources[f"timestamp[{unit}]:empty"] = pa.array([],
pa.timestamp(unit))
- # Timestamp with timezone
- sources["timestamp[us,tz=UTC]:standard"] = pa.array(
- [dt1, dt2], pa.timestamp("us", tz="UTC")
- )
- sources["timestamp[us,tz=UTC]:nullable"] = pa.array(
- [dt1, None], pa.timestamp("us", tz="UTC")
- )
- sources["timestamp[us,tz=UTC]:empty"] = pa.array([],
pa.timestamp("us", tz="UTC"))
+ """
+ Reuse the base's temporal group (the types this argument targets),
then add
+ coercion-specific overflow rows and a few non-temporal controls.
+ """
+ sources = self._temporal_sources()
+
# Overflow: coercion to nanoseconds has a valid range (~1677-2262); a
# far-future second-resolution timestamp cannot fit and should error.
sources["timestamp[s]:overflow"] = pa.array(
[datetime.datetime(2500, 1, 1)], pa.timestamp("s")
)
-
- # =====================================================================
- # Duration types (also coerced to nanoseconds)
- # =====================================================================
- td1 = datetime.timedelta(days=1)
- td2 = datetime.timedelta(hours=2, minutes=30)
- for unit in ["s", "ms", "us", "ns"]:
- sources[f"duration[{unit}]:standard"] = pa.array([td1, td2],
pa.duration(unit))
- sources[f"duration[{unit}]:nullable"] = pa.array([td1, None],
pa.duration(unit))
- sources[f"duration[{unit}]:empty"] = pa.array([],
pa.duration(unit))
# Overflow: a duration beyond ~292 years exceeds the int64 nanosecond
# range. Unlike timestamp overflow (which raises), coercing it silently
# wraps around to a bogus value; this row pins that behavior.
@@ -248,43 +122,7 @@ class
PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
[datetime.timedelta(days=300 * 365)], pa.duration("s")
)
- # =====================================================================
- # Date types. With the default date_as_object=True, pandas yields an
- # object-dtype Series of datetime.date, so coerce_temporal_nanoseconds
- # has nothing to coerce; the "date_as_object=False" column forces the
- # numeric datetime64[ns] path where the argument actually takes effect.
- # =====================================================================
- d1 = datetime.date(2024, 1, 1)
- d2 = datetime.date(2024, 6, 15)
- sources["date32:standard"] = pa.array([d1, d2], pa.date32())
- sources["date32:nullable"] = pa.array([d1, None], pa.date32())
- sources["date32:empty"] = pa.array([], pa.date32())
- sources["date64:standard"] = pa.array([d1, d2], pa.date64())
- sources["date64:nullable"] = pa.array([d1, None], pa.date64())
- sources["date64:empty"] = pa.array([], pa.date64())
-
- # =====================================================================
- # Time types (control: pandas yields object-dtype datetime.time, no
- # native time-of-day dtype, so coerce_temporal_nanoseconds is a no-op)
- # =====================================================================
- t1 = datetime.time(12, 30, 0)
- t2 = datetime.time(18, 45, 30)
- sources["time32[s]:standard"] = pa.array([t1, t2], pa.time32("s"))
- sources["time32[s]:nullable"] = pa.array([t1, None], pa.time32("s"))
- sources["time32[s]:empty"] = pa.array([], pa.time32("s"))
- sources["time32[ms]:standard"] = pa.array([t1, t2], pa.time32("ms"))
- sources["time32[ms]:nullable"] = pa.array([t1, None], pa.time32("ms"))
- sources["time32[ms]:empty"] = pa.array([], pa.time32("ms"))
- sources["time64[us]:standard"] = pa.array([t1, t2], pa.time64("us"))
- sources["time64[us]:nullable"] = pa.array([t1, None], pa.time64("us"))
- sources["time64[us]:empty"] = pa.array([], pa.time64("us"))
- sources["time64[ns]:standard"] = pa.array([t1, t2], pa.time64("ns"))
- sources["time64[ns]:nullable"] = pa.array([t1, None], pa.time64("ns"))
- sources["time64[ns]:empty"] = pa.array([], pa.time64("ns"))
-
- # =====================================================================
- # Non-temporal controls (unaffected by coerce_temporal_nanoseconds)
- # =====================================================================
+ # Non-temporal controls (unaffected by coerce_temporal_nanoseconds).
sources["int64:standard"] = pa.array([0, 1, -1], pa.int64())
sources["int64:nullable"] = pa.array([0, 1, None], pa.int64())
sources["float64:standard"] = pa.array([0.0, 1.5, -1.5], pa.float64())
@@ -395,19 +233,18 @@ class
PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
def _build_source_arrays(self):
"""
- Reuse every row of the sibling default test, then add the layout
variants
- that only matter for zero-copy.
+ Reuse the base's full inventory, then add the layout variants that only
+ matter for zero-copy.
- Sharing the row set keeps both golden files pinning the same types, and
- means a type added to the default test is covered here automatically.
+ Sharing the row set keeps this and the default golden pinning the same
types,
+ and a type added to the base is covered here automatically.
A slice still views a contiguous no-null region, so it stays zero-copy
even
though its data starts partway into the parent buffer. A single chunk
is
zero-copy, but multiple chunks must be concatenated into one contiguous
numpy buffer -- a copy. Multi-chunk is the common shape in real
PySpark,
where each partition arrives as its own chunk.
"""
- default_tests =
test_pyarrow_arrow_to_pandas_default.PyArrowArrayToPandasDefaultTests
- sources = default_tests._build_source_arrays(self)
+ sources = super()._build_source_arrays()
sources["int64:sliced"] = pa.array(list(range(10)),
pa.int64()).slice(2, 3)
sources["int64:sliced-with-null"] = pa.array([1, 2, None, 4, 5],
pa.int64()).slice(1, 3)
@@ -418,6 +255,86 @@ class
PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
return sources
+ @staticmethod
+ def _arrow_buffers(arrow_obj):
+ """
+ Every non-null buffer backing ``arrow_obj``.
+
+ A ChunkedArray has no buffers of its own -- its data lives in its
chunks --
+ so it is expanded first. ``None`` entries (e.g. an absent validity
bitmap)
+ are skipped.
+ """
+ if isinstance(arrow_obj, pa.ChunkedArray):
+ chunks = [arrow_obj.chunk(i) for i in range(arrow_obj.num_chunks)]
+ else:
+ chunks = [arrow_obj]
+
+ buffers = []
+ for chunk in chunks:
+ for buffer in chunk.buffers():
+ if buffer is not None:
+ buffers.append(buffer)
+ return buffers
+
+ @classmethod
+ def _verify_zero_copy(cls, arr, **to_pandas_kwargs) -> str:
+ """
+ Independently verify whether ``to_pandas`` reused ``arr``'s buffers,
+ instead of trusting PyArrow's own ``zero_copy_only`` verdict.
+
+ The check inspects whatever storage pandas returned rather than
assuming a
+ numpy-backed Series, so it stays correct as pandas moves more dtypes to
+ Arrow-backed storage.
+
+ Returns ``"zero-copy"``, ``"partial-copy"``, ``"copied"``, or
+ ``"ERR@<ExceptionClass>"``.
+ """
+ try:
+ series = arr.to_pandas(**to_pandas_kwargs)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
+ backing_array = series.array
+
+ # Arrow-backed result: compare buffer addresses, reading the stored
data
+ # back through the public __arrow_array__ protocol. to_numpy() would
+ # materialize a copy here and wrongly report no sharing. Keying on the
+ # protocol rather than on ArrowDtype also covers dtypes that are
+ # Arrow-backed without being ArrowDtype, such as pandas 3's string.
+ #
+ # The result has several buffers (validity, offsets, values) and can
borrow
+ # some while allocating others, so count how many it borrowed rather
than
+ # stopping at the first match. The result's buffers are the
denominator
+ # because the question is what pandas had to allocate.
+ if hasattr(backing_array, "__arrow_array__"):
+ source_addresses = {buffer.address for buffer in
cls._arrow_buffers(arr)}
+ stored_buffers = cls._arrow_buffers(pa.array(backing_array))
+ borrowed = 0
+ for buffer in stored_buffers:
+ if buffer.address in source_addresses:
+ borrowed += 1
+ if borrowed == len(stored_buffers):
+ return "zero-copy"
+ return "partial-copy" if borrowed else "copied"
+
+ # numpy-backed result: np.shares_memory accounts for slice offsets, so
it
+ # is robust where raw address equality is not.
+ pandas_values = series.to_numpy()
+ for buffer in cls._arrow_buffers(arr):
+ if np.shares_memory(np.frombuffer(buffer, dtype=np.uint8),
pandas_values):
+ return "zero-copy"
+
+ # Zero bytes cannot overlap, so an empty result is zero-copy if the
numpy
+ # array still borrows the Arrow array (numpy's .base is its memory
owner).
+ if len(arr) == 0:
+ owner = pandas_values
+ while (base := getattr(owner, "base", None)) is not None:
+ if base is arr:
+ return "zero-copy"
+ owner = base
+
+ return "copied"
+
# Output column for the default zero_copy_only=False: PyArrow silently
copies
# when a view is not possible, so this always succeeds and records the
dtype.
COL_ZERO_COPY_OFF = "zero_copy_only=False"
@@ -575,13 +492,11 @@ class
PyArrowArrayToPandasIntegerObjectNullsTests(_PyArrowToPandasTestBase):
def _build_source_arrays(self):
"""
- Reuse every row of the sibling default test, then add the integer
variants
- that only matter for this argument: the shared rows' nullable values
are
- small enough to survive ``float64``, and their nested rows are missing
whole
- sub-lists rather than a single integer inside one.
+ Reuse the base's full inventory, then add the integer variants it does
not
+ reach: its nullable values are small enough to survive ``float64``,
and its
+ nested rows are missing whole sub-lists rather than a single integer
inside one.
"""
- default_tests =
test_pyarrow_arrow_to_pandas_default.PyArrowArrayToPandasDefaultTests
- sources = default_tests._build_source_arrays(self)
+ sources = super()._build_source_arrays()
# Each width's min and max alongside a null. The shared rows use
small values,
# which float64 represents exactly; only at 64 bits does the range
exceed its
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
