This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new b0200d1bae4d [SPARK-58720][PYTHON][TEST] Add tests for
pa.Table.to_pandas
b0200d1bae4d is described below
commit b0200d1bae4d877e3f8814dfb4c7ad7fe9aaa03d
Author: Spenser Sun <[email protected]>
AuthorDate: Thu Aug 13 15:37:41 2026 +0800
[SPARK-58720][PYTHON][TEST] Add tests for pa.Table.to_pandas
### What changes were proposed in this pull request?
Adds golden-file drift tests for `pa.Table.to_pandas()` under the
SPARK-54936 umbrella. New file
`python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py` with
two classes:
- `PyArrowTableToPandasDefaultTests` -- multi-column DataFrame assembly and
the empty-table edges (0-column and 0-row) under a bare `to_pandas()`.
- `PyArrowTableToPandasCoerceTemporalTests` --
`coerce_temporal_nanoseconds=True` with `date_as_object` at its default and
`=False`, plus timestamp/duration overflow rows.
Spark calls `pa.Table.to_pandas()` in only two places: the 0-column early
return in `python/pyspark/sql/pandas/conversion.py` (which passes
`coerce_temporal_nanoseconds` and `date_as_object`) and a bare whole-Table
conversion of a SQL command result in
`python/pyspark/sql/connect/client/core.py`. Those are the only arguments Spark
ever passes to a Table, which is why the tests cover just the bare call and
those two flags; every other `to_pandas` flag
(`self_destruct`/`split_blocks`/`u [...]
`Table.to_pandas()` returns a `DataFrame` while
`Array`/`ChunkedArray.to_pandas()` return a `Series`, so the golden files have
to render two shapes:
- The input-table anchor column needs a `pa.Table` renderer, so
`repr_value` in the shared `python/pyspark/testing/goldenutils.py` gains a
`pa.Table` branch (`repr_arrow_table_value`). Without it a `pa.Table` fell
through to the generic `str(value)` formatter, which is verbose and, being
PyArrow's own `__repr__`, could change across versions and produce spurious
golden diffs. The new helper renders each column with PyArrow's stable scalar
formatting plus the Arrow schema. This is the [...]
- The converted-DataFrame result column is formatted by a small helper
local to the test (`_repr_dataframe`), which serializes per-column via
`tolist()`. It deliberately does not go through the shared `repr_pandas_value`
(which uses `to_json()`): `to_json()` defaults to an epoch-nanosecond date
format, so it raises `OverflowError` on out-of-nanosecond-range datetimes --
e.g. the year-9999 date rows, which under the default `date_as_object=True`
come back as an object column of Python [...]
### Why are the changes needed?
Part of SPARK-54936, which pins upstream PyArrow / pandas conversion
behavior so that a library version bump changing it fails loudly in CI instead
of silently returning wrong data. `pa.Table.to_pandas` is the last `to_pandas`
conversion Spark relies on without golden coverage (`Array` and `ChunkedArray`
are already covered; a `RecordBatch` is converted column-by-column rather than
via `RecordBatch.to_pandas`).
### Does this PR introduce _any_ user-facing change?
No. Test-only, plus one test-infrastructure helper in `goldenutils.py` (the
`pa.Table` renderer).
### How was this patch tested?
New golden tests, validated across a PyArrow 18-25 x pandas 2/3 sweep (16
combinations), with version-legitimate differences recorded as
`LooseVersion`-guarded `overrides`. The sweep is Linux/x86 only, so an ARM
follow-up may be needed once CI runs the full matrix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #57939 from Spenserrrr/table-to-pandas-tests.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit df1646c0ab325a631e6b00bf67be18d5955cefe2)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
dev/sparktestsupport/modules.py | 1 +
python/pyspark/testing/goldenutils.py | 27 ++
.../pyarrow/golden_pyarrow_table_to_pandas.csv | 13 +
.../pyarrow/golden_pyarrow_table_to_pandas.md | 14 +
...den_pyarrow_table_to_pandas_coerce_temporal.csv | 8 +
...lden_pyarrow_table_to_pandas_coerce_temporal.md | 9 +
.../pyarrow/test_pyarrow_table_to_pandas.py | 343 +++++++++++++++++++++
7 files changed, 415 insertions(+)
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 445642e59618..f5edb21c09fb 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -545,6 +545,7 @@ pyspark_core = Module(
"pyspark.tests.upstream.pyarrow.test_pyarrow_ignore_timezone",
"pyspark.tests.upstream.pyarrow.test_pyarrow_scalar_type_coercion",
"pyspark.tests.upstream.pyarrow.test_pyarrow_scalar_type_inference",
+ "pyspark.tests.upstream.pyarrow.test_pyarrow_table_to_pandas",
"pyspark.tests.upstream.pyarrow.test_pyarrow_type_coercion",
],
)
diff --git a/python/pyspark/testing/goldenutils.py
b/python/pyspark/testing/goldenutils.py
index d0a191e9fe9a..95b7da956ff3 100644
--- a/python/pyspark/testing/goldenutils.py
+++ b/python/pyspark/testing/goldenutils.py
@@ -337,6 +337,30 @@ class GoldenFileTestMixin:
v_str = v_str[:max_len]
return f"{v_str}@{cls.repr_type(value.type)}"
+ @classmethod
+ def repr_arrow_table_value(cls, value: Any, max_len: int = 32) -> str:
+ """
+ Format a PyArrow Table for golden file.
+
+ Renders each column with PyArrow's scalar formatting (as
``repr_arrow_value``
+ does for an Array), keyed by column name, plus the Arrow schema.
+
+ Returns
+ -------
+ str
+ "{col: [val1, val2, None], ...}@Table[name: type, ...]"
+ """
+ columns = []
+ for name, column in zip(value.column_names, value.columns):
+ # Escape NULL bytes so the value can be safely stored in CSV files.
+ elements = [str(scalar).replace("\x00", "\\0") for scalar in
column]
+ columns.append(f"{name}: [" + ", ".join(elements) + "]")
+ v_str = "{" + ", ".join(columns) + "}"
+ if max_len > 0:
+ v_str = v_str[:max_len]
+ schema = ", ".join(f"{f.name}: {cls.repr_type(f.type)}" for f in
value.schema)
+ return f"{v_str}@Table[{schema}]"
+
@classmethod
def repr_pandas_value(cls, value: Any, max_len: int = 32) -> str:
"""
@@ -404,6 +428,7 @@ class GoldenFileTestMixin:
based on the value's type.
- PyArrow Array/ChunkedArray -> repr_arrow_value
+ - PyArrow Table -> repr_arrow_table_value
- pandas DataFrame -> repr_pandas_value
- numpy ndarray -> repr_numpy_value
- Everything else -> repr_python_value
@@ -422,6 +447,8 @@ class GoldenFileTestMixin:
"""
if have_pyarrow and isinstance(value, (pa.Array, pa.ChunkedArray)):
return cls.repr_arrow_value(value, max_len)
+ if have_pyarrow and isinstance(value, pa.Table):
+ return cls.repr_arrow_table_value(value, max_len)
if have_pandas and isinstance(value, pd.DataFrame):
return cls.repr_pandas_value(value, max_len)
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas.csv
new file mode 100644
index 000000000000..bdb4eb03b486
--- /dev/null
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas.csv
@@ -0,0 +1,13 @@
+test case pyarrow table pandas dataframe
+empty:0-columns {}@Table[] {}@Dataframe[]
+empty:columns-no-rows {i: [], s: []}@Table[i: int64, s: string] {'i':
[], 's': []}@Dataframe[i int64, s object]
+single-column {i: [1, 2, None]}@Table[i: int64] {'i': [1.0, 2.0,
nan]}@Dataframe[i float64]
+single-column:string {s: [hello, world, None]}@Table[s: string] {'s':
['hello', 'world', None]}@Dataframe[s object]
+multi-column:mixed-scalar {i: [1, 2, None], s: [a, b, None], f: [1.5,
2.5, 3.5], b: [True, False, None]}@Table[i: int64, s: string, f: float64, b:
bool] {'i': [1.0, 2.0, nan], 's': ['a', 'b', None], 'f': [1.5, 2.5, 3.5], 'b':
[True, False, None]}@Dataframe[i float64, s object, f float64, b object]
+multi-column:all-null {i: [None, None], s: [None, None]}@Table[i: int64, s:
string] {'i': [nan, nan], 's': [None, None]}@Dataframe[i float64, s object]
+multi-column:nested {lst: [[1, 2], [3], None], st: [[('x', 1)], None,
[('x', 3)]]}@Table[lst: list<item: int64>, st: struct<x: int64>] {'lst':
[array([1, 2]), array([3]), None], 'st': [{'x': 1}, None, {'x':
3}]}@Dataframe[lst object, st object]
+temporal:timestamp {ts: [2020-01-01 05:30:00, 2021-06-15
23:59:00]}@Table[ts: timestamp[us]] {'ts': [Timestamp('2020-01-01
05:30:00'), Timestamp('2021-06-15 23:59:00')]}@Dataframe[ts datetime64[us]]
+temporal:date32 {d: [2020-01-01, 2021-06-15]}@Table[d: date32[day]]
{'d': [datetime.date(2020, 1, 1), datetime.date(2021, 6, 15)]}@Dataframe[d
object]
+temporal:date64 {d: [2020-01-01, 2021-06-15]}@Table[d: date64[ms]]
{'d': [datetime.date(2020, 1, 1), datetime.date(2021, 6, 15)]}@Dataframe[d
object]
+temporal:date32-far-future {d: [9999-12-31]}@Table[d: date32[day]] {'d':
[datetime.date(9999, 12, 31)]}@Dataframe[d object]
+temporal:multi-column-mix {ts: [2020-01-01 05:30:00, 2020-01-01
05:30:00], d: [2020-01-01, 9999-12-31], i: [1, 2]}@Table[ts: timestamp[us], d:
date32[day], i: int64] {'ts': [Timestamp('2020-01-01 05:30:00'),
Timestamp('2020-01-01 05:30:00')], 'd': [datetime.date(2020, 1, 1),
datetime.date(9999, 12, 31)], 'i': [1, 2]}@Dataframe[ts datetime64[us], d
object, i int64]
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas.md
new file mode 100644
index 000000000000..cafeca290702
--- /dev/null
+++ b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas.md
@@ -0,0 +1,14 @@
+| test case | pyarrow table
| pandas dataframe
|
+|----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| empty:0-columns | {}@Table[]
| {}@Dataframe[]
|
+| empty:columns-no-rows | {i: [], s: []}@Table[i: int64, s: string]
| {'i': [], 's': []}@Dataframe[i int64, s object]
|
+| single-column | {i: [1, 2, None]}@Table[i: int64]
| {'i': [1.0, 2.0, nan]}@Dataframe[i float64]
|
+| single-column:string | {s: [hello, world, None]}@Table[s: string]
| {'s': ['hello', 'world', None]}@Dataframe[s object]
|
+| multi-column:mixed-scalar | {i: [1, 2, None], s: [a, b, None], f: [1.5,
2.5, 3.5], b: [True, False, None]}@Table[i: int64, s: string, f: float64, b:
bool] | {'i': [1.0, 2.0, nan], 's': ['a', 'b', None], 'f': [1.5,
2.5, 3.5], 'b': [True, False, None]}@Dataframe[i float64, s object, f float64,
b object] |
+| multi-column:all-null | {i: [None, None], s: [None, None]}@Table[i:
int64, s: string]
| {'i': [nan, nan], 's': [None, None]}@Dataframe[i float64, s
object]
|
+| multi-column:nested | {lst: [[1, 2], [3], None], st: [[('x', 1)],
None, [('x', 3)]]}@Table[lst: list<item: int64>, st: struct<x: int64>]
| {'lst': [array([1, 2]), array([3]), None], 'st': [{'x': 1},
None, {'x': 3}]}@Dataframe[lst object, st object]
|
+| temporal:timestamp | {ts: [2020-01-01 05:30:00, 2021-06-15
23:59:00]}@Table[ts: timestamp[us]]
| {'ts': [Timestamp('2020-01-01 05:30:00'),
Timestamp('2021-06-15 23:59:00')]}@Dataframe[ts datetime64[us]]
|
+| temporal:date32 | {d: [2020-01-01, 2021-06-15]}@Table[d:
date32[day]]
| {'d': [datetime.date(2020, 1, 1), datetime.date(2021, 6,
15)]}@Dataframe[d object]
|
+| temporal:date64 | {d: [2020-01-01, 2021-06-15]}@Table[d:
date64[ms]]
| {'d': [datetime.date(2020, 1, 1), datetime.date(2021, 6,
15)]}@Dataframe[d object]
|
+| temporal:date32-far-future | {d: [9999-12-31]}@Table[d: date32[day]]
| {'d': [datetime.date(9999, 12, 31)]}@Dataframe[d object]
|
+| temporal:multi-column-mix | {ts: [2020-01-01 05:30:00, 2020-01-01
05:30:00], d: [2020-01-01, 9999-12-31], i: [1, 2]}@Table[ts: timestamp[us], d:
date32[day], i: int64] | {'ts': [Timestamp('2020-01-01 05:30:00'),
Timestamp('2020-01-01 05:30:00')], 'd': [datetime.date(2020, 1, 1),
datetime.date(9999, 12, 31)], 'i': [1, 2]}@Dataframe[ts datetime64[us], d
object, i int64] |
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas_coerce_temporal.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas_coerce_temporal.csv
new file mode 100644
index 000000000000..706bd3c5bff1
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas_coerce_temporal.csv
@@ -0,0 +1,8 @@
+test case pyarrow table pandas dataframe pandas dataframe
(date_as_object=False)
+temporal:timestamp {ts: [2020-01-01 05:30:00, 2021-06-15
23:59:00]}@Table[ts: timestamp[us]] {'ts': [Timestamp('2020-01-01
05:30:00'), Timestamp('2021-06-15 23:59:00')]}@Dataframe[ts datetime64[ns]]
{'ts': [Timestamp('2020-01-01 05:30:00'), Timestamp('2021-06-15
23:59:00')]}@Dataframe[ts datetime64[ns]]
+temporal:date32 {d: [2020-01-01, 2021-06-15]}@Table[d: date32[day]]
{'d': [datetime.date(2020, 1, 1), datetime.date(2021, 6, 15)]}@Dataframe[d
object] {'d': [Timestamp('2020-01-01 00:00:00'), Timestamp('2021-06-15
00:00:00')]}@Dataframe[d datetime64[ns]]
+temporal:date64 {d: [2020-01-01, 2021-06-15]}@Table[d: date64[ms]]
{'d': [datetime.date(2020, 1, 1), datetime.date(2021, 6, 15)]}@Dataframe[d
object] {'d': [Timestamp('2020-01-01 00:00:00'), Timestamp('2021-06-15
00:00:00')]}@Dataframe[d datetime64[ns]]
+temporal:date32-far-future {d: [9999-12-31]}@Table[d: date32[day]] {'d':
[datetime.date(9999, 12, 31)]}@Dataframe[d object] {'d':
[Timestamp('1816-03-29 05:56:08.066277376')]}@Dataframe[d datetime64[ns]]
+temporal:multi-column-mix {ts: [2020-01-01 05:30:00, 2020-01-01
05:30:00], d: [2020-01-01, 9999-12-31], i: [1, 2]}@Table[ts: timestamp[us], d:
date32[day], i: int64] {'ts': [Timestamp('2020-01-01 05:30:00'),
Timestamp('2020-01-01 05:30:00')], 'd': [datetime.date(2020, 1, 1),
datetime.date(9999, 12, 31)], 'i': [1, 2]}@Dataframe[ts datetime64[ns], d
object, i int64] {'ts': [Timestamp('2020-01-01 05:30:00'),
Timestamp('2020-01-01 05:30:00')], 'd': [Timestamp('2020-01-01 00:00:00'),
Timestam [...]
+temporal:timestamp-overflow {ts: [2500-01-01 00:00:00]}@Table[ts:
timestamp[s]] ERR@ArrowInvalid ERR@ArrowInvalid
+temporal:duration-overflow {dur: [109500 days, 0:00:00]}@Table[dur:
duration[s]] {'dur': [Timedelta('-104004 days
+00:25:26.290448384')]}@Dataframe[dur timedelta64[ns]] {'dur':
[Timedelta('-104004 days +00:25:26.290448384')]}@Dataframe[dur timedelta64[ns]]
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas_coerce_temporal.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas_coerce_temporal.md
new file mode 100644
index 000000000000..7730a2ec0ab7
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_table_to_pandas_coerce_temporal.md
@@ -0,0 +1,9 @@
+| test case | pyarrow table
| pandas dataframe
| pandas dataframe
(date_as_object=False)
[...]
+|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------
[...]
+| temporal:timestamp | {ts: [2020-01-01 05:30:00, 2021-06-15
23:59:00]}@Table[ts: timestamp[us]]
| {'ts': [Timestamp('2020-01-01 05:30:00'),
Timestamp('2021-06-15 23:59:00')]}@Dataframe[ts datetime64[ns]]
| {'ts': [Timestamp('2020-01-01 05:30:00'), Timestamp('2021-06-15
23:59:00')]}@Dataframe[ts datetime64[ns]] [...]
+| temporal:date32 | {d: [2020-01-01, 2021-06-15]}@Table[d:
date32[day]]
| {'d': [datetime.date(2020, 1, 1), datetime.date(2021, 6,
15)]}@Dataframe[d object]
| {'d':
[Timestamp('2020-01-01 00:00:00'), Timestamp('2021-06-15
00:00:00')]}@Dataframe[d datetime64[ns]] [...]
+| temporal:date64 | {d: [2020-01-01, 2021-06-15]}@Table[d:
date64[ms]]
| {'d': [datetime.date(2020, 1, 1), datetime.date(2021, 6,
15)]}@Dataframe[d object]
| {'d':
[Timestamp('2020-01-01 00:00:00'), Timestamp('2021-06-15
00:00:00')]}@Dataframe[d datetime64[ns]] [...]
+| temporal:date32-far-future | {d: [9999-12-31]}@Table[d: date32[day]]
| {'d': [datetime.date(9999, 12, 31)]}@Dataframe[d object]
| {'d':
[Timestamp('1816-03-29 05:56:08.066277376')]}@Dataframe[d datetime64[ns]]
[...]
+| temporal:multi-column-mix | {ts: [2020-01-01 05:30:00, 2020-01-01
05:30:00], d: [2020-01-01, 9999-12-31], i: [1, 2]}@Table[ts: timestamp[us], d:
date32[day], i: int64] | {'ts': [Timestamp('2020-01-01 05:30:00'),
Timestamp('2020-01-01 05:30:00')], 'd': [datetime.date(2020, 1, 1),
datetime.date(9999, 12, 31)], 'i': [1, 2]}@Dataframe[ts datetime64[ns], d
object, i int64] | {'ts': [Timestamp('2020-01-01 05:30:00'),
Timestamp('2020-01-01 05:30:00')], 'd': [Timestamp('2020-01-01 00:00:00') [...]
+| temporal:timestamp-overflow | {ts: [2500-01-01 00:00:00]}@Table[ts:
timestamp[s]]
| ERR@ArrowInvalid
|
ERR@ArrowInvalid
[...]
+| temporal:duration-overflow | {dur: [109500 days, 0:00:00]}@Table[dur:
duration[s]]
| {'dur': [Timedelta('-104004 days
+00:25:26.290448384')]}@Dataframe[dur timedelta64[ns]]
| {'dur': [Timedelta('-104004 days
+00:25:26.290448384')]}@Dataframe[dur timedelta64[ns]]
[...]
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
new file mode 100644
index 000000000000..42977bea5bf5
--- /dev/null
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
@@ -0,0 +1,343 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Tests for PyArrow Table.to_pandas() using golden file comparison.
+
+Unlike Array/ChunkedArray.to_pandas() (which returns a Series and is covered by
+test_pyarrow_arrow_to_pandas_{default,non_default}.py), Table.to_pandas()
returns a
+DataFrame. Its per-column conversion matches the Array tests, so this file
pins the
+genuinely Table-specific behavior instead: multi-column DataFrame assembly and
the
+empty-table edges (0 columns / 0 rows). Spark calls Table.to_pandas() at
+python/pyspark/sql/pandas/conversion.py:255 (the 0-column path) and, in Spark
Connect,
+at python/pyspark/sql/connect/client/core.py:1423 (a bare whole-Table
conversion).
+
+## Golden File Cell Format
+
+Each cell uses the value@type format:
+- pyarrow Table: "{col: [val1, val2, None], ...}@Table[name: type, ...]"
+- pandas DataFrame: "{col: [values], ...}@Dataframe[name dtype, ...]"
+- Error: "ERR@ExceptionClassName"
+
+Values are formatted via tolist() for stable, Python-native representation.
+
+## Regenerating Golden Files
+
+Set SPARK_GENERATE_GOLDEN_FILES=1 before running:
+
+ SPARK_GENERATE_GOLDEN_FILES=1 python -m pytest \\
+ python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
+"""
+
+import datetime
+import unittest
+
+from pyspark.loose_version import LooseVersion
+from pyspark.testing.utils import (
+ have_pyarrow,
+ have_pandas,
+ pyarrow_requirement_message,
+ pandas_requirement_message,
+)
+from pyspark.testing.goldenutils import GoldenFileTestMixin
+
+if have_pandas:
+ import pandas as pd
+if have_pyarrow:
+ import pyarrow as pa
+
+
+class _PyArrowTableToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
+ """
+ Shared machinery for pa.Table.to_pandas() golden file tests.
+
+ Holds the conversion helper and the source-table inventory, split into
group
+ methods that these and the (temporal-flag) tests reuse. Defines no
``test_*`` of
+ its own.
+ """
+
+ @staticmethod
+ def _repr_dataframe(df) -> str:
+ """
+ Format a pandas DataFrame result as a golden-file cell: per-column
tolist()
+ for a stable, Python-native representation, mirroring goldenutils'
+ ``repr_pandas_series_value`` for the Series case.
+
+ This deliberately does NOT go through
``repr_value``/``repr_pandas_value``
+ (which use ``to_json``). With the default date_as_object=True a
far-future
+ date (year 9999) comes back as an object column of Python
``datetime.date``
+ objects, and ``DataFrame.to_json`` overflows converting each to an
epoch
+ nanosecond int (OverflowError); tolist() returns the objects as-is, so
+ nothing overflows.
+ """
+ body = str({name: col.tolist() for name, col in
df.items()}).replace("\n", " ")
+ schema = ", ".join(f"{t} {d.name}" for t, d in df.dtypes.items())
+ return f"{body}@Dataframe[{schema}]"
+
+ def _to_pandas_cell(self, table, **to_pandas_kwargs) -> str:
+ """
+ Convert ``table`` via ``to_pandas(**to_pandas_kwargs)`` and format the
result
+ as a golden-file cell, returning ``ERR@<ExceptionClass>`` if the
conversion
+ raises. Only the conversion is guarded: a formatting error is a test
bug, not
+ a conversion signal, so it propagates instead of masquerading as
``ERR@``.
+ """
+ try:
+ pdf = table.to_pandas(**to_pandas_kwargs)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+ return self._repr_dataframe(pdf)
+
+ def _structural_tables(self):
+ """Assembly shapes and empty edges (no temporal coercion involved)."""
+ sources = {}
+
+ # =====================================================================
+ # Empty edges (shapes an Array/ChunkedArray cannot be)
+ # =====================================================================
+ sources["empty:0-columns"] = pa.table({})
+ sources["empty:columns-no-rows"] = pa.table(
+ {"i": pa.array([], pa.int64()), "s": pa.array([], pa.string())}
+ )
+
+ # =====================================================================
+ # Multi-column assembly
+ # =====================================================================
+ sources["single-column"] = pa.table({"i": pa.array([1, 2, None],
pa.int64())})
+ sources["single-column:string"] = pa.table(
+ {"s": pa.array(["hello", "world", None], pa.string())}
+ )
+ sources["multi-column:mixed-scalar"] = pa.table(
+ {
+ "i": pa.array([1, 2, None], pa.int64()),
+ "s": pa.array(["a", "b", None], pa.string()),
+ "f": pa.array([1.5, 2.5, 3.5], pa.float64()),
+ "b": pa.array([True, False, None], pa.bool_()),
+ }
+ )
+ sources["multi-column:all-null"] = pa.table(
+ {
+ "i": pa.array([None, None], pa.int64()),
+ "s": pa.array([None, None], pa.string()),
+ }
+ )
+ sources["multi-column:nested"] = pa.table(
+ {
+ "lst": pa.array([[1, 2], [3], None], pa.list_(pa.int64())),
+ "st": pa.array([{"x": 1}, None, {"x": 3}], pa.struct([("x",
pa.int64())])),
+ }
+ )
+
+ return sources
+
+ def _temporal_tables(self):
+ """
+ Tables with temporal columns. Shared with the temporal-flag tests,
where
+ coerce_temporal_nanoseconds / date_as_object actually move cells.
+ """
+ sources = {}
+
+ sources["temporal:timestamp"] = pa.table(
+ {
+ "ts": pa.array(
+ [
+ datetime.datetime(2020, 1, 1, 5, 30),
+ datetime.datetime(2021, 6, 15, 23, 59),
+ ],
+ pa.timestamp("us"),
+ )
+ }
+ )
+ sources["temporal:date32"] = pa.table(
+ {"d": pa.array([datetime.date(2020, 1, 1), datetime.date(2021, 6,
15)], pa.date32())}
+ )
+ sources["temporal:date64"] = pa.table(
+ {"d": pa.array([datetime.date(2020, 1, 1), datetime.date(2021, 6,
15)], pa.date64())}
+ )
+ sources["temporal:date32-far-future"] = pa.table(
+ {"d": pa.array([datetime.date(9999, 12, 31)], pa.date32())}
+ )
+ sources["temporal:multi-column-mix"] = pa.table(
+ {
+ "ts": pa.array([datetime.datetime(2020, 1, 1, 5, 30)] * 2,
pa.timestamp("us")),
+ "d": pa.array(
+ [datetime.date(2020, 1, 1), datetime.date(9999, 12, 31)],
pa.date32()
+ ),
+ "i": pa.array([1, 2], pa.int64()),
+ }
+ )
+
+ return sources
+
+ def _build_all_tables(self):
+ """Build an ordered dict of named source PyArrow tables for testing."""
+ sources = {}
+ for group in [
+ self._structural_tables(),
+ self._temporal_tables(),
+ ]:
+ sources.update(group)
+ return sources
+
+
[email protected](
+ not have_pyarrow or not have_pandas,
+ pyarrow_requirement_message or pandas_requirement_message,
+)
+class PyArrowTableToPandasDefaultTests(_PyArrowTableToPandasTestBase):
+ """
+ Tests pa.Table.to_pandas() with default arguments via golden file
comparison.
+
+ Pins multi-column DataFrame assembly (combined schema, column names,
coexisting
+ dtypes) and the empty-table edges.
+ """
+
+ def test_to_pandas_default(self):
+ """Test pa.Table.to_pandas() with default arguments against golden
file."""
+ sources = self._build_all_tables()
+ row_names = list(sources.keys())
+ col_names = ["pyarrow table", "pandas dataframe"]
+
+ overrides = {}
+ pandas_3_plus = LooseVersion(pd.__version__) >= LooseVersion("3.0.0")
+ pyarrow_19_plus = LooseVersion(pa.__version__) >=
LooseVersion("19.0.0")
+
+ # On pandas 3 with PyArrow 19+, Arrow string columns convert to the
"str"
+ # dtype (missing values become nan) rather than object dtype (None).
This
+ # boundary differs from the Array path
(test_pyarrow_arrow_to_pandas_*),
+ # whose empty string columns switch only at PyArrow 24.
+ if pandas_3_plus and pyarrow_19_plus:
+ overrides.update(
+ {
+ ("single-column:string", "pandas dataframe"): (
+ "{'s': ['hello', 'world', nan]}@Dataframe[s str]"
+ )
+ }
+ )
+ overrides.update(
+ {
+ ("multi-column:mixed-scalar", "pandas dataframe"): (
+ "{'i': [1.0, 2.0, nan], 's': ['a', 'b', nan], "
+ "'f': [1.5, 2.5, 3.5], 'b': [True, False, None]}"
+ "@Dataframe[i float64, s str, f float64, b object]"
+ )
+ }
+ )
+ overrides.update(
+ {
+ ("multi-column:all-null", "pandas dataframe"): (
+ "{'i': [nan, nan], 's': [nan, nan]}@Dataframe[i
float64, s str]"
+ )
+ }
+ )
+ overrides.update(
+ {
+ ("empty:columns-no-rows", "pandas dataframe"): (
+ "{'i': [], 's': []}@Dataframe[i int64, s str]"
+ )
+ }
+ )
+
+ def compute_cell(row_name, col_name):
+ table = sources[row_name]
+ if col_name == "pyarrow table":
+ return self.repr_value(table, max_len=0)
+ else:
+ return self._to_pandas_cell(table)
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+ golden_file_prefix="golden_pyarrow_table_to_pandas",
+ index_name="test case",
+ overrides=overrides,
+ )
+
+
[email protected](
+ not have_pyarrow or not have_pandas,
+ pyarrow_requirement_message or pandas_requirement_message,
+)
+class PyArrowTableToPandasCoerceTemporalTests(_PyArrowTableToPandasTestBase):
+ """
+ Tests pa.Table.to_pandas(coerce_temporal_nanoseconds=True) via golden file
comparison.
+
+ Reuses the shared temporal tables plus a coercion overflow row, recorded
under two
+ columns: the default date_as_object=True (dates stay object, unaffected by
coercion)
+ and date_as_object=False (the datetime64[ns] path where coercion applies,
including
+ the far-future overflow).
+ """
+
+ # to_pandas(coerce_temporal_nanoseconds=True); date_as_object at its
default (True).
+ COL_PANDAS = "pandas dataframe"
+
+ # to_pandas(coerce_temporal_nanoseconds=True, date_as_object=False) -- the
only path
+ # on which coercion observably affects date columns.
+ COL_PANDAS_DATE_AS_OBJECT_FALSE = "pandas dataframe (date_as_object=False)"
+
+ def test_to_pandas_coerce_temporal_nanoseconds(self):
+ """Test pa.Table.to_pandas(coerce_temporal_nanoseconds=True) against
golden file."""
+ sources = self._temporal_tables()
+ # Coercion to nanoseconds has a valid range (~1677-2262); a far-future
+ # second-resolution timestamp column cannot fit and raises.
+ sources["temporal:timestamp-overflow"] = pa.table(
+ {"ts": pa.array([datetime.datetime(2500, 1, 1)],
pa.timestamp("s"))}
+ )
+ # A duration beyond ~292 years also exceeds the int64 nanosecond
range, but
+ # unlike timestamp overflow, coercion wraps it silently to a bogus
value.
+ sources["temporal:duration-overflow"] = pa.table(
+ {"dur": pa.array([datetime.timedelta(days=300 * 365)],
pa.duration("s"))}
+ )
+ row_names = list(sources.keys())
+ col_names = [
+ "pyarrow table",
+ self.COL_PANDAS,
+ self.COL_PANDAS_DATE_AS_OBJECT_FALSE,
+ ]
+
+ # Version-specific expected values go here, keyed by (row, col), when
a newer
+ # pandas/PyArrow legitimately changes a cell. Add a
LooseVersion-guarded block
+ # for each known drift.
+ overrides: dict[tuple[str, str], str] = {}
+
+ def compute_cell(row_name, col_name):
+ table = sources[row_name]
+ if col_name == "pyarrow table":
+ return self.repr_value(table, max_len=0)
+ elif col_name == self.COL_PANDAS:
+ return self._to_pandas_cell(table,
coerce_temporal_nanoseconds=True)
+ elif col_name == self.COL_PANDAS_DATE_AS_OBJECT_FALSE:
+ return self._to_pandas_cell(
+ table, coerce_temporal_nanoseconds=True,
date_as_object=False
+ )
+ else:
+ raise ValueError(f"unknown column: {col_name}")
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+
golden_file_prefix="golden_pyarrow_table_to_pandas_coerce_temporal",
+ index_name="test case",
+ overrides=overrides,
+ )
+
+
+if __name__ == "__main__":
+ from pyspark.testing import main
+
+ main()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
