This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new de97fd485994 [SPARK-58720][PYTHON][TEST][FOLLOWUP] Move the pandas
DataFrame golden repr into GoldenFileTestMixin
de97fd485994 is described below
commit de97fd4859948856ae0cb63045407758e0688ff0
Author: Spenser Sun <[email protected]>
AuthorDate: Fri Aug 14 07:33:58 2026 +0000
[SPARK-58720][PYTHON][TEST][FOLLOWUP] Move the pandas DataFrame golden repr
into GoldenFileTestMixin
### What changes were proposed in this pull request?
Follow-up to #57939 (SPARK-58720). That PR added `pa.Table.to_pandas`
golden-file tests with a local `_repr_dataframe` helper that formats a pandas
DataFrame result via per-column `tolist()`, deliberately bypassing
`GoldenFileTestMixin.repr_pandas_value` (which used `DataFrame.to_json`). This
PR promotes that formatter into the shared mixin so there is a single DataFrame
repr:
- `GoldenFileTestMixin.repr_pandas_value` now renders each column with
`tolist()` (byte-identical to the old local `_repr_dataframe`) instead of
`DataFrame.to_json`.
- `test_pyarrow_table_to_pandas.py` drops its local `_repr_dataframe` and
formats results through `repr_value(pdf, max_len=0)`.
- `test_pandas_udf_return_type`'s golden files (`pandas_2` and `pandas_3`)
are regenerated. It is the only other test that renders a pandas DataFrame
through `repr_value` — the `pd.DataFrame({"_1": [1, 2]})` column header. The
regeneration changes only that header; all coercion result cells are unchanged.
### Why are the changes needed?
`DataFrame.to_json` defaults to an epoch date format that serializes
temporal values to an int64 nanosecond count. That overflows on
out-of-nanosecond-range dates (e.g. year 9999 under the default
`date_as_object=True`, which returns an object column of Python
`datetime.date`) and misreads non-nanosecond units on pandas 2 (so the same
cell drifts between pandas 2 and 3). `tolist()` bypasses the epoch machinery
entirely, giving a stable, version-independent representation — exactly wha
[...]
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
- `test_pyarrow_table_to_pandas` passes unchanged against its committed
golden files, confirming the switch to `repr_value` is byte-identical
(golden-inert).
- `test_pandas_udf_return_type` was regenerated and re-run in compare mode
under both pandas 2 and pandas 3; the golden diff is the single DataFrame
column header, with every data cell byte-identical.
- The full `python/pyspark/tests/upstream/pyarrow` suite and the
`python/pyspark/sql/tests/coercion` suite pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
This pull request and its description were written by Isaac.
Closes #57999 from Spenserrrr/unify-dataframe-repr.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
...golden_pandas_udf_return_type_coercion_base.csv | 2 +-
.../golden_pandas_udf_return_type_coercion_base.md | 34 +++++++++++-----------
...golden_pandas_udf_return_type_coercion_base.csv | 2 +-
.../golden_pandas_udf_return_type_coercion_base.md | 34 +++++++++++-----------
python/pyspark/testing/goldenutils.py | 10 +++++--
.../pyarrow/test_pyarrow_table_to_pandas.py | 20 +------------
6 files changed, 45 insertions(+), 57 deletions(-)
diff --git
a/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv
b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv
index 4f5d13daaf10..ae47d24a24db 100644
---
a/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv
+++
b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv
@@ -1,4 +1,4 @@
-SQL Type \ Value@Type [None, None]@list [True, False]@list ['a',
'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list
[{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16]
[1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1
2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1.
2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1.
2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j
2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
+SQL Type \ Value@Type [None, None]@list [True, False]@list ['a',
'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list
[{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16]
[1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1
2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1.
2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1.
2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j
2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
boolean [None, None] [True, False] X X X X
[True, True] [True, True] [True, True] [True, True] [True, True]
[True, True] [True, True] [True, True] X [True, True] [True,
True] X X X X X X X X
tinyint [None, None] [1, 0] X [12, 34] [1, 2] X
[1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2]
[1, 2] X X X X X X X X
smallint [None, None] [1, 0] X [12, 34] [1, 2] X
[1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2]
[1, 2] X X X X X X X X
diff --git
a/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md
b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md
index f8d7f11310fa..54a0d16389d2 100644
---
a/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md
+++
b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md
@@ -1,17 +1,17 @@
-| SQL Type \ Value@Type | [None, None]@list | [True, False]@list |
['a', 'b']@list | ['12', '34']@list | [Decimal('1'), Decimal('2')]@list
| [{'a': 1}, {'b': 2}]@list | [1 2]@ndarray[int8] | [1
2]@ndarray[int16] | [1 2]@ndarray[int32] | [1
2]@ndarray[int64] | [1 2]@ndarray[uint8] | [1
2]@ndarray[uint16] | [1 2]@ndarray[uint32] | [1
2]@ndarray[uint64] | [1. 2.]@ndarray[float16] | [1. 2.]@ [...]
-|--------------------------|---------------------|----------------------|-------------------|---------------------|------------------------------------------|-----------------------------|------------------------------|------------------------------|------------------------------------------|-----------------------------------------|------------------------------|------------------------------|------------------------------|-------------------------|----------------------------|---------
[...]
-| boolean | [None, None] | [True, False] | X
| X | X |
X | [True, True] | [True, True]
| [True, True] | [True, True]
| [True, True] | [True, True] |
[True, True] | [True, True] | X
| [True, T [...]
-| tinyint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2] |
X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] |
[1, 2] | [1, 2] | [1, 2]
| [1, 2] [...]
-| smallint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2] |
X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] |
[1, 2] | [1, 2] | [1, 2]
| [1, 2] [...]
-| int | [None, None] | [1, 0] | X
| [12, 34] | [1, 2] |
X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] |
[1, 2] | [1, 2] | [1, 2]
| [1, 2] [...]
-| bigint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2] |
X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] |
[1, 2] | [1, 2] | [1, 2]
| [1, 2] [...]
-| string | [None, None] | X |
['a', 'b'] | ['12', '34'] | X
| X | X | X
| X | X
| X | X
| X | X | X
| X [...]
-| date | [None, None] | X | X
| X | [datetime.date(1970, 1, 2), datetime.dat |
X | X | X
| [datetime.date(1970, 1, 2), datetime.dat | X
| X | X |
X | X | X
| X [...]
-| timestamp | [None, None] | X | X
| X | [datetime.datetime(1969, 12, 31, 16, 0, |
X | X | X
| X | [datetime.datetime(1969,
12, 31, 16, 0, | X | X |
X | X | X
| X [...]
-| float | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0] |
X | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] |
[1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2. [...]
-| double | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0] |
X | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] |
[1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2. [...]
-| array<int> | [None, None] | X | X
| [[1, 2], [3, 4]] | X |
X | X | X
| X | X
| X | X |
X | X | X
| X [...]
-| binary | [None, None] | [b'\x01', b''] |
[b'a', b'b'] | [b'12', b'34'] | X
| X | [b'\x01', b'\x02'] | [b'\x01',
b'\x02'] | [b'\x01', b'\x02'] | [b'\x01',
b'\x02'] | [b'\x01', b'\x02'] | [b'\x01',
b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] |
[b'', b''] | [b'', b' [...]
-| decimal(10,0) | [None, None] | X | X
| X | [Decimal('1'), Decimal('2')] |
X | [Decimal('1'), Decimal('2')] | [Decimal('1'),
Decimal('2')] | [Decimal('1'), Decimal('2')] | X
| [Decimal('1'), Decimal('2')] | [Decimal('1'),
Decimal('2')] | [Decimal('1'), Decimal('2')] | X | X
| [Decimal [...]
-| map<string,int> | [None, None] | X | X
| X | X |
[{'a': 1}, {'b': 2}] | X | X
| X | X
| X | X |
X | X | X
| X [...]
-| struct<_1:int> | X | X | X
| X | X |
X | X | X
| X | X
| X | X |
X | X | X
| X [...]
\ No newline at end of file
+| SQL Type \ Value@Type | [None, None]@list | [True, False]@list |
['a', 'b']@list | ['12', '34']@list | [Decimal('1'), Decimal('2')]@list
| [{'a': 1}, {'b': 2}]@list | [1 2]@ndarray[int8] | [1
2]@ndarray[int16] | [1 2]@ndarray[int32] | [1
2]@ndarray[int64] | [1 2]@ndarray[uint8] | [1
2]@ndarray[uint16] | [1 2]@ndarray[uint32] | [1
2]@ndarray[uint64] | [1. 2.]@ndarray[float16] | [1. 2.]@ [...]
+|--------------------------|---------------------|----------------------|-------------------|---------------------|------------------------------------------|-----------------------------|------------------------------|------------------------------|------------------------------------------|-----------------------------------------|------------------------------|------------------------------|------------------------------|-------------------------|----------------------------|---------
[...]
+| boolean | [None, None] | [True, False] | X
| X | X |
X | [True, True] | [True, True]
| [True, True] | [True, True]
| [True, True] | [True, True] |
[True, True] | [True, True] | X
| [True, T [...]
+| tinyint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2] |
X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] |
[1, 2] | [1, 2] | [1, 2]
| [1, 2] [...]
+| smallint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2] |
X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] |
[1, 2] | [1, 2] | [1, 2]
| [1, 2] [...]
+| int | [None, None] | [1, 0] | X
| [12, 34] | [1, 2] |
X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] |
[1, 2] | [1, 2] | [1, 2]
| [1, 2] [...]
+| bigint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2] |
X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] |
[1, 2] | [1, 2] | [1, 2]
| [1, 2] [...]
+| string | [None, None] | X |
['a', 'b'] | ['12', '34'] | X
| X | X | X
| X | X
| X | X
| X | X | X
| X [...]
+| date | [None, None] | X | X
| X | [datetime.date(1970, 1, 2), datetime.dat |
X | X | X
| [datetime.date(1970, 1, 2), datetime.dat | X
| X | X |
X | X | X
| X [...]
+| timestamp | [None, None] | X | X
| X | [datetime.datetime(1969, 12, 31, 16, 0, |
X | X | X
| X | [datetime.datetime(1969,
12, 31, 16, 0, | X | X |
X | X | X
| X [...]
+| float | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0] |
X | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] |
[1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2. [...]
+| double | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0] |
X | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] |
[1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2. [...]
+| array<int> | [None, None] | X | X
| [[1, 2], [3, 4]] | X |
X | X | X
| X | X
| X | X |
X | X | X
| X [...]
+| binary | [None, None] | [b'\x01', b''] |
[b'a', b'b'] | [b'12', b'34'] | X
| X | [b'\x01', b'\x02'] | [b'\x01',
b'\x02'] | [b'\x01', b'\x02'] | [b'\x01',
b'\x02'] | [b'\x01', b'\x02'] | [b'\x01',
b'\x02'] | [b'\x01', b'\x02'] | [b'\x01', b'\x02'] |
[b'', b''] | [b'', b' [...]
+| decimal(10,0) | [None, None] | X | X
| X | [Decimal('1'), Decimal('2')] |
X | [Decimal('1'), Decimal('2')] | [Decimal('1'),
Decimal('2')] | [Decimal('1'), Decimal('2')] | X
| [Decimal('1'), Decimal('2')] | [Decimal('1'),
Decimal('2')] | [Decimal('1'), Decimal('2')] | X | X
| [Decimal [...]
+| map<string,int> | [None, None] | X | X
| X | X |
[{'a': 1}, {'b': 2}] | X | X
| X | X
| X | X |
X | X | X
| X [...]
+| struct<_1:int> | X | X | X
| X | X |
X | X | X
| X | X
| X | X |
X | X | X
| X [...]
\ No newline at end of file
diff --git
a/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
index 5e12cc57c84a..0b955d06a552 100644
---
a/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
+++
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
@@ -1,4 +1,4 @@
-SQL Type \ Value@Type [None, None]@list [True, False]@list ['a',
'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list
[{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16]
[1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1
2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1.
2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1.
2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j
2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
+SQL Type \ Value@Type [None, None]@list [True, False]@list ['a',
'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list
[{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16]
[1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1
2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1.
2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1.
2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j
2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
boolean [None, None] [True, False] X X X X
[True, True] [True, True] [True, True] [True, True] [True, True]
[True, True] [True, True] [True, True] X [True, True] [True,
True] X X X X X X X X
tinyint [None, None] [1, 0] X [12, 34] [1, 2] X
[1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2]
[1, 2] X X X X X X X X
smallint [None, None] [1, 0] X [12, 34] [1, 2] X
[1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2]
[1, 2] X X X X X X X X
diff --git
a/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
index aa57ca6eec96..257ae0e7548c 100644
---
a/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
+++
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
@@ -1,17 +1,17 @@
-| SQL Type \ Value@Type | [None, None]@list | [True, False]@list |
['a', 'b']@list | ['12', '34']@list | [Decimal('1'),
Decimal('2')]@list | [{'a': 1}, {'b': 2}]@list | [1 2]@ndarray[int8]
| [1 2]@ndarray[int16] | [1 2]@ndarray[int32]
| [1 2]@ndarray[int64] | [1 2]@ndarray[uint8] | [1
2]@ndarray[uint16] | [1 2]@ndarray[uint32] | [1
2]@ndarray[uint64] | [1. 2.]@ndarray[float16] [...]
-|--------------------------|---------------------|----------------------|-------------------|--------------------------------|------------------------------------------|-----------------------------|------------------------------|------------------------------|------------------------------------------|-----------------------------------------|------------------------------|------------------------------|------------------------------|-------------------------|---------------------------
[...]
-| boolean | [None, None] | [True, False] | X
| X | X
| X | [True, True] | [True,
True] | [True, True] | [True, True]
| [True, True] | [True, True]
| [True, True] | [True, True] | X
[...]
-| tinyint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
-| smallint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
-| int | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
-| bigint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
-| string | [None, None] | X |
['a', 'b'] | ['12', '34'] | X
| X | X | X
| X | X
| X | X
| X | X | X
[...]
-| date | [None, None] | X | X
| X | [datetime.date(1970, 1, 2),
datetime.dat | X | X | X
| [datetime.date(1970, 1, 2), datetime.dat | X
| X | X
| X | X | X
[...]
-| timestamp | [None, None] | X | X
| X | [datetime.datetime(1969, 12,
31, 16, 0, | X | X | X
| X |
[datetime.datetime(1969, 12, 31, 16, 0, | X | X
| X | X
| X [...]
-| float | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0]
| X | [1.0, 2.0] | [1.0,
2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
[...]
-| double | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0]
| X | [1.0, 2.0] | [1.0,
2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
[...]
-| array<int> | [None, None] | X | X
| [[1, 2], [3, 4]] | X
| X | X | X
| X | X
| X | X
| X | X | X
[...]
-| binary | [None, None] | [b'\x01', b''] |
[b'a', b'b'] | [b'12', b'34'] | X
| X | [b'\x01', b'\x02'] |
[b'\x01', b'\x02'] | [b'\x01', b'\x02'] |
[b'\x01', b'\x02'] | [b'\x01', b'\x02'] |
[b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01',
b'\x02'] | [b'', b''] [...]
-| decimal(10,0) | [None, None] | X | X
| [Decimal('12'), Decimal('34')] | [Decimal('1'), Decimal('2')]
| X | [Decimal('1'), Decimal('2')] |
[Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X
| [Decimal('1'), Decimal('2')] |
[Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X
| X [...]
-| map<string,int> | [None, None] | X | X
| X | X
| [{'a': 1}, {'b': 2}] | X | X
| X | X
| X | X
| X | X | X
[...]
-| struct<_1:int> | X | X | X
| X | X
| X | X | X
| X | X
| X | X
| X | X | X
[...]
\ No newline at end of file
+| SQL Type \ Value@Type | [None, None]@list | [True, False]@list |
['a', 'b']@list | ['12', '34']@list | [Decimal('1'),
Decimal('2')]@list | [{'a': 1}, {'b': 2}]@list | [1 2]@ndarray[int8]
| [1 2]@ndarray[int16] | [1 2]@ndarray[int32]
| [1 2]@ndarray[int64] | [1 2]@ndarray[uint8] | [1
2]@ndarray[uint16] | [1 2]@ndarray[uint32] | [1
2]@ndarray[uint64] | [1. 2.]@ndarray[float16] [...]
+|--------------------------|---------------------|----------------------|-------------------|--------------------------------|------------------------------------------|-----------------------------|------------------------------|------------------------------|------------------------------------------|-----------------------------------------|------------------------------|------------------------------|------------------------------|-------------------------|---------------------------
[...]
+| boolean | [None, None] | [True, False] | X
| X | X
| X | [True, True] | [True,
True] | [True, True] | [True, True]
| [True, True] | [True, True]
| [True, True] | [True, True] | X
[...]
+| tinyint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
+| smallint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
+| int | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
+| bigint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
+| string | [None, None] | X |
['a', 'b'] | ['12', '34'] | X
| X | X | X
| X | X
| X | X
| X | X | X
[...]
+| date | [None, None] | X | X
| X | [datetime.date(1970, 1, 2),
datetime.dat | X | X | X
| [datetime.date(1970, 1, 2), datetime.dat | X
| X | X
| X | X | X
[...]
+| timestamp | [None, None] | X | X
| X | [datetime.datetime(1969, 12,
31, 16, 0, | X | X | X
| X |
[datetime.datetime(1969, 12, 31, 16, 0, | X | X
| X | X
| X [...]
+| float | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0]
| X | [1.0, 2.0] | [1.0,
2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
[...]
+| double | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0]
| X | [1.0, 2.0] | [1.0,
2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
[...]
+| array<int> | [None, None] | X | X
| [[1, 2], [3, 4]] | X
| X | X | X
| X | X
| X | X
| X | X | X
[...]
+| binary | [None, None] | [b'\x01', b''] |
[b'a', b'b'] | [b'12', b'34'] | X
| X | [b'\x01', b'\x02'] |
[b'\x01', b'\x02'] | [b'\x01', b'\x02'] |
[b'\x01', b'\x02'] | [b'\x01', b'\x02'] |
[b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01',
b'\x02'] | [b'', b''] [...]
+| decimal(10,0) | [None, None] | X | X
| [Decimal('12'), Decimal('34')] | [Decimal('1'), Decimal('2')]
| X | [Decimal('1'), Decimal('2')] |
[Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X
| [Decimal('1'), Decimal('2')] |
[Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X
| X [...]
+| map<string,int> | [None, None] | X | X
| X | X
| [{'a': 1}, {'b': 2}] | X | X
| X | X
| X | X
| X | X | X
[...]
+| struct<_1:int> | X | X | X
| X | X
| X | X | X
| X | X
| X | X
| X | X | X
[...]
\ No newline at end of file
diff --git a/python/pyspark/testing/goldenutils.py
b/python/pyspark/testing/goldenutils.py
index 95b7da956ff3..7f2c889954ed 100644
--- a/python/pyspark/testing/goldenutils.py
+++ b/python/pyspark/testing/goldenutils.py
@@ -366,6 +366,12 @@ class GoldenFileTestMixin:
"""
Format a pandas DataFrame for golden file.
+ Renders each column with tolist() (as ``repr_pandas_series_value``
does for a
+ Series), keyed by column name, plus the schema. tolist() gives a stable
+ Python-native representation and avoids ``DataFrame.to_json``'s epoch
date
+ serialization, which overflows on out-of-nanosecond-range dates (year
9999
+ with the default date_as_object=True) and misreads non-ns units on
pandas 2.
+
Parameters
----------
value : pd.DataFrame
@@ -376,9 +382,9 @@ class GoldenFileTestMixin:
Returns
-------
str
- "value@Dataframe[schema]"
+ "{col: [val1, val2], ...}@Dataframe[schema]"
"""
- v_str = value.to_json().replace("\n", " ")
+ v_str = str({name: col.tolist() for name, col in
value.items()}).replace("\n", " ")
if max_len > 0:
v_str = v_str[:max_len]
simple_schema = ", ".join([f"{t} {d.name}" for t, d in
value.dtypes.items()])
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
index 42977bea5bf5..8c5175961184 100644
--- a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
+++ b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py
@@ -70,24 +70,6 @@ class _PyArrowTableToPandasTestBase(GoldenFileTestMixin,
unittest.TestCase):
its own.
"""
- @staticmethod
- def _repr_dataframe(df) -> str:
- """
- Format a pandas DataFrame result as a golden-file cell: per-column
tolist()
- for a stable, Python-native representation, mirroring goldenutils'
- ``repr_pandas_series_value`` for the Series case.
-
- This deliberately does NOT go through
``repr_value``/``repr_pandas_value``
- (which use ``to_json``). With the default date_as_object=True a
far-future
- date (year 9999) comes back as an object column of Python
``datetime.date``
- objects, and ``DataFrame.to_json`` overflows converting each to an
epoch
- nanosecond int (OverflowError); tolist() returns the objects as-is, so
- nothing overflows.
- """
- body = str({name: col.tolist() for name, col in
df.items()}).replace("\n", " ")
- schema = ", ".join(f"{t} {d.name}" for t, d in df.dtypes.items())
- return f"{body}@Dataframe[{schema}]"
-
def _to_pandas_cell(self, table, **to_pandas_kwargs) -> str:
"""
Convert ``table`` via ``to_pandas(**to_pandas_kwargs)`` and format the
result
@@ -99,7 +81,7 @@ class _PyArrowTableToPandasTestBase(GoldenFileTestMixin,
unittest.TestCase):
pdf = table.to_pandas(**to_pandas_kwargs)
except Exception as e:
return f"ERR@{type(e).__name__}"
- return self._repr_dataframe(pdf)
+ return self.repr_value(pdf, max_len=0)
def _structural_tables(self):
"""Assembly shapes and empty edges (no temporal coercion involved)."""
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
