This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5a9792961eba [SPARK-54946][PYTHON][TEST] Add tests for
pa.Array.to_pandas with coerce_temporal_nanoseconds
5a9792961eba is described below
commit 5a9792961eba9da641dd5a2aa18a09aa1f1fd1d8
Author: Spenser Sun <[email protected]>
AuthorDate: Fri Jul 24 18:13:55 2026 +0000
[SPARK-54946][PYTHON][TEST] Add tests for pa.Array.to_pandas with
coerce_temporal_nanoseconds
### What changes were proposed in this pull request?
This PR adds golden-file tests that pin the behavior of PyArrow's
`Array.to_pandas()` with non-default arguments, starting with
`coerce_temporal_nanoseconds=True`.
- New test:
`python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py`
- New golden files:
`golden_pyarrow_arrow_to_pandas_coerce_temporal.{csv,md}`
- Registered the module in `dev/sparktestsupport/modules.py`.
**Structure.** The file is organized as a shared `_PyArrowToPandasTestBase`
and one subclass per flag, where each subclass brings its own source arrays and
test method(s) and produces its own golden file. `coerce_temporal_nanoseconds`
is the first such subclass; a future flag like `zero_copy` (SPARK-54945) which
needs a different row set can be added as another subclass in this same file
rather than a new file. `test_pyarrow_arrow_to_pandas_default.py` stays
separate.
**Rows.** Limited to the temporal types the argument affects (`timestamp`
and `duration` in units s/ms/us/ns, tz-aware `timestamp`, `date`, `time`), plus
overflow cases and a few non-temporal control rows (`int64`/`float64`/`string`)
to show the argument leaves non-temporal types unaffected.
**Columns.** The golden file records, per source array, the resulting
pandas Series dtype and values under two output columns:
`coerce_temporal_nanoseconds=True` (default `date_as_object=True`) and
`coerce_temporal_nanoseconds=True, date_as_object=False`. The second column is
added as a column rather than as extra rows because the matrix applies one
conversion uniformly across all rows.
**Recorded behavior** (generated with pandas 2.3.3, pyarrow 24.0.0, numpy
2.4.1):
- `timestamp[s|ms|us|ns]` all coerce to `datetime64[ns]`.
- tz-aware `timestamp[us, tz=UTC]` coerces to `datetime64[ns, UTC]`.
- `duration[s|ms|us|ns]` all coerce to `timedelta64[ns]`.
- `date32`/`date64` stay `object` (`datetime.date`) by default, but become
`datetime64[ns]` with `date_as_object=False`; `time32`/`time64` stay `object`.
- A far-future second-resolution timestamp (year 2500) exceeds the
nanosecond range and raises `ArrowInvalid`, whereas an out-of-range
`duration[s]` silently wraps to a bogus value â both are pinned.
The recorded cells were verified to be identical across pyarrow 18-25 (each
paired with a pandas/numpy that version supports), so no version-specific
overrides are needed for the temporal cells.
### Why are the changes needed?
This is part of SPARK-54936 ("Monitor behaviour changes from upstream").
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New golden-file test. Generated the golden files with
`SPARK_GENERATE_GOLDEN_FILES=1` (pandas 2.3.3, pyarrow 24.0.0, numpy 2.4.1),
reviewed every cell, then confirmed the test passes in comparison mode:
python -m pytest
python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_coerce_temporal.py
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57435 from Spenserrrr/coerce-temporal-nanoseconds-tests.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
dev/sparktestsupport/modules.py | 1 +
...den_pyarrow_arrow_to_pandas_coerce_temporal.csv | 52 ++++
...lden_pyarrow_arrow_to_pandas_coerce_temporal.md | 53 ++++
.../test_pyarrow_arrow_to_pandas_non_default.py | 327 +++++++++++++++++++++
4 files changed, 433 insertions(+)
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 3edebcee6537..77aa43eb8d94 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -524,6 +524,7 @@ pyspark_core = Module(
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_cast",
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_type_inference",
"pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_default",
+
"pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_non_default",
"pyspark.tests.upstream.pyarrow.test_pyarrow_ignore_timezone",
"pyspark.tests.upstream.pyarrow.test_pyarrow_scalar_type_coercion",
"pyspark.tests.upstream.pyarrow.test_pyarrow_scalar_type_inference",
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
new file mode 100644
index 000000000000..d4a8df1a2a62
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
@@ -0,0 +1,52 @@
+test case pyarrow array pandas series pandas series
(date_as_object=False)
+timestamp[s]:standard [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[s]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
+timestamp[s]:nullable [2024-01-01 12:00:00, None]@timestamp[s]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
+timestamp[s]:empty []@timestamp[s] []@Series[datetime64[ns]]
[]@Series[datetime64[ns]]
+timestamp[ms]:standard [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ms] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]]
+timestamp[ms]:nullable [2024-01-01 12:00:00, None]@timestamp[ms]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
+timestamp[ms]:empty []@timestamp[ms] []@Series[datetime64[ns]]
[]@Series[datetime64[ns]]
+timestamp[us]:standard [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]]
+timestamp[us]:nullable [2024-01-01 12:00:00, None]@timestamp[us]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
+timestamp[us]:empty []@timestamp[us] []@Series[datetime64[ns]]
[]@Series[datetime64[ns]]
+timestamp[ns]:standard [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ns] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]]
+timestamp[ns]:nullable [2024-01-01 12:00:00, None]@timestamp[ns]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
+timestamp[ns]:empty []@timestamp[ns] []@Series[datetime64[ns]]
[]@Series[datetime64[ns]]
+timestamp[us,tz=UTC]:standard [2024-01-01 12:00:00+00:00, 2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]]
+timestamp[us,tz=UTC]:nullable [2024-01-01 12:00:00+00:00, None]@timestamp[us,
tz=UTC] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'),
NaT]@Series[datetime64[ns, UTC]] [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), NaT]@Series[datetime64[ns, UTC]]
+timestamp[us,tz=UTC]:empty []@timestamp[us, tz=UTC]
[]@Series[datetime64[ns, UTC]] []@Series[datetime64[ns, UTC]]
+timestamp[s]:overflow [2500-01-01 00:00:00]@timestamp[s]
ERR@ArrowInvalid ERR@ArrowInvalid
+duration[s]:standard [1 day, 0:00:00, 2:30:00]@duration[s] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]]
+duration[s]:nullable [1 day, 0:00:00, None]@duration[s] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[ns]] [Timedelta('1 days
00:00:00'), NaT]@Series[timedelta64[ns]]
+duration[s]:empty []@duration[s] []@Series[timedelta64[ns]]
[]@Series[timedelta64[ns]]
+duration[ms]:standard [1 day, 0:00:00, 2:30:00]@duration[ms] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]]
+duration[ms]:nullable [1 day, 0:00:00, None]@duration[ms] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[ns]] [Timedelta('1 days
00:00:00'), NaT]@Series[timedelta64[ns]]
+duration[ms]:empty []@duration[ms] []@Series[timedelta64[ns]]
[]@Series[timedelta64[ns]]
+duration[us]:standard [1 day, 0:00:00, 2:30:00]@duration[us] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]]
+duration[us]:nullable [1 day, 0:00:00, None]@duration[us] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[ns]] [Timedelta('1 days
00:00:00'), NaT]@Series[timedelta64[ns]]
+duration[us]:empty []@duration[us] []@Series[timedelta64[ns]]
[]@Series[timedelta64[ns]]
+duration[ns]:standard [1 days 00:00:00, 0 days 02:30:00]@duration[ns]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
+duration[ns]:nullable [1 days 00:00:00, None]@duration[ns] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[ns]] [Timedelta('1 days
00:00:00'), NaT]@Series[timedelta64[ns]]
+duration[ns]:empty []@duration[ns] []@Series[timedelta64[ns]]
[]@Series[timedelta64[ns]]
+duration[s]:overflow [109500 days, 0:00:00]@duration[s]
[Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
[Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
+date32:standard [2024-01-01, 2024-06-15]@date32[day]
[datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
[Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]]
+date32:nullable [2024-01-01, None]@date32[day] [datetime.date(2024, 1,
1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
+date32:empty []@date32[day] []@Series[object]
[]@Series[datetime64[ns]]
+date64:standard [2024-01-01, 2024-06-15]@date64[ms]
[datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
[Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]]
+date64:nullable [2024-01-01, None]@date64[ms] [datetime.date(2024, 1,
1), None]@Series[object] [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
+date64:empty []@date64[ms] []@Series[object]
[]@Series[datetime64[ns]]
+time32[s]:standard [12:30:00, 18:45:30]@time32[s] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object]
+time32[s]:nullable [12:30:00, None]@time32[s] [datetime.time(12, 30),
None]@Series[object] [datetime.time(12, 30), None]@Series[object]
+time32[s]:empty []@time32[s] []@Series[object]
[]@Series[object]
+time32[ms]:standard [12:30:00, 18:45:30]@time32[ms] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object]
+time32[ms]:nullable [12:30:00, None]@time32[ms] [datetime.time(12, 30),
None]@Series[object] [datetime.time(12, 30), None]@Series[object]
+time32[ms]:empty []@time32[ms] []@Series[object]
[]@Series[object]
+time64[us]:standard [12:30:00, 18:45:30]@time64[us] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object]
+time64[us]:nullable [12:30:00, None]@time64[us] [datetime.time(12, 30),
None]@Series[object] [datetime.time(12, 30), None]@Series[object]
+time64[us]:empty []@time64[us] []@Series[object]
[]@Series[object]
+time64[ns]:standard [12:30:00, 18:45:30]@time64[ns] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object]
+time64[ns]:nullable [12:30:00, None]@time64[ns] [datetime.time(12, 30),
None]@Series[object] [datetime.time(12, 30), None]@Series[object]
+time64[ns]:empty []@time64[ns] []@Series[object]
[]@Series[object]
+int64:standard [0, 1, -1]@int64 [0, 1, -1]@Series[int64] [0, 1,
-1]@Series[int64]
+int64:nullable [0, 1, None]@int64 [0.0, 1.0, nan]@Series[float64] [0.0,
1.0, nan]@Series[float64]
+float64:standard [0.0, 1.5, -1.5]@float64 [0.0, 1.5,
-1.5]@Series[float64] [0.0, 1.5, -1.5]@Series[float64]
+string:standard [hello, world, ]@string ['hello', 'world',
'']@Series[object] ['hello', 'world', '']@Series[object]
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
new file mode 100644
index 000000000000..45eec337f059
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
@@ -0,0 +1,53 @@
+| test case | pyarrow array
| pandas series
| pandas series (date_as_object=False)
|
+|-------------------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|
+| timestamp[s]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[s] | [Timestamp('2024-01-01
12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
|
+| timestamp[s]:nullable | [2024-01-01 12:00:00, None]@timestamp[s]
| [Timestamp('2024-01-01 12:00:00'),
NaT]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
|
+| timestamp[s]:empty | []@timestamp[s]
| []@Series[datetime64[ns]]
| []@Series[datetime64[ns]]
|
+| timestamp[ms]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ms] | [Timestamp('2024-01-01
12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
|
+| timestamp[ms]:nullable | [2024-01-01 12:00:00, None]@timestamp[ms]
| [Timestamp('2024-01-01 12:00:00'),
NaT]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
|
+| timestamp[ms]:empty | []@timestamp[ms]
| []@Series[datetime64[ns]]
| []@Series[datetime64[ns]]
|
+| timestamp[us]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] | [Timestamp('2024-01-01
12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
|
+| timestamp[us]:nullable | [2024-01-01 12:00:00, None]@timestamp[us]
| [Timestamp('2024-01-01 12:00:00'),
NaT]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
|
+| timestamp[us]:empty | []@timestamp[us]
| []@Series[datetime64[ns]]
| []@Series[datetime64[ns]]
|
+| timestamp[ns]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ns] | [Timestamp('2024-01-01
12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
|
+| timestamp[ns]:nullable | [2024-01-01 12:00:00, None]@timestamp[ns]
| [Timestamp('2024-01-01 12:00:00'),
NaT]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
|
+| timestamp[ns]:empty | []@timestamp[ns]
| []@Series[datetime64[ns]]
| []@Series[datetime64[ns]]
|
+| timestamp[us,tz=UTC]:standard | [2024-01-01 12:00:00+00:00, 2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] | [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] | [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] |
+| timestamp[us,tz=UTC]:nullable | [2024-01-01 12:00:00+00:00,
None]@timestamp[us, tz=UTC] | [Timestamp('2024-01-01
12:00:00+0000', tz='UTC'), NaT]@Series[datetime64[ns, UTC]]
| [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'),
NaT]@Series[datetime64[ns, UTC]] |
+| timestamp[us,tz=UTC]:empty | []@timestamp[us, tz=UTC]
| []@Series[datetime64[ns, UTC]]
| []@Series[datetime64[ns, UTC]]
|
+| timestamp[s]:overflow | [2500-01-01 00:00:00]@timestamp[s]
| ERR@ArrowInvalid
| ERR@ArrowInvalid
|
+| duration[s]:standard | [1 day, 0:00:00, 2:30:00]@duration[s]
| [Timedelta('1 days 00:00:00'), Timedelta('0
days 02:30:00')]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] |
+| duration[s]:nullable | [1 day, 0:00:00, None]@duration[s]
| [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]]
|
+| duration[s]:empty | []@duration[s]
| []@Series[timedelta64[ns]]
| []@Series[timedelta64[ns]]
|
+| duration[ms]:standard | [1 day, 0:00:00, 2:30:00]@duration[ms]
| [Timedelta('1 days 00:00:00'), Timedelta('0
days 02:30:00')]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] |
+| duration[ms]:nullable | [1 day, 0:00:00, None]@duration[ms]
| [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]]
|
+| duration[ms]:empty | []@duration[ms]
| []@Series[timedelta64[ns]]
| []@Series[timedelta64[ns]]
|
+| duration[us]:standard | [1 day, 0:00:00, 2:30:00]@duration[us]
| [Timedelta('1 days 00:00:00'), Timedelta('0
days 02:30:00')]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] |
+| duration[us]:nullable | [1 day, 0:00:00, None]@duration[us]
| [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]]
|
+| duration[us]:empty | []@duration[us]
| []@Series[timedelta64[ns]]
| []@Series[timedelta64[ns]]
|
+| duration[ns]:standard | [1 days 00:00:00, 0 days
02:30:00]@duration[ns] | [Timedelta('1 days
00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] |
+| duration[ns]:nullable | [1 days 00:00:00, None]@duration[ns]
| [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]]
| [Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]]
|
+| duration[ns]:empty | []@duration[ns]
| []@Series[timedelta64[ns]]
| []@Series[timedelta64[ns]]
|
+| duration[s]:overflow | [109500 days, 0:00:00]@duration[s]
| [Timedelta('-104004 days
+00:25:26.290448384')]@Series[timedelta64[ns]]
| [Timedelta('-104004 days
+00:25:26.290448384')]@Series[timedelta64[ns]]
|
+| date32:standard | [2024-01-01, 2024-06-15]@date32[day]
| [datetime.date(2024, 1, 1),
datetime.date(2024, 6, 15)]@Series[object]
| [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]] |
+| date32:nullable | [2024-01-01, None]@date32[day]
| [datetime.date(2024, 1, 1),
None]@Series[object]
| [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
|
+| date32:empty | []@date32[day]
| []@Series[object]
| []@Series[datetime64[ns]]
|
+| date64:standard | [2024-01-01, 2024-06-15]@date64[ms]
| [datetime.date(2024, 1, 1),
datetime.date(2024, 6, 15)]@Series[object]
| [Timestamp('2024-01-01 00:00:00'), Timestamp('2024-06-15
00:00:00')]@Series[datetime64[ns]] |
+| date64:nullable | [2024-01-01, None]@date64[ms]
| [datetime.date(2024, 1, 1),
None]@Series[object]
| [Timestamp('2024-01-01 00:00:00'),
NaT]@Series[datetime64[ns]]
|
+| date64:empty | []@date64[ms]
| []@Series[object]
| []@Series[datetime64[ns]]
|
+| time32[s]:standard | [12:30:00, 18:45:30]@time32[s]
| [datetime.time(12, 30), datetime.time(18, 45,
30)]@Series[object]
| [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object]
|
+| time32[s]:nullable | [12:30:00, None]@time32[s]
| [datetime.time(12, 30), None]@Series[object]
| [datetime.time(12, 30), None]@Series[object]
|
+| time32[s]:empty | []@time32[s]
| []@Series[object]
| []@Series[object]
|
+| time32[ms]:standard | [12:30:00, 18:45:30]@time32[ms]
| [datetime.time(12, 30), datetime.time(18, 45,
30)]@Series[object]
| [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object]
|
+| time32[ms]:nullable | [12:30:00, None]@time32[ms]
| [datetime.time(12, 30), None]@Series[object]
| [datetime.time(12, 30), None]@Series[object]
|
+| time32[ms]:empty | []@time32[ms]
| []@Series[object]
| []@Series[object]
|
+| time64[us]:standard | [12:30:00, 18:45:30]@time64[us]
| [datetime.time(12, 30), datetime.time(18, 45,
30)]@Series[object]
| [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object]
|
+| time64[us]:nullable | [12:30:00, None]@time64[us]
| [datetime.time(12, 30), None]@Series[object]
| [datetime.time(12, 30), None]@Series[object]
|
+| time64[us]:empty | []@time64[us]
| []@Series[object]
| []@Series[object]
|
+| time64[ns]:standard | [12:30:00, 18:45:30]@time64[ns]
| [datetime.time(12, 30), datetime.time(18, 45,
30)]@Series[object]
| [datetime.time(12, 30), datetime.time(18, 45, 30)]@Series[object]
|
+| time64[ns]:nullable | [12:30:00, None]@time64[ns]
| [datetime.time(12, 30), None]@Series[object]
| [datetime.time(12, 30), None]@Series[object]
|
+| time64[ns]:empty | []@time64[ns]
| []@Series[object]
| []@Series[object]
|
+| int64:standard | [0, 1, -1]@int64
| [0, 1, -1]@Series[int64]
| [0, 1, -1]@Series[int64]
|
+| int64:nullable | [0, 1, None]@int64
| [0.0, 1.0, nan]@Series[float64]
| [0.0, 1.0, nan]@Series[float64]
|
+| float64:standard | [0.0, 1.5, -1.5]@float64
| [0.0, 1.5, -1.5]@Series[float64]
| [0.0, 1.5, -1.5]@Series[float64]
|
+| string:standard | [hello, world, ]@string
| ['hello', 'world', '']@Series[object]
| ['hello', 'world', '']@Series[object]
|
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
new file mode 100644
index 000000000000..6587f5deb4da
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
@@ -0,0 +1,327 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Tests for PyArrow Array.to_pandas() with non-default arguments, using golden
file
+comparison.
+
+PySpark relies on these arguments in production: when building pandas objects
from
+Arrow data (see ``python/pyspark/sql/pandas/conversion.py``), it calls
+``to_pandas(coerce_temporal_nanoseconds=True)`` (and, when applicable,
+``date_as_object=True``) so that the Arrow path produces the same
+nanosecond-resolution ``datetime64[ns]`` / ``timedelta64[ns]`` values as the
+non-Arrow path. These tests record how each Arrow type behaves under those
+arguments so CI fails loudly if the behavior drifts across pandas/PyArrow/NumPy
+upgrades.
+
+The shared ``_PyArrowToPandasTestBase`` holds the golden-file matrix driver and
+the per-cell conversion helper. Each concrete test class supplies its own
source
+arrays (the rows relevant to the argument under test) and one test method per
+argument combination (each producing its own golden file). New ``to_pandas``
+arguments (e.g. ``zero_copy_only``) can be added as additional classes without
+touching the existing ones.
+
+## Golden File Cell Format
+
+Each cell uses the value@type format:
+- numpy ndarray: "python_list_repr@ndarray[dtype]"
+- pandas Series: "python_list_repr@Series[dtype]"
+- Error: "ERR@ExceptionClassName"
+
+Values are formatted via tolist() for stable, Python-native representation.
+
+## Regenerating Golden Files
+
+Set SPARK_GENERATE_GOLDEN_FILES=1 before running:
+
+ SPARK_GENERATE_GOLDEN_FILES=1 python -m pytest \\
+
python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
+
+## PyArrow and pandas Version Compatibility
+
+The golden files capture behavior for specific PyArrow and pandas versions.
+Regenerate when upgrading either dependency, as to_pandas() behavior may
change.
+The committed golden files were generated with pandas 2.3.3, pyarrow 24.0.0,
and
+numpy 2.4.1.
+"""
+
+import datetime
+import inspect
+import os
+import unittest
+from typing import Callable, List, Optional
+
+from pyspark.loose_version import LooseVersion
+from pyspark.testing.utils import (
+ have_pyarrow,
+ have_pandas,
+ have_numpy,
+ pyarrow_requirement_message,
+ pandas_requirement_message,
+ numpy_requirement_message,
+)
+from pyspark.testing.goldenutils import GoldenFileTestMixin
+
+if have_pandas:
+ import pandas as pd
+if have_pyarrow:
+ import pyarrow as pa
+
+
+class _PyArrowToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
+ """
+ Shared machinery for pa.Array.to_pandas() golden file tests.
+
+ Concrete subclasses provide their own ``_build_source_arrays`` (the rows)
and
+ one or more ``test_*`` methods that call
``compare_or_generate_golden_matrix``.
+ This base defines no ``test_*`` methods, so it contributes no tests itself.
+ """
+
+ def compare_or_generate_golden_matrix(
+ self,
+ row_names: List[str],
+ col_names: List[str],
+ compute_cell: Callable[[str, str], str],
+ golden_file_prefix: str,
+ index_name: str = "source \\ target",
+ overrides: Optional[dict[tuple[str, str], str]] = None,
+ ) -> None:
+ """
+ Run a matrix of computations and compare against (or generate) a
golden file.
+
+ 1. If SPARK_GENERATE_GOLDEN_FILES=1, compute every cell, build a
+ DataFrame, and save it as the new golden CSV / Markdown file.
+ 2. Otherwise, load the existing golden file and assert that every cell
+ matches the freshly computed value.
+ """
+ generating = self.is_generating_golden()
+
+ test_dir = os.path.dirname(inspect.getfile(type(self)))
+ golden_csv = os.path.join(test_dir, f"{golden_file_prefix}.csv")
+ golden_md = os.path.join(test_dir, f"{golden_file_prefix}.md")
+
+ golden = None
+ if not generating:
+ golden = self.load_golden_csv(golden_csv)
+
+ errors = []
+ results = {}
+
+ for row_name in row_names:
+ for col_name in col_names:
+ result = compute_cell(row_name, col_name)
+ results[(row_name, col_name)] = result
+
+ if not generating:
+ if overrides and (row_name, col_name) in overrides:
+ expected = overrides[(row_name, col_name)]
+ else:
+ expected = golden.loc[row_name, col_name]
+ if expected != result:
+ errors.append(
+ f"{row_name} -> {col_name}: expected '{expected}',
got '{result}'"
+ )
+
+ if generating:
+ import pandas as pd
+
+ index = pd.Index(row_names, name=index_name)
+ df = pd.DataFrame(index=index)
+ for col_name in col_names:
+ df[col_name] = [results[(row, col_name)] for row in row_names]
+ self.save_golden(df, golden_csv, golden_md)
+ else:
+ self.assertEqual(
+ len(errors),
+ 0,
+ f"\n{len(errors)} golden file mismatches:\n" +
"\n".join(errors),
+ )
+
+ def _to_pandas_cell(self, arr, **to_pandas_kwargs) -> str:
+ """
+ Convert ``arr`` via ``to_pandas(**to_pandas_kwargs)`` and format the
+ result as a golden-file cell, returning ``ERR@<ExceptionClass>`` if the
+ conversion raises.
+ """
+ try:
+ return self.repr_value(arr.to_pandas(**to_pandas_kwargs),
max_len=0)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
+
[email protected](
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
+)
+class PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
+ """
+ Tests pa.Array.to_pandas(coerce_temporal_nanoseconds=True) via golden file
comparison.
+
+ Covers the temporal Arrow types the argument affects (timestamp and
duration in
+ units s/ms/us/ns, tz-aware timestamp, date, and time), plus overflow cases
and
+ a few non-temporal control rows. Each type is tested without nulls, with
a null,
+ and empty.
+
+ The date rows are recorded under two output columns: the default
+ ``date_as_object=True`` (object-dtype ``datetime.date``, which coercion
does not
+ affect) and ``date_as_object=False`` (the numeric ``datetime64[ns]`` path
where
+ coercion takes effect).
+ """
+
+ def _build_source_arrays(self):
+ """Build an ordered dict of named source PyArrow arrays for testing."""
+ sources = {}
+
+ # =====================================================================
+ # Timestamp types (the primary target of coerce_temporal_nanoseconds)
+ # =====================================================================
+ dt1 = datetime.datetime(2024, 1, 1, 12, 0, 0)
+ dt2 = datetime.datetime(2024, 6, 15, 18, 30, 0)
+ for unit in ["s", "ms", "us", "ns"]:
+ sources[f"timestamp[{unit}]:standard"] = pa.array([dt1, dt2],
pa.timestamp(unit))
+ sources[f"timestamp[{unit}]:nullable"] = pa.array([dt1, None],
pa.timestamp(unit))
+ sources[f"timestamp[{unit}]:empty"] = pa.array([],
pa.timestamp(unit))
+ # Timestamp with timezone
+ sources["timestamp[us,tz=UTC]:standard"] = pa.array(
+ [dt1, dt2], pa.timestamp("us", tz="UTC")
+ )
+ sources["timestamp[us,tz=UTC]:nullable"] = pa.array(
+ [dt1, None], pa.timestamp("us", tz="UTC")
+ )
+ sources["timestamp[us,tz=UTC]:empty"] = pa.array([],
pa.timestamp("us", tz="UTC"))
+ # Overflow: coercion to nanoseconds has a valid range (~1677-2262); a
+ # far-future second-resolution timestamp cannot fit and should error.
+ sources["timestamp[s]:overflow"] = pa.array(
+ [datetime.datetime(2500, 1, 1)], pa.timestamp("s")
+ )
+
+ # =====================================================================
+ # Duration types (also coerced to nanoseconds)
+ # =====================================================================
+ td1 = datetime.timedelta(days=1)
+ td2 = datetime.timedelta(hours=2, minutes=30)
+ for unit in ["s", "ms", "us", "ns"]:
+ sources[f"duration[{unit}]:standard"] = pa.array([td1, td2],
pa.duration(unit))
+ sources[f"duration[{unit}]:nullable"] = pa.array([td1, None],
pa.duration(unit))
+ sources[f"duration[{unit}]:empty"] = pa.array([],
pa.duration(unit))
+ # Overflow: a duration beyond ~292 years exceeds the int64 nanosecond
+ # range. Unlike timestamp overflow (which raises), coercing it silently
+ # wraps around to a bogus value; this row pins that behavior.
+ sources["duration[s]:overflow"] = pa.array(
+ [datetime.timedelta(days=300 * 365)], pa.duration("s")
+ )
+
+ # =====================================================================
+ # Date types. With the default date_as_object=True, pandas yields an
+ # object-dtype Series of datetime.date, so coerce_temporal_nanoseconds
+ # has nothing to coerce; the "date_as_object=False" column forces the
+ # numeric datetime64[ns] path where the argument actually takes effect.
+ # =====================================================================
+ d1 = datetime.date(2024, 1, 1)
+ d2 = datetime.date(2024, 6, 15)
+ sources["date32:standard"] = pa.array([d1, d2], pa.date32())
+ sources["date32:nullable"] = pa.array([d1, None], pa.date32())
+ sources["date32:empty"] = pa.array([], pa.date32())
+ sources["date64:standard"] = pa.array([d1, d2], pa.date64())
+ sources["date64:nullable"] = pa.array([d1, None], pa.date64())
+ sources["date64:empty"] = pa.array([], pa.date64())
+
+ # =====================================================================
+ # Time types (control: pandas yields object-dtype datetime.time, no
+ # native time-of-day dtype, so coerce_temporal_nanoseconds is a no-op)
+ # =====================================================================
+ t1 = datetime.time(12, 30, 0)
+ t2 = datetime.time(18, 45, 30)
+ sources["time32[s]:standard"] = pa.array([t1, t2], pa.time32("s"))
+ sources["time32[s]:nullable"] = pa.array([t1, None], pa.time32("s"))
+ sources["time32[s]:empty"] = pa.array([], pa.time32("s"))
+ sources["time32[ms]:standard"] = pa.array([t1, t2], pa.time32("ms"))
+ sources["time32[ms]:nullable"] = pa.array([t1, None], pa.time32("ms"))
+ sources["time32[ms]:empty"] = pa.array([], pa.time32("ms"))
+ sources["time64[us]:standard"] = pa.array([t1, t2], pa.time64("us"))
+ sources["time64[us]:nullable"] = pa.array([t1, None], pa.time64("us"))
+ sources["time64[us]:empty"] = pa.array([], pa.time64("us"))
+ sources["time64[ns]:standard"] = pa.array([t1, t2], pa.time64("ns"))
+ sources["time64[ns]:nullable"] = pa.array([t1, None], pa.time64("ns"))
+ sources["time64[ns]:empty"] = pa.array([], pa.time64("ns"))
+
+ # =====================================================================
+ # Non-temporal controls (unaffected by coerce_temporal_nanoseconds)
+ # =====================================================================
+ sources["int64:standard"] = pa.array([0, 1, -1], pa.int64())
+ sources["int64:nullable"] = pa.array([0, 1, None], pa.int64())
+ sources["float64:standard"] = pa.array([0.0, 1.5, -1.5], pa.float64())
+ sources["string:standard"] = pa.array(["hello", "world", ""],
pa.string())
+
+ return sources
+
+ # Output column recording to_pandas(coerce_temporal_nanoseconds=True).
+ # date_as_object stays at its default (True).
+ COL_PANDAS = "pandas series"
+
+ # Output column recording to_pandas(coerce_temporal_nanoseconds=True,
+ # date_as_object=False). date_as_object=False is the only path on which
+ # coercion observably affects date types.
+ COL_PANDAS_DATE_AS_OBJECT_FALSE = "pandas series (date_as_object=False)"
+
+ def test_to_pandas_coerce_temporal_nanoseconds(self):
+ """Test pa.Array.to_pandas(coerce_temporal_nanoseconds=True) against
golden file."""
+ sources = self._build_source_arrays()
+ row_names = list(sources.keys())
+ col_names = [
+ "pyarrow array",
+ self.COL_PANDAS,
+ self.COL_PANDAS_DATE_AS_OBJECT_FALSE,
+ ]
+
+ # Version-specific expected values go here, keyed by (row, col), when a
+ # newer pandas/PyArrow/NumPy legitimately changes a cell's output.
+ # Add a LooseVersion-guarded block for each known drift.
+ overrides: dict[tuple[str, str], str] = {}
+ # Pandas 3 uses its dedicated string dtype for non-empty Arrow string
arrays.
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ str_expected = "['hello', 'world', '']@Series[str]"
+ overrides[("string:standard", self.COL_PANDAS)] = str_expected
+ overrides[("string:standard",
self.COL_PANDAS_DATE_AS_OBJECT_FALSE)] = str_expected
+
+ def compute_cell(row_name, col_name):
+ arr = sources[row_name]
+ if col_name == "pyarrow array":
+ return self.repr_value(arr, max_len=0)
+ elif col_name == self.COL_PANDAS:
+ return self._to_pandas_cell(arr,
coerce_temporal_nanoseconds=True)
+ elif col_name == self.COL_PANDAS_DATE_AS_OBJECT_FALSE:
+ return self._to_pandas_cell(
+ arr, coerce_temporal_nanoseconds=True, date_as_object=False
+ )
+ else:
+ raise ValueError(f"unknown column: {col_name}")
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+
golden_file_prefix="golden_pyarrow_arrow_to_pandas_coerce_temporal",
+ index_name="test case",
+ overrides=overrides,
+ )
+
+
+if __name__ == "__main__":
+ from pyspark.testing import main
+
+ main()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
