This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 15f12ebf1892 [SPARK-54945][PYTHON][TEST] Add tests for
pa.Array.to_pandas with zero copy
15f12ebf1892 is described below
commit 15f12ebf1892ea503c69e3acfc62a4592ca81e91
Author: Spenser Sun <[email protected]>
AuthorDate: Tue Aug 4 13:43:53 2026 +0800
[SPARK-54945][PYTHON][TEST] Add tests for pa.Array.to_pandas with zero copy
### What changes were proposed in this pull request?
This PR adds a golden-file test that monitors the zero-copy behavior of
`pa.Array.to_pandas`, under the SPARK-54936 umbrella for detecting upstream
PyArrow/pandas behavior drift.
It adds `PyArrowArrayToPandasZeroCopyTests` as a new subclass in the
existing
`python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py`
(reusing the shared `_PyArrowToPandasTestBase` and the centralized
golden-matrix driver), plus a `_verify_zero_copy` helper on the base class.
Three things are recorded per source array:
- **`zero_copy_only=False`** — the default, where PyArrow silently copies
when a view is not possible; records the resulting dtype.
- **`zero_copy_only=True`** — PyArrow's own verdict: `Series[dtype]` when
zero-copy, or `ERRArrowInvalid` when a copy is required.
- **`verified zero-copy`** — an *independent* check of whether the buffers
were actually reused, rather than trusting the flag. It inspects whatever
storage pandas returned: for Arrow-backed results it compares buffer addresses
via the public `__arrow_array__` protocol (keying on that protocol rather than
`isinstance(dtype, pd.ArrowDtype)`, since pandas 3's string dtype is
Arrow-backed without being `ArrowDtype`); otherwise it uses `np.shares_memory`,
which accounts for slice offsets.
An Arrow array is several buffers (validity, offsets, values), so a
result can borrow some while allocating others. The check therefore counts how
many of the *result's* buffers were borrowed and reports `zero-copy` when all
were, `partial-copy` when only some were, and `copied` when none were — rather
than declaring zero-copy on the first buffer that matches, which would let a
conversion reuse a cheap validity bitmap while reallocating the values and
still pass. This is not hypothe [...]
The row set reuses the sibling `test_pyarrow_arrow_to_pandas_default.py`
row set directly, so both golden files pin the same Arrow types, and appends
the layout variants that only matter for zero-copy: sliced (offset) arrays and
single- vs multi-chunk ChunkedArrays.
The two verification columns are recorded as data rather than asserted
equal, because they legitimately disagree: a tz-aware timestamp reports
`zero_copy_only=True` while pandas materializes it into a `DatetimeTZDtype`
array that reuses no Arrow buffer. `to_pandas` is two stages — the
Arrow-to-numpy step is genuinely zero-copy, but the pandas wrapper built
afterwards copies — and the flag only describes the first stage. Pinning both
makes that visible instead of hiding it behind an as [...]
**Scope note:** this PR covers the default (NumPy-backed) path. The
Arrow-backed path (`types_mapper=pd.ArrowDtype`) is a separate argument and
will be a follow-up.
### Why are the changes needed?
PySpark relies on `pa.Array.to_pandas` throughout its conversion layer
(`python/pyspark/sql/pandas/conversion.py`), and whether a given Arrow type
converts without copying its buffers directly affects the memory and latency of
`toPandas` and pandas UDFs. Pinning this behavior in a golden file lets CI fail
loudly if it drifts across PyArrow/pandas/NumPy upgrades, instead of silently
regressing performance.
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
This is itself a test. It runs without a Spark session (it exercises
PyArrow/pandas directly). Verified both with `SPARK_GENERATE_GOLDEN_FILES=1`
(regenerating and reviewing every cell) and without the flag (comparing against
the committed golden file).
Validated against the committed golden across 16 environments — PyArrow 18,
19, 20, 21, 22, 23, 24 and 25, each with pandas 2 and pandas 3 — all passing.
The pandas 3 differences are pinned with version-guarded `overrides` in two
tiers: pandas 3 alone only changes the string dtype, while pandas 3 together
with PyArrow >= 24 is what makes the conversion Arrow-backed, so that is the
only regime where buffers are reused at all and where `partial-copy` can arise.
The golden file itself is [...]
ruff check and format pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57624 from Spenserrrr/zero-copy-only-tests.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../golden_pyarrow_arrow_to_pandas_zero_copy.csv | 126 +++++++++++
.../golden_pyarrow_arrow_to_pandas_zero_copy.md | 127 +++++++++++
.../test_pyarrow_arrow_to_pandas_non_default.py | 235 +++++++++++++++++++++
3 files changed, 488 insertions(+)
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
new file mode 100644
index 000000000000..f2648eccf094
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
@@ -0,0 +1,126 @@
+test case pyarrow array zero_copy_only=False zero_copy_only=True
verified zero-copy
+int8:standard [0, 1, -1, 127, -128]@int8 [0, 1, -1, 127,
-128]@Series[int8] [0, 1, -1, 127, -128]@Series[int8] zero-copy
+int8:nullable [0, 1, None]@int8 [0.0, 1.0, nan]@Series[float64]
ERR@ArrowInvalid copied
+int8:empty []@int8 []@Series[int8] []@Series[int8] zero-copy
+int16:standard [0, 1, -1, 32767, -32768]@int16 [0, 1, -1, 32767,
-32768]@Series[int16] [0, 1, -1, 32767, -32768]@Series[int16] zero-copy
+int16:nullable [0, 1, None]@int16 [0.0, 1.0, nan]@Series[float64]
ERR@ArrowInvalid copied
+int16:empty []@int16 []@Series[int16] []@Series[int16]
zero-copy
+int32:standard [0, 1, -1, 2147483647, -2147483648]@int32 [0, 1, -1,
2147483647, -2147483648]@Series[int32] [0, 1, -1, 2147483647,
-2147483648]@Series[int32] zero-copy
+int32:nullable [0, 1, None]@int32 [0.0, 1.0, nan]@Series[float64]
ERR@ArrowInvalid copied
+int32:empty []@int32 []@Series[int32] []@Series[int32]
zero-copy
+int64:standard [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64
[0, 1, -1, 9223372036854775807, -9223372036854775808]@Series[int64] [0, 1,
-1, 9223372036854775807, -9223372036854775808]@Series[int64] zero-copy
+int64:nullable [0, 1, None]@int64 [0.0, 1.0, nan]@Series[float64]
ERR@ArrowInvalid copied
+int64:empty []@int64 []@Series[int64] []@Series[int64]
zero-copy
+uint8:standard [0, 1, 255]@uint8 [0, 1, 255]@Series[uint8] [0, 1,
255]@Series[uint8] zero-copy
+uint8:nullable [0, 1, None]@uint8 [0.0, 1.0, nan]@Series[float64]
ERR@ArrowInvalid copied
+uint8:empty []@uint8 []@Series[uint8] []@Series[uint8]
zero-copy
+uint16:standard [0, 1, 65535]@uint16 [0, 1, 65535]@Series[uint16]
[0, 1, 65535]@Series[uint16] zero-copy
+uint16:nullable [0, 1, None]@uint16 [0.0, 1.0, nan]@Series[float64]
ERR@ArrowInvalid copied
+uint16:empty []@uint16 []@Series[uint16] []@Series[uint16]
zero-copy
+uint32:standard [0, 1, 4294967295]@uint32 [0, 1,
4294967295]@Series[uint32] [0, 1, 4294967295]@Series[uint32]
zero-copy
+uint32:nullable [0, 1, None]@uint32 [0.0, 1.0, nan]@Series[float64]
ERR@ArrowInvalid copied
+uint32:empty []@uint32 []@Series[uint32] []@Series[uint32]
zero-copy
+uint64:standard [0, 1, 18446744073709551615]@uint64 [0, 1,
18446744073709551615]@Series[uint64] [0, 1,
18446744073709551615]@Series[uint64] zero-copy
+uint64:nullable [0, 1, None]@uint64 [0.0, 1.0, nan]@Series[float64]
ERR@ArrowInvalid copied
+uint64:empty []@uint64 []@Series[uint64] []@Series[uint64]
zero-copy
+float32:standard [0.0, 1.5, -1.5]@float32 [0.0, 1.5,
-1.5]@Series[float32] [0.0, 1.5, -1.5]@Series[float32] zero-copy
+float32:nullable [0.0, 1.5, None]@float32 [0.0, 1.5,
nan]@Series[float32] ERR@ArrowInvalid copied
+float32:empty []@float32 []@Series[float32] []@Series[float32]
zero-copy
+float64:standard [0.0, 1.5, -1.5]@float64 [0.0, 1.5,
-1.5]@Series[float64] [0.0, 1.5, -1.5]@Series[float64] zero-copy
+float64:nullable [0.0, 1.5, None]@float64 [0.0, 1.5,
nan]@Series[float64] ERR@ArrowInvalid copied
+float64:special [nan, inf, -inf]@float64 [nan, inf,
-inf]@Series[float64] [nan, inf, -inf]@Series[float64] zero-copy
+float64:empty []@float64 []@Series[float64] []@Series[float64]
zero-copy
+bool:standard [True, False, True]@bool [True, False,
True]@Series[bool] ERR@ArrowInvalid copied
+bool:nullable [True, False, None]@bool [True, False,
None]@Series[object] ERR@ArrowInvalid copied
+bool:empty []@bool []@Series[bool] ERR@ArrowInvalid copied
+string:standard [hello, world, ]@string ['hello', 'world',
'']@Series[object] ERR@ArrowInvalid copied
+string:nullable [hello, None, world]@string ['hello', None,
'world']@Series[object] ERR@ArrowInvalid copied
+string:empty []@string []@Series[object] ERR@ArrowInvalid
copied
+large_string:standard [hello, world]@large_string ['hello',
'world']@Series[object] ERR@ArrowInvalid copied
+large_string:nullable [hello, None]@large_string ['hello',
None]@Series[object] ERR@ArrowInvalid copied
+large_string:empty []@large_string []@Series[object]
ERR@ArrowInvalid copied
+binary:standard [b'hello', b'world']@binary [b'hello',
b'world']@Series[object] ERR@ArrowInvalid copied
+binary:nullable [b'hello', None]@binary [b'hello', None]@Series[object]
ERR@ArrowInvalid copied
+binary:empty []@binary []@Series[object] ERR@ArrowInvalid
copied
+large_binary:standard [b'hello', b'world']@large_binary [b'hello',
b'world']@Series[object] ERR@ArrowInvalid copied
+large_binary:nullable [b'hello', None]@large_binary [b'hello',
None]@Series[object] ERR@ArrowInvalid copied
+large_binary:empty []@large_binary []@Series[object]
ERR@ArrowInvalid copied
+decimal128:standard [1.23, 4.56, -7.89]@decimal128(5, 2)
[Decimal('1.23'), Decimal('4.56'), Decimal('-7.89')]@Series[object]
ERR@ArrowInvalid copied
+decimal128:nullable [1.23, None, 4.56]@decimal128(5, 2)
[Decimal('1.23'), None, Decimal('4.56')]@Series[object] ERR@ArrowInvalid
copied
+decimal128:empty []@decimal128(5, 2) []@Series[object]
ERR@ArrowInvalid copied
+date32:standard [2024-01-01, 2024-06-15]@date32[day]
[datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
ERR@ArrowInvalid copied
+date32:nullable [2024-01-01, None]@date32[day] [datetime.date(2024, 1,
1), None]@Series[object] ERR@ArrowInvalid copied
+date32:empty []@date32[day] []@Series[object] ERR@ArrowInvalid
copied
+date64:standard [2024-01-01, 2024-06-15]@date64[ms]
[datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
ERR@ArrowInvalid copied
+date64:nullable [2024-01-01, None]@date64[ms] [datetime.date(2024, 1,
1), None]@Series[object] ERR@ArrowInvalid copied
+date64:empty []@date64[ms] []@Series[object] ERR@ArrowInvalid
copied
+timestamp[s]:standard [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[s]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[s]] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[s]] zero-copy
+timestamp[s]:nullable [2024-01-01 12:00:00, None]@timestamp[s]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[s]]
ERR@ArrowInvalid copied
+timestamp[s]:empty []@timestamp[s] []@Series[datetime64[s]]
[]@Series[datetime64[s]] zero-copy
+timestamp[ms]:standard [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ms] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ms]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ms]] zero-copy
+timestamp[ms]:nullable [2024-01-01 12:00:00, None]@timestamp[ms]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ms]]
ERR@ArrowInvalid copied
+timestamp[ms]:empty []@timestamp[ms] []@Series[datetime64[ms]]
[]@Series[datetime64[ms]] zero-copy
+timestamp[us]:standard [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[us]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[us]] zero-copy
+timestamp[us]:nullable [2024-01-01 12:00:00, None]@timestamp[us]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[us]]
ERR@ArrowInvalid copied
+timestamp[us]:empty []@timestamp[us] []@Series[datetime64[us]]
[]@Series[datetime64[us]] zero-copy
+timestamp[ns]:standard [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ns] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] zero-copy
+timestamp[ns]:nullable [2024-01-01 12:00:00, None]@timestamp[ns]
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
ERR@ArrowInvalid copied
+timestamp[ns]:empty []@timestamp[ns] []@Series[datetime64[ns]]
[]@Series[datetime64[ns]] zero-copy
+timestamp[us,tz=UTC]:standard [2024-01-01 12:00:00+00:00, 2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[us, UTC]] [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[us, UTC]] copied
+timestamp[us,tz=UTC]:nullable [2024-01-01 12:00:00+00:00, None]@timestamp[us,
tz=UTC] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'),
NaT]@Series[datetime64[us, UTC]] ERR@ArrowInvalid copied
+timestamp[us,tz=UTC]:empty []@timestamp[us, tz=UTC]
[]@Series[datetime64[us, UTC]] []@Series[datetime64[us, UTC]] copied
+duration[s]:standard [1 day, 0:00:00, 2:30:00]@duration[s] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[s]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[s]] zero-copy
+duration[s]:nullable [1 day, 0:00:00, None]@duration[s] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[s]] ERR@ArrowInvalid copied
+duration[s]:empty []@duration[s] []@Series[timedelta64[s]]
[]@Series[timedelta64[s]] zero-copy
+duration[ms]:standard [1 day, 0:00:00, 2:30:00]@duration[ms] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[ms]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ms]] zero-copy
+duration[ms]:nullable [1 day, 0:00:00, None]@duration[ms] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[ms]] ERR@ArrowInvalid copied
+duration[ms]:empty []@duration[ms] []@Series[timedelta64[ms]]
[]@Series[timedelta64[ms]] zero-copy
+duration[us]:standard [1 day, 0:00:00, 2:30:00]@duration[us] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[timedelta64[us]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[us]] zero-copy
+duration[us]:nullable [1 day, 0:00:00, None]@duration[us] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[us]] ERR@ArrowInvalid copied
+duration[us]:empty []@duration[us] []@Series[timedelta64[us]]
[]@Series[timedelta64[us]] zero-copy
+duration[ns]:standard [1 days 00:00:00, 0 days 02:30:00]@duration[ns]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:30:00')]@Series[timedelta64[ns]] zero-copy
+duration[ns]:nullable [1 days 00:00:00, None]@duration[ns] [Timedelta('1
days 00:00:00'), NaT]@Series[timedelta64[ns]] ERR@ArrowInvalid copied
+duration[ns]:empty []@duration[ns] []@Series[timedelta64[ns]]
[]@Series[timedelta64[ns]] zero-copy
+time32[s]:standard [12:30:00, 18:45:30]@time32[s] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] ERR@ArrowInvalid copied
+time32[s]:nullable [12:30:00, None]@time32[s] [datetime.time(12, 30),
None]@Series[object] ERR@ArrowInvalid copied
+time32[s]:empty []@time32[s] []@Series[object]
ERR@ArrowInvalid copied
+time32[ms]:standard [12:30:00, 18:45:30]@time32[ms] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] ERR@ArrowInvalid copied
+time32[ms]:nullable [12:30:00, None]@time32[ms] [datetime.time(12, 30),
None]@Series[object] ERR@ArrowInvalid copied
+time32[ms]:empty []@time32[ms] []@Series[object]
ERR@ArrowInvalid copied
+time64[us]:standard [12:30:00, 18:45:30]@time64[us] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] ERR@ArrowInvalid copied
+time64[us]:nullable [12:30:00, None]@time64[us] [datetime.time(12, 30),
None]@Series[object] ERR@ArrowInvalid copied
+time64[us]:empty []@time64[us] []@Series[object]
ERR@ArrowInvalid copied
+time64[ns]:standard [12:30:00, 18:45:30]@time64[ns] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[object] ERR@ArrowInvalid copied
+time64[ns]:nullable [12:30:00, None]@time64[ns] [datetime.time(12, 30),
None]@Series[object] ERR@ArrowInvalid copied
+time64[ns]:empty []@time64[ns] []@Series[object]
ERR@ArrowInvalid copied
+null:standard [None, None, None]@null [None, None, None]@Series[object]
ERR@ArrowInvalid copied
+null:empty []@null []@Series[object] ERR@ArrowInvalid copied
+list<int64>:standard [[1, 2], [3, 4, 5]]@list<item: int64> [array([1, 2]),
array([3, 4, 5])]@Series[object] ERR@ArrowInvalid copied
+list<int64>:nullable [[1, 2], None, [3]]@list<item: int64> [array([1, 2]),
None, array([3])]@Series[object] ERR@ArrowInvalid copied
+list<int64>:empty []@list<item: int64> []@Series[object]
ERR@ArrowInvalid copied
+list<string>:standard [['a', 'b'], ['c']]@list<item: string> [array(['a',
'b'], dtype=object), array(['c'], dtype=object)]@Series[object]
ERR@ArrowInvalid copied
+large_list<int64>:standard [[1, 2], [3, 4]]@large_list<item: int64>
[array([1, 2]), array([3, 4])]@Series[object] ERR@ArrowInvalid copied
+large_list<int64>:empty []@large_list<item: int64>
[]@Series[object] ERR@ArrowInvalid copied
+fixed_size_list<int64>[3]:standard [[1, 2, 3], [4, 5,
6]]@fixed_size_list<item: int64>[3] [array([1, 2, 3]), array([4, 5,
6])]@Series[object] ERR@ArrowInvalid copied
+fixed_size_list<int64>[3]:empty []@fixed_size_list<item: int64>[3]
[]@Series[object] ERR@ArrowInvalid copied
+struct:standard [[('x', 1), ('y', 'a')], [('x', 2), ('y',
'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[object] ERR@ArrowInvalid copied
+struct:nullable [[('x', 1), ('y', 'a')], None]@struct<x: int64, y:
string> [{'x': 1, 'y': 'a'}, None]@Series[object] ERR@ArrowInvalid
copied
+struct:empty []@struct<x: int64, y: string> []@Series[object]
ERR@ArrowInvalid copied
+map<string,int64>:standard [[('a', 1), ('b', 2)], [('c', 3)]]@map<string,
int64> [[('a', 1), ('b', 2)], [('c', 3)]]@Series[object]
ERR@ArrowInvalid copied
+map<string,int64>:empty []@map<string, int64> []@Series[object]
ERR@ArrowInvalid copied
+list<list<int64>>:standard [[[1, 2], [3]], [[4, 5, 6]]]@list<item:
list<item: int64>> [array([array([1, 2]), array([3])], dtype=object),
array([array([4, 5, 6])], dtype=object)]@Series[object] ERR@ArrowInvalid
copied
+list<struct>:standard [[{'x': 1}, {'x': 2}], [{'x': 3}]]@list<item: struct<x:
int64>> [array([{'x': 1}, {'x': 2}], dtype=object), array([{'x': 3}],
dtype=object)]@Series[object] ERR@ArrowInvalid copied
+list<map<string,int64>>:standard [[[('a', 1)], [('b', 2)]], [[('c',
3)]]]@list<item: map<string, int64>> [array([list([('a', 1)]), list([('b',
2)])], dtype=object), array([list([('c', 3)])], dtype=object)]@Series[object]
ERR@ArrowInvalid copied
+struct<struct>:standard [[('outer', {'inner': 1})], [('outer',
{'inner': 2})]]@struct<outer: struct<inner: int64>> [{'outer': {'inner':
1}}, {'outer': {'inner': 2}}]@Series[object] ERR@ArrowInvalid
copied
+struct<list<int64>>:standard [[('items', [1, 2, 3])], [('items', [4,
5])]]@struct<items: list<item: int64>> [{'items': array([1, 2, 3])}, {'items':
array([4, 5])}]@Series[object] ERR@ArrowInvalid copied
+struct<map<string,int64>>:standard [[('mapping', [('a', 1)])],
[('mapping', [('b', 2)])]]@struct<mapping: map<string, int64>>
[{'mapping': [('a', 1)]}, {'mapping': [('b', 2)]}]@Series[object]
ERR@ArrowInvalid copied
+map<string,list<int64>>:standard [[('a', [1, 2]), ('b', [3])], [('c',
[4, 5, 6])]]@map<string, list<item: int64>> [[('a', array([1, 2])),
('b', array([3]))], [('c', array([4, 5, 6]))]]@Series[object]
ERR@ArrowInvalid copied
+map<string,struct>:standard [[('a', {'v': 1}), ('b', {'v': 2})], [('c',
{'v': 3})]]@map<string, struct<v: int64>> [[('a', {'v': 1}), ('b', {'v':
2})], [('c', {'v': 3})]]@Series[object] ERR@ArrowInvalid copied
+map<string,map<string,int64>>:standard [[('a', [('x', 1)]), ('b', [('y',
2)])], [('c', [('z', 3)])]]@map<string, map<string, int64>> [[('a', [('x',
1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[object]
ERR@ArrowInvalid copied
+dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string,
indices=int32, ordered=0> ['a', 'b', 'a', 'b']@Series[category]
ERR@ArrowInvalid copied
+dictionary<int32,string>:nullable [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', nan,
'a']@Series[category] ERR@ArrowInvalid copied
+dictionary<int32,string>:empty []@dictionary<values=string, indices=int32,
ordered=0> []@Series[category] ERR@ArrowInvalid copied
+int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64] [2, 3, 4]@Series[int64]
zero-copy
+int64:sliced-with-null [2, None, 4]@int64 [2.0, nan, 4.0]@Series[float64]
ERR@ArrowInvalid copied
+int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2,
3]@Series[int64] zero-copy
+int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64]
ERR@ArrowInvalid copied
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
new file mode 100644
index 000000000000..b1954bdceb27
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
@@ -0,0 +1,127 @@
+| test case | pyarrow array
|
zero_copy_only=False
| zero_copy_only=True
| verified zero-copy |
+|----------------------------------------|-----------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|----------------------|
+| int8:standard | [0, 1, -1, 127, -128]@int8
| [0, 1, -1, 127,
-128]@Series[int8]
| [0, 1, -1, 127, -128]@Series[int8]
| zero-copy |
+| int8:nullable | [0, 1, None]@int8
| [0.0, 1.0,
nan]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| int8:empty | []@int8
| []@Series[int8]
| []@Series[int8]
|
zero-copy |
+| int16:standard | [0, 1, -1, 32767, -32768]@int16
| [0, 1, -1, 32767,
-32768]@Series[int16]
| [0, 1, -1, 32767, -32768]@Series[int16]
|
zero-copy |
+| int16:nullable | [0, 1, None]@int16
| [0.0, 1.0,
nan]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| int16:empty | []@int16
| []@Series[int16]
| []@Series[int16]
|
zero-copy |
+| int32:standard | [0, 1, -1, 2147483647,
-2147483648]@int32 | [0, 1,
-1, 2147483647, -2147483648]@Series[int32]
| [0, 1, -1, 2147483647,
-2147483648]@Series[int32]
| zero-copy |
+| int32:nullable | [0, 1, None]@int32
| [0.0, 1.0,
nan]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| int32:empty | []@int32
| []@Series[int32]
| []@Series[int32]
|
zero-copy |
+| int64:standard | [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 | [0, 1, -1,
9223372036854775807, -9223372036854775808]@Series[int64]
| [0, 1, -1, 9223372036854775807,
-9223372036854775808]@Series[int64]
| zero-copy |
+| int64:nullable | [0, 1, None]@int64
| [0.0, 1.0,
nan]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| int64:empty | []@int64
| []@Series[int64]
| []@Series[int64]
|
zero-copy |
+| uint8:standard | [0, 1, 255]@uint8
| [0, 1,
255]@Series[uint8]
| [0, 1, 255]@Series[uint8]
| zero-copy |
+| uint8:nullable | [0, 1, None]@uint8
| [0.0, 1.0,
nan]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| uint8:empty | []@uint8
| []@Series[uint8]
| []@Series[uint8]
|
zero-copy |
+| uint16:standard | [0, 1, 65535]@uint16
| [0, 1,
65535]@Series[uint16]
| [0, 1, 65535]@Series[uint16]
| zero-copy |
+| uint16:nullable | [0, 1, None]@uint16
| [0.0, 1.0,
nan]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| uint16:empty | []@uint16
| []@Series[uint16]
| []@Series[uint16]
|
zero-copy |
+| uint32:standard | [0, 1, 4294967295]@uint32
| [0, 1,
4294967295]@Series[uint32]
| [0, 1, 4294967295]@Series[uint32]
| zero-copy |
+| uint32:nullable | [0, 1, None]@uint32
| [0.0, 1.0,
nan]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| uint32:empty | []@uint32
| []@Series[uint32]
| []@Series[uint32]
|
zero-copy |
+| uint64:standard | [0, 1, 18446744073709551615]@uint64
| [0, 1,
18446744073709551615]@Series[uint64]
| [0, 1,
18446744073709551615]@Series[uint64]
| zero-copy |
+| uint64:nullable | [0, 1, None]@uint64
| [0.0, 1.0,
nan]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| uint64:empty | []@uint64
| []@Series[uint64]
| []@Series[uint64]
|
zero-copy |
+| float32:standard | [0.0, 1.5, -1.5]@float32
| [0.0, 1.5,
-1.5]@Series[float32]
| [0.0, 1.5, -1.5]@Series[float32]
| zero-copy |
+| float32:nullable | [0.0, 1.5, None]@float32
| [0.0, 1.5,
nan]@Series[float32]
| ERR@ArrowInvalid
| copied |
+| float32:empty | []@float32
| []@Series[float32]
| []@Series[float32]
|
zero-copy |
+| float64:standard | [0.0, 1.5, -1.5]@float64
| [0.0, 1.5,
-1.5]@Series[float64]
| [0.0, 1.5, -1.5]@Series[float64]
| zero-copy |
+| float64:nullable | [0.0, 1.5, None]@float64
| [0.0, 1.5,
nan]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| float64:special | [nan, inf, -inf]@float64
| [nan, inf,
-inf]@Series[float64]
| [nan, inf, -inf]@Series[float64]
| zero-copy |
+| float64:empty | []@float64
| []@Series[float64]
| []@Series[float64]
|
zero-copy |
+| bool:standard | [True, False, True]@bool
| [True, False,
True]@Series[bool]
| ERR@ArrowInvalid
| copied |
+| bool:nullable | [True, False, None]@bool
| [True, False,
None]@Series[object]
| ERR@ArrowInvalid
| copied |
+| bool:empty | []@bool
| []@Series[bool]
| ERR@ArrowInvalid
|
copied |
+| string:standard | [hello, world, ]@string
| ['hello', 'world',
'']@Series[object]
| ERR@ArrowInvalid
|
copied |
+| string:nullable | [hello, None, world]@string
| ['hello', None,
'world']@Series[object]
| ERR@ArrowInvalid
| copied |
+| string:empty | []@string
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| large_string:standard | [hello, world]@large_string
| ['hello',
'world']@Series[object]
| ERR@ArrowInvalid
| copied |
+| large_string:nullable | [hello, None]@large_string
| ['hello',
None]@Series[object]
| ERR@ArrowInvalid
| copied |
+| large_string:empty | []@large_string
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| binary:standard | [b'hello', b'world']@binary
| [b'hello',
b'world']@Series[object]
| ERR@ArrowInvalid
| copied |
+| binary:nullable | [b'hello', None]@binary
| [b'hello',
None]@Series[object]
| ERR@ArrowInvalid
| copied |
+| binary:empty | []@binary
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| large_binary:standard | [b'hello', b'world']@large_binary
| [b'hello',
b'world']@Series[object]
| ERR@ArrowInvalid
| copied |
+| large_binary:nullable | [b'hello', None]@large_binary
| [b'hello',
None]@Series[object]
| ERR@ArrowInvalid
| copied |
+| large_binary:empty | []@large_binary
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| decimal128:standard | [1.23, 4.56, -7.89]@decimal128(5,
2) | [Decimal('1.23'),
Decimal('4.56'), Decimal('-7.89')]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| decimal128:nullable | [1.23, None, 4.56]@decimal128(5, 2)
| [Decimal('1.23'),
None, Decimal('4.56')]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| decimal128:empty | []@decimal128(5, 2)
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| date32:standard | [2024-01-01,
2024-06-15]@date32[day]
| [datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
| ERR@ArrowInvalid
| copied |
+| date32:nullable | [2024-01-01, None]@date32[day]
|
[datetime.date(2024, 1, 1), None]@Series[object]
| ERR@ArrowInvalid
| copied |
+| date32:empty | []@date32[day]
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| date64:standard | [2024-01-01, 2024-06-15]@date64[ms]
|
[datetime.date(2024, 1, 1), datetime.date(2024, 6, 15)]@Series[object]
| ERR@ArrowInvalid
| copied |
+| date64:nullable | [2024-01-01, None]@date64[ms]
|
[datetime.date(2024, 1, 1), None]@Series[object]
| ERR@ArrowInvalid
| copied |
+| date64:empty | []@date64[ms]
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| timestamp[s]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[s] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[s]] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[s]] |
zero-copy |
+| timestamp[s]:nullable | [2024-01-01 12:00:00,
None]@timestamp[s] |
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[s]]
| ERR@ArrowInvalid
| copied |
+| timestamp[s]:empty | []@timestamp[s]
|
[]@Series[datetime64[s]]
| []@Series[datetime64[s]]
| zero-copy |
+| timestamp[ms]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ms] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ms]] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ms]] |
zero-copy |
+| timestamp[ms]:nullable | [2024-01-01 12:00:00,
None]@timestamp[ms] |
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ms]]
| ERR@ArrowInvalid
| copied |
+| timestamp[ms]:empty | []@timestamp[ms]
|
[]@Series[datetime64[ms]]
| []@Series[datetime64[ms]]
| zero-copy |
+| timestamp[us]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[us]] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[us]] |
zero-copy |
+| timestamp[us]:nullable | [2024-01-01 12:00:00,
None]@timestamp[us] |
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[us]]
| ERR@ArrowInvalid
| copied |
+| timestamp[us]:empty | []@timestamp[us]
|
[]@Series[datetime64[us]]
| []@Series[datetime64[us]]
| zero-copy |
+| timestamp[ns]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ns] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] |
zero-copy |
+| timestamp[ns]:nullable | [2024-01-01 12:00:00,
None]@timestamp[ns] |
[Timestamp('2024-01-01 12:00:00'), NaT]@Series[datetime64[ns]]
| ERR@ArrowInvalid
| copied |
+| timestamp[ns]:empty | []@timestamp[ns]
|
[]@Series[datetime64[ns]]
| []@Series[datetime64[ns]]
| zero-copy |
+| timestamp[us,tz=UTC]:standard | [2024-01-01 12:00:00+00:00,
2024-06-15 18:30:00+00:00]@timestamp[us, tz=UTC] |
[Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15
18:30:00+0000', tz='UTC')]@Series[datetime64[us, UTC]] | [Timestamp('2024-01-01
12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[us, UTC]] | copied |
+| timestamp[us,tz=UTC]:nullable | [2024-01-01 12:00:00+00:00,
None]@timestamp[us, tz=UTC] |
[Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), NaT]@Series[datetime64[us,
UTC]] | ERR@ArrowInvalid
| copied |
+| timestamp[us,tz=UTC]:empty | []@timestamp[us, tz=UTC]
|
[]@Series[datetime64[us, UTC]]
| []@Series[datetime64[us, UTC]]
| copied |
+| duration[s]:standard | [1 day, 0:00:00,
2:30:00]@duration[s] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[s]] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[s]] |
zero-copy |
+| duration[s]:nullable | [1 day, 0:00:00, None]@duration[s]
| [Timedelta('1 days
00:00:00'), NaT]@Series[timedelta64[s]]
| ERR@ArrowInvalid
|
copied |
+| duration[s]:empty | []@duration[s]
|
[]@Series[timedelta64[s]]
| []@Series[timedelta64[s]]
| zero-copy |
+| duration[ms]:standard | [1 day, 0:00:00,
2:30:00]@duration[ms] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ms]] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ms]] |
zero-copy |
+| duration[ms]:nullable | [1 day, 0:00:00, None]@duration[ms]
| [Timedelta('1 days
00:00:00'), NaT]@Series[timedelta64[ms]]
| ERR@ArrowInvalid
|
copied |
+| duration[ms]:empty | []@duration[ms]
|
[]@Series[timedelta64[ms]]
| []@Series[timedelta64[ms]]
| zero-copy |
+| duration[us]:standard | [1 day, 0:00:00,
2:30:00]@duration[us] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[us]] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[us]] |
zero-copy |
+| duration[us]:nullable | [1 day, 0:00:00, None]@duration[us]
| [Timedelta('1 days
00:00:00'), NaT]@Series[timedelta64[us]]
| ERR@ArrowInvalid
|
copied |
+| duration[us]:empty | []@duration[us]
|
[]@Series[timedelta64[us]]
| []@Series[timedelta64[us]]
| zero-copy |
+| duration[ns]:standard | [1 days 00:00:00, 0 days
02:30:00]@duration[ns] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[timedelta64[ns]] |
zero-copy |
+| duration[ns]:nullable | [1 days 00:00:00,
None]@duration[ns] |
[Timedelta('1 days 00:00:00'), NaT]@Series[timedelta64[ns]]
| ERR@ArrowInvalid
| copied |
+| duration[ns]:empty | []@duration[ns]
|
[]@Series[timedelta64[ns]]
| []@Series[timedelta64[ns]]
| zero-copy |
+| time32[s]:standard | [12:30:00, 18:45:30]@time32[s]
| [datetime.time(12,
30), datetime.time(18, 45, 30)]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time32[s]:nullable | [12:30:00, None]@time32[s]
| [datetime.time(12,
30), None]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time32[s]:empty | []@time32[s]
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time32[ms]:standard | [12:30:00, 18:45:30]@time32[ms]
| [datetime.time(12,
30), datetime.time(18, 45, 30)]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time32[ms]:nullable | [12:30:00, None]@time32[ms]
| [datetime.time(12,
30), None]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time32[ms]:empty | []@time32[ms]
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time64[us]:standard | [12:30:00, 18:45:30]@time64[us]
| [datetime.time(12,
30), datetime.time(18, 45, 30)]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time64[us]:nullable | [12:30:00, None]@time64[us]
| [datetime.time(12,
30), None]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time64[us]:empty | []@time64[us]
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time64[ns]:standard | [12:30:00, 18:45:30]@time64[ns]
| [datetime.time(12,
30), datetime.time(18, 45, 30)]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time64[ns]:nullable | [12:30:00, None]@time64[ns]
| [datetime.time(12,
30), None]@Series[object]
| ERR@ArrowInvalid
|
copied |
+| time64[ns]:empty | []@time64[ns]
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| null:standard | [None, None, None]@null
| [None, None,
None]@Series[object]
| ERR@ArrowInvalid
| copied |
+| null:empty | []@null
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| list<int64>:standard | [[1, 2], [3, 4, 5]]@list<item:
int64> | [array([1,
2]), array([3, 4, 5])]@Series[object]
| ERR@ArrowInvalid
| copied |
+| list<int64>:nullable | [[1, 2], None, [3]]@list<item:
int64> | [array([1,
2]), None, array([3])]@Series[object]
| ERR@ArrowInvalid
| copied |
+| list<int64>:empty | []@list<item: int64>
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| list<string>:standard | [['a', 'b'], ['c']]@list<item:
string> | [array(['a',
'b'], dtype=object), array(['c'], dtype=object)]@Series[object]
| ERR@ArrowInvalid
| copied |
+| large_list<int64>:standard | [[1, 2], [3, 4]]@large_list<item:
int64> | [array([1, 2]),
array([3, 4])]@Series[object]
| ERR@ArrowInvalid
| copied |
+| large_list<int64>:empty | []@large_list<item: int64>
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| fixed_size_list<int64>[3]:standard | [[1, 2, 3], [4, 5,
6]]@fixed_size_list<item: int64>[3] |
[array([1, 2, 3]), array([4, 5, 6])]@Series[object]
| ERR@ArrowInvalid
| copied |
+| fixed_size_list<int64>[3]:empty | []@fixed_size_list<item: int64>[3]
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| struct:standard | [[('x', 1), ('y', 'a')], [('x', 2),
('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y':
'a'}, {'x': 2, 'y': 'b'}]@Series[object]
| ERR@ArrowInvalid
| copied |
+| struct:nullable | [[('x', 1), ('y', 'a')],
None]@struct<x: int64, y: string> | [{'x':
1, 'y': 'a'}, None]@Series[object]
| ERR@ArrowInvalid
| copied |
+| struct:empty | []@struct<x: int64, y: string>
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| map<string,int64>:standard | [[('a', 1), ('b', 2)], [('c',
3)]]@map<string, int64> | [[('a', 1),
('b', 2)], [('c', 3)]]@Series[object]
| ERR@ArrowInvalid
| copied |
+| map<string,int64>:empty | []@map<string, int64>
| []@Series[object]
| ERR@ArrowInvalid
|
copied |
+| list<list<int64>>:standard | [[[1, 2], [3]], [[4, 5,
6]]]@list<item: list<item: int64>> |
[array([array([1, 2]), array([3])], dtype=object), array([array([4, 5, 6])],
dtype=object)]@Series[object] | ERR@ArrowInvalid
| copied |
+| list<struct>:standard | [[{'x': 1}, {'x': 2}], [{'x':
3}]]@list<item: struct<x: int64>> | [array([{'x':
1}, {'x': 2}], dtype=object), array([{'x': 3}], dtype=object)]@Series[object]
| ERR@ArrowInvalid
| copied |
+| list<map<string,int64>>:standard | [[[('a', 1)], [('b', 2)]], [[('c',
3)]]]@list<item: map<string, int64>> |
[array([list([('a', 1)]), list([('b', 2)])], dtype=object), array([list([('c',
3)])], dtype=object)]@Series[object] | ERR@ArrowInvalid
| copied |
+| struct<struct>:standard | [[('outer', {'inner': 1})],
[('outer', {'inner': 2})]]@struct<outer: struct<inner: int64>> | [{'outer':
{'inner': 1}}, {'outer': {'inner': 2}}]@Series[object]
| ERR@ArrowInvalid
| copied |
+| struct<list<int64>>:standard | [[('items', [1, 2, 3])], [('items',
[4, 5])]]@struct<items: list<item: int64>> | [{'items':
array([1, 2, 3])}, {'items': array([4, 5])}]@Series[object]
| ERR@ArrowInvalid
| copied |
+| struct<map<string,int64>>:standard | [[('mapping', [('a', 1)])],
[('mapping', [('b', 2)])]]@struct<mapping: map<string, int64>> |
[{'mapping': [('a', 1)]}, {'mapping': [('b', 2)]}]@Series[object]
| ERR@ArrowInvalid
| copied |
+| map<string,list<int64>>:standard | [[('a', [1, 2]), ('b', [3])],
[('c', [4, 5, 6])]]@map<string, list<item: int64>> | [[('a',
array([1, 2])), ('b', array([3]))], [('c', array([4, 5, 6]))]]@Series[object]
| ERR@ArrowInvalid
| copied |
+| map<string,struct>:standard | [[('a', {'v': 1}), ('b', {'v':
2})], [('c', {'v': 3})]]@map<string, struct<v: int64>> | [[('a', {'v':
1}), ('b', {'v': 2})], [('c', {'v': 3})]]@Series[object]
| ERR@ArrowInvalid
| copied |
+| map<string,map<string,int64>>:standard | [[('a', [('x', 1)]), ('b', [('y',
2)])], [('c', [('z', 3)])]]@map<string, map<string, int64>> | [[('a', [('x',
1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[object]
| ERR@ArrowInvalid
| copied |
+| dictionary<int32,string>:standard | [a, b, a,
b]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', 'a', 'b']@Series[category]
| ERR@ArrowInvalid
| copied |
+| dictionary<int32,string>:nullable | [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', nan, 'a']@Series[category]
| ERR@ArrowInvalid
| copied |
+| dictionary<int32,string>:empty | []@dictionary<values=string,
indices=int32, ordered=0> |
[]@Series[category]
| ERR@ArrowInvalid
| copied |
+| int64:sliced | [2, 3, 4]@int64
| [2, 3,
4]@Series[int64]
| [2, 3, 4]@Series[int64]
| zero-copy |
+| int64:sliced-with-null | [2, None, 4]@int64
| [2.0, nan,
4.0]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| int64:single-chunk | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64]
| [1, 2, 3]@Series[int64]
| zero-copy |
+| int64:multi-chunk | [1, 2, 3, 4]@int64
| [1, 2, 3,
4]@Series[int64]
| ERR@ArrowInvalid
| copied |
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
index ddfc8f759e26..ed660f163aab 100644
---
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
@@ -73,10 +73,16 @@ from pyspark.testing.utils import (
)
from pyspark.testing.goldenutils import GoldenFileTestMixin
+# Imported as a module, not `from ... import
PyArrowArrayToPandasDefaultTests`, so
+# that unittest does not collect and re-run the default test class from this
module.
+from pyspark.tests.upstream.pyarrow import test_pyarrow_arrow_to_pandas_default
+
if have_pandas:
import pandas as pd
if have_pyarrow:
import pyarrow as pa
+if have_numpy:
+ import numpy as np
class _PyArrowToPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
@@ -99,6 +105,86 @@ class _PyArrowToPandasTestBase(GoldenFileTestMixin,
unittest.TestCase):
except Exception as e:
return f"ERR@{type(e).__name__}"
+ @staticmethod
+ def _arrow_buffers(arrow_obj):
+ """
+ Every non-null buffer backing ``arrow_obj``.
+
+ A ChunkedArray has no buffers of its own -- its data lives in its
chunks --
+ so it is expanded first. ``None`` entries (e.g. an absent validity
bitmap)
+ are skipped.
+ """
+ if isinstance(arrow_obj, pa.ChunkedArray):
+ chunks = [arrow_obj.chunk(i) for i in range(arrow_obj.num_chunks)]
+ else:
+ chunks = [arrow_obj]
+
+ buffers = []
+ for chunk in chunks:
+ for buffer in chunk.buffers():
+ if buffer is not None:
+ buffers.append(buffer)
+ return buffers
+
+ @classmethod
+ def _verify_zero_copy(cls, arr, **to_pandas_kwargs) -> str:
+ """
+ Independently verify whether ``to_pandas`` reused ``arr``'s buffers,
+ instead of trusting PyArrow's own ``zero_copy_only`` verdict.
+
+ The check inspects whatever storage pandas returned rather than
assuming a
+ numpy-backed Series, so it stays correct as pandas moves more dtypes to
+ Arrow-backed storage.
+
+ Returns ``"zero-copy"``, ``"partial-copy"``, ``"copied"``, or
+ ``"ERR@<ExceptionClass>"``.
+ """
+ try:
+ series = arr.to_pandas(**to_pandas_kwargs)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
+ backing_array = series.array
+
+ # Arrow-backed result: compare buffer addresses, reading the stored
data
+ # back through the public __arrow_array__ protocol. to_numpy() would
+ # materialize a copy here and wrongly report no sharing. Keying on the
+ # protocol rather than on ArrowDtype also covers dtypes that are
+ # Arrow-backed without being ArrowDtype, such as pandas 3's string.
+ #
+ # The result has several buffers (validity, offsets, values) and can
borrow
+ # some while allocating others, so count how many it borrowed rather
than
+ # stopping at the first match. The result's buffers are the
denominator
+ # because the question is what pandas had to allocate.
+ if hasattr(backing_array, "__arrow_array__"):
+ source_addresses = {buffer.address for buffer in
cls._arrow_buffers(arr)}
+ stored_buffers = cls._arrow_buffers(pa.array(backing_array))
+ borrowed = 0
+ for buffer in stored_buffers:
+ if buffer.address in source_addresses:
+ borrowed += 1
+ if borrowed == len(stored_buffers):
+ return "zero-copy"
+ return "partial-copy" if borrowed else "copied"
+
+ # numpy-backed result: np.shares_memory accounts for slice offsets, so
it
+ # is robust where raw address equality is not.
+ pandas_values = series.to_numpy()
+ for buffer in cls._arrow_buffers(arr):
+ if np.shares_memory(np.frombuffer(buffer, dtype=np.uint8),
pandas_values):
+ return "zero-copy"
+
+ # Zero bytes cannot overlap, so an empty result is zero-copy if the
numpy
+ # array still borrows the Arrow array (numpy's .base is its memory
owner).
+ if len(arr) == 0:
+ owner = pandas_values
+ while (base := getattr(owner, "base", None)) is not None:
+ if base is arr:
+ return "zero-copy"
+ owner = base
+
+ return "copied"
+
@unittest.skipIf(
not have_pyarrow or not have_pandas or not have_numpy,
@@ -258,6 +344,155 @@ class
PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
)
[email protected](
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
+)
+class PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
+ """
+ Tests pa.Array.to_pandas(zero_copy_only=True) via golden file comparison.
+
+ PySpark converts Arrow data to (numpy-backed) pandas objects throughout its
+ conversion layer (``python/pyspark/sql/pandas/conversion.py``); whether a
+ given Arrow type can make that conversion WITHOUT copying its buffers
+ directly affects the memory and latency of ``toPandas`` and pandas UDFs.
+ ``zero_copy_only=True`` makes PyArrow raise ``ArrowInvalid`` instead of
+ silently copying, so it is the natural probe for "is this conversion
+ zero-copy?". These tests record, per Arrow type, whether the conversion is
+ zero-copy so CI fails loudly if that changes across pandas/PyArrow/NumPy
+ upgrades.
+
+ Three output columns are recorded for each source array:
+
+ - ``zero_copy_only=False``: the default, where PyArrow silently copies
when a
+ view is not possible. This always succeeds, and records the resulting
dtype.
+ - ``zero_copy_only=True``: PyArrow's own verdict -- ``Series[dtype]`` when
the
+ conversion is zero-copy, or ``ERR@ArrowInvalid`` when a copy is required.
+ - ``verified zero-copy``: an INDEPENDENT check of whether the conversion
+ actually reused the Arrow buffers, rather than trusting PyArrow's flag.
+ ``zero-copy`` when every buffer the result needs was borrowed, ``copied``
+ when none were, and ``partial-copy`` when only some were (e.g. the values
+ reused but the offsets reallocated). The last two columns are expected
to
+ agree, but are recorded separately rather than asserted equal, because
they
+ do not always: a tz-aware timestamp reports ``zero_copy_only=True`` while
+ pandas materializes it into a ``DatetimeTZDtype`` array that shares
nothing.
+ Pinning both as data makes such disagreements visible instead of hiding
them
+ behind an assertion.
+
+ The row set mirrors ``test_pyarrow_arrow_to_pandas_default.py``'s rows
exactly
+ -- every Arrow type it covers, in its standard / nullable / empty variants
--
+ so both golden files pin the same types, and appends the layout variants
that
+ only matter for zero-copy: sliced (offset) arrays and single- vs
multi-chunk
+ ChunkedArrays.
+ """
+
+ def _build_source_arrays(self):
+ """
+ Reuse every row of the sibling default test, then add the layout
variants
+ that only matter for zero-copy.
+
+ Sharing the row set keeps both golden files pinning the same types, and
+ means a type added to the default test is covered here automatically.
+ A slice still views a contiguous no-null region, so it stays zero-copy
even
+ though its data starts partway into the parent buffer. A single chunk
is
+ zero-copy, but multiple chunks must be concatenated into one contiguous
+ numpy buffer -- a copy. Multi-chunk is the common shape in real
PySpark,
+ where each partition arrives as its own chunk.
+ """
+ default_tests =
test_pyarrow_arrow_to_pandas_default.PyArrowArrayToPandasDefaultTests
+ sources = default_tests._build_source_arrays(self)
+
+ sources["int64:sliced"] = pa.array(list(range(10)),
pa.int64()).slice(2, 3)
+ sources["int64:sliced-with-null"] = pa.array([1, 2, None, 4, 5],
pa.int64()).slice(1, 3)
+ sources["int64:single-chunk"] = pa.chunked_array([pa.array([1, 2, 3],
pa.int64())])
+ sources["int64:multi-chunk"] = pa.chunked_array(
+ [pa.array([1, 2], pa.int64()), pa.array([3, 4], pa.int64())]
+ )
+
+ return sources
+
+ # Output column for the default zero_copy_only=False: PyArrow silently
copies
+ # when a view is not possible, so this always succeeds and records the
dtype.
+ COL_ZERO_COPY_OFF = "zero_copy_only=False"
+
+ # Output column for zero_copy_only=True: PyArrow raises ArrowInvalid
instead
+ # of copying, so this records its verdict on whether a view was possible.
+ COL_ZERO_COPY_ON = "zero_copy_only=True"
+
+ # Output column independently verifying that buffers were actually reused.
+ COL_VERIFIED = "verified zero-copy"
+
+ def test_to_pandas_zero_copy_only(self):
+ """Test pa.Array.to_pandas(zero_copy_only=True/False) against golden
file."""
+ sources = self._build_source_arrays()
+ row_names = list(sources.keys())
+ col_names = [
+ "pyarrow array",
+ self.COL_ZERO_COPY_OFF,
+ self.COL_ZERO_COPY_ON,
+ self.COL_VERIFIED,
+ ]
+
+ # Version-specific expected values go here, keyed by (row, col), when a
+ # newer pandas/PyArrow/NumPy legitimately changes a cell's output.
+ overrides: dict[tuple[str, str], str] = {}
+ # Pandas 3 renders non-empty Arrow string arrays with its dedicated
string
+ # dtype, so the copying conversion reports Series[str] instead of
object.
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ # Pandas stores that dtype as large_string, so `string` keeps its
values
+ # but has its 32-bit offsets rebuilt as 64-bit, while
`large_string`
+ # already matches and passes through untouched.
+ non_empty_strings = [
+ ("string:standard", "['hello', 'world', '']@Series[str]",
"partial-copy"),
+ ("string:nullable", "['hello', nan, 'world']@Series[str]",
"partial-copy"),
+ ("large_string:standard", "['hello', 'world']@Series[str]",
"zero-copy"),
+ ("large_string:nullable", "['hello', nan]@Series[str]",
"zero-copy"),
+ ]
+ for row, expected, _ in non_empty_strings:
+ overrides[(row, self.COL_ZERO_COPY_OFF)] = expected
+
+ # Only from PyArrow 24 is that string conversion actually
Arrow-backed,
+ # so zero_copy_only succeeds and the buffers are genuinely reused.
On
+ # PyArrow < 24 it still materializes, so the pandas 2 expectations
+ # (ERR@ArrowInvalid / copied) remain correct and need no override.
+ if LooseVersion(pa.__version__) >= LooseVersion("24.0.0"):
+ for row, expected, verified in non_empty_strings:
+ overrides[(row, self.COL_ZERO_COPY_ON)] = expected
+ overrides[(row, self.COL_VERIFIED)] = verified
+ # Empty arrays also gain the string dtype in PyArrow 24.
Offsets are
+ # the only buffer an empty result has, so rebuilding them
shares
+ # nothing at all -- fully copied rather than partial.
+ for row, verified in [
+ ("string:empty", "copied"),
+ ("large_string:empty", "zero-copy"),
+ ]:
+ overrides[(row, self.COL_ZERO_COPY_OFF)] = "[]@Series[str]"
+ overrides[(row, self.COL_ZERO_COPY_ON)] = "[]@Series[str]"
+ overrides[(row, self.COL_VERIFIED)] = verified
+
+ def compute_cell(row_name, col_name):
+ arr = sources[row_name]
+ if col_name == "pyarrow array":
+ return self.repr_value(arr, max_len=0)
+ elif col_name == self.COL_ZERO_COPY_OFF:
+ return self._to_pandas_cell(arr, zero_copy_only=False)
+ elif col_name == self.COL_ZERO_COPY_ON:
+ return self._to_pandas_cell(arr, zero_copy_only=True)
+ elif col_name == self.COL_VERIFIED:
+ return self._verify_zero_copy(arr)
+ else:
+ raise ValueError(f"unknown column: {col_name}")
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+ golden_file_prefix="golden_pyarrow_arrow_to_pandas_zero_copy",
+ index_name="test case",
+ overrides=overrides,
+ )
+
+
if __name__ == "__main__":
from pyspark.testing import main
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]