This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7f8403aa1f79 [SPARK-58656][PYTHON][TEST][FOLLOWUP] Add tests for
pa.ChunkedArray.to_pandas
7f8403aa1f79 is described below
commit 7f8403aa1f793c569eb8cc05eab917541ba4f143
Author: Spenser Sun <[email protected]>
AuthorDate: Tue Aug 11 11:41:35 2026 +0800
[SPARK-58656][PYTHON][TEST][FOLLOWUP] Add tests for
pa.ChunkedArray.to_pandas
### What changes were proposed in this pull request?
This is a follow-up to SPARK-58656, which centralized the `to_pandas`
golden test inventory into a shared `_PyArrowToPandasTestBase`. It extends that
inventory to cover `pa.ChunkedArray.to_pandas` across the existing golden
tests, plus the memory-tuning arguments that only apply on the chunked
(Table->columns) path.
Concretely:
- Adds a `_chunked_sources()` group method to the shared base covering the
chunk-count axis (zero-chunk, empty-chunk, single-chunk, and multi-chunk
variants) across representative types (int64, float64, string, list, struct).
Because the base's `_build_source_arrays()` unions all groups, these rows flow
into the default, zero_copy, zero_copy_arrow_backed, and integer_object_nulls
goldens automatically, so each records how chunking interacts with its argument.
- Consolidates the chunk rows that previously lived separately in the
`ZeroCopy` and `IntegerObjectNulls` subclasses into the shared group.
- Adds `PyArrowChunkedArrayToPandasMemoryFlagsTests`, pinning `to_pandas`
under the `self_destruct` / `split_blocks` / `use_threads` bundle that
`toPandas()` uses. The three are recorded as one column because Spark only ever
sets them together, and a final column records that the source ChunkedArray
stays readable after `self_destruct=True` (freeing is an optimization, not a
contract).
- Adds chunked `timestamp[us]` rows to the coerce_temporal test, where
per-chunk unit coercion is uniquely observable.
Spark converts a `pa.ChunkedArray` (not a plain Array) on the
Table->columns path in `python/pyspark/sql/pandas/conversion.py`
(`df.toPandas()`, the empty-dataset branch, and the pandas-UDF paths after
`combine_chunks()`), so these tests pin the upstream behavior PySpark depends
on there.
### Why are the changes needed?
These golden tests act as a drift canary for upstream PyArrow/pandas
behavior under umbrella SPARK-54936. `pa.ChunkedArray.to_pandas` was previously
covered only incidentally by a few scattered rows; this makes the coverage
systematic so a behavior change in a future PyArrow/pandas release fails CI
loudly instead of silently changing what PySpark returns.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
New and updated golden-file tests, verified across a sweep of PyArrow 18-25
x pandas 2 and 3 (16 combinations), all passing. Existing goldens whose row set
changed were regenerated and diffed to confirm the moved chunk cells are
value-identical.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
This pull request and its description were written by Isaac.
Closes #57909 from Spenserrrr/chunkedarray-tests.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
...den_pyarrow_arrow_to_pandas_coerce_temporal.csv | 2 +
...lden_pyarrow_arrow_to_pandas_coerce_temporal.md | 2 +
.../golden_pyarrow_arrow_to_pandas_default.csv | 12 ++
.../golden_pyarrow_arrow_to_pandas_default.md | 14 +-
...yarrow_arrow_to_pandas_integer_object_nulls.csv | 13 +-
...pyarrow_arrow_to_pandas_integer_object_nulls.md | 13 +-
.../golden_pyarrow_arrow_to_pandas_zero_copy.csv | 14 +-
.../golden_pyarrow_arrow_to_pandas_zero_copy.md | 16 +-
...rrow_arrow_to_pandas_zero_copy_arrow_backed.csv | 14 +-
...arrow_arrow_to_pandas_zero_copy_arrow_backed.md | 16 +-
...yarrow_chunked_array_to_pandas_memory_flags.csv | 11 ++
...pyarrow_chunked_array_to_pandas_memory_flags.md | 12 ++
.../test_pyarrow_arrow_to_pandas_default.py | 72 +++++++-
.../test_pyarrow_arrow_to_pandas_non_default.py | 199 ++++++++++++++++++---
14 files changed, 370 insertions(+), 40 deletions(-)
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
index 4f8439cd1d2e..afc22fdfac99 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.csv
@@ -46,6 +46,8 @@ time64[ns]:nullable [12:30:00, None]@time64[ns]
[datetime.time(12, 30), None]@Se
time64[ns]:empty []@time64[ns] []@Series[object]
[]@Series[object]
timestamp[s]:overflow [2500-01-01 00:00:00]@timestamp[s]
ERR@ArrowInvalid ERR@ArrowInvalid
duration[s]:overflow [109500 days, 0:00:00]@duration[s]
[Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
[Timedelta('-104004 days +00:25:26.290448384')]@Series[timedelta64[ns]]
+timestamp[us]:single-chunk [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]]
+timestamp[us]:multi-chunk [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]]
int64:standard [0, 1, -1]@int64 [0, 1, -1]@Series[int64] [0, 1,
-1]@Series[int64]
int64:nullable [0, 1, None]@int64 [0.0, 1.0, nan]@Series[float64] [0.0,
1.0, nan]@Series[float64]
float64:standard [0.0, 1.5, -1.5]@float64 [0.0, 1.5,
-1.5]@Series[float64] [0.0, 1.5, -1.5]@Series[float64]
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
index e6b3a061c97d..3e3e500fc52e 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_coerce_temporal.md
@@ -47,6 +47,8 @@
| time64[ns]:empty | []@time64[ns]
| []@Series[object]
| []@Series[object]
|
| timestamp[s]:overflow | [2500-01-01 00:00:00]@timestamp[s]
| ERR@ArrowInvalid
| ERR@ArrowInvalid
|
| duration[s]:overflow | [109500 days, 0:00:00]@duration[s]
| [Timedelta('-104004 days
+00:25:26.290448384')]@Series[timedelta64[ns]]
| [Timedelta('-104004 days
+00:25:26.290448384')]@Series[timedelta64[ns]]
|
+| timestamp[us]:single-chunk | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] | [Timestamp('2024-01-01
12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
|
+| timestamp[us]:multi-chunk | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] | [Timestamp('2024-01-01
12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
| [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[datetime64[ns]]
|
| int64:standard | [0, 1, -1]@int64
| [0, 1, -1]@Series[int64]
| [0, 1, -1]@Series[int64]
|
| int64:nullable | [0, 1, None]@int64
| [0.0, 1.0, nan]@Series[float64]
| [0.0, 1.0, nan]@Series[float64]
|
| float64:standard | [0.0, 1.5, -1.5]@float64
| [0.0, 1.5, -1.5]@Series[float64]
| [0.0, 1.5, -1.5]@Series[float64]
|
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.csv
index 13450e5634ed..5015d7c396f8 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.csv
@@ -120,3 +120,15 @@ map<string,map<string,int64>>:standard [[('a', [('x',
1)]), ('b', [('y', 2)])],
dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string,
indices=int32, ordered=0> ['a', 'b', 'a', 'b']@Series[category]
dictionary<int32,string>:nullable [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', nan,
'a']@Series[category]
dictionary<int32,string>:empty []@dictionary<values=string, indices=int32,
ordered=0> []@Series[category]
+int64:zero-chunk []@int64 []@Series[int64]
+int64:empty-chunk []@int64 []@Series[int64]
+int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64]
+int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64]
+int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan,
2.0]@Series[float64]
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2, 3]@Series[int64]
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5, 3.5]@Series[float64]
+string:single-chunk [a, b]@string ['a', 'b']@Series[object]
+string:multi-chunk [a, b, c]@string ['a', 'b', 'c']@Series[object]
+string:multi-chunk-nullable [a, None, c]@string ['a', None,
'c']@Series[object]
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64>
[array([1, 2]), array([3]), array([4])]@Series[object]
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y',
'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[object]
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.md
index 04debdc77e03..48d561f5963a 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_default.md
@@ -120,4 +120,16 @@
| map<string,map<string,int64>>:standard | [[('a', [('x', 1)]), ('b', [('y',
2)])], [('c', [('z', 3)])]]@map<string, map<string, int64>> | [[('a', [('x',
1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[object]
|
| dictionary<int32,string>:standard | [a, b, a,
b]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', 'a', 'b']@Series[category]
|
| dictionary<int32,string>:nullable | [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', nan, 'a']@Series[category]
|
-| dictionary<int32,string>:empty | []@dictionary<values=string,
indices=int32, ordered=0> |
[]@Series[category]
|
\ No newline at end of file
+| dictionary<int32,string>:empty | []@dictionary<values=string,
indices=int32, ordered=0> |
[]@Series[category]
|
+| int64:zero-chunk | []@int64
| []@Series[int64]
|
+| int64:empty-chunk | []@int64
| []@Series[int64]
|
+| int64:single-chunk | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64]
|
+| int64:multi-chunk | [1, 2, 3, 4]@int64
| [1, 2, 3,
4]@Series[int64]
|
+| int64:multi-chunk-nullable | [1, None, 2]@int64
| [1.0, nan,
2.0]@Series[float64]
|
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64]
|
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64
| [1.5, 2.5,
3.5]@Series[float64]
|
+| string:single-chunk | [a, b]@string
| ['a',
'b']@Series[object]
|
+| string:multi-chunk | [a, b, c]@string
| ['a', 'b',
'c']@Series[object]
|
+| string:multi-chunk-nullable | [a, None, c]@string
| ['a', None,
'c']@Series[object]
|
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item:
int64> | [array([1,
2]), array([3]), array([4])]@Series[object]
|
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2),
('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y':
'a'}, {'x': 2, 'y': 'b'}]@Series[object]
|
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.csv
index e28e175f4107..e9ef08af1bd0 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.csv
@@ -120,6 +120,18 @@ map<string,map<string,int64>>:standard [[('a', [('x',
1)]), ('b', [('y', 2)])],
dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string,
indices=int32, ordered=0> ['a', 'b', 'a', 'b']@Series[category] ['a',
'b', 'a', 'b']@Series[category] ['a', 'b', 'a', 'b']@Series[category]
dictionary<int32,string>:nullable [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', nan,
'a']@Series[category] ['a', 'b', nan, 'a']@Series[category] ['a', 'b', nan,
'a']@Series[category]
dictionary<int32,string>:empty []@dictionary<values=string, indices=int32,
ordered=0> []@Series[category] []@Series[category] []@Series[category]
+int64:zero-chunk []@int64 []@Series[int64]
[]@Series[int64] []@Series[int64]
+int64:empty-chunk []@int64 []@Series[int64]
[]@Series[int64] []@Series[int64]
+int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2,
3]@Series[int64] [1, 2, 3]@Series[int64]
+int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64]
[1, 2, 3, 4]@Series[int64] [1, 2, 3, 4]@Series[int64]
+int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan,
2.0]@Series[float64] [1, None, 2]@Series[object] [1, None, 2]@Series[object]
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2,
3]@Series[int64] [1, 2, 3]@Series[int64]
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5, 3.5]@Series[float64]
[1.5, 2.5, 3.5]@Series[float64] [1.5, 2.5, 3.5]@Series[float64]
+string:single-chunk [a, b]@string ['a', 'b']@Series[object] ['a',
'b']@Series[object] ['a', 'b']@Series[object]
+string:multi-chunk [a, b, c]@string ['a', 'b', 'c']@Series[object]
['a', 'b', 'c']@Series[object] ['a', 'b', 'c']@Series[object]
+string:multi-chunk-nullable [a, None, c]@string ['a', None,
'c']@Series[object] ['a', None, 'c']@Series[object] ['a', None,
'c']@Series[object]
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64>
[array([1, 2]), array([3]), array([4])]@Series[object] [array([1, 2]),
array([3]), array([4])]@Series[object] [array([1, 2]), array([3]),
array([4])]@Series[object]
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y',
'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[object] [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object]
[{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object]
int8:extremes-nullable [127, -128, None]@int8 [127.0, -128.0,
nan]@Series[float64] [127, -128, None]@Series[object] [127, -128,
None]@Series[object]
int16:extremes-nullable [32767, -32768, None]@int16 [32767.0,
-32768.0, nan]@Series[float64] [32767, -32768, None]@Series[object]
[32767, -32768, None]@Series[object]
int32:extremes-nullable [2147483647, -2147483648, None]@int32
[2147483647.0, -2147483648.0, nan]@Series[float64] [2147483647,
-2147483648, None]@Series[object] [2147483647, -2147483648,
None]@Series[object]
@@ -129,7 +141,6 @@ uint16:extremes-nullable [65535, 0, None]@uint16
[65535.0, 0.0, nan]@Series[floa
uint32:extremes-nullable [4294967295, 0, None]@uint32 [4294967295.0,
0.0, nan]@Series[float64] [4294967295, 0, None]@Series[object]
[4294967295, 0, None]@Series[object]
uint64:extremes-nullable [18446744073709551615, 0, None]@uint64
[1.8446744073709552e+19, 0.0, nan]@Series[float64] [18446744073709551615,
0, None]@Series[object] [18446744073709551615, 0, None]@Series[object]
int64:all-null [None, None]@int64 [nan, nan]@Series[float64] [None,
None]@Series[object] [None, None]@Series[object]
-int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan,
2.0]@Series[float64] [1, None, 2]@Series[object] [1, None, 2]@Series[object]
list<int64>:null-element [[1, None], [2, 3]]@list<item: int64>
[array([ 1., nan]), array([2., 3.])]@Series[object] [array([1, None],
dtype=object), array([2, 3], dtype=object)]@Series[object] [array([1, None],
dtype=object), array([2, 3], dtype=object)]@Series[object]
list<int64>:null-element-extreme [[9223372036854775807,
None]]@list<item: int64> [array([9.22337204e+18,
nan])]@Series[object] [array([9223372036854775807, None],
dtype=object)]@Series[object] [array([9223372036854775807, None],
dtype=object)]@Series[object]
large_list<int64>:null-element [[1, None]]@large_list<item: int64>
[array([ 1., nan])]@Series[object] [array([1, None],
dtype=object)]@Series[object] [array([1, None], dtype=object)]@Series[object]
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.md
index 9545c071aa08..661016dc3df1 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_integer_object_nulls.md
@@ -121,6 +121,18 @@
| dictionary<int32,string>:standard | [a, b, a,
b]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', 'a', 'b']@Series[category]
| ['a', 'b', 'a',
'b']@Series[category]
| ['a', 'b', 'a', 'b']@Series[category]
[...]
| dictionary<int32,string>:nullable | [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', nan, 'a']@Series[category]
| ['a', 'b', nan,
'a']@Series[category]
| ['a', 'b', nan, 'a']@Series[category]
[...]
| dictionary<int32,string>:empty | []@dictionary<values=string,
indices=int32, ordered=0> |
[]@Series[category]
| []@Series[category]
| []@Series[category]
[...]
+| int64:zero-chunk | []@int64
| []@Series[int64]
| []@Series[int64]
|
[]@Series[int64]
[...]
+| int64:empty-chunk | []@int64
| []@Series[int64]
| []@Series[int64]
|
[]@Series[int64]
[...]
+| int64:single-chunk | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64]
| [1, 2, 3]@Series[int64]
| [1, 2, 3]@Series[int64]
[...]
+| int64:multi-chunk | [1, 2, 3, 4]@int64
| [1, 2, 3,
4]@Series[int64]
| [1, 2, 3, 4]@Series[int64]
| [1, 2, 3, 4]@Series[int64]
[...]
+| int64:multi-chunk-nullable | [1, None, 2]@int64
| [1.0, nan,
2.0]@Series[float64]
| [1, None, 2]@Series[object]
| [1, None, 2]@Series[object]
[...]
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64]
| [1, 2, 3]@Series[int64]
| [1, 2, 3]@Series[int64]
[...]
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64
| [1.5, 2.5,
3.5]@Series[float64]
| [1.5, 2.5, 3.5]@Series[float64]
| [1.5, 2.5, 3.5]@Series[float64]
[...]
+| string:single-chunk | [a, b]@string
| ['a',
'b']@Series[object]
| ['a', 'b']@Series[object]
| ['a', 'b']@Series[object]
[...]
+| string:multi-chunk | [a, b, c]@string
| ['a', 'b',
'c']@Series[object]
| ['a', 'b', 'c']@Series[object]
| ['a', 'b', 'c']@Series[object]
[...]
+| string:multi-chunk-nullable | [a, None, c]@string
| ['a', None,
'c']@Series[object]
| ['a', None, 'c']@Series[object]
| ['a', None, 'c']@Series[object]
[...]
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item:
int64> | [array([1,
2]), array([3]), array([4])]@Series[object]
| [array([1, 2]), array([3]),
array([4])]@Series[object]
| [array([1, 2]), array([3]), array([4])]@Series[object]
[...]
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2),
('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y':
'a'}, {'x': 2, 'y': 'b'}]@Series[object]
| [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[object]
| [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object]
[...]
| int8:extremes-nullable | [127, -128, None]@int8
| [127.0, -128.0,
nan]@Series[float64]
| [127, -128, None]@Series[object]
| [127, -128, None]@Series[object]
[...]
| int16:extremes-nullable | [32767, -32768, None]@int16
| [32767.0, -32768.0,
nan]@Series[float64]
| [32767, -32768, None]@Series[object]
|
[32767, -32768, None]@Series[object]
[...]
| int32:extremes-nullable | [2147483647, -2147483648,
None]@int32 |
[2147483647.0, -2147483648.0, nan]@Series[float64]
| [2147483647, -2147483648,
None]@Series[object]
| [2147483647, -2147483648, None]@Series[object]
[...]
@@ -130,7 +142,6 @@
| uint32:extremes-nullable | [4294967295, 0, None]@uint32
| [4294967295.0, 0.0,
nan]@Series[float64]
| [4294967295, 0, None]@Series[object]
|
[4294967295, 0, None]@Series[object]
[...]
| uint64:extremes-nullable | [18446744073709551615, 0,
None]@uint64 |
[1.8446744073709552e+19, 0.0, nan]@Series[float64]
| [18446744073709551615, 0,
None]@Series[object]
| [18446744073709551615, 0, None]@Series[object]
[...]
| int64:all-null | [None, None]@int64
| [nan,
nan]@Series[float64]
| [None, None]@Series[object]
| [None, None]@Series[object]
[...]
-| int64:multi-chunk-nullable | [1, None, 2]@int64
| [1.0, nan,
2.0]@Series[float64]
| [1, None, 2]@Series[object]
| [1, None, 2]@Series[object]
[...]
| list<int64>:null-element | [[1, None], [2, 3]]@list<item:
int64> | [array([ 1.,
nan]), array([2., 3.])]@Series[object]
| [array([1, None], dtype=object), array([2,
3], dtype=object)]@Series[object]
| [array([1, None], dtype=object), array([2, 3],
dtype=object)]@Series[object] [...]
| list<int64>:null-element-extreme | [[9223372036854775807,
None]]@list<item: int64> |
[array([9.22337204e+18, nan])]@Series[object]
| [array([9223372036854775807,
None], dtype=object)]@Series[object]
| [array([9223372036854775807, None],
dtype=object)]@Series[object] [...]
| large_list<int64>:null-element | [[1, None]]@large_list<item: int64>
| [array([ 1.,
nan])]@Series[object]
| [array([1, None],
dtype=object)]@Series[object]
| [array([1, None], dtype=object)]@Series[object]
[...]
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
index f2648eccf094..f587c3753a27 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.csv
@@ -120,7 +120,17 @@ map<string,map<string,int64>>:standard [[('a', [('x',
1)]), ('b', [('y', 2)])],
dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string,
indices=int32, ordered=0> ['a', 'b', 'a', 'b']@Series[category]
ERR@ArrowInvalid copied
dictionary<int32,string>:nullable [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', nan,
'a']@Series[category] ERR@ArrowInvalid copied
dictionary<int32,string>:empty []@dictionary<values=string, indices=int32,
ordered=0> []@Series[category] ERR@ArrowInvalid copied
-int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64] [2, 3, 4]@Series[int64]
zero-copy
-int64:sliced-with-null [2, None, 4]@int64 [2.0, nan, 4.0]@Series[float64]
ERR@ArrowInvalid copied
+int64:zero-chunk []@int64 []@Series[int64]
ERR@ArrowInvalid copied
+int64:empty-chunk []@int64 []@Series[int64]
[]@Series[int64] zero-copy
int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2,
3]@Series[int64] zero-copy
int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64]
ERR@ArrowInvalid copied
+int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan,
2.0]@Series[float64] ERR@ArrowInvalid copied
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2, 3]@Series[int64]
ERR@ArrowInvalid copied
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5, 3.5]@Series[float64]
ERR@ArrowInvalid copied
+string:single-chunk [a, b]@string ['a', 'b']@Series[object]
ERR@ArrowInvalid copied
+string:multi-chunk [a, b, c]@string ['a', 'b', 'c']@Series[object]
ERR@ArrowInvalid copied
+string:multi-chunk-nullable [a, None, c]@string ['a', None,
'c']@Series[object] ERR@ArrowInvalid copied
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64>
[array([1, 2]), array([3]), array([4])]@Series[object] ERR@ArrowInvalid
copied
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y',
'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[object] ERR@ArrowInvalid copied
+int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64] [2, 3, 4]@Series[int64]
zero-copy
+int64:sliced-with-null [2, None, 4]@int64 [2.0, nan, 4.0]@Series[float64]
ERR@ArrowInvalid copied
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
index b1954bdceb27..1091d1e5d3d6 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy.md
@@ -121,7 +121,17 @@
| dictionary<int32,string>:standard | [a, b, a,
b]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', 'a', 'b']@Series[category]
| ERR@ArrowInvalid
| copied |
| dictionary<int32,string>:nullable | [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', nan, 'a']@Series[category]
| ERR@ArrowInvalid
| copied |
| dictionary<int32,string>:empty | []@dictionary<values=string,
indices=int32, ordered=0> |
[]@Series[category]
| ERR@ArrowInvalid
| copied |
-| int64:sliced | [2, 3, 4]@int64
| [2, 3,
4]@Series[int64]
| [2, 3, 4]@Series[int64]
| zero-copy |
-| int64:sliced-with-null | [2, None, 4]@int64
| [2.0, nan,
4.0]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| int64:zero-chunk | []@int64
| []@Series[int64]
| ERR@ArrowInvalid
|
copied |
+| int64:empty-chunk | []@int64
| []@Series[int64]
| []@Series[int64]
|
zero-copy |
| int64:single-chunk | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64]
| [1, 2, 3]@Series[int64]
| zero-copy |
-| int64:multi-chunk | [1, 2, 3, 4]@int64
| [1, 2, 3,
4]@Series[int64]
| ERR@ArrowInvalid
| copied |
\ No newline at end of file
+| int64:multi-chunk | [1, 2, 3, 4]@int64
| [1, 2, 3,
4]@Series[int64]
| ERR@ArrowInvalid
| copied |
+| int64:multi-chunk-nullable | [1, None, 2]@int64
| [1.0, nan,
2.0]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64]
| ERR@ArrowInvalid
| copied |
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64
| [1.5, 2.5,
3.5]@Series[float64]
| ERR@ArrowInvalid
| copied |
+| string:single-chunk | [a, b]@string
| ['a',
'b']@Series[object]
| ERR@ArrowInvalid
| copied |
+| string:multi-chunk | [a, b, c]@string
| ['a', 'b',
'c']@Series[object]
| ERR@ArrowInvalid
| copied |
+| string:multi-chunk-nullable | [a, None, c]@string
| ['a', None,
'c']@Series[object]
| ERR@ArrowInvalid
| copied |
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item:
int64> | [array([1,
2]), array([3]), array([4])]@Series[object]
| ERR@ArrowInvalid
| copied |
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2),
('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y':
'a'}, {'x': 2, 'y': 'b'}]@Series[object]
| ERR@ArrowInvalid
| copied |
+| int64:sliced | [2, 3, 4]@int64
| [2, 3,
4]@Series[int64]
| [2, 3, 4]@Series[int64]
| zero-copy |
+| int64:sliced-with-null | [2, None, 4]@int64
| [2.0, nan,
4.0]@Series[float64]
| ERR@ArrowInvalid
| copied |
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
index 11f2831ca359..55a84ce933e3 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
@@ -120,7 +120,17 @@ map<string,map<string,int64>>:standard [[('a', [('x',
1)]), ('b', [('y', 2)])],
dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string,
indices=int32, ordered=0> ['a', 'b', 'a',
'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]]
['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] zero-copy
dictionary<int32,string>:nullable [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', <NA>,
'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]]
['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] zero-copy
dictionary<int32,string>:empty []@dictionary<values=string, indices=int32,
ordered=0> []@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] []@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] zero-copy
-int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64[pyarrow]] [2, 3,
4]@Series[int64[pyarrow]] zero-copy
-int64:sliced-with-null [2, None, 4]@int64 [2, <NA>,
4]@Series[int64[pyarrow]] [2, <NA>, 4]@Series[int64[pyarrow]] zero-copy
+int64:zero-chunk []@int64 []@Series[int64[pyarrow]]
[]@Series[int64[pyarrow]] zero-copy
+int64:empty-chunk []@int64 []@Series[int64[pyarrow]]
[]@Series[int64[pyarrow]] zero-copy
int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64[pyarrow]]
[1, 2, 3]@Series[int64[pyarrow]] zero-copy
int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3,
4]@Series[int64[pyarrow]] [1, 2, 3, 4]@Series[int64[pyarrow]] zero-copy
+int64:multi-chunk-nullable [1, None, 2]@int64 [1, <NA>,
2]@Series[int64[pyarrow]] [1, <NA>, 2]@Series[int64[pyarrow]] zero-copy
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2,
3]@Series[int64[pyarrow]] [1, 2, 3]@Series[int64[pyarrow]]
zero-copy
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5,
3.5]@Series[double[pyarrow]] [1.5, 2.5, 3.5]@Series[double[pyarrow]] zero-copy
+string:single-chunk [a, b]@string ['a', 'b']@Series[string[pyarrow]]
['a', 'b']@Series[string[pyarrow]] zero-copy
+string:multi-chunk [a, b, c]@string ['a', 'b',
'c']@Series[string[pyarrow]] ['a', 'b', 'c']@Series[string[pyarrow]] zero-copy
+string:multi-chunk-nullable [a, None, c]@string ['a', <NA>,
'c']@Series[string[pyarrow]] ['a', <NA>, 'c']@Series[string[pyarrow]]
zero-copy
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64> [[1,
2], [3], [4]]@Series[list<item: int64>[pyarrow]] [[1, 2], [3],
[4]]@Series[list<item: int64>[pyarrow]] zero-copy
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y',
'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[struct<x: int64, y: string>[pyarrow]] [{'x': 1, 'y': 'a'}, {'x':
2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] zero-copy
+int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64[pyarrow]] [2, 3,
4]@Series[int64[pyarrow]] zero-copy
+int64:sliced-with-null [2, None, 4]@int64 [2, <NA>,
4]@Series[int64[pyarrow]] [2, <NA>, 4]@Series[int64[pyarrow]] zero-copy
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
index ac177461a2ed..daa25761c7b4 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
@@ -121,7 +121,17 @@
| dictionary<int32,string>:standard | [a, b, a,
b]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] | ['a',
'b', 'a', 'b']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] |
zero-copy |
| dictionary<int32,string>:nullable | [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] | ['a',
'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] | zero-copy
|
| dictionary<int32,string>:empty | []@dictionary<values=string,
indices=int32, ordered=0> |
[]@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]]
|
[]@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]]
| zero-copy
|
-| int64:sliced | [2, 3, 4]@int64
| [2, 3,
4]@Series[int64[pyarrow]]
| [2, 3,
4]@Series[int64[pyarrow]]
| zero-copy |
-| int64:sliced-with-null | [2, None, 4]@int64
| [2, <NA>,
4]@Series[int64[pyarrow]]
| [2, <NA>,
4]@Series[int64[pyarrow]]
| zero-copy |
+| int64:zero-chunk | []@int64
|
[]@Series[int64[pyarrow]]
|
[]@Series[int64[pyarrow]]
| zero-copy
|
+| int64:empty-chunk | []@int64
|
[]@Series[int64[pyarrow]]
|
[]@Series[int64[pyarrow]]
| zero-copy
|
| int64:single-chunk | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64[pyarrow]]
| [1, 2,
3]@Series[int64[pyarrow]]
| zero-copy |
-| int64:multi-chunk | [1, 2, 3, 4]@int64
| [1, 2, 3,
4]@Series[int64[pyarrow]]
| [1, 2, 3,
4]@Series[int64[pyarrow]]
| zero-copy |
\ No newline at end of file
+| int64:multi-chunk | [1, 2, 3, 4]@int64
| [1, 2, 3,
4]@Series[int64[pyarrow]]
| [1, 2, 3,
4]@Series[int64[pyarrow]]
| zero-copy |
+| int64:multi-chunk-nullable | [1, None, 2]@int64
| [1, <NA>,
2]@Series[int64[pyarrow]]
| [1, <NA>,
2]@Series[int64[pyarrow]]
| zero-copy |
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64[pyarrow]]
| [1, 2,
3]@Series[int64[pyarrow]]
| zero-copy |
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64
| [1.5, 2.5,
3.5]@Series[double[pyarrow]]
| [1.5, 2.5,
3.5]@Series[double[pyarrow]]
| zero-copy |
+| string:single-chunk | [a, b]@string
| ['a',
'b']@Series[string[pyarrow]]
| ['a',
'b']@Series[string[pyarrow]]
| zero-copy |
+| string:multi-chunk | [a, b, c]@string
| ['a', 'b',
'c']@Series[string[pyarrow]]
| ['a', 'b',
'c']@Series[string[pyarrow]]
| zero-copy |
+| string:multi-chunk-nullable | [a, None, c]@string
| ['a', <NA>,
'c']@Series[string[pyarrow]]
| ['a', <NA>,
'c']@Series[string[pyarrow]]
| zero-copy |
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item:
int64> | [[1, 2], [3],
[4]]@Series[list<item: int64>[pyarrow]]
| [[1, 2], [3],
[4]]@Series[list<item: int64>[pyarrow]]
| zero-copy |
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2),
('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y':
'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]]
| [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[struct<x: int64, y: string>[pyarrow]]
| zero-copy |
+| int64:sliced | [2, 3, 4]@int64
| [2, 3,
4]@Series[int64[pyarrow]]
| [2, 3,
4]@Series[int64[pyarrow]]
| zero-copy |
+| int64:sliced-with-null | [2, None, 4]@int64
| [2, <NA>,
4]@Series[int64[pyarrow]]
| [2, <NA>,
4]@Series[int64[pyarrow]]
| zero-copy |
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.csv
new file mode 100644
index 000000000000..7ab114cfbf89
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.csv
@@ -0,0 +1,11 @@
+test case pyarrow array default spark memory options source readable
after self_destruct
+int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2,
3]@Series[int64] readable
+int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3, 4]@Series[int64]
[1, 2, 3, 4]@Series[int64] readable
+int64:multi-chunk-nullable [1, None, 2]@int64 [1.0, nan,
2.0]@Series[float64] [1.0, nan, 2.0]@Series[float64] readable
+int64:multi-chunk-with-empty [1, 2, 3]@int64 [1, 2, 3]@Series[int64] [1, 2,
3]@Series[int64] readable
+int64:empty-chunk []@int64 []@Series[int64]
[]@Series[int64] readable
+float64:multi-chunk [1.5, 2.5, 3.5]@float64 [1.5, 2.5, 3.5]@Series[float64]
[1.5, 2.5, 3.5]@Series[float64] readable
+string:multi-chunk [a, b, c]@string ['a', 'b', 'c']@Series[object]
['a', 'b', 'c']@Series[object] readable
+string:multi-chunk-nullable [a, None, c]@string ['a', None,
'c']@Series[object] ['a', None, 'c']@Series[object] readable
+list<int64>:multi-chunk [[1, 2], [3], [4]]@list<item: int64>
[array([1, 2]), array([3]), array([4])]@Series[object] [array([1, 2]),
array([3]), array([4])]@Series[object] readable
+struct:multi-chunk [[('x', 1), ('y', 'a')], [('x', 2), ('y',
'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[object] [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object]
readable
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.md
new file mode 100644
index 000000000000..e41a51201015
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_chunked_array_to_pandas_memory_flags.md
@@ -0,0 +1,12 @@
+| test case | pyarrow array
| default
| spark memory options | source
readable after self_destruct |
+|------------------------------|------------------------------------------------------------------------------|---------------------------------------------------------|---------------------------------------------------------|---------------------------------------|
+| int64:single-chunk | [1, 2, 3]@int64
| [1, 2, 3]@Series[int64]
| [1, 2, 3]@Series[int64] | readable
|
+| int64:multi-chunk | [1, 2, 3, 4]@int64
| [1, 2, 3, 4]@Series[int64]
| [1, 2, 3, 4]@Series[int64] | readable
|
+| int64:multi-chunk-nullable | [1, None, 2]@int64
| [1.0, nan, 2.0]@Series[float64]
| [1.0, nan, 2.0]@Series[float64] | readable
|
+| int64:multi-chunk-with-empty | [1, 2, 3]@int64
| [1, 2, 3]@Series[int64]
| [1, 2, 3]@Series[int64] | readable
|
+| int64:empty-chunk | []@int64
| []@Series[int64]
| []@Series[int64] | readable
|
+| float64:multi-chunk | [1.5, 2.5, 3.5]@float64
| [1.5, 2.5, 3.5]@Series[float64]
| [1.5, 2.5, 3.5]@Series[float64] | readable
|
+| string:multi-chunk | [a, b, c]@string
| ['a', 'b', 'c']@Series[object]
| ['a', 'b', 'c']@Series[object] | readable
|
+| string:multi-chunk-nullable | [a, None, c]@string
| ['a', None, 'c']@Series[object]
| ['a', None, 'c']@Series[object] | readable
|
+| list<int64>:multi-chunk | [[1, 2], [3], [4]]@list<item: int64>
| [array([1, 2]), array([3]),
array([4])]@Series[object] | [array([1, 2]), array([3]),
array([4])]@Series[object] | readable |
+| struct:multi-chunk | [[('x', 1), ('y', 'a')], [('x', 2), ('y',
'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[object] | [{'x': 1, 'y': 'a'}, {'x': 2, 'y': 'b'}]@Series[object]
| readable |
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
index a7e2b6e056d1..68c9c4e5de51 100644
---
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_default.py
@@ -80,8 +80,9 @@ class _PyArrowToPandasTestBase(GoldenFileTestMixin,
unittest.TestCase):
Shared machinery for pa.Array.to_pandas() golden file tests.
Holds the conversion helper and the source-array inventory, split into
type-family
- group methods reusable whole (via ``_build_source_arrays``) or one at a
time.
- Defines no ``test_*`` of its own.
+ group methods reusable whole (via ``_build_source_arrays``) or one at a
time, plus a
+ ``_chunked_sources`` layout group of ChunkedArrays (chunk count is
orthogonal to
+ type). Defines no ``test_*`` of its own.
"""
def _to_pandas_cell(self, arr, **to_pandas_kwargs) -> str:
@@ -358,6 +359,69 @@ class _PyArrowToPandasTestBase(GoldenFileTestMixin,
unittest.TestCase):
return sources
+ def _chunked_sources(self):
+ """
+ ChunkedArray inputs covering the chunk-count axis.
+
+ pa.Array and pa.ChunkedArray share one ``to_pandas``, and Spark feeds a
+ ChunkedArray into it on the Table->columns path
+ (``python/pyspark/sql/pandas/conversion.py``): ``df.toPandas()``
builds one per
+ partition (no ``combine_chunks``), the empty-dataset branch uses
+ ``empty_table()`` (a single empty chunk), and the UDF paths
``combine_chunks()``
+ first (one chunk). Chunk count is orthogonal to type, so these rows
reuse a few
+ representative types and vary only how the values are split.
+ """
+ sources = {}
+
+ # =====================================================================
+ # Numeric chunk layouts (int64: the type most Spark columns reduce to)
+ # =====================================================================
+ sources["int64:zero-chunk"] = pa.chunked_array([], type=pa.int64())
+ sources["int64:empty-chunk"] = pa.chunked_array([pa.array([],
pa.int64())])
+ sources["int64:single-chunk"] = pa.chunked_array([pa.array([1, 2, 3],
pa.int64())])
+ sources["int64:multi-chunk"] = pa.chunked_array(
+ [pa.array([1, 2], pa.int64()), pa.array([3, 4], pa.int64())]
+ )
+ # Null in one chunk, not the other. Reachable via cogrouped
applyInPandas, the
+ # one UDF path that does not combine_chunks() first.
+ sources["int64:multi-chunk-nullable"] = pa.chunked_array(
+ [pa.array([1, None], pa.int64()), pa.array([2], pa.int64())]
+ )
+ # Empty chunk between two non-empty ones: concatenation must skip it
cleanly.
+ sources["int64:multi-chunk-with-empty"] = pa.chunked_array(
+ [pa.array([1, 2], pa.int64()), pa.array([], pa.int64()),
pa.array([3], pa.int64())]
+ )
+ sources["float64:multi-chunk"] = pa.chunked_array(
+ [pa.array([1.5, 2.5], pa.float64()), pa.array([3.5], pa.float64())]
+ )
+
+ # =====================================================================
+ # Variable-width chunk layouts (offset buffers, not just values)
+ # =====================================================================
+ sources["string:single-chunk"] = pa.chunked_array([pa.array(["a",
"b"], pa.string())])
+ sources["string:multi-chunk"] = pa.chunked_array(
+ [pa.array(["a", "b"], pa.string()), pa.array(["c"], pa.string())]
+ )
+ sources["string:multi-chunk-nullable"] = pa.chunked_array(
+ [pa.array(["a", None], pa.string()), pa.array(["c"], pa.string())]
+ )
+
+ # =====================================================================
+ # Nested chunk layouts (child arrays split across chunks)
+ # =====================================================================
+ sources["list<int64>:multi-chunk"] = pa.chunked_array(
+ [pa.array([[1, 2], [3]], pa.list_(pa.int64())), pa.array([[4]],
pa.list_(pa.int64()))]
+ )
+ struct_type = pa.struct([("x", pa.int64()), ("y", pa.string())])
+ sources["struct:multi-chunk"] = pa.chunked_array(
+ [
+ pa.array([{"x": 1, "y": "a"}], struct_type),
+ pa.array([{"x": 2, "y": "b"}], struct_type),
+ ]
+ )
+
+ return sources
+
def _build_source_arrays(self):
"""Build an ordered dict of named source PyArrow arrays for testing."""
sources = {}
@@ -367,6 +431,7 @@ class _PyArrowToPandasTestBase(GoldenFileTestMixin,
unittest.TestCase):
self._temporal_sources(),
self._nested_sources(),
self._dictionary_sources(),
+ self._chunked_sources(),
]:
sources.update(group)
return sources
@@ -400,6 +465,9 @@ class
PyArrowArrayToPandasDefaultTests(_PyArrowToPandasTestBase):
("string:nullable", "pandas series"): ("['hello', nan,
'world']@Series[str]"),
("large_string:standard", "pandas series"): ("['hello',
'world']@Series[str]"),
("large_string:nullable", "pandas series"): "['hello',
nan]@Series[str]",
+ ("string:single-chunk", "pandas series"): "['a',
'b']@Series[str]",
+ ("string:multi-chunk", "pandas series"): "['a', 'b',
'c']@Series[str]",
+ ("string:multi-chunk-nullable", "pandas series"): "['a',
nan, 'c']@Series[str]",
}
)
# PyArrow 24 extends the pandas string dtype conversion to empty
arrays.
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
index b619a443912b..84d711156f03 100644
---
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
@@ -106,7 +106,7 @@ class
PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
def _build_source_arrays(self):
"""
Reuse the base's temporal group (the types this argument targets),
then add
- coercion-specific overflow rows and a few non-temporal controls.
+ coercion-specific overflow rows, chunked timestamps, and non-temporal
controls.
"""
sources = self._temporal_sources()
@@ -122,6 +122,18 @@ class
PyArrowArrayToPandasCoerceTemporalTests(_PyArrowToPandasTestBase):
[datetime.timedelta(days=300 * 365)], pa.duration("s")
)
+ # Chunked timestamps: the base's _chunked_sources has no temporal
rows, so pin
+ # here that a chunked (non-ns) timestamp coerces to the same
datetime64[ns] as
+ # the equivalent contiguous Array.
+ dt1 = datetime.datetime(2024, 1, 1, 12, 0, 0)
+ dt2 = datetime.datetime(2024, 6, 15, 18, 30, 0)
+ sources["timestamp[us]:single-chunk"] = pa.chunked_array(
+ [pa.array([dt1, dt2], pa.timestamp("us"))]
+ )
+ sources["timestamp[us]:multi-chunk"] = pa.chunked_array(
+ [pa.array([dt1], pa.timestamp("us")), pa.array([dt2],
pa.timestamp("us"))]
+ )
+
# Non-temporal controls (unaffected by coerce_temporal_nanoseconds).
sources["int64:standard"] = pa.array([0, 1, -1], pa.int64())
sources["int64:nullable"] = pa.array([0, 1, None], pa.int64())
@@ -218,10 +230,10 @@ class
PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
behind an assertion.
The row set mirrors ``test_pyarrow_arrow_to_pandas_default.py``'s rows
exactly
- -- every Arrow type it covers, in its standard / nullable / empty variants
--
- so both golden files pin the same types, and appends the layout variants
that
- only matter for zero-copy: sliced (offset) arrays and single- vs
multi-chunk
- ChunkedArrays.
+ -- every Arrow type it covers, in its standard / nullable / empty
variants, plus
+ the shared ChunkedArray layouts -- so both golden files pin the same
types, and
+ appends the one layout variant that only matters for zero-copy: sliced
(offset)
+ arrays.
A second test method repeats those rows with
``types_mapper=pd.ArrowDtype``, which
asks pandas to keep pointing at the Arrow buffers instead of materializing
them
@@ -233,25 +245,16 @@ class
PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
def _build_source_arrays(self):
"""
- Reuse the base's full inventory, then add the layout variants that only
- matter for zero-copy.
-
- Sharing the row set keeps this and the default golden pinning the same
types,
- and a type added to the base is covered here automatically.
- A slice still views a contiguous no-null region, so it stays zero-copy
even
- though its data starts partway into the parent buffer. A single chunk
is
- zero-copy, but multiple chunks must be concatenated into one contiguous
- numpy buffer -- a copy. Multi-chunk is the common shape in real
PySpark,
- where each partition arrives as its own chunk.
+ Reuse the base's full inventory, then add the sliced layout variants
that only
+ matter for zero-copy: a slice views a contiguous no-null region, so it
stays
+ zero-copy even though its data starts partway into the parent buffer.
The
+ chunk-count variants live in the base's ``_chunked_sources`` so every
golden
+ records their zero-copy behavior, not just this one.
"""
sources = super()._build_source_arrays()
sources["int64:sliced"] = pa.array(list(range(10)),
pa.int64()).slice(2, 3)
sources["int64:sliced-with-null"] = pa.array([1, 2, None, 4, 5],
pa.int64()).slice(1, 3)
- sources["int64:single-chunk"] = pa.chunked_array([pa.array([1, 2, 3],
pa.int64())])
- sources["int64:multi-chunk"] = pa.chunked_array(
- [pa.array([1, 2], pa.int64()), pa.array([3, 4], pa.int64())]
- )
return sources
@@ -371,6 +374,9 @@ class
PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
("string:nullable", "['hello', nan, 'world']@Series[str]",
"partial-copy"),
("large_string:standard", "['hello', 'world']@Series[str]",
"zero-copy"),
("large_string:nullable", "['hello', nan]@Series[str]",
"zero-copy"),
+ ("string:single-chunk", "['a', 'b']@Series[str]",
"partial-copy"),
+ ("string:multi-chunk", "['a', 'b', 'c']@Series[str]",
"partial-copy"),
+ ("string:multi-chunk-nullable", "['a', nan, 'c']@Series[str]",
"partial-copy"),
]
for row, expected, _ in non_empty_strings:
overrides[(row, self.COL_ZERO_COPY_OFF)] = expected
@@ -519,12 +525,6 @@ class
PyArrowArrayToPandasIntegerObjectNullsTests(_PyArrowToPandasTestBase):
# Every value is null, so there is no integer left to convert.
sources["int64:all-null"] = pa.array([None, None], pa.int64())
- # Values split across buffers, with the null in one chunk and not the
other.
- # Reachable in production: cogrouped applyInPandas is the one UDF path
that does
- # not call combine_chunks() first.
- sources["int64:multi-chunk-nullable"] = pa.chunked_array(
- [pa.array([1, None], pa.int64()), pa.array([2], pa.int64())]
- )
# Nested types whose null is an integer ELEMENT, not a missing
sub-list: the
# shared rows only cover the latter, which this argument does not
affect.
@@ -600,6 +600,9 @@ class
PyArrowArrayToPandasIntegerObjectNullsTests(_PyArrowToPandasTestBase):
override_outputs("string:nullable", "['hello', nan,
'world']@Series[str]")
override_outputs("large_string:standard", "['hello',
'world']@Series[str]")
override_outputs("large_string:nullable", "['hello',
nan]@Series[str]")
+ override_outputs("string:single-chunk", "['a', 'b']@Series[str]")
+ override_outputs("string:multi-chunk", "['a', 'b',
'c']@Series[str]")
+ override_outputs("string:multi-chunk-nullable", "['a', nan,
'c']@Series[str]")
# Empty ones stay object until PyArrow 24, so the baseline holds
before that.
# Spark supports PyArrow 18+, so both branches are reachable.
@@ -630,6 +633,152 @@ class
PyArrowArrayToPandasIntegerObjectNullsTests(_PyArrowToPandasTestBase):
)
[email protected](
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
+)
+class PyArrowChunkedArrayToPandasMemoryFlagsTests(_PyArrowToPandasTestBase):
+ """
+ Tests pa.ChunkedArray.to_pandas() under the memory-tuning arguments
+ ``self_destruct`` / ``split_blocks`` / ``use_threads`` via golden file
comparison.
+
+ Spark sets all three together for ``df.toPandas()`` when Arrow
self-destruct is
+ enabled (``python/pyspark/sql/pandas/conversion.py``), freeing each
column's buffers
+ as it is converted to keep peak memory near one column. They form one
unit --
+ freeing (``self_destruct``) needs per-column blocks (``split_blocks``) and
+ single-threaded conversion (``use_threads=False``) -- so this golden
records the
+ bundle (``spark memory options``) against ``default`` rather than a column
per flag.
+ The path always converts a ``pa.ChunkedArray`` (``Table.column(i)``),
hence the rows.
+
+ The arguments tune HOW the conversion runs, not WHAT it produces, so the
bundle is
+ expected to match the default. Pinning that equivalence catches drift: if
these
+ arguments ever alter the output -- or ``self_destruct`` actually consumes
the input,
+ which today it does not at this level -- a cell moves. A final column
records whether
+ the source is still readable after ``self_destruct=True``, since Spark
treats freeing
+ as an optional optimization, not a contract. Each cell rebuilds its
source from a
+ factory so a destructive ``self_destruct`` cannot corrupt another cell.
+ """
+
+ def _memory_flag_source_factories(self):
+ """
+ Named factories, each returning a FRESH ChunkedArray per call, since
+ ``self_destruct=True`` may consume its input. Rows span the
chunk-count axis
+ across numeric, variable-width, and nested types (buffer layout
differs by type).
+ """
+ struct_type = pa.struct([("x", pa.int64()), ("y", pa.string())])
+ return {
+ "int64:single-chunk": lambda: pa.chunked_array([pa.array([1, 2,
3], pa.int64())]),
+ "int64:multi-chunk": lambda: pa.chunked_array(
+ [pa.array([1, 2], pa.int64()), pa.array([3, 4], pa.int64())]
+ ),
+ "int64:multi-chunk-nullable": lambda: pa.chunked_array(
+ [pa.array([1, None], pa.int64()), pa.array([2], pa.int64())]
+ ),
+ "int64:multi-chunk-with-empty": lambda: pa.chunked_array(
+ [pa.array([1, 2], pa.int64()), pa.array([], pa.int64()),
pa.array([3], pa.int64())]
+ ),
+ "int64:empty-chunk": lambda: pa.chunked_array([pa.array([],
pa.int64())]),
+ "float64:multi-chunk": lambda: pa.chunked_array(
+ [pa.array([1.5, 2.5], pa.float64()), pa.array([3.5],
pa.float64())]
+ ),
+ "string:multi-chunk": lambda: pa.chunked_array(
+ [pa.array(["a", "b"], pa.string()), pa.array(["c"],
pa.string())]
+ ),
+ "string:multi-chunk-nullable": lambda: pa.chunked_array(
+ [pa.array(["a", None], pa.string()), pa.array(["c"],
pa.string())]
+ ),
+ "list<int64>:multi-chunk": lambda: pa.chunked_array(
+ [
+ pa.array([[1, 2], [3]], pa.list_(pa.int64())),
+ pa.array([[4]], pa.list_(pa.int64())),
+ ]
+ ),
+ "struct:multi-chunk": lambda: pa.chunked_array(
+ [
+ pa.array([{"x": 1, "y": "a"}], struct_type),
+ pa.array([{"x": 2, "y": "b"}], struct_type),
+ ]
+ ),
+ }
+
+ # Default (no memory arguments) -- the baseline the bundle should match.
+ COL_DEFAULT = "default"
+ # All three flags as convert_arrow_table_to_pandas passes them together.
+ COL_SPARK_MEMORY_OPTIONS = "spark memory options"
+ # Whether the source ChunkedArray is still readable after
self_destruct=True.
+ COL_SOURCE_READABLE = "source readable after self_destruct"
+
+ # Kept as one dict so the column cannot drift from the call site it mirrors
+ # (python/pyspark/sql/pandas/conversion.py).
+ SPARK_MEMORY_OPTIONS = {
+ "self_destruct": True,
+ "split_blocks": True,
+ "use_threads": False,
+ }
+
+ def _source_readable_after_self_destruct(self, factory) -> str:
+ """
+ Convert a fresh source with ``self_destruct=True``, then report
whether it is
+ still readable: ``readable``, or ``unreadable@<ExceptionClass>`` if
freed.
+ """
+ arr = factory()
+ try:
+ arr.to_pandas(self_destruct=True)
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+ try:
+ arr.to_pylist()
+ return "readable"
+ except Exception as e:
+ return f"unreadable@{type(e).__name__}"
+
+ def test_to_pandas_memory_flags(self):
+ """Test pa.ChunkedArray.to_pandas() under the memory-tuning
arguments."""
+ factories = self._memory_flag_source_factories()
+ row_names = list(factories.keys())
+ col_names = [
+ "pyarrow array",
+ self.COL_DEFAULT,
+ self.COL_SPARK_MEMORY_OPTIONS,
+ self.COL_SOURCE_READABLE,
+ ]
+
+ # Version-specific expected values go here, keyed by (row, col), when
a newer
+ # pandas/PyArrow/NumPy legitimately changes a cell's output. These
arguments do
+ # not touch strings, so a string row shifts in both value columns at
once.
+ overrides: dict[tuple[str, str], str] = {}
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ value_cols = [self.COL_DEFAULT, self.COL_SPARK_MEMORY_OPTIONS]
+ for row, expected in [
+ ("string:multi-chunk", "['a', 'b', 'c']@Series[str]"),
+ ("string:multi-chunk-nullable", "['a', nan, 'c']@Series[str]"),
+ ]:
+ for col in value_cols:
+ overrides[(row, col)] = expected
+
+ def compute_cell(row_name, col_name):
+ factory = factories[row_name]
+ if col_name == "pyarrow array":
+ return self.repr_value(factory(), max_len=0)
+ elif col_name == self.COL_DEFAULT:
+ return self._to_pandas_cell(factory())
+ elif col_name == self.COL_SPARK_MEMORY_OPTIONS:
+ return self._to_pandas_cell(factory(),
**self.SPARK_MEMORY_OPTIONS)
+ elif col_name == self.COL_SOURCE_READABLE:
+ return self._source_readable_after_self_destruct(factory)
+ else:
+ raise ValueError(f"unknown column: {col_name}")
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+
golden_file_prefix="golden_pyarrow_chunked_array_to_pandas_memory_flags",
+ index_name="test case",
+ overrides=overrides,
+ )
+
+
if __name__ == "__main__":
from pyspark.testing import main
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
