This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new a6e75dae63de [SPARK-58545][PYTHON][TEST] Add tests for
pa.Array.to_pandas with types_mapper
a6e75dae63de is described below
commit a6e75dae63dee1322a9415419a1faad2285d73f2
Author: Spenser Sun <[email protected]>
AuthorDate: Wed Aug 5 09:37:20 2026 +0000
[SPARK-58545][PYTHON][TEST] Add tests for pa.Array.to_pandas with
types_mapper
### What changes were proposed in this pull request?
This PR adds a golden-file test pinning the behavior of
`pa.Array.to_pandas(types_mapper=pd.ArrowDtype)`, under the umbrella
[SPARK-54936](https://issues.apache.org/jira/browse/SPARK-54936) (monitor
behavior changes from upstream). It is a new test method on the existing
`PyArrowArrayToPandasZeroCopyTests`, so it reuses that class's row set and the
`_verify_zero_copy` / `_arrow_buffers` helpers rather than adding new machinery:
- `test_to_pandas_zero_copy_only_arrow_backed`, with the new golden file
`golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.{csv,md}` (125 rows x 4
columns).
- Columns: `pyarrow array` | `types_mapper=pd.ArrowDtype,
zero_copy_only=False` | `types_mapper=pd.ArrowDtype, zero_copy_only=True` |
`verified zero-copy`.
Both `zero_copy_only` states are recorded even though the flag makes no
observable difference on this backend today. Omitting a column because it looks
redundant would bake that assumption into a monitoring test; recording both
means a version where the flag starts to matter shows up as a diff instead of
passing silently.
No `dev/sparktestsupport/modules.py` change is needed, since this is a new
method in an already-registered test module.
### Why are the changes needed?
PySpark calls this argument in production, at
`ArrowArrayToPandasConversion.convert_numpy`
(`python/pyspark/sql/conversion.py`), for `ByteType` / `ShortType` /
`IntegerType` / `LongType` when
`spark.sql.execution.pythonUDF.pandas.preferIntExtensionDtype` is enabled. That
config exists because the integer dtype handed to a pandas UDF otherwise
depends on whether the current batch happens to contain a null (SPARK-55788),
so a change in how `types_mapper` maps Arrow types would change wh [...]
The existing goldens cover `to_pandas` on the NumPy backend. Asking pandas
for Arrow-backed storage is a different conversion, and the point of that
backend is that it can keep pointing at the Arrow buffers instead of
materializing them into NumPy. Recording it in its own golden file lets the two
be read side by side, and every row in this golden is `zero-copy`: strings,
nested types and multi-chunk arrays, which
`golden_pyarrow_arrow_to_pandas_zero_copy` records as copied on the NumP [...]
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
New golden-file test. Generated with `SPARK_GENERATE_GOLDEN_FILES=1`, then
every cell was reviewed before being committed.
The whole `python/pyspark/tests/upstream/pyarrow/` directory passes (49
passed, 1 skipped), and regeneration is byte-identical, so the golden is
deterministic.
Validated across **PyArrow 18-25 x pandas 2 and pandas 3 = 16/16
combinations pass**, each in a fresh venv running the committed test file
against the committed golden.
All 16 pass with an empty `overrides` dict, which is worth noting because
the sibling methods in this file do need a pandas-3 override for their string
rows. `types_mapper=pd.ArrowDtype` selects the dtype explicitly, so pandas'
default never applies and the pandas-3 change to it does not reach these cells:
the default string conversion goes `object` (pandas 2) -> `str` (pandas 3),
while `types_mapper` reads `string[pyarrow]` on both. This was checked cell by
cell, not just on the stri [...]
Since this golden is uniformly `zero-copy`, the `verified zero-copy` column
was sanity-checked rather than trusted: the empty rows perform a real buffer
comparison rather than a vacuous zero-length one, and multi-chunk arrays
preserve both chunk addresses instead of being combined. The same helper
produces 86 `copied` rows in `golden_pyarrow_arrow_to_pandas_zero_copy`, so it
is not rubber-stamping.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Closes #57771 from Spenserrrr/types-mapper-tests.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit 055c0b9647b5a28ca5d590bdd2ce0cc2a32e3c98)
Signed-off-by: Uros Bojanic <[email protected]>
---
...rrow_arrow_to_pandas_zero_copy_arrow_backed.csv | 126 ++++++++++++++++++++
...arrow_arrow_to_pandas_zero_copy_arrow_backed.md | 127 +++++++++++++++++++++
.../test_pyarrow_arrow_to_pandas_non_default.py | 50 ++++++++
3 files changed, 303 insertions(+)
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
new file mode 100644
index 000000000000..11f2831ca359
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.csv
@@ -0,0 +1,126 @@
+test case pyarrow array types_mapper=pd.ArrowDtype,
zero_copy_only=False types_mapper=pd.ArrowDtype, zero_copy_only=True
verified zero-copy
+int8:standard [0, 1, -1, 127, -128]@int8 [0, 1, -1, 127,
-128]@Series[int8[pyarrow]] [0, 1, -1, 127, -128]@Series[int8[pyarrow]]
zero-copy
+int8:nullable [0, 1, None]@int8 [0, 1, <NA>]@Series[int8[pyarrow]]
[0, 1, <NA>]@Series[int8[pyarrow]] zero-copy
+int8:empty []@int8 []@Series[int8[pyarrow]]
[]@Series[int8[pyarrow]] zero-copy
+int16:standard [0, 1, -1, 32767, -32768]@int16 [0, 1, -1, 32767,
-32768]@Series[int16[pyarrow]] [0, 1, -1, 32767,
-32768]@Series[int16[pyarrow]] zero-copy
+int16:nullable [0, 1, None]@int16 [0, 1, <NA>]@Series[int16[pyarrow]]
[0, 1, <NA>]@Series[int16[pyarrow]] zero-copy
+int16:empty []@int16 []@Series[int16[pyarrow]]
[]@Series[int16[pyarrow]] zero-copy
+int32:standard [0, 1, -1, 2147483647, -2147483648]@int32 [0, 1, -1,
2147483647, -2147483648]@Series[int32[pyarrow]] [0, 1, -1, 2147483647,
-2147483648]@Series[int32[pyarrow]] zero-copy
+int32:nullable [0, 1, None]@int32 [0, 1, <NA>]@Series[int32[pyarrow]]
[0, 1, <NA>]@Series[int32[pyarrow]] zero-copy
+int32:empty []@int32 []@Series[int32[pyarrow]]
[]@Series[int32[pyarrow]] zero-copy
+int64:standard [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64
[0, 1, -1, 9223372036854775807, -9223372036854775808]@Series[int64[pyarrow]]
[0, 1, -1, 9223372036854775807, -9223372036854775808]@Series[int64[pyarrow]]
zero-copy
+int64:nullable [0, 1, None]@int64 [0, 1, <NA>]@Series[int64[pyarrow]]
[0, 1, <NA>]@Series[int64[pyarrow]] zero-copy
+int64:empty []@int64 []@Series[int64[pyarrow]]
[]@Series[int64[pyarrow]] zero-copy
+uint8:standard [0, 1, 255]@uint8 [0, 1, 255]@Series[uint8[pyarrow]]
[0, 1, 255]@Series[uint8[pyarrow]] zero-copy
+uint8:nullable [0, 1, None]@uint8 [0, 1, <NA>]@Series[uint8[pyarrow]]
[0, 1, <NA>]@Series[uint8[pyarrow]] zero-copy
+uint8:empty []@uint8 []@Series[uint8[pyarrow]]
[]@Series[uint8[pyarrow]] zero-copy
+uint16:standard [0, 1, 65535]@uint16 [0, 1,
65535]@Series[uint16[pyarrow]] [0, 1, 65535]@Series[uint16[pyarrow]]
zero-copy
+uint16:nullable [0, 1, None]@uint16 [0, 1,
<NA>]@Series[uint16[pyarrow]] [0, 1, <NA>]@Series[uint16[pyarrow]]
zero-copy
+uint16:empty []@uint16 []@Series[uint16[pyarrow]]
[]@Series[uint16[pyarrow]] zero-copy
+uint32:standard [0, 1, 4294967295]@uint32 [0, 1,
4294967295]@Series[uint32[pyarrow]] [0, 1,
4294967295]@Series[uint32[pyarrow]] zero-copy
+uint32:nullable [0, 1, None]@uint32 [0, 1,
<NA>]@Series[uint32[pyarrow]] [0, 1, <NA>]@Series[uint32[pyarrow]]
zero-copy
+uint32:empty []@uint32 []@Series[uint32[pyarrow]]
[]@Series[uint32[pyarrow]] zero-copy
+uint64:standard [0, 1, 18446744073709551615]@uint64 [0, 1,
18446744073709551615]@Series[uint64[pyarrow]] [0, 1,
18446744073709551615]@Series[uint64[pyarrow]] zero-copy
+uint64:nullable [0, 1, None]@uint64 [0, 1,
<NA>]@Series[uint64[pyarrow]] [0, 1, <NA>]@Series[uint64[pyarrow]]
zero-copy
+uint64:empty []@uint64 []@Series[uint64[pyarrow]]
[]@Series[uint64[pyarrow]] zero-copy
+float32:standard [0.0, 1.5, -1.5]@float32 [0.0, 1.5,
-1.5]@Series[float[pyarrow]] [0.0, 1.5, -1.5]@Series[float[pyarrow]] zero-copy
+float32:nullable [0.0, 1.5, None]@float32 [0.0, 1.5,
<NA>]@Series[float[pyarrow]] [0.0, 1.5, <NA>]@Series[float[pyarrow]] zero-copy
+float32:empty []@float32 []@Series[float[pyarrow]]
[]@Series[float[pyarrow]] zero-copy
+float64:standard [0.0, 1.5, -1.5]@float64 [0.0, 1.5,
-1.5]@Series[double[pyarrow]] [0.0, 1.5, -1.5]@Series[double[pyarrow]]
zero-copy
+float64:nullable [0.0, 1.5, None]@float64 [0.0, 1.5,
<NA>]@Series[double[pyarrow]] [0.0, 1.5, <NA>]@Series[double[pyarrow]]
zero-copy
+float64:special [nan, inf, -inf]@float64 [nan, inf,
-inf]@Series[double[pyarrow]] [nan, inf, -inf]@Series[double[pyarrow]]
zero-copy
+float64:empty []@float64 []@Series[double[pyarrow]]
[]@Series[double[pyarrow]] zero-copy
+bool:standard [True, False, True]@bool [True, False,
True]@Series[bool[pyarrow]] [True, False, True]@Series[bool[pyarrow]]
zero-copy
+bool:nullable [True, False, None]@bool [True, False,
<NA>]@Series[bool[pyarrow]] [True, False, <NA>]@Series[bool[pyarrow]]
zero-copy
+bool:empty []@bool []@Series[bool[pyarrow]]
[]@Series[bool[pyarrow]] zero-copy
+string:standard [hello, world, ]@string ['hello', 'world',
'']@Series[string[pyarrow]] ['hello', 'world', '']@Series[string[pyarrow]]
zero-copy
+string:nullable [hello, None, world]@string ['hello', <NA>,
'world']@Series[string[pyarrow]] ['hello', <NA>,
'world']@Series[string[pyarrow]] zero-copy
+string:empty []@string []@Series[string[pyarrow]]
[]@Series[string[pyarrow]] zero-copy
+large_string:standard [hello, world]@large_string ['hello',
'world']@Series[large_string[pyarrow]] ['hello',
'world']@Series[large_string[pyarrow]] zero-copy
+large_string:nullable [hello, None]@large_string ['hello',
<NA>]@Series[large_string[pyarrow]] ['hello',
<NA>]@Series[large_string[pyarrow]] zero-copy
+large_string:empty []@large_string []@Series[large_string[pyarrow]]
[]@Series[large_string[pyarrow]] zero-copy
+binary:standard [b'hello', b'world']@binary [b'hello',
b'world']@Series[binary[pyarrow]] [b'hello',
b'world']@Series[binary[pyarrow]] zero-copy
+binary:nullable [b'hello', None]@binary [b'hello',
<NA>]@Series[binary[pyarrow]] [b'hello', <NA>]@Series[binary[pyarrow]]
zero-copy
+binary:empty []@binary []@Series[binary[pyarrow]]
[]@Series[binary[pyarrow]] zero-copy
+large_binary:standard [b'hello', b'world']@large_binary [b'hello',
b'world']@Series[large_binary[pyarrow]] [b'hello',
b'world']@Series[large_binary[pyarrow]] zero-copy
+large_binary:nullable [b'hello', None]@large_binary [b'hello',
<NA>]@Series[large_binary[pyarrow]] [b'hello',
<NA>]@Series[large_binary[pyarrow]] zero-copy
+large_binary:empty []@large_binary []@Series[large_binary[pyarrow]]
[]@Series[large_binary[pyarrow]] zero-copy
+decimal128:standard [1.23, 4.56, -7.89]@decimal128(5, 2)
[Decimal('1.23'), Decimal('4.56'), Decimal('-7.89')]@Series[decimal128(5,
2)[pyarrow]] [Decimal('1.23'), Decimal('4.56'),
Decimal('-7.89')]@Series[decimal128(5, 2)[pyarrow]] zero-copy
+decimal128:nullable [1.23, None, 4.56]@decimal128(5, 2)
[Decimal('1.23'), <NA>, Decimal('4.56')]@Series[decimal128(5, 2)[pyarrow]]
[Decimal('1.23'), <NA>, Decimal('4.56')]@Series[decimal128(5, 2)[pyarrow]]
zero-copy
+decimal128:empty []@decimal128(5, 2) []@Series[decimal128(5,
2)[pyarrow]] []@Series[decimal128(5, 2)[pyarrow]] zero-copy
+date32:standard [2024-01-01, 2024-06-15]@date32[day]
[datetime.date(2024, 1, 1), datetime.date(2024, 6,
15)]@Series[date32[day][pyarrow]] [datetime.date(2024, 1, 1),
datetime.date(2024, 6, 15)]@Series[date32[day][pyarrow]] zero-copy
+date32:nullable [2024-01-01, None]@date32[day] [datetime.date(2024, 1,
1), <NA>]@Series[date32[day][pyarrow]] [datetime.date(2024, 1, 1),
<NA>]@Series[date32[day][pyarrow]] zero-copy
+date32:empty []@date32[day] []@Series[date32[day][pyarrow]]
[]@Series[date32[day][pyarrow]] zero-copy
+date64:standard [2024-01-01, 2024-06-15]@date64[ms]
[datetime.date(2024, 1, 1), datetime.date(2024, 6,
15)]@Series[date64[ms][pyarrow]] [datetime.date(2024, 1, 1),
datetime.date(2024, 6, 15)]@Series[date64[ms][pyarrow]] zero-copy
+date64:nullable [2024-01-01, None]@date64[ms] [datetime.date(2024, 1,
1), <NA>]@Series[date64[ms][pyarrow]] [datetime.date(2024, 1, 1),
<NA>]@Series[date64[ms][pyarrow]] zero-copy
+date64:empty []@date64[ms] []@Series[date64[ms][pyarrow]]
[]@Series[date64[ms][pyarrow]] zero-copy
+timestamp[s]:standard [2024-01-01 12:00:00, 2024-06-15 18:30:00]@timestamp[s]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[s][pyarrow]] [Timestamp('2024-01-01
12:00:00'), Timestamp('2024-06-15 18:30:00')]@Series[timestamp[s][pyarrow]]
zero-copy
+timestamp[s]:nullable [2024-01-01 12:00:00, None]@timestamp[s]
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[s][pyarrow]]
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[s][pyarrow]]
zero-copy
+timestamp[s]:empty []@timestamp[s] []@Series[timestamp[s][pyarrow]]
[]@Series[timestamp[s][pyarrow]] zero-copy
+timestamp[ms]:standard [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ms] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ms][pyarrow]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[ms][pyarrow]] zero-copy
+timestamp[ms]:nullable [2024-01-01 12:00:00, None]@timestamp[ms]
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ms][pyarrow]]
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ms][pyarrow]]
zero-copy
+timestamp[ms]:empty []@timestamp[ms]
[]@Series[timestamp[ms][pyarrow]] []@Series[timestamp[ms][pyarrow]]
zero-copy
+timestamp[us]:standard [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[timestamp[us][pyarrow]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[us][pyarrow]] zero-copy
+timestamp[us]:nullable [2024-01-01 12:00:00, None]@timestamp[us]
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[us][pyarrow]]
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[us][pyarrow]]
zero-copy
+timestamp[us]:empty []@timestamp[us]
[]@Series[timestamp[us][pyarrow]] []@Series[timestamp[us][pyarrow]]
zero-copy
+timestamp[ns]:standard [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ns] [Timestamp('2024-01-01 12:00:00'),
Timestamp('2024-06-15 18:30:00')]@Series[timestamp[ns][pyarrow]]
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[ns][pyarrow]] zero-copy
+timestamp[ns]:nullable [2024-01-01 12:00:00, None]@timestamp[ns]
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ns][pyarrow]]
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ns][pyarrow]]
zero-copy
+timestamp[ns]:empty []@timestamp[ns]
[]@Series[timestamp[ns][pyarrow]] []@Series[timestamp[ns][pyarrow]]
zero-copy
+timestamp[us,tz=UTC]:standard [2024-01-01 12:00:00+00:00, 2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] [Timestamp('2024-01-01 12:00:00+0000',
tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[timestamp[us, tz=UTC][pyarrow]] [Timestamp('2024-01-01
12:00:00+0000', tz='UTC'), Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[timestamp[us, tz=UTC][pyarrow]] zero-copy
+timestamp[us,tz=UTC]:nullable [2024-01-01 12:00:00+00:00, None]@timestamp[us,
tz=UTC] [Timestamp('2024-01-01 12:00:00+0000', tz='UTC'),
<NA>]@Series[timestamp[us, tz=UTC][pyarrow]] [Timestamp('2024-01-01
12:00:00+0000', tz='UTC'), <NA>]@Series[timestamp[us, tz=UTC][pyarrow]]
zero-copy
+timestamp[us,tz=UTC]:empty []@timestamp[us, tz=UTC]
[]@Series[timestamp[us, tz=UTC][pyarrow]] []@Series[timestamp[us,
tz=UTC][pyarrow]] zero-copy
+duration[s]:standard [1 day, 0:00:00, 2:30:00]@duration[s] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[s][pyarrow]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[s][pyarrow]] zero-copy
+duration[s]:nullable [1 day, 0:00:00, None]@duration[s] [Timedelta('1
days 00:00:00'), <NA>]@Series[duration[s][pyarrow]] [Timedelta('1 days
00:00:00'), <NA>]@Series[duration[s][pyarrow]] zero-copy
+duration[s]:empty []@duration[s] []@Series[duration[s][pyarrow]]
[]@Series[duration[s][pyarrow]] zero-copy
+duration[ms]:standard [1 day, 0:00:00, 2:30:00]@duration[ms] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[ms][pyarrow]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[ms][pyarrow]] zero-copy
+duration[ms]:nullable [1 day, 0:00:00, None]@duration[ms] [Timedelta('1
days 00:00:00'), <NA>]@Series[duration[ms][pyarrow]] [Timedelta('1 days
00:00:00'), <NA>]@Series[duration[ms][pyarrow]] zero-copy
+duration[ms]:empty []@duration[ms] []@Series[duration[ms][pyarrow]]
[]@Series[duration[ms][pyarrow]] zero-copy
+duration[us]:standard [1 day, 0:00:00, 2:30:00]@duration[us] [Timedelta('1
days 00:00:00'), Timedelta('0 days 02:30:00')]@Series[duration[us][pyarrow]]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[us][pyarrow]] zero-copy
+duration[us]:nullable [1 day, 0:00:00, None]@duration[us] [Timedelta('1
days 00:00:00'), <NA>]@Series[duration[us][pyarrow]] [Timedelta('1 days
00:00:00'), <NA>]@Series[duration[us][pyarrow]] zero-copy
+duration[us]:empty []@duration[us] []@Series[duration[us][pyarrow]]
[]@Series[duration[us][pyarrow]] zero-copy
+duration[ns]:standard [1 days 00:00:00, 0 days 02:30:00]@duration[ns]
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[ns][pyarrow]] [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:30:00')]@Series[duration[ns][pyarrow]] zero-copy
+duration[ns]:nullable [1 days 00:00:00, None]@duration[ns] [Timedelta('1
days 00:00:00'), <NA>]@Series[duration[ns][pyarrow]] [Timedelta('1 days
00:00:00'), <NA>]@Series[duration[ns][pyarrow]] zero-copy
+duration[ns]:empty []@duration[ns] []@Series[duration[ns][pyarrow]]
[]@Series[duration[ns][pyarrow]] zero-copy
+time32[s]:standard [12:30:00, 18:45:30]@time32[s] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time32[s][pyarrow]] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time32[s][pyarrow]] zero-copy
+time32[s]:nullable [12:30:00, None]@time32[s] [datetime.time(12, 30),
<NA>]@Series[time32[s][pyarrow]] [datetime.time(12, 30),
<NA>]@Series[time32[s][pyarrow]] zero-copy
+time32[s]:empty []@time32[s] []@Series[time32[s][pyarrow]]
[]@Series[time32[s][pyarrow]] zero-copy
+time32[ms]:standard [12:30:00, 18:45:30]@time32[ms] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time32[ms][pyarrow]] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time32[ms][pyarrow]] zero-copy
+time32[ms]:nullable [12:30:00, None]@time32[ms] [datetime.time(12, 30),
<NA>]@Series[time32[ms][pyarrow]] [datetime.time(12, 30),
<NA>]@Series[time32[ms][pyarrow]] zero-copy
+time32[ms]:empty []@time32[ms] []@Series[time32[ms][pyarrow]]
[]@Series[time32[ms][pyarrow]] zero-copy
+time64[us]:standard [12:30:00, 18:45:30]@time64[us] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time64[us][pyarrow]] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time64[us][pyarrow]] zero-copy
+time64[us]:nullable [12:30:00, None]@time64[us] [datetime.time(12, 30),
<NA>]@Series[time64[us][pyarrow]] [datetime.time(12, 30),
<NA>]@Series[time64[us][pyarrow]] zero-copy
+time64[us]:empty []@time64[us] []@Series[time64[us][pyarrow]]
[]@Series[time64[us][pyarrow]] zero-copy
+time64[ns]:standard [12:30:00, 18:45:30]@time64[ns] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time64[ns][pyarrow]] [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time64[ns][pyarrow]] zero-copy
+time64[ns]:nullable [12:30:00, None]@time64[ns] [datetime.time(12, 30),
<NA>]@Series[time64[ns][pyarrow]] [datetime.time(12, 30),
<NA>]@Series[time64[ns][pyarrow]] zero-copy
+time64[ns]:empty []@time64[ns] []@Series[time64[ns][pyarrow]]
[]@Series[time64[ns][pyarrow]] zero-copy
+null:standard [None, None, None]@null [<NA>, <NA>,
<NA>]@Series[null[pyarrow]] [<NA>, <NA>, <NA>]@Series[null[pyarrow]]
zero-copy
+null:empty []@null []@Series[null[pyarrow]]
[]@Series[null[pyarrow]] zero-copy
+list<int64>:standard [[1, 2], [3, 4, 5]]@list<item: int64> [[1, 2], [3, 4,
5]]@Series[list<item: int64>[pyarrow]] [[1, 2], [3, 4, 5]]@Series[list<item:
int64>[pyarrow]] zero-copy
+list<int64>:nullable [[1, 2], None, [3]]@list<item: int64> [[1, 2], <NA>,
[3]]@Series[list<item: int64>[pyarrow]] [[1, 2], <NA>, [3]]@Series[list<item:
int64>[pyarrow]] zero-copy
+list<int64>:empty []@list<item: int64> []@Series[list<item:
int64>[pyarrow]] []@Series[list<item: int64>[pyarrow]] zero-copy
+list<string>:standard [['a', 'b'], ['c']]@list<item: string> [['a', 'b'],
['c']]@Series[list<item: string>[pyarrow]] [['a', 'b'],
['c']]@Series[list<item: string>[pyarrow]] zero-copy
+large_list<int64>:standard [[1, 2], [3, 4]]@large_list<item: int64>
[[1, 2], [3, 4]]@Series[large_list<item: int64>[pyarrow]] [[1, 2], [3,
4]]@Series[large_list<item: int64>[pyarrow]] zero-copy
+large_list<int64>:empty []@large_list<item: int64>
[]@Series[large_list<item: int64>[pyarrow]] []@Series[large_list<item:
int64>[pyarrow]] zero-copy
+fixed_size_list<int64>[3]:standard [[1, 2, 3], [4, 5,
6]]@fixed_size_list<item: int64>[3] [[1, 2, 3], [4, 5,
6]]@Series[fixed_size_list<item: int64>[3][pyarrow]] [[1, 2, 3], [4, 5,
6]]@Series[fixed_size_list<item: int64>[3][pyarrow]] zero-copy
+fixed_size_list<int64>[3]:empty []@fixed_size_list<item: int64>[3]
[]@Series[fixed_size_list<item: int64>[3][pyarrow]]
[]@Series[fixed_size_list<item: int64>[3][pyarrow]] zero-copy
+struct:standard [[('x', 1), ('y', 'a')], [('x', 2), ('y',
'b')]]@struct<x: int64, y: string> [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[struct<x: int64, y: string>[pyarrow]] [{'x': 1, 'y': 'a'}, {'x':
2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]] zero-copy
+struct:nullable [[('x', 1), ('y', 'a')], None]@struct<x: int64, y:
string> [{'x': 1, 'y': 'a'}, <NA>]@Series[struct<x: int64, y:
string>[pyarrow]] [{'x': 1, 'y': 'a'}, <NA>]@Series[struct<x: int64, y:
string>[pyarrow]] zero-copy
+struct:empty []@struct<x: int64, y: string> []@Series[struct<x: int64, y:
string>[pyarrow]] []@Series[struct<x: int64, y: string>[pyarrow]] zero-copy
+map<string,int64>:standard [[('a', 1), ('b', 2)], [('c', 3)]]@map<string,
int64> [[('a', 1), ('b', 2)], [('c', 3)]]@Series[map<string, int64>[pyarrow]]
[[('a', 1), ('b', 2)], [('c', 3)]]@Series[map<string, int64>[pyarrow]]
zero-copy
+map<string,int64>:empty []@map<string, int64> []@Series[map<string,
int64>[pyarrow]] []@Series[map<string, int64>[pyarrow]] zero-copy
+list<list<int64>>:standard [[[1, 2], [3]], [[4, 5, 6]]]@list<item:
list<item: int64>> [[[1, 2], [3]], [[4, 5, 6]]]@Series[list<item:
list<item: int64>>[pyarrow]] [[[1, 2], [3]], [[4, 5, 6]]]@Series[list<item:
list<item: int64>>[pyarrow]] zero-copy
+list<struct>:standard [[{'x': 1}, {'x': 2}], [{'x': 3}]]@list<item: struct<x:
int64>> [[{'x': 1}, {'x': 2}], [{'x': 3}]]@Series[list<item: struct<x:
int64>>[pyarrow]] [[{'x': 1}, {'x': 2}], [{'x': 3}]]@Series[list<item:
struct<x: int64>>[pyarrow]] zero-copy
+list<map<string,int64>>:standard [[[('a', 1)], [('b', 2)]], [[('c',
3)]]]@list<item: map<string, int64>> [[[('a', 1)], [('b', 2)]], [[('c',
3)]]]@Series[list<item: map<string, int64>>[pyarrow]] [[[('a', 1)],
[('b', 2)]], [[('c', 3)]]]@Series[list<item: map<string, int64>>[pyarrow]]
zero-copy
+struct<struct>:standard [[('outer', {'inner': 1})], [('outer',
{'inner': 2})]]@struct<outer: struct<inner: int64>> [{'outer': {'inner':
1}}, {'outer': {'inner': 2}}]@Series[struct<outer: struct<inner:
int64>>[pyarrow]] [{'outer': {'inner': 1}}, {'outer': {'inner':
2}}]@Series[struct<outer: struct<inner: int64>>[pyarrow]] zero-copy
+struct<list<int64>>:standard [[('items', [1, 2, 3])], [('items', [4,
5])]]@struct<items: list<item: int64>> [{'items': [1, 2, 3]}, {'items': [4,
5]}]@Series[struct<items: list<item: int64>>[pyarrow]] [{'items': [1, 2,
3]}, {'items': [4, 5]}]@Series[struct<items: list<item: int64>>[pyarrow]]
zero-copy
+struct<map<string,int64>>:standard [[('mapping', [('a', 1)])],
[('mapping', [('b', 2)])]]@struct<mapping: map<string, int64>>
[{'mapping': [('a', 1)]}, {'mapping': [('b', 2)]}]@Series[struct<mapping:
map<string, int64>>[pyarrow]] [{'mapping': [('a', 1)]}, {'mapping': [('b',
2)]}]@Series[struct<mapping: map<string, int64>>[pyarrow]] zero-copy
+map<string,list<int64>>:standard [[('a', [1, 2]), ('b', [3])], [('c',
[4, 5, 6])]]@map<string, list<item: int64>> [[('a', [1, 2]), ('b',
[3])], [('c', [4, 5, 6])]]@Series[map<string, list<item: int64>>[pyarrow]]
[[('a', [1, 2]), ('b', [3])], [('c', [4, 5, 6])]]@Series[map<string,
list<item: int64>>[pyarrow]] zero-copy
+map<string,struct>:standard [[('a', {'v': 1}), ('b', {'v': 2})], [('c',
{'v': 3})]]@map<string, struct<v: int64>> [[('a', {'v': 1}), ('b', {'v':
2})], [('c', {'v': 3})]]@Series[map<string, struct<v: int64>>[pyarrow]]
[[('a', {'v': 1}), ('b', {'v': 2})], [('c', {'v': 3})]]@Series[map<string,
struct<v: int64>>[pyarrow]] zero-copy
+map<string,map<string,int64>>:standard [[('a', [('x', 1)]), ('b', [('y',
2)])], [('c', [('z', 3)])]]@map<string, map<string, int64>> [[('a', [('x',
1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[map<string, map<string,
int64>>[pyarrow]] [[('a', [('x', 1)]), ('b', [('y', 2)])], [('c', [('z',
3)])]]@Series[map<string, map<string, int64>>[pyarrow]] zero-copy
+dictionary<int32,string>:standard [a, b, a, b]@dictionary<values=string,
indices=int32, ordered=0> ['a', 'b', 'a',
'b']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]]
['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] zero-copy
+dictionary<int32,string>:nullable [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0> ['a', 'b', <NA>,
'a']@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]]
['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] zero-copy
+dictionary<int32,string>:empty []@dictionary<values=string, indices=int32,
ordered=0> []@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] []@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] zero-copy
+int64:sliced [2, 3, 4]@int64 [2, 3, 4]@Series[int64[pyarrow]] [2, 3,
4]@Series[int64[pyarrow]] zero-copy
+int64:sliced-with-null [2, None, 4]@int64 [2, <NA>,
4]@Series[int64[pyarrow]] [2, <NA>, 4]@Series[int64[pyarrow]] zero-copy
+int64:single-chunk [1, 2, 3]@int64 [1, 2, 3]@Series[int64[pyarrow]]
[1, 2, 3]@Series[int64[pyarrow]] zero-copy
+int64:multi-chunk [1, 2, 3, 4]@int64 [1, 2, 3,
4]@Series[int64[pyarrow]] [1, 2, 3, 4]@Series[int64[pyarrow]] zero-copy
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
new file mode 100644
index 000000000000..ac177461a2ed
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed.md
@@ -0,0 +1,127 @@
+| test case | pyarrow array
|
types_mapper=pd.ArrowDtype, zero_copy_only=False
|
types_mapper=pd.ArrowDtype, zero_copy_only=True
| verified zero-copy
|
+|----------------------------------------|-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|----------------------|
+| int8:standard | [0, 1, -1, 127, -128]@int8
| [0, 1, -1, 127,
-128]@Series[int8[pyarrow]]
| [0, 1, -1, 127,
-128]@Series[int8[pyarrow]]
| zero-copy |
+| int8:nullable | [0, 1, None]@int8
| [0, 1,
<NA>]@Series[int8[pyarrow]]
| [0, 1,
<NA>]@Series[int8[pyarrow]]
| zero-copy |
+| int8:empty | []@int8
|
[]@Series[int8[pyarrow]]
|
[]@Series[int8[pyarrow]]
| zero-copy
|
+| int16:standard | [0, 1, -1, 32767, -32768]@int16
| [0, 1, -1, 32767,
-32768]@Series[int16[pyarrow]]
| [0, 1, -1, 32767,
-32768]@Series[int16[pyarrow]]
| zero-copy |
+| int16:nullable | [0, 1, None]@int16
| [0, 1,
<NA>]@Series[int16[pyarrow]]
| [0, 1,
<NA>]@Series[int16[pyarrow]]
| zero-copy |
+| int16:empty | []@int16
|
[]@Series[int16[pyarrow]]
|
[]@Series[int16[pyarrow]]
| zero-copy
|
+| int32:standard | [0, 1, -1, 2147483647,
-2147483648]@int32 | [0, 1,
-1, 2147483647, -2147483648]@Series[int32[pyarrow]]
| [0, 1, -1, 2147483647,
-2147483648]@Series[int32[pyarrow]]
| zero-copy |
+| int32:nullable | [0, 1, None]@int32
| [0, 1,
<NA>]@Series[int32[pyarrow]]
| [0, 1,
<NA>]@Series[int32[pyarrow]]
| zero-copy |
+| int32:empty | []@int32
|
[]@Series[int32[pyarrow]]
|
[]@Series[int32[pyarrow]]
| zero-copy
|
+| int64:standard | [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 | [0, 1, -1,
9223372036854775807, -9223372036854775808]@Series[int64[pyarrow]]
| [0, 1, -1,
9223372036854775807, -9223372036854775808]@Series[int64[pyarrow]]
| zero-copy |
+| int64:nullable | [0, 1, None]@int64
| [0, 1,
<NA>]@Series[int64[pyarrow]]
| [0, 1,
<NA>]@Series[int64[pyarrow]]
| zero-copy |
+| int64:empty | []@int64
|
[]@Series[int64[pyarrow]]
|
[]@Series[int64[pyarrow]]
| zero-copy
|
+| uint8:standard | [0, 1, 255]@uint8
| [0, 1,
255]@Series[uint8[pyarrow]]
| [0, 1,
255]@Series[uint8[pyarrow]]
| zero-copy |
+| uint8:nullable | [0, 1, None]@uint8
| [0, 1,
<NA>]@Series[uint8[pyarrow]]
| [0, 1,
<NA>]@Series[uint8[pyarrow]]
| zero-copy |
+| uint8:empty | []@uint8
|
[]@Series[uint8[pyarrow]]
|
[]@Series[uint8[pyarrow]]
| zero-copy
|
+| uint16:standard | [0, 1, 65535]@uint16
| [0, 1,
65535]@Series[uint16[pyarrow]]
| [0, 1,
65535]@Series[uint16[pyarrow]]
| zero-copy |
+| uint16:nullable | [0, 1, None]@uint16
| [0, 1,
<NA>]@Series[uint16[pyarrow]]
| [0, 1,
<NA>]@Series[uint16[pyarrow]]
| zero-copy |
+| uint16:empty | []@uint16
|
[]@Series[uint16[pyarrow]]
|
[]@Series[uint16[pyarrow]]
| zero-copy
|
+| uint32:standard | [0, 1, 4294967295]@uint32
| [0, 1,
4294967295]@Series[uint32[pyarrow]]
| [0, 1,
4294967295]@Series[uint32[pyarrow]]
| zero-copy |
+| uint32:nullable | [0, 1, None]@uint32
| [0, 1,
<NA>]@Series[uint32[pyarrow]]
| [0, 1,
<NA>]@Series[uint32[pyarrow]]
| zero-copy |
+| uint32:empty | []@uint32
|
[]@Series[uint32[pyarrow]]
|
[]@Series[uint32[pyarrow]]
| zero-copy
|
+| uint64:standard | [0, 1, 18446744073709551615]@uint64
| [0, 1,
18446744073709551615]@Series[uint64[pyarrow]]
| [0, 1,
18446744073709551615]@Series[uint64[pyarrow]]
| zero-copy |
+| uint64:nullable | [0, 1, None]@uint64
| [0, 1,
<NA>]@Series[uint64[pyarrow]]
| [0, 1,
<NA>]@Series[uint64[pyarrow]]
| zero-copy |
+| uint64:empty | []@uint64
|
[]@Series[uint64[pyarrow]]
|
[]@Series[uint64[pyarrow]]
| zero-copy
|
+| float32:standard | [0.0, 1.5, -1.5]@float32
| [0.0, 1.5,
-1.5]@Series[float[pyarrow]]
| [0.0, 1.5,
-1.5]@Series[float[pyarrow]]
| zero-copy |
+| float32:nullable | [0.0, 1.5, None]@float32
| [0.0, 1.5,
<NA>]@Series[float[pyarrow]]
| [0.0, 1.5,
<NA>]@Series[float[pyarrow]]
| zero-copy |
+| float32:empty | []@float32
|
[]@Series[float[pyarrow]]
|
[]@Series[float[pyarrow]]
| zero-copy
|
+| float64:standard | [0.0, 1.5, -1.5]@float64
| [0.0, 1.5,
-1.5]@Series[double[pyarrow]]
| [0.0, 1.5,
-1.5]@Series[double[pyarrow]]
| zero-copy |
+| float64:nullable | [0.0, 1.5, None]@float64
| [0.0, 1.5,
<NA>]@Series[double[pyarrow]]
| [0.0, 1.5,
<NA>]@Series[double[pyarrow]]
| zero-copy |
+| float64:special | [nan, inf, -inf]@float64
| [nan, inf,
-inf]@Series[double[pyarrow]]
| [nan, inf,
-inf]@Series[double[pyarrow]]
| zero-copy |
+| float64:empty | []@float64
|
[]@Series[double[pyarrow]]
|
[]@Series[double[pyarrow]]
| zero-copy
|
+| bool:standard | [True, False, True]@bool
| [True, False,
True]@Series[bool[pyarrow]]
| [True, False,
True]@Series[bool[pyarrow]]
| zero-copy |
+| bool:nullable | [True, False, None]@bool
| [True, False,
<NA>]@Series[bool[pyarrow]]
| [True, False,
<NA>]@Series[bool[pyarrow]]
| zero-copy |
+| bool:empty | []@bool
|
[]@Series[bool[pyarrow]]
|
[]@Series[bool[pyarrow]]
| zero-copy
|
+| string:standard | [hello, world, ]@string
| ['hello', 'world',
'']@Series[string[pyarrow]]
| ['hello', 'world',
'']@Series[string[pyarrow]]
| zero-copy |
+| string:nullable | [hello, None, world]@string
| ['hello', <NA>,
'world']@Series[string[pyarrow]]
| ['hello', <NA>,
'world']@Series[string[pyarrow]]
| zero-copy |
+| string:empty | []@string
|
[]@Series[string[pyarrow]]
|
[]@Series[string[pyarrow]]
| zero-copy
|
+| large_string:standard | [hello, world]@large_string
| ['hello',
'world']@Series[large_string[pyarrow]]
| ['hello',
'world']@Series[large_string[pyarrow]]
| zero-copy |
+| large_string:nullable | [hello, None]@large_string
| ['hello',
<NA>]@Series[large_string[pyarrow]]
| ['hello',
<NA>]@Series[large_string[pyarrow]]
| zero-copy |
+| large_string:empty | []@large_string
|
[]@Series[large_string[pyarrow]]
|
[]@Series[large_string[pyarrow]]
| zero-copy
|
+| binary:standard | [b'hello', b'world']@binary
| [b'hello',
b'world']@Series[binary[pyarrow]]
| [b'hello',
b'world']@Series[binary[pyarrow]]
| zero-copy |
+| binary:nullable | [b'hello', None]@binary
| [b'hello',
<NA>]@Series[binary[pyarrow]]
| [b'hello',
<NA>]@Series[binary[pyarrow]]
| zero-copy |
+| binary:empty | []@binary
|
[]@Series[binary[pyarrow]]
|
[]@Series[binary[pyarrow]]
| zero-copy
|
+| large_binary:standard | [b'hello', b'world']@large_binary
| [b'hello',
b'world']@Series[large_binary[pyarrow]]
| [b'hello',
b'world']@Series[large_binary[pyarrow]]
| zero-copy |
+| large_binary:nullable | [b'hello', None]@large_binary
| [b'hello',
<NA>]@Series[large_binary[pyarrow]]
| [b'hello',
<NA>]@Series[large_binary[pyarrow]]
| zero-copy |
+| large_binary:empty | []@large_binary
|
[]@Series[large_binary[pyarrow]]
|
[]@Series[large_binary[pyarrow]]
| zero-copy
|
+| decimal128:standard | [1.23, 4.56, -7.89]@decimal128(5,
2) | [Decimal('1.23'),
Decimal('4.56'), Decimal('-7.89')]@Series[decimal128(5, 2)[pyarrow]]
| [Decimal('1.23'), Decimal('4.56'),
Decimal('-7.89')]@Series[decimal128(5, 2)[pyarrow]]
| zero-copy |
+| decimal128:nullable | [1.23, None, 4.56]@decimal128(5, 2)
| [Decimal('1.23'),
<NA>, Decimal('4.56')]@Series[decimal128(5, 2)[pyarrow]]
| [Decimal('1.23'), <NA>,
Decimal('4.56')]@Series[decimal128(5, 2)[pyarrow]]
| zero-copy |
+| decimal128:empty | []@decimal128(5, 2)
|
[]@Series[decimal128(5, 2)[pyarrow]]
|
[]@Series[decimal128(5, 2)[pyarrow]]
| zero-copy
|
+| date32:standard | [2024-01-01,
2024-06-15]@date32[day]
| [datetime.date(2024, 1, 1), datetime.date(2024, 6,
15)]@Series[date32[day][pyarrow]]
| [datetime.date(2024, 1, 1), datetime.date(2024, 6,
15)]@Series[date32[day][pyarrow]]
| zero-copy |
+| date32:nullable | [2024-01-01, None]@date32[day]
|
[datetime.date(2024, 1, 1), <NA>]@Series[date32[day][pyarrow]]
|
[datetime.date(2024, 1, 1), <NA>]@Series[date32[day][pyarrow]]
| zero-copy
|
+| date32:empty | []@date32[day]
|
[]@Series[date32[day][pyarrow]]
|
[]@Series[date32[day][pyarrow]]
| zero-copy
|
+| date64:standard | [2024-01-01, 2024-06-15]@date64[ms]
|
[datetime.date(2024, 1, 1), datetime.date(2024, 6,
15)]@Series[date64[ms][pyarrow]]
| [datetime.date(2024, 1, 1), datetime.date(2024, 6,
15)]@Series[date64[ms][pyarrow]]
| zero-copy |
+| date64:nullable | [2024-01-01, None]@date64[ms]
|
[datetime.date(2024, 1, 1), <NA>]@Series[date64[ms][pyarrow]]
|
[datetime.date(2024, 1, 1), <NA>]@Series[date64[ms][pyarrow]]
| zero-copy
|
+| date64:empty | []@date64[ms]
|
[]@Series[date64[ms][pyarrow]]
|
[]@Series[date64[ms][pyarrow]]
| zero-copy
|
+| timestamp[s]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[s] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[s][pyarrow]]
| [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[s][pyarrow]]
| zero-copy |
+| timestamp[s]:nullable | [2024-01-01 12:00:00,
None]@timestamp[s] |
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[s][pyarrow]]
|
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[s][pyarrow]]
| zero-copy
|
+| timestamp[s]:empty | []@timestamp[s]
|
[]@Series[timestamp[s][pyarrow]]
|
[]@Series[timestamp[s][pyarrow]]
| zero-copy
|
+| timestamp[ms]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ms] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[ms][pyarrow]]
| [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[ms][pyarrow]]
| zero-copy |
+| timestamp[ms]:nullable | [2024-01-01 12:00:00,
None]@timestamp[ms] |
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ms][pyarrow]]
|
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ms][pyarrow]]
| zero-copy
|
+| timestamp[ms]:empty | []@timestamp[ms]
|
[]@Series[timestamp[ms][pyarrow]]
|
[]@Series[timestamp[ms][pyarrow]]
| zero-copy
|
+| timestamp[us]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[us] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[us][pyarrow]]
| [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[us][pyarrow]]
| zero-copy |
+| timestamp[us]:nullable | [2024-01-01 12:00:00,
None]@timestamp[us] |
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[us][pyarrow]]
|
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[us][pyarrow]]
| zero-copy
|
+| timestamp[us]:empty | []@timestamp[us]
|
[]@Series[timestamp[us][pyarrow]]
|
[]@Series[timestamp[us][pyarrow]]
| zero-copy
|
+| timestamp[ns]:standard | [2024-01-01 12:00:00, 2024-06-15
18:30:00]@timestamp[ns] |
[Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[ns][pyarrow]]
| [Timestamp('2024-01-01 12:00:00'), Timestamp('2024-06-15
18:30:00')]@Series[timestamp[ns][pyarrow]]
| zero-copy |
+| timestamp[ns]:nullable | [2024-01-01 12:00:00,
None]@timestamp[ns] |
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ns][pyarrow]]
|
[Timestamp('2024-01-01 12:00:00'), <NA>]@Series[timestamp[ns][pyarrow]]
| zero-copy
|
+| timestamp[ns]:empty | []@timestamp[ns]
|
[]@Series[timestamp[ns][pyarrow]]
|
[]@Series[timestamp[ns][pyarrow]]
| zero-copy
|
+| timestamp[us,tz=UTC]:standard | [2024-01-01 12:00:00+00:00,
2024-06-15 18:30:00+00:00]@timestamp[us, tz=UTC] |
[Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15
18:30:00+0000', tz='UTC')]@Series[timestamp[us, tz=UTC][pyarrow]] |
[Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), Timestamp('2024-06-15
18:30:00+0000', tz='UTC')]@Series[timestamp[us, tz=UTC][pyarrow]] | zero-copy
|
+| timestamp[us,tz=UTC]:nullable | [2024-01-01 12:00:00+00:00,
None]@timestamp[us, tz=UTC] |
[Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), <NA>]@Series[timestamp[us,
tz=UTC][pyarrow]] |
[Timestamp('2024-01-01 12:00:00+0000', tz='UTC'), <NA>]@Series[timestamp[us,
tz=UTC][pyarrow]] | zero-copy
|
+| timestamp[us,tz=UTC]:empty | []@timestamp[us, tz=UTC]
|
[]@Series[timestamp[us, tz=UTC][pyarrow]]
|
[]@Series[timestamp[us, tz=UTC][pyarrow]]
| zero-copy
|
+| duration[s]:standard | [1 day, 0:00:00,
2:30:00]@duration[s] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[s][pyarrow]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[s][pyarrow]]
| zero-copy |
+| duration[s]:nullable | [1 day, 0:00:00, None]@duration[s]
| [Timedelta('1 days
00:00:00'), <NA>]@Series[duration[s][pyarrow]]
| [Timedelta('1 days 00:00:00'),
<NA>]@Series[duration[s][pyarrow]]
| zero-copy |
+| duration[s]:empty | []@duration[s]
|
[]@Series[duration[s][pyarrow]]
|
[]@Series[duration[s][pyarrow]]
| zero-copy
|
+| duration[ms]:standard | [1 day, 0:00:00,
2:30:00]@duration[ms] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[ms][pyarrow]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[ms][pyarrow]]
| zero-copy |
+| duration[ms]:nullable | [1 day, 0:00:00, None]@duration[ms]
| [Timedelta('1 days
00:00:00'), <NA>]@Series[duration[ms][pyarrow]]
| [Timedelta('1 days 00:00:00'),
<NA>]@Series[duration[ms][pyarrow]]
| zero-copy |
+| duration[ms]:empty | []@duration[ms]
|
[]@Series[duration[ms][pyarrow]]
|
[]@Series[duration[ms][pyarrow]]
| zero-copy
|
+| duration[us]:standard | [1 day, 0:00:00,
2:30:00]@duration[us] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[us][pyarrow]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[us][pyarrow]]
| zero-copy |
+| duration[us]:nullable | [1 day, 0:00:00, None]@duration[us]
| [Timedelta('1 days
00:00:00'), <NA>]@Series[duration[us][pyarrow]]
| [Timedelta('1 days 00:00:00'),
<NA>]@Series[duration[us][pyarrow]]
| zero-copy |
+| duration[us]:empty | []@duration[us]
|
[]@Series[duration[us][pyarrow]]
|
[]@Series[duration[us][pyarrow]]
| zero-copy
|
+| duration[ns]:standard | [1 days 00:00:00, 0 days
02:30:00]@duration[ns] |
[Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[ns][pyarrow]]
| [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:30:00')]@Series[duration[ns][pyarrow]]
| zero-copy |
+| duration[ns]:nullable | [1 days 00:00:00,
None]@duration[ns] |
[Timedelta('1 days 00:00:00'), <NA>]@Series[duration[ns][pyarrow]]
| [Timedelta('1 days
00:00:00'), <NA>]@Series[duration[ns][pyarrow]]
| zero-copy |
+| duration[ns]:empty | []@duration[ns]
|
[]@Series[duration[ns][pyarrow]]
|
[]@Series[duration[ns][pyarrow]]
| zero-copy
|
+| time32[s]:standard | [12:30:00, 18:45:30]@time32[s]
| [datetime.time(12,
30), datetime.time(18, 45, 30)]@Series[time32[s][pyarrow]]
| [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time32[s][pyarrow]]
| zero-copy |
+| time32[s]:nullable | [12:30:00, None]@time32[s]
| [datetime.time(12,
30), <NA>]@Series[time32[s][pyarrow]]
| [datetime.time(12, 30),
<NA>]@Series[time32[s][pyarrow]]
| zero-copy |
+| time32[s]:empty | []@time32[s]
|
[]@Series[time32[s][pyarrow]]
|
[]@Series[time32[s][pyarrow]]
| zero-copy
|
+| time32[ms]:standard | [12:30:00, 18:45:30]@time32[ms]
| [datetime.time(12,
30), datetime.time(18, 45, 30)]@Series[time32[ms][pyarrow]]
| [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time32[ms][pyarrow]]
| zero-copy |
+| time32[ms]:nullable | [12:30:00, None]@time32[ms]
| [datetime.time(12,
30), <NA>]@Series[time32[ms][pyarrow]]
| [datetime.time(12, 30),
<NA>]@Series[time32[ms][pyarrow]]
| zero-copy |
+| time32[ms]:empty | []@time32[ms]
|
[]@Series[time32[ms][pyarrow]]
|
[]@Series[time32[ms][pyarrow]]
| zero-copy
|
+| time64[us]:standard | [12:30:00, 18:45:30]@time64[us]
| [datetime.time(12,
30), datetime.time(18, 45, 30)]@Series[time64[us][pyarrow]]
| [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time64[us][pyarrow]]
| zero-copy |
+| time64[us]:nullable | [12:30:00, None]@time64[us]
| [datetime.time(12,
30), <NA>]@Series[time64[us][pyarrow]]
| [datetime.time(12, 30),
<NA>]@Series[time64[us][pyarrow]]
| zero-copy |
+| time64[us]:empty | []@time64[us]
|
[]@Series[time64[us][pyarrow]]
|
[]@Series[time64[us][pyarrow]]
| zero-copy
|
+| time64[ns]:standard | [12:30:00, 18:45:30]@time64[ns]
| [datetime.time(12,
30), datetime.time(18, 45, 30)]@Series[time64[ns][pyarrow]]
| [datetime.time(12, 30),
datetime.time(18, 45, 30)]@Series[time64[ns][pyarrow]]
| zero-copy |
+| time64[ns]:nullable | [12:30:00, None]@time64[ns]
| [datetime.time(12,
30), <NA>]@Series[time64[ns][pyarrow]]
| [datetime.time(12, 30),
<NA>]@Series[time64[ns][pyarrow]]
| zero-copy |
+| time64[ns]:empty | []@time64[ns]
|
[]@Series[time64[ns][pyarrow]]
|
[]@Series[time64[ns][pyarrow]]
| zero-copy
|
+| null:standard | [None, None, None]@null
| [<NA>, <NA>,
<NA>]@Series[null[pyarrow]]
| [<NA>, <NA>,
<NA>]@Series[null[pyarrow]]
| zero-copy |
+| null:empty | []@null
|
[]@Series[null[pyarrow]]
|
[]@Series[null[pyarrow]]
| zero-copy
|
+| list<int64>:standard | [[1, 2], [3, 4, 5]]@list<item:
int64> | [[1, 2], [3,
4, 5]]@Series[list<item: int64>[pyarrow]]
| [[1, 2], [3, 4,
5]]@Series[list<item: int64>[pyarrow]]
| zero-copy |
+| list<int64>:nullable | [[1, 2], None, [3]]@list<item:
int64> | [[1, 2], <NA>,
[3]]@Series[list<item: int64>[pyarrow]]
| [[1, 2], <NA>,
[3]]@Series[list<item: int64>[pyarrow]]
| zero-copy |
+| list<int64>:empty | []@list<item: int64>
|
[]@Series[list<item: int64>[pyarrow]]
|
[]@Series[list<item: int64>[pyarrow]]
| zero-copy
|
+| list<string>:standard | [['a', 'b'], ['c']]@list<item:
string> | [['a', 'b'],
['c']]@Series[list<item: string>[pyarrow]]
| [['a', 'b'],
['c']]@Series[list<item: string>[pyarrow]]
| zero-copy |
+| large_list<int64>:standard | [[1, 2], [3, 4]]@large_list<item:
int64> | [[1, 2], [3,
4]]@Series[large_list<item: int64>[pyarrow]]
| [[1, 2], [3,
4]]@Series[large_list<item: int64>[pyarrow]]
| zero-copy |
+| large_list<int64>:empty | []@large_list<item: int64>
|
[]@Series[large_list<item: int64>[pyarrow]]
|
[]@Series[large_list<item: int64>[pyarrow]]
| zero-copy
|
+| fixed_size_list<int64>[3]:standard | [[1, 2, 3], [4, 5,
6]]@fixed_size_list<item: int64>[3] |
[[1, 2, 3], [4, 5, 6]]@Series[fixed_size_list<item: int64>[3][pyarrow]]
| [[1, 2, 3], [4, 5,
6]]@Series[fixed_size_list<item: int64>[3][pyarrow]]
| zero-copy |
+| fixed_size_list<int64>[3]:empty | []@fixed_size_list<item: int64>[3]
|
[]@Series[fixed_size_list<item: int64>[3][pyarrow]]
|
[]@Series[fixed_size_list<item: int64>[3][pyarrow]]
| zero-copy
|
+| struct:standard | [[('x', 1), ('y', 'a')], [('x', 2),
('y', 'b')]]@struct<x: int64, y: string> | [{'x': 1, 'y':
'a'}, {'x': 2, 'y': 'b'}]@Series[struct<x: int64, y: string>[pyarrow]]
| [{'x': 1, 'y': 'a'}, {'x': 2, 'y':
'b'}]@Series[struct<x: int64, y: string>[pyarrow]]
| zero-copy |
+| struct:nullable | [[('x', 1), ('y', 'a')],
None]@struct<x: int64, y: string> | [{'x':
1, 'y': 'a'}, <NA>]@Series[struct<x: int64, y: string>[pyarrow]]
| [{'x': 1, 'y': 'a'},
<NA>]@Series[struct<x: int64, y: string>[pyarrow]]
| zero-copy |
+| struct:empty | []@struct<x: int64, y: string>
| []@Series[struct<x:
int64, y: string>[pyarrow]]
| []@Series[struct<x: int64, y:
string>[pyarrow]]
| zero-copy |
+| map<string,int64>:standard | [[('a', 1), ('b', 2)], [('c',
3)]]@map<string, int64> | [[('a', 1),
('b', 2)], [('c', 3)]]@Series[map<string, int64>[pyarrow]]
| [[('a', 1), ('b', 2)], [('c',
3)]]@Series[map<string, int64>[pyarrow]]
| zero-copy |
+| map<string,int64>:empty | []@map<string, int64>
|
[]@Series[map<string, int64>[pyarrow]]
|
[]@Series[map<string, int64>[pyarrow]]
| zero-copy
|
+| list<list<int64>>:standard | [[[1, 2], [3]], [[4, 5,
6]]]@list<item: list<item: int64>> | [[[1,
2], [3]], [[4, 5, 6]]]@Series[list<item: list<item: int64>>[pyarrow]]
| [[[1, 2], [3]], [[4, 5,
6]]]@Series[list<item: list<item: int64>>[pyarrow]]
| zero-copy |
+| list<struct>:standard | [[{'x': 1}, {'x': 2}], [{'x':
3}]]@list<item: struct<x: int64>> | [[{'x': 1},
{'x': 2}], [{'x': 3}]]@Series[list<item: struct<x: int64>>[pyarrow]]
| [[{'x': 1}, {'x': 2}], [{'x':
3}]]@Series[list<item: struct<x: int64>>[pyarrow]]
| zero-copy |
+| list<map<string,int64>>:standard | [[[('a', 1)], [('b', 2)]], [[('c',
3)]]]@list<item: map<string, int64>> | [[[('a', 1)],
[('b', 2)]], [[('c', 3)]]]@Series[list<item: map<string, int64>>[pyarrow]]
| [[[('a', 1)], [('b', 2)]],
[[('c', 3)]]]@Series[list<item: map<string, int64>>[pyarrow]]
| zero-copy |
+| struct<struct>:standard | [[('outer', {'inner': 1})],
[('outer', {'inner': 2})]]@struct<outer: struct<inner: int64>> | [{'outer':
{'inner': 1}}, {'outer': {'inner': 2}}]@Series[struct<outer: struct<inner:
int64>>[pyarrow]] | [{'outer': {'inner': 1}},
{'outer': {'inner': 2}}]@Series[struct<outer: struct<inner: int64>>[pyarrow]]
| zero-copy |
+| struct<list<int64>>:standard | [[('items', [1, 2, 3])], [('items',
[4, 5])]]@struct<items: list<item: int64>> | [{'items': [1, 2,
3]}, {'items': [4, 5]}]@Series[struct<items: list<item: int64>>[pyarrow]]
| [{'items': [1, 2, 3]}, {'items': [4,
5]}]@Series[struct<items: list<item: int64>>[pyarrow]]
| zero-copy |
+| struct<map<string,int64>>:standard | [[('mapping', [('a', 1)])],
[('mapping', [('b', 2)])]]@struct<mapping: map<string, int64>> |
[{'mapping': [('a', 1)]}, {'mapping': [('b', 2)]}]@Series[struct<mapping:
map<string, int64>>[pyarrow]] | [{'mapping':
[('a', 1)]}, {'mapping': [('b', 2)]}]@Series[struct<mapping: map<string,
int64>>[pyarrow]] | zero-copy |
+| map<string,list<int64>>:standard | [[('a', [1, 2]), ('b', [3])],
[('c', [4, 5, 6])]]@map<string, list<item: int64>> | [[('a', [1,
2]), ('b', [3])], [('c', [4, 5, 6])]]@Series[map<string, list<item:
int64>>[pyarrow]] | [[('a', [1, 2]),
('b', [3])], [('c', [4, 5, 6])]]@Series[map<string, list<item:
int64>>[pyarrow]] | zero-copy
|
+| map<string,struct>:standard | [[('a', {'v': 1}), ('b', {'v':
2})], [('c', {'v': 3})]]@map<string, struct<v: int64>> | [[('a', {'v':
1}), ('b', {'v': 2})], [('c', {'v': 3})]]@Series[map<string, struct<v:
int64>>[pyarrow]] | [[('a', {'v': 1}), ('b',
{'v': 2})], [('c', {'v': 3})]]@Series[map<string, struct<v: int64>>[pyarrow]]
| zero-copy |
+| map<string,map<string,int64>>:standard | [[('a', [('x', 1)]), ('b', [('y',
2)])], [('c', [('z', 3)])]]@map<string, map<string, int64>> | [[('a', [('x',
1)]), ('b', [('y', 2)])], [('c', [('z', 3)])]]@Series[map<string, map<string,
int64>>[pyarrow]] | [[('a', [('x', 1)]), ('b',
[('y', 2)])], [('c', [('z', 3)])]]@Series[map<string, map<string,
int64>>[pyarrow]] | zero-copy |
+| dictionary<int32,string>:standard | [a, b, a,
b]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', 'a', 'b']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] | ['a',
'b', 'a', 'b']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] |
zero-copy |
+| dictionary<int32,string>:nullable | [a, b, None,
a]@dictionary<values=string, indices=int32, ordered=0>
| ['a', 'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] | ['a',
'b', <NA>, 'a']@Series[dictionary<values=string, indices=int32,
ordered=0>[pyarrow]] | zero-copy
|
+| dictionary<int32,string>:empty | []@dictionary<values=string,
indices=int32, ordered=0> |
[]@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]]
|
[]@Series[dictionary<values=string, indices=int32, ordered=0>[pyarrow]]
| zero-copy
|
+| int64:sliced | [2, 3, 4]@int64
| [2, 3,
4]@Series[int64[pyarrow]]
| [2, 3,
4]@Series[int64[pyarrow]]
| zero-copy |
+| int64:sliced-with-null | [2, None, 4]@int64
| [2, <NA>,
4]@Series[int64[pyarrow]]
| [2, <NA>,
4]@Series[int64[pyarrow]]
| zero-copy |
+| int64:single-chunk | [1, 2, 3]@int64
| [1, 2,
3]@Series[int64[pyarrow]]
| [1, 2,
3]@Series[int64[pyarrow]]
| zero-copy |
+| int64:multi-chunk | [1, 2, 3, 4]@int64
| [1, 2, 3,
4]@Series[int64[pyarrow]]
| [1, 2, 3,
4]@Series[int64[pyarrow]]
| zero-copy |
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
index 742b7db7389f..7e0dc3a0823a 100644
---
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_arrow_to_pandas_non_default.py
@@ -384,6 +384,13 @@ class
PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
so both golden files pin the same types, and appends the layout variants
that
only matter for zero-copy: sliced (offset) arrays and single- vs
multi-chunk
ChunkedArrays.
+
+ A second test method repeats those rows with
``types_mapper=pd.ArrowDtype``, which
+ asks pandas to keep pointing at the Arrow buffers instead of materializing
them
+ into NumPy -- avoiding the copy is the point of that backend. Its own
golden file
+ records the result, so the two can be read side by side. PySpark takes
this path
+ in ``ArrowArrayToPandasConversion.convert_numpy``
+ (``python/pyspark/sql/conversion.py``).
"""
def _build_source_arrays(self):
@@ -492,6 +499,49 @@ class
PyArrowArrayToPandasZeroCopyTests(_PyArrowToPandasTestBase):
overrides=overrides,
)
+ # Output columns for the Arrow-backed conversion. Both zero_copy_only
states are
+ # recorded even though the flag is expected to make no difference here
+ COL_ARROW_ZERO_COPY_OFF = "types_mapper=pd.ArrowDtype,
zero_copy_only=False"
+ COL_ARROW_ZERO_COPY_ON = "types_mapper=pd.ArrowDtype, zero_copy_only=True"
+ COL_ARROW_VERIFIED = "verified zero-copy"
+
+ def test_to_pandas_zero_copy_only_arrow_backed(self):
+ """Test pa.Array.to_pandas(types_mapper=pd.ArrowDtype) against golden
file."""
+ sources = self._build_source_arrays()
+ row_names = list(sources.keys())
+ col_names = [
+ "pyarrow array",
+ self.COL_ARROW_ZERO_COPY_OFF,
+ self.COL_ARROW_ZERO_COPY_ON,
+ self.COL_ARROW_VERIFIED,
+ ]
+
+ # Version-specific expected values go here, keyed by (row, col), when a
+ # newer pandas/PyArrow/NumPy legitimately changes a cell's output.
+ overrides: dict[tuple[str, str], str] = {}
+
+ def compute_cell(row_name, col_name):
+ arr = sources[row_name]
+ if col_name == "pyarrow array":
+ return self.repr_value(arr, max_len=0)
+ elif col_name == self.COL_ARROW_ZERO_COPY_OFF:
+ return self._to_pandas_cell(arr, types_mapper=pd.ArrowDtype,
zero_copy_only=False)
+ elif col_name == self.COL_ARROW_ZERO_COPY_ON:
+ return self._to_pandas_cell(arr, types_mapper=pd.ArrowDtype,
zero_copy_only=True)
+ elif col_name == self.COL_ARROW_VERIFIED:
+ return self._verify_zero_copy(arr, types_mapper=pd.ArrowDtype)
+ else:
+ raise ValueError(f"unknown column: {col_name}")
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+
golden_file_prefix="golden_pyarrow_arrow_to_pandas_zero_copy_arrow_backed",
+ index_name="test case",
+ overrides=overrides,
+ )
+
@unittest.skipIf(
not have_pyarrow or not have_pandas or not have_numpy,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
