This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new b34b6cdef0e2 [SPARK-58657][PYTHON][TEST] Add tests for
pa.Array.from_pandas with the mask argument
b34b6cdef0e2 is described below
commit b34b6cdef0e20ee2772f5280270209959c5046bc
Author: Spenser Sun <[email protected]>
AuthorDate: Tue Aug 11 11:43:20 2026 +0800
[SPARK-58657][PYTHON][TEST] Add tests for pa.Array.from_pandas with the
mask argument
### What changes were proposed in this pull request?
This PR adds a golden-file test pinning the behavior of
`pa.Array.from_pandas()` under the `mask` argument, part of the umbrella effort
([SPARK-54936](https://issues.apache.org/jira/browse/SPARK-54936)) to monitor
upstream PyArrow/pandas behavior that PySpark's pandas <-> Arrow conversion
layer depends on. These tests run without a Spark session and act as a drift
canary: if a library upgrade changes the primitive's behavior, the golden
comparison fails loudly at the exact cell.
The changes:
- New `test_pyarrow_array_from_pandas_non_default.py` with
`PyArrowArrayFromPandasMaskTests`, which subclasses
`_PyArrowFromPandasTestBase` (from the default test) to reuse its source Series
inventory, and records `golden_pyarrow_array_from_pandas_mask.{csv,md}` (98
rows x 3 columns: `pandas series | mask=None | mask=isnull()`).
- A small behavior-preserving refactor of the merged default test,
`test_pyarrow_array_from_pandas_default.py`:
- the source-Series inventory and `repr_from_pandas_result` were already
lifted into a test-free base `_PyArrowFromPandasTestBase`, with
`PyArrowArrayFromPandasDefaultTests` as a thin subclass, so the non-default
tests can subclass the base directly (no unbound-call trick, and the base
carries no `test_*` to be re-collected);
- the shared `_from_pandas_cell(series, **kwargs)` cell formatter is now
a method on that base, used by both the default and the new mask test.
- Registered the new module in `dev/sparktestsupport/modules.py`.
PySpark does not pass `mask` freely; it derives it from how the Series is
stored (`python/pyspark/sql/conversion.py:435`,
`python/pyspark/sql/pandas/conversion.py:113`):
```python
mask = None if hasattr(series.array, "__arrow_array__") else series.isnull()
```
The golden records both `mask=None` and `mask=isnull()` so the two regimes
stay observable:
- **numpy-backed** dtypes do not implement `__arrow_array__`, so PySpark
passes `mask=series.isnull()`. `from_pandas` also infers nulls from the Series
at `mask=None`, so the two columns agree; the test pins that agreement.
- **protocol** dtypes (implementing `__arrow_array__` -- the nullable
extension, `string[python]`, and `[pyarrow]` dtypes) return a finished Arrow
array with their own validity bitmap, so PyArrow rejects a caller-supplied mask
with `ValueError` on the argument's mere presence (even an all-False no-op mask
raises). PySpark passes `mask=None` for them, and `mask=isnull()` records
`ERRValueError`.
Because `mask` is fixed by the input row's dtype, it is a column pair
rather than an independent matrix dimension.
### Why are the changes needed?
PySpark calls `pa.Array.from_pandas(series, mask=mask, ...)` on the pandas
-> Arrow path (`createDataFrame(pandas_df)` and every pandas UDF's return
value). The `mask` argument's two-regime behavior -- silently accepted on
numpy-backed input, rejected on protocol input -- is upstream behavior PySpark
relies on but does not itself test. A golden test makes any future drift (for
example, protocol dtypes beginning to accept a mask, or `mask=None` no longer
inferring nulls) fail visibly i [...]
### Does this PR introduce _any_ user-facing change?
No. This adds tests only.
### How was this patch tested?
New golden-file test, run with and without `SPARK_GENERATE_GOLDEN_FILES=1`
(regeneration is byte-identical). Validated across the full support matrix --
PyArrow 18, 19, 20, 21, 22, 23, 24, 25 x pandas 2 and 3 (16/16 combinations
pass), each in a fresh virtualenv running the committed test against the
committed golden. `ruff check` and `ruff format --check` are clean. The
pandas-3 differences are recorded as version-guarded `overrides` (notably
`string:inferred`, whose dtype becomes th [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic), model Claude Opus 4.8
Closes #57862 from Spenserrrr/from-pandas-mask-tests.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 4c86e4c3f1adea1e78c9776c3372f144051081c9)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
dev/sparktestsupport/modules.py | 1 +
.../golden_pyarrow_array_from_pandas_mask.csv | 99 ++++++++++++
.../golden_pyarrow_array_from_pandas_mask.md | 100 ++++++++++++
.../test_pyarrow_array_from_pandas_default.py | 38 +++--
.../test_pyarrow_array_from_pandas_non_default.py | 167 +++++++++++++++++++++
5 files changed, 393 insertions(+), 12 deletions(-)
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 2ac49a4019eb..f4e3e5877054 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -538,6 +538,7 @@ pyspark_core = Module(
# unittests for upstream projects
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_cast",
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_from_pandas_default",
+
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_from_pandas_non_default",
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_type_inference",
"pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_default",
"pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_non_default",
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.csv
new file mode 100644
index 000000000000..7af02dca017f
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.csv
@@ -0,0 +1,99 @@
+test case pandas series mask=None mask=isnull()
+int8:standard [0, 1, -1, 127, -128]@Series[int8] [0, 1, -1, 127,
-128]@int8 [0, 1, -1, 127, -128]@int8
+int8:empty []@Series[int8] []@int8 []@int8
+int16:standard [0, 1, -1, 32767, -32768]@Series[int16] [0, 1, -1, 32767,
-32768]@int16 [0, 1, -1, 32767, -32768]@int16
+int16:empty []@Series[int16] []@int16 []@int16
+int32:standard [0, 1, -1, 2147483647, -2147483648]@Series[int32] [0, 1,
-1, 2147483647, -2147483648]@int32 [0, 1, -1, 2147483647,
-2147483648]@int32
+int32:empty []@Series[int32] []@int32 []@int32
+int64:standard [0, 1, -1, 9223372036854775807,
-9223372036854775808]@Series[int64] [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64
+int64:empty []@Series[int64] []@int64 []@int64
+int64:nullable [0.0, 1.0, nan]@Series[float64] [0.0, 1.0, None]@float64
[0.0, 1.0, None]@float64
+uint8:standard [0, 1, 255]@Series[uint8] [0, 1, 255]@uint8 [0, 1,
255]@uint8
+uint16:standard [0, 1, 65535]@Series[uint16] [0, 1, 65535]@uint16
[0, 1, 65535]@uint16
+uint32:standard [0, 1, 4294967295]@Series[uint32] [0, 1,
4294967295]@uint32 [0, 1, 4294967295]@uint32
+uint64:standard [0, 1, 18446744073709551615]@Series[uint64] [0, 1,
18446744073709551615]@uint64 [0, 1, 18446744073709551615]@uint64
+float32:standard [0.0, 1.5, -1.5]@Series[float32] [0.0, 1.5,
-1.5]@float32 [0.0, 1.5, -1.5]@float32
+float32:nullable [0.0, nan, 1.5]@Series[float32] [0.0, None,
1.5]@float32 [0.0, None, 1.5]@float32
+float32:empty []@Series[float32] []@float32 []@float32
+float64:standard [0.0, 1.5, -1.5]@Series[float64] [0.0, 1.5,
-1.5]@float64 [0.0, 1.5, -1.5]@float64
+float64:nullable [0.0, nan, 1.5]@Series[float64] [0.0, None,
1.5]@float64 [0.0, None, 1.5]@float64
+float64:empty []@Series[float64] []@float64 []@float64
+bool:standard [True, False, True]@Series[bool] [True, False,
True]@bool [True, False, True]@bool
+bool:empty []@Series[bool] []@bool []@bool
+object:string ['hello', 'world', '']@Series[object] [hello, world, ]@string
[hello, world, ]@string
+object:string-nullable ['hello', None, 'world']@Series[object] [hello, None,
world]@string [hello, None, world]@string
+string:inferred ['hello', 'world']@Series[object] [hello,
world]@string [hello, world]@string
+object:bytes [b'hello', b'world']@Series[object] [b'hello',
b'world']@binary [b'hello', b'world']@binary
+object:empty []@Series[object] []@null []@null
+object:all-null [None, None]@Series[object] [None, None]@null
[None, None]@null
+object:decimal [Decimal('1.50'), Decimal('-2.25')]@Series[object] [1.50,
-2.25]@decimal128(3, 2) [1.50, -2.25]@decimal128(3, 2)
+list<int64>:standard [[1, 2], [3]]@Series[object] [[1, 2],
[3]]@list<item: int64> [[1, 2], [3]]@list<item: int64>
+list<int64>:nullable [[1, 2], None]@Series[object] [[1, 2],
None]@list<item: int64> [[1, 2], None]@list<item: int64>
+list<int64>:null-element [[1, None], [3]]@Series[object] [[1, None],
[3]]@list<item: int64> [[1, None], [3]]@list<item: int64>
+list<string>:standard [['a', 'b'], ['c']]@Series[object] [['a', 'b'],
['c']]@list<item: string> [['a', 'b'], ['c']]@list<item: string>
+list<list<int64>>:standard [[[1, 2], [3]], [[4]]]@Series[object] [[[1,
2], [3]], [[4]]]@list<item: list<item: int64>> [[[1, 2], [3]],
[[4]]]@list<item: list<item: int64>>
+list<struct>:standard [[{'a': 1}], [{'a': 2}]]@Series[object] [[{'a': 1}],
[{'a': 2}]]@list<item: struct<a: int64>> [[{'a': 1}], [{'a': 2}]]@list<item:
struct<a: int64>>
+struct:standard [{'a': 1, 'b': 'x'}]@Series[object] [[('a', 1),
('b', 'x')]]@struct<a: int64, b: string> [[('a', 1), ('b', 'x')]]@struct<a:
int64, b: string>
+struct:nullable [{'a': 1, 'b': 'x'}, None]@Series[object] [[('a',
1), ('b', 'x')], None]@struct<a: int64, b: string> [[('a', 1), ('b',
'x')], None]@struct<a: int64, b: string>
+struct<struct>:standard [{'a': {'b': 1}}]@Series[object] [[('a',
{'b': 1})]]@struct<a: struct<b: int64>> [[('a', {'b': 1})]]@struct<a: struct<b:
int64>>
+struct<list<int64>>:standard [{'a': [1, 2]}]@Series[object] [[('a', [1,
2])]]@struct<a: list<item: int64>> [[('a', [1, 2])]]@struct<a: list<item:
int64>>
+datetime64[ns]:standard [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] [2024-06-15 18:30:00]@timestamp[ns]
[2024-06-15 18:30:00]@timestamp[ns]
+datetime64[ns]:nullable [Timestamp('2024-06-15 18:30:00'),
NaT]@Series[datetime64[ns]] [2024-06-15 18:30:00, None]@timestamp[ns]
[2024-06-15 18:30:00, None]@timestamp[ns]
+datetime64[ns]:empty []@Series[datetime64[ns]] []@timestamp[ns]
[]@timestamp[ns]
+datetime64[ns,tz]:standard [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] [2024-06-15
18:30:00+00:00]@timestamp[ns, tz=UTC] [2024-06-15
18:30:00+00:00]@timestamp[ns, tz=UTC]
+datetime64[ns,tz]:nullable [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC'), NaT]@Series[datetime64[ns, UTC]] [2024-06-15 18:30:00+00:00,
None]@timestamp[ns, tz=UTC] [2024-06-15 18:30:00+00:00, None]@timestamp[ns,
tz=UTC]
+timedelta64[ns]:standard [Timedelta('1 days 00:00:00'), Timedelta('0
days 02:00:00')]@Series[timedelta64[ns]] [1 days 00:00:00, 0 days
02:00:00]@duration[ns] [1 days 00:00:00, 0 days 02:00:00]@duration[ns]
+timedelta64[ns]:nullable [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]] [1 days 00:00:00, None]@duration[ns] [1
days 00:00:00, None]@duration[ns]
+datetime64[us]:standard [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[us]] [2024-06-15 18:30:00]@timestamp[us]
[2024-06-15 18:30:00]@timestamp[us]
+datetime64[us]:nullable [Timestamp('2024-06-15 18:30:00'),
NaT]@Series[datetime64[us]] [2024-06-15 18:30:00, None]@timestamp[us]
[2024-06-15 18:30:00, None]@timestamp[us]
+datetime64[us]:empty []@Series[datetime64[us]] []@timestamp[us]
[]@timestamp[us]
+datetime64[us,tz]:standard [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[us, UTC]] [2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] [2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC]
+datetime64[us,tz]:nullable [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC'), NaT]@Series[datetime64[us, UTC]] [2024-06-15 18:30:00+00:00,
None]@timestamp[us, tz=UTC] [2024-06-15 18:30:00+00:00, None]@timestamp[us,
tz=UTC]
+timedelta64[us]:standard [Timedelta('1 days 00:00:00'), Timedelta('0
days 02:00:00')]@Series[timedelta64[us]] [1 day, 0:00:00,
2:00:00]@duration[us] [1 day, 0:00:00, 2:00:00]@duration[us]
+timedelta64[us]:nullable [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[us]] [1 day, 0:00:00, None]@duration[us] [1
day, 0:00:00, None]@duration[us]
+datetime64:inferred [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] [2024-06-15 18:30:00]@timestamp[ns]
[2024-06-15 18:30:00]@timestamp[ns]
+timedelta64:inferred [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:00:00')]@Series[timedelta64[ns]] [1 days 00:00:00, 0 days
02:00:00]@duration[ns] [1 days 00:00:00, 0 days 02:00:00]@duration[ns]
+datetime64[us]:out-of-ns-range [Timestamp('1500-01-01
00:00:00')]@Series[datetime64[us]] [1500-01-01 00:00:00]@timestamp[us]
[1500-01-01 00:00:00]@timestamp[us]
+date:standard [datetime.date(2024, 6, 15)]@Series[object]
[2024-06-15]@date32[day] [2024-06-15]@date32[day]
+time:standard [datetime.time(18, 30, 45)]@Series[object]
[18:30:45]@time64[us] [18:30:45]@time64[us]
+object:datetime [datetime.datetime(2024, 6, 15, 18, 30)]@Series[object]
[2024-06-15 18:30:00]@timestamp[us] [2024-06-15 18:30:00]@timestamp[us]
+object:datetime-sub-us [Timestamp('2024-01-01
00:00:00.000000123')]@Series[object] [2024-01-01 00:00:00]@timestamp[us]
[2024-01-01 00:00:00]@timestamp[us]
+object:timedelta [datetime.timedelta(days=1,
seconds=7200)]@Series[object] [1 day, 2:00:00]@duration[us] [1 day,
2:00:00]@duration[us]
+category:standard ['a', 'b', 'a']@Series[category] [a, b,
a]@dictionary<values=string, indices=int8, ordered=0> [a, b,
a]@dictionary<values=string, indices=int8, ordered=0>
+category:nullable ['a', nan, 'b']@Series[category] [a, None,
b]@dictionary<values=string, indices=int8, ordered=0> [a, None,
b]@dictionary<values=string, indices=int8, ordered=0>
+Int8:standard [0, 1, 127, -128]@Series[Int8] [0, 1, 127, -128]@int8
ERR@ValueError
+Int8:nullable [0, 1, <NA>]@Series[Int8] [0, 1, None]@int8
ERR@ValueError
+Int16:standard [0, 1, 32767, -32768]@Series[Int16] [0, 1, 32767,
-32768]@int16 ERR@ValueError
+Int16:nullable [0, 1, <NA>]@Series[Int16] [0, 1, None]@int16
ERR@ValueError
+Int32:standard [0, 1, 2147483647, -2147483648]@Series[Int32] [0, 1,
2147483647, -2147483648]@int32 ERR@ValueError
+Int32:nullable [0, 1, <NA>]@Series[Int32] [0, 1, None]@int32
ERR@ValueError
+Int64:standard [0, 1, 9223372036854775807, -9223372036854775808]@Series[Int64]
[0, 1, 9223372036854775807, -9223372036854775808]@int64 ERR@ValueError
+Int64:nullable [0, 1, <NA>]@Series[Int64] [0, 1, None]@int64
ERR@ValueError
+UInt64:standard [0, 1, 18446744073709551615]@Series[UInt64] [0, 1,
18446744073709551615]@uint64 ERR@ValueError
+Int64:empty []@Series[Int64] []@int64 ERR@ValueError
+Int64:all-null [<NA>, <NA>]@Series[Int64] [None, None]@int64
ERR@ValueError
+Float64:standard [0.0, 1.5]@Series[Float64] [0.0, 1.5]@float64
ERR@ValueError
+Float64:nullable [0.0, <NA>]@Series[Float64] [0.0, None]@float64
ERR@ValueError
+boolean:standard [True, False]@Series[boolean] [True, False]@bool
ERR@ValueError
+boolean:nullable [True, <NA>]@Series[boolean] [True, None]@bool
ERR@ValueError
+string[python]:standard ['hello', 'world']@Series[string] [hello,
world]@string ERR@ValueError
+string[python]:nullable ['hello', <NA>]@Series[string] [hello,
None]@string ERR@ValueError
+string[python]:empty []@Series[string] []@string ERR@ValueError
+int64[pyarrow]:standard [0, 1, -1]@Series[int64[pyarrow]] [0, 1,
-1]@int64 ERR@ValueError
+int64[pyarrow]:nullable [0, 1, <NA>]@Series[int64[pyarrow]] [0, 1,
None]@int64 ERR@ValueError
+int64[pyarrow]:empty []@Series[int64[pyarrow]] []@int64
ERR@ValueError
+double[pyarrow]:nullable [0.0, <NA>]@Series[double[pyarrow]] [0.0,
None]@float64 ERR@ValueError
+bool[pyarrow]:nullable [True, <NA>]@Series[bool[pyarrow]] [True,
None]@bool ERR@ValueError
+string[pyarrow]:standard ['hello', 'world']@Series[string] [hello,
world]@large_string ERR@ValueError
+string[pyarrow]:nullable ['hello', <NA>]@Series[string] [hello,
None]@large_string ERR@ValueError
+string[pyarrow]:empty []@Series[string] []@large_string ERR@ValueError
+timestamp[us][pyarrow]:standard [Timestamp('2024-01-01
12:00:00')]@Series[timestamp[us][pyarrow]] [2024-01-01
12:00:00]@timestamp[us] ERR@ValueError
+int64[pyarrow]:single-chunk [1, 2]@Series[int64[pyarrow]] [1, 2]@int64
ERR@ValueError
+int64[pyarrow]:multi-chunk [1, 2, 3]@Series[int64[pyarrow]] [1, 2,
3]@chunked<int64> ERR@ValueError
+int64:overflow [300, 1]@Series[int64] [300, 1]@int64 [300, 1]@int64
+float64:fractional [1.5, 2.5]@Series[float64] [1.5, 2.5]@float64
[1.5, 2.5]@float64
+float64:infinity [inf, 1.0]@Series[float64] [inf, 1.0]@float64
[inf, 1.0]@float64
+float64:precision [1.1234567890123]@Series[float64]
[1.1234567890123]@float64 [1.1234567890123]@float64
+datetime64[ns]:sub-us [Timestamp('2024-01-01
00:00:00.000000123')]@Series[datetime64[ns]] [2024-01-01
00:00:00.000000123]@timestamp[ns] [2024-01-01
00:00:00.000000123]@timestamp[ns]
+object:date-then-datetime [datetime.date(2024, 1, 1),
datetime.datetime(2024, 1, 1, 5, 30)]@Series[object] [2024-01-01,
2024-01-01]@date32[day] [2024-01-01, 2024-01-01]@date32[day]
+object:datetime-then-date [datetime.datetime(2024, 1, 1, 5, 30),
datetime.date(2024, 1, 1)]@Series[object] ERR@ArrowTypeError
ERR@ArrowTypeError
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.md
new file mode 100644
index 000000000000..8b3c5ff0f85d
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.md
@@ -0,0 +1,100 @@
+| test case | pandas series
| mask=None
| mask=isnull()
|
+|---------------------------------|--------------------------------------------------------------------------------------|-----------------------------------------------------------------|-----------------------------------------------------------------|
+| int8:standard | [0, 1, -1, 127, -128]@Series[int8]
| [0, 1, -1, 127, -128]@int8
| [0, 1, -1, 127, -128]@int8
|
+| int8:empty | []@Series[int8]
| []@int8
| []@int8
|
+| int16:standard | [0, 1, -1, 32767, -32768]@Series[int16]
| [0, 1, -1, 32767, -32768]@int16
| [0, 1, -1, 32767, -32768]@int16
|
+| int16:empty | []@Series[int16]
| []@int16
| []@int16
|
+| int32:standard | [0, 1, -1, 2147483647,
-2147483648]@Series[int32] | [0, 1, -1,
2147483647, -2147483648]@int32 | [0, 1, -1, 2147483647,
-2147483648]@int32 |
+| int32:empty | []@Series[int32]
| []@int32
| []@int32
|
+| int64:standard | [0, 1, -1, 9223372036854775807,
-9223372036854775808]@Series[int64] | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 |
+| int64:empty | []@Series[int64]
| []@int64
| []@int64
|
+| int64:nullable | [0.0, 1.0, nan]@Series[float64]
| [0.0, 1.0, None]@float64
| [0.0, 1.0, None]@float64
|
+| uint8:standard | [0, 1, 255]@Series[uint8]
| [0, 1, 255]@uint8
| [0, 1, 255]@uint8
|
+| uint16:standard | [0, 1, 65535]@Series[uint16]
| [0, 1, 65535]@uint16
| [0, 1, 65535]@uint16
|
+| uint32:standard | [0, 1, 4294967295]@Series[uint32]
| [0, 1, 4294967295]@uint32
| [0, 1, 4294967295]@uint32
|
+| uint64:standard | [0, 1,
18446744073709551615]@Series[uint64] |
[0, 1, 18446744073709551615]@uint64 | [0, 1,
18446744073709551615]@uint64 |
+| float32:standard | [0.0, 1.5, -1.5]@Series[float32]
| [0.0, 1.5, -1.5]@float32
| [0.0, 1.5, -1.5]@float32
|
+| float32:nullable | [0.0, nan, 1.5]@Series[float32]
| [0.0, None, 1.5]@float32
| [0.0, None, 1.5]@float32
|
+| float32:empty | []@Series[float32]
| []@float32
| []@float32
|
+| float64:standard | [0.0, 1.5, -1.5]@Series[float64]
| [0.0, 1.5, -1.5]@float64
| [0.0, 1.5, -1.5]@float64
|
+| float64:nullable | [0.0, nan, 1.5]@Series[float64]
| [0.0, None, 1.5]@float64
| [0.0, None, 1.5]@float64
|
+| float64:empty | []@Series[float64]
| []@float64
| []@float64
|
+| bool:standard | [True, False, True]@Series[bool]
| [True, False, True]@bool
| [True, False, True]@bool
|
+| bool:empty | []@Series[bool]
| []@bool
| []@bool
|
+| object:string | ['hello', 'world', '']@Series[object]
| [hello, world, ]@string
| [hello, world, ]@string
|
+| object:string-nullable | ['hello', None, 'world']@Series[object]
| [hello, None, world]@string
| [hello, None, world]@string
|
+| string:inferred | ['hello', 'world']@Series[object]
| [hello, world]@string
| [hello, world]@string
|
+| object:bytes | [b'hello', b'world']@Series[object]
| [b'hello', b'world']@binary
| [b'hello', b'world']@binary
|
+| object:empty | []@Series[object]
| []@null
| []@null
|
+| object:all-null | [None, None]@Series[object]
| [None, None]@null
| [None, None]@null
|
+| object:decimal | [Decimal('1.50'),
Decimal('-2.25')]@Series[object] | [1.50,
-2.25]@decimal128(3, 2) | [1.50,
-2.25]@decimal128(3, 2) |
+| list<int64>:standard | [[1, 2], [3]]@Series[object]
| [[1, 2], [3]]@list<item: int64>
| [[1, 2], [3]]@list<item: int64>
|
+| list<int64>:nullable | [[1, 2], None]@Series[object]
| [[1, 2], None]@list<item: int64>
| [[1, 2], None]@list<item: int64>
|
+| list<int64>:null-element | [[1, None], [3]]@Series[object]
| [[1, None], [3]]@list<item: int64>
| [[1, None], [3]]@list<item: int64>
|
+| list<string>:standard | [['a', 'b'], ['c']]@Series[object]
| [['a', 'b'], ['c']]@list<item:
string> | [['a', 'b'], ['c']]@list<item: string>
|
+| list<list<int64>>:standard | [[[1, 2], [3]], [[4]]]@Series[object]
| [[[1, 2], [3]], [[4]]]@list<item:
list<item: int64>> | [[[1, 2], [3]], [[4]]]@list<item: list<item:
int64>> |
+| list<struct>:standard | [[{'a': 1}], [{'a': 2}]]@Series[object]
| [[{'a': 1}], [{'a': 2}]]@list<item:
struct<a: int64>> | [[{'a': 1}], [{'a': 2}]]@list<item: struct<a:
int64>> |
+| struct:standard | [{'a': 1, 'b': 'x'}]@Series[object]
| [[('a', 1), ('b', 'x')]]@struct<a:
int64, b: string> | [[('a', 1), ('b', 'x')]]@struct<a: int64, b:
string> |
+| struct:nullable | [{'a': 1, 'b': 'x'}, None]@Series[object]
| [[('a', 1), ('b', 'x')],
None]@struct<a: int64, b: string> | [[('a', 1), ('b', 'x')],
None]@struct<a: int64, b: string> |
+| struct<struct>:standard | [{'a': {'b': 1}}]@Series[object]
| [[('a', {'b': 1})]]@struct<a:
struct<b: int64>> | [[('a', {'b': 1})]]@struct<a: struct<b:
int64>> |
+| struct<list<int64>>:standard | [{'a': [1, 2]}]@Series[object]
| [[('a', [1, 2])]]@struct<a:
list<item: int64>> | [[('a', [1, 2])]]@struct<a: list<item:
int64>> |
+| datetime64[ns]:standard | [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] | [2024-06-15
18:30:00]@timestamp[ns] | [2024-06-15
18:30:00]@timestamp[ns] |
+| datetime64[ns]:nullable | [Timestamp('2024-06-15 18:30:00'),
NaT]@Series[datetime64[ns]] | [2024-06-15 18:30:00,
None]@timestamp[ns] | [2024-06-15 18:30:00,
None]@timestamp[ns] |
+| datetime64[ns]:empty | []@Series[datetime64[ns]]
| []@timestamp[ns]
| []@timestamp[ns]
|
+| datetime64[ns,tz]:standard | [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] | [2024-06-15
18:30:00+00:00]@timestamp[ns, tz=UTC] | [2024-06-15
18:30:00+00:00]@timestamp[ns, tz=UTC] |
+| datetime64[ns,tz]:nullable | [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC'), NaT]@Series[datetime64[ns, UTC]] | [2024-06-15 18:30:00+00:00,
None]@timestamp[ns, tz=UTC] | [2024-06-15 18:30:00+00:00,
None]@timestamp[ns, tz=UTC] |
+| timedelta64[ns]:standard | [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:00:00')]@Series[timedelta64[ns]] | [1 days 00:00:00, 0
days 02:00:00]@duration[ns] | [1 days 00:00:00, 0 days
02:00:00]@duration[ns] |
+| timedelta64[ns]:nullable | [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]] | [1 days 00:00:00,
None]@duration[ns] | [1 days 00:00:00,
None]@duration[ns] |
+| datetime64[us]:standard | [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[us]] | [2024-06-15
18:30:00]@timestamp[us] | [2024-06-15
18:30:00]@timestamp[us] |
+| datetime64[us]:nullable | [Timestamp('2024-06-15 18:30:00'),
NaT]@Series[datetime64[us]] | [2024-06-15 18:30:00,
None]@timestamp[us] | [2024-06-15 18:30:00,
None]@timestamp[us] |
+| datetime64[us]:empty | []@Series[datetime64[us]]
| []@timestamp[us]
| []@timestamp[us]
|
+| datetime64[us,tz]:standard | [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[us, UTC]] | [2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] | [2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] |
+| datetime64[us,tz]:nullable | [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC'), NaT]@Series[datetime64[us, UTC]] | [2024-06-15 18:30:00+00:00,
None]@timestamp[us, tz=UTC] | [2024-06-15 18:30:00+00:00,
None]@timestamp[us, tz=UTC] |
+| timedelta64[us]:standard | [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:00:00')]@Series[timedelta64[us]] | [1 day, 0:00:00,
2:00:00]@duration[us] | [1 day, 0:00:00,
2:00:00]@duration[us] |
+| timedelta64[us]:nullable | [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[us]] | [1 day, 0:00:00,
None]@duration[us] | [1 day, 0:00:00,
None]@duration[us] |
+| datetime64:inferred | [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] | [2024-06-15
18:30:00]@timestamp[ns] | [2024-06-15
18:30:00]@timestamp[ns] |
+| timedelta64:inferred | [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:00:00')]@Series[timedelta64[ns]] | [1 days 00:00:00, 0
days 02:00:00]@duration[ns] | [1 days 00:00:00, 0 days
02:00:00]@duration[ns] |
+| datetime64[us]:out-of-ns-range | [Timestamp('1500-01-01
00:00:00')]@Series[datetime64[us]] | [1500-01-01
00:00:00]@timestamp[us] | [1500-01-01
00:00:00]@timestamp[us] |
+| date:standard | [datetime.date(2024, 6,
15)]@Series[object] |
[2024-06-15]@date32[day] |
[2024-06-15]@date32[day] |
+| time:standard | [datetime.time(18, 30, 45)]@Series[object]
| [18:30:45]@time64[us]
| [18:30:45]@time64[us]
|
+| object:datetime | [datetime.datetime(2024, 6, 15, 18,
30)]@Series[object] | [2024-06-15
18:30:00]@timestamp[us] | [2024-06-15
18:30:00]@timestamp[us] |
+| object:datetime-sub-us | [Timestamp('2024-01-01
00:00:00.000000123')]@Series[object] | [2024-01-01
00:00:00]@timestamp[us] | [2024-01-01
00:00:00]@timestamp[us] |
+| object:timedelta | [datetime.timedelta(days=1,
seconds=7200)]@Series[object] | [1 day,
2:00:00]@duration[us] | [1 day,
2:00:00]@duration[us] |
+| category:standard | ['a', 'b', 'a']@Series[category]
| [a, b, a]@dictionary<values=string,
indices=int8, ordered=0> | [a, b, a]@dictionary<values=string, indices=int8,
ordered=0> |
+| category:nullable | ['a', nan, 'b']@Series[category]
| [a, None,
b]@dictionary<values=string, indices=int8, ordered=0> | [a, None,
b]@dictionary<values=string, indices=int8, ordered=0> |
+| Int8:standard | [0, 1, 127, -128]@Series[Int8]
| [0, 1, 127, -128]@int8
| ERR@ValueError
|
+| Int8:nullable | [0, 1, <NA>]@Series[Int8]
| [0, 1, None]@int8
| ERR@ValueError
|
+| Int16:standard | [0, 1, 32767, -32768]@Series[Int16]
| [0, 1, 32767, -32768]@int16
| ERR@ValueError
|
+| Int16:nullable | [0, 1, <NA>]@Series[Int16]
| [0, 1, None]@int16
| ERR@ValueError
|
+| Int32:standard | [0, 1, 2147483647,
-2147483648]@Series[Int32] | [0, 1,
2147483647, -2147483648]@int32 | ERR@ValueError
|
+| Int32:nullable | [0, 1, <NA>]@Series[Int32]
| [0, 1, None]@int32
| ERR@ValueError
|
+| Int64:standard | [0, 1, 9223372036854775807,
-9223372036854775808]@Series[Int64] | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ValueError
|
+| Int64:nullable | [0, 1, <NA>]@Series[Int64]
| [0, 1, None]@int64
| ERR@ValueError
|
+| UInt64:standard | [0, 1,
18446744073709551615]@Series[UInt64] |
[0, 1, 18446744073709551615]@uint64 |
ERR@ValueError |
+| Int64:empty | []@Series[Int64]
| []@int64
| ERR@ValueError
|
+| Int64:all-null | [<NA>, <NA>]@Series[Int64]
| [None, None]@int64
| ERR@ValueError
|
+| Float64:standard | [0.0, 1.5]@Series[Float64]
| [0.0, 1.5]@float64
| ERR@ValueError
|
+| Float64:nullable | [0.0, <NA>]@Series[Float64]
| [0.0, None]@float64
| ERR@ValueError
|
+| boolean:standard | [True, False]@Series[boolean]
| [True, False]@bool
| ERR@ValueError
|
+| boolean:nullable | [True, <NA>]@Series[boolean]
| [True, None]@bool
| ERR@ValueError
|
+| string[python]:standard | ['hello', 'world']@Series[string]
| [hello, world]@string
| ERR@ValueError
|
+| string[python]:nullable | ['hello', <NA>]@Series[string]
| [hello, None]@string
| ERR@ValueError
|
+| string[python]:empty | []@Series[string]
| []@string
| ERR@ValueError
|
+| int64[pyarrow]:standard | [0, 1, -1]@Series[int64[pyarrow]]
| [0, 1, -1]@int64
| ERR@ValueError
|
+| int64[pyarrow]:nullable | [0, 1, <NA>]@Series[int64[pyarrow]]
| [0, 1, None]@int64
| ERR@ValueError
|
+| int64[pyarrow]:empty | []@Series[int64[pyarrow]]
| []@int64
| ERR@ValueError
|
+| double[pyarrow]:nullable | [0.0, <NA>]@Series[double[pyarrow]]
| [0.0, None]@float64
| ERR@ValueError
|
+| bool[pyarrow]:nullable | [True, <NA>]@Series[bool[pyarrow]]
| [True, None]@bool
| ERR@ValueError
|
+| string[pyarrow]:standard | ['hello', 'world']@Series[string]
| [hello, world]@large_string
| ERR@ValueError
|
+| string[pyarrow]:nullable | ['hello', <NA>]@Series[string]
| [hello, None]@large_string
| ERR@ValueError
|
+| string[pyarrow]:empty | []@Series[string]
| []@large_string
| ERR@ValueError
|
+| timestamp[us][pyarrow]:standard | [Timestamp('2024-01-01
12:00:00')]@Series[timestamp[us][pyarrow]] | [2024-01-01
12:00:00]@timestamp[us] | ERR@ValueError
|
+| int64[pyarrow]:single-chunk | [1, 2]@Series[int64[pyarrow]]
| [1, 2]@int64
| ERR@ValueError
|
+| int64[pyarrow]:multi-chunk | [1, 2, 3]@Series[int64[pyarrow]]
| [1, 2, 3]@chunked<int64>
| ERR@ValueError
|
+| int64:overflow | [300, 1]@Series[int64]
| [300, 1]@int64
| [300, 1]@int64
|
+| float64:fractional | [1.5, 2.5]@Series[float64]
| [1.5, 2.5]@float64
| [1.5, 2.5]@float64
|
+| float64:infinity | [inf, 1.0]@Series[float64]
| [inf, 1.0]@float64
| [inf, 1.0]@float64
|
+| float64:precision | [1.1234567890123]@Series[float64]
| [1.1234567890123]@float64
| [1.1234567890123]@float64
|
+| datetime64[ns]:sub-us | [Timestamp('2024-01-01
00:00:00.000000123')]@Series[datetime64[ns]] | [2024-01-01
00:00:00.000000123]@timestamp[ns] | [2024-01-01
00:00:00.000000123]@timestamp[ns] |
+| object:date-then-datetime | [datetime.date(2024, 1, 1),
datetime.datetime(2024, 1, 1, 5, 30)]@Series[object] | [2024-01-01,
2024-01-01]@date32[day] | [2024-01-01,
2024-01-01]@date32[day] |
+| object:datetime-then-date | [datetime.datetime(2024, 1, 1, 5, 30),
datetime.date(2024, 1, 1)]@Series[object] | ERR@ArrowTypeError
| ERR@ArrowTypeError
|
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
index 56122eabf513..125ab62789e4 100644
---
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
@@ -89,16 +89,15 @@ if have_numpy:
import numpy as np
[email protected](
- not have_pyarrow or not have_pandas or not have_numpy,
- pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
-)
-class PyArrowArrayFromPandasDefaultTests(GoldenFileTestMixin,
unittest.TestCase):
+class _PyArrowFromPandasTestBase(GoldenFileTestMixin, unittest.TestCase):
"""
- Tests pa.Array.from_pandas() with default arguments via golden file
comparison.
+ Shared machinery for pa.Array.from_pandas() golden file tests.
- Three group methods build the source Series so the non-default tests can
reuse the whole
- inventory or one group. Groups are disjoint and their order fixes the row
order.
+ Owns the source Series inventory: three disjoint group methods unioned by
+ ``_build_source_arrays``, whose order fixes the row order. The default
test below and
+ the non-default tests in ``test_pyarrow_array_from_pandas_non_default.py``
subclass this
+ to reuse the whole inventory or one group, along with
``repr_from_pandas_result``. This
+ base defines no ``test_*`` methods, so it contributes no tests itself.
"""
@staticmethod
@@ -115,6 +114,16 @@ class
PyArrowArrayFromPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase)
return f"{values}@chunked<{arrow_type}>"
return rendered
+ def _from_pandas_cell(self, series, **from_pandas_kwargs) -> str:
+ """
+ Convert ``series`` via ``from_pandas(**from_pandas_kwargs)`` and
format the result
+ as a golden-file cell, returning ``ERR@<ExceptionClass>`` if the
conversion raises.
+ """
+ try:
+ return self.repr_from_pandas_result(pa.Array.from_pandas(series,
**from_pandas_kwargs))
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
def _numpy_backed_sources(self):
"""
Series whose storage is numpy, so ``hasattr(series.array,
"__arrow_array__")`` is
@@ -352,6 +361,14 @@ class
PyArrowArrayFromPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase)
sources.update(group)
return sources
+
[email protected](
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
+)
+class PyArrowArrayFromPandasDefaultTests(_PyArrowFromPandasTestBase):
+ """Tests pa.Array.from_pandas() with default arguments via golden file
comparison."""
+
def test_from_pandas_default(self):
"""Test pa.Array.from_pandas() with default arguments against golden
file."""
sources = self._build_source_arrays()
@@ -392,10 +409,7 @@ class
PyArrowArrayFromPandasDefaultTests(GoldenFileTestMixin, unittest.TestCase)
if col_name == "pandas series":
return self.repr_value(series, max_len=0)
else:
- try:
- return
self.repr_from_pandas_result(pa.Array.from_pandas(series))
- except Exception as e:
- return f"ERR@{type(e).__name__}"
+ return self._from_pandas_cell(series)
self.compare_or_generate_golden_matrix(
row_names=row_names,
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_non_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_non_default.py
new file mode 100644
index 000000000000..6439e189ff43
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_non_default.py
@@ -0,0 +1,167 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Tests for PyArrow Array.from_pandas() with non-default arguments, using golden
file
+comparison.
+
+PySpark passes ``pa.Array.from_pandas(series, mask=mask, type=arrow_type,
safe=safecheck)``
+to convert a pandas Series into an Arrow array. The bare-argument behavior is
covered by
+``test_pyarrow_array_from_pandas_default.py``, whose source Series inventory
+(``_PyArrowFromPandasTestBase``) is reused here; this file records the
non-default
+arguments so CI fails loudly if they drift across pandas/PyArrow/NumPy
upgrades.
+
+## The ``mask`` argument
+
+PySpark derives ``mask`` from how the Series is stored
+(``conversion.py:435``, ``pandas/conversion.py:113``)::
+
+ mask = None if hasattr(series.array, "__arrow_array__") else
series.isnull()
+
+- **numpy-backed** dtypes take ``mask=series.isnull()``, which agrees with the
nulls
+ ``from_pandas`` infers at ``mask=None``; this test pins that agreement.
+- **protocol** dtypes (implementing ``__arrow_array__``) return a finished
Arrow array with
+ its own validity bitmap, so PyArrow rejects any mask -- even an all-False
no-op -- with
+ ``ValueError``; PySpark passes ``mask=None``.
+
+So ``mask`` is fixed by the row's dtype, hence a column pair (``mask=None`` vs
+``mask=isnull()``) rather than a matrix dimension.
+
+## Golden File Cell Format
+
+Each cell uses the value@type format:
+- pandas Series: "python_list_repr@Series[dtype]"
+- PyArrow Array: "python_list_repr@arrow_type"
+- PyArrow ChunkedArray: "python_list_repr@chunked<arrow_type>"
+- Error: "ERR@ExceptionClassName"
+
+## Regenerating Golden Files
+
+Set SPARK_GENERATE_GOLDEN_FILES=1 before running:
+
+ SPARK_GENERATE_GOLDEN_FILES=1 python -m pytest \\
+
python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_non_default.py
+
+## PyArrow and pandas Version Compatibility
+
+The golden files capture behavior for specific PyArrow and pandas versions.
+Regenerate when upgrading either dependency, as from_pandas() behavior may
change.
+The committed golden files were generated with pandas 2.3.3, pyarrow 24.0.0,
and
+numpy 2.4.1.
+"""
+
+import unittest
+
+from pyspark.loose_version import LooseVersion
+from pyspark.testing.utils import (
+ have_pyarrow,
+ have_pandas,
+ have_numpy,
+ pyarrow_requirement_message,
+ pandas_requirement_message,
+ numpy_requirement_message,
+)
+
+# Imported as a module, not `from ... import _PyArrowFromPandasTestBase`, so
that the
+# non-default classes pick up only the shared base -- the default test class
is not
+# re-collected here.
+from pyspark.tests.upstream.pyarrow import
test_pyarrow_array_from_pandas_default
+
+if have_pandas:
+ import pandas as pd
+
+
[email protected](
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
+)
+class PyArrowArrayFromPandasMaskTests(
+ test_pyarrow_array_from_pandas_default._PyArrowFromPandasTestBase
+):
+ """
+ Tests pa.Array.from_pandas(series, mask=...) via golden file comparison.
+
+ Reuses the full source inventory. numpy-backed rows accept a mask and
record equal
+ results for mask=None and mask=isnull(); protocol rows reject a mask and
record
+ ERR@ValueError for mask=isnull().
+ """
+
+ COL_MASK_NONE = "mask=None"
+ COL_MASK_ISNULL = "mask=isnull()"
+
+ def test_from_pandas_mask(self):
+ """Test pa.Array.from_pandas() with the mask argument against golden
file."""
+ sources = self._build_source_arrays()
+ row_names = list(sources.keys())
+ col_names = ["pandas series", self.COL_MASK_NONE, self.COL_MASK_ISNULL]
+
+ # Version-specific expected values go here, keyed by (row, col), when
a newer
+ # pandas/PyArrow/NumPy legitimately changes a cell's output.
+ overrides: dict[tuple[str, str], str] = {}
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ # The deliberately unpinned rows infer microseconds on pandas 3
where pandas 2
+ # infers nanoseconds, and a categorical's values inherit the new
str dtype's
+ # large_string backing. These stay numpy-backed, so both mask
columns move
+ # together and accept the mask.
+ overrides[("datetime64:inferred", "pandas series")] = (
+ "[Timestamp('2024-06-15 18:30:00')]@Series[datetime64[us]]"
+ )
+ overrides[("timedelta64:inferred", "pandas series")] = (
+ "[Timedelta('1 days 00:00:00'), "
+ "Timedelta('0 days 02:00:00')]@Series[timedelta64[us]]"
+ )
+ for col in (self.COL_MASK_NONE, self.COL_MASK_ISNULL):
+ overrides[("datetime64:inferred", col)] = "[2024-06-15
18:30:00]@timestamp[us]"
+ overrides[("timedelta64:inferred", col)] = "[1 day, 0:00:00,
2:00:00]@duration[us]"
+ overrides[("category:standard", col)] = (
+ "[a, b, a]@dictionary<values=large_string, indices=int8,
ordered=0>"
+ )
+ overrides[("category:nullable", col)] = (
+ "[a, None, b]@dictionary<values=large_string,
indices=int8, ordered=0>"
+ )
+ # string:inferred is object (numpy-backed) on pandas 2 but the
dedicated str
+ # dtype (large_string, __arrow_array__) on pandas 3, so it moves
from a
+ # mask-accepting row to a protocol row: mask=isnull() flips to
ERR@ValueError.
+ overrides[("string:inferred", "pandas series")] = "['hello',
'world']@Series[str]"
+ overrides[("string:inferred", self.COL_MASK_NONE)] = "[hello,
world]@large_string"
+ overrides[("string:inferred", self.COL_MASK_ISNULL)] =
"ERR@ValueError"
+
+ def compute_cell(row_name, col_name):
+ series = sources[row_name]
+ if col_name == "pandas series":
+ return self.repr_value(series, max_len=0)
+ elif col_name == self.COL_MASK_NONE:
+ return self._from_pandas_cell(series, mask=None)
+ elif col_name == self.COL_MASK_ISNULL:
+ return self._from_pandas_cell(series, mask=series.isnull())
+ else:
+ raise ValueError(f"unknown column: {col_name}")
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+ golden_file_prefix="golden_pyarrow_array_from_pandas_mask",
+ index_name="test case",
+ overrides=overrides,
+ )
+
+
+if __name__ == "__main__":
+ from pyspark.testing import main
+
+ main()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
