This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new c9fc60f1bb76 [SPARK-58602][PYTHON][TEST] Add tests for
pa.Array.from_pandas with default arguments
c9fc60f1bb76 is described below
commit c9fc60f1bb766d2e70f7bba6aa49933d516ba318
Author: Spenser Sun <[email protected]>
AuthorDate: Fri Aug 7 13:43:26 2026 +0800
[SPARK-58602][PYTHON][TEST] Add tests for pa.Array.from_pandas with default
arguments
### What changes were proposed in this pull request?
This PR adds a golden-file test pinning the behavior of
`pa.Array.from_pandas` with default arguments (no `mask`, no `type`, no
`safe`), under the SPARK-54936 umbrella for monitoring upstream library
behavior changes.
New file
`python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py`
with `PyArrowArrayFromPandasDefaultTests`, plus the golden files
`golden_pyarrow_array_from_pandas_default.{csv,md}` (97 rows x 2 columns) and
one line in `dev/sparktestsupport/modules.py`.
The source Series are built by three group methods, so the follow-up
non-default tests can reuse the whole inventory or one group:
- `_numpy_backed_sources()` -- numpy storage, so `hasattr(series.array,
"__arrow_array__")` is `False` and PySpark passes `mask=series.isnull()`.
- `_protocol_sources()` -- dtypes implementing `__arrow_array__`, where
PySpark passes `mask=None` because supplying a mask raises. The protocol means
"can export Arrow", not "is stored as Arrow": `Int64` is numpy values plus a
byte mask and `string[python]` is an object ndarray, yet both implement it
alongside the genuinely Arrow-backed `[pyarrow]` dtypes. PySpark does not
distinguish them -- `conversion.py:435` branches only on the protocol -- so
they share one group.
- `_coercion_sources()` -- values that are lossy or ambiguous once a target
type is requested. Their unconverted results are recorded here to give the
follow-up type tests a baseline to be read against.
`from_pandas` does not always return a `pa.Array` -- for a Series backed by
a chunked Arrow array it returns a `pa.ChunkedArray`. A small helper therefore
outputs those cells as `chunked<type>`, since `repr_value` reports the element
type and would render the two identically. Checking who consumes that return
value then showed one place that accepts only an `Array`: the pandas UDF return
path assembles its result with `pa.RecordBatch.from_arrays`, so such a UDF
fails with a raw `TypeE [...]
### Why are the changes needed?
`from_pandas` is the pandas -> Arrow direction, which the previously merged
golden tests under SPARK-54936 do not cover -- they are all Arrow -> pandas.
PySpark depends on it at three production call sites:
`python/pyspark/sql/conversion.py:443` and `:474` (the pandas UDF return leg,
legacy and non-legacy), and `python/pyspark/sql/pandas/conversion.py:117`
(`create_arrow_array_from_pandas`, used by `createDataFrame(pandas_df)`). A
silent change here becomes wrong data on the JVM side [...]
Two behaviors the golden pins that are easy to miss:
- **pandas 3 changed its default temporal resolution.**
`pd.Series([datetime])` infers `datetime64[ns]` on pandas 2 and
`datetime64[us]` on pandas 3, and `Categorical.categories` become `str` (so the
dictionary's values become `large_string`). The rows pin each resolution
explicitly and add deliberately unpinned `:inferred` rows whose only job is to
record which default pandas picks.
- **`object` dtype and `datetime64[ns]` disagree.** In object dtype pyarrow
takes the unit from `datetime.datetime`'s own microsecond resolution rather
than from a numpy dtype, so the same values give `timestamp[us]` instead of
`timestamp[ns]` -- and a sub-microsecond value is dropped silently, while an
explicit `type=pa.timestamp("us")` raises `ArrowInvalid` on that same loss.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
New golden-file test. It runs without a Spark session, so it exercises
PyArrow and pandas directly.
Validated across **16 environments -- PyArrow 18/19/20/21/22/23/24/25 x
pandas 2 and pandas 3** -- with a fresh virtualenv per combination, running the
committed test file against the committed golden file. All 16 pass. Six cells
legitimately differ on pandas 3 and are carried in the `overrides` dict rather
than by regenerating; disabling the `LooseVersion` guard makes pandas 3 fail on
exactly those six while pandas 2 still passes, so the overrides are
load-bearing.
Also verified: the test passes without `SPARK_GENERATE_GOLDEN_FILES`,
regeneration is byte-identical, and `ruff check` / `ruff format --check` /
`mypy` are clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Closes #57830 from Spenserrrr/from-pandas-tests.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 4c4ce5ff0663d77f08b524cc2488c19c09099f35)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
dev/sparktestsupport/modules.py | 1 +
.../golden_pyarrow_array_from_pandas_default.csv | 99 +++++
.../golden_pyarrow_array_from_pandas_default.md | 100 +++++
.../test_pyarrow_array_from_pandas_default.py | 413 +++++++++++++++++++++
4 files changed, 613 insertions(+)
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 79e2e5a7bc2e..1f1e4604f6d2 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -537,6 +537,7 @@ pyspark_core = Module(
"pyspark.tests.test_zero_copy_byte_stream",
# unittests for upstream projects
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_cast",
+
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_from_pandas_default",
"pyspark.tests.upstream.pyarrow.test_pyarrow_array_type_inference",
"pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_default",
"pyspark.tests.upstream.pyarrow.test_pyarrow_arrow_to_pandas_non_default",
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.csv
new file mode 100644
index 000000000000..725784fe911d
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.csv
@@ -0,0 +1,99 @@
+test case pandas series arrow array
+int8:standard [0, 1, -1, 127, -128]@Series[int8] [0, 1, -1, 127,
-128]@int8
+int8:empty []@Series[int8] []@int8
+int16:standard [0, 1, -1, 32767, -32768]@Series[int16] [0, 1, -1, 32767,
-32768]@int16
+int16:empty []@Series[int16] []@int16
+int32:standard [0, 1, -1, 2147483647, -2147483648]@Series[int32] [0, 1,
-1, 2147483647, -2147483648]@int32
+int32:empty []@Series[int32] []@int32
+int64:standard [0, 1, -1, 9223372036854775807,
-9223372036854775808]@Series[int64] [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64
+int64:empty []@Series[int64] []@int64
+int64:nullable [0.0, 1.0, nan]@Series[float64] [0.0, 1.0, None]@float64
+uint8:standard [0, 1, 255]@Series[uint8] [0, 1, 255]@uint8
+uint16:standard [0, 1, 65535]@Series[uint16] [0, 1, 65535]@uint16
+uint32:standard [0, 1, 4294967295]@Series[uint32] [0, 1,
4294967295]@uint32
+uint64:standard [0, 1, 18446744073709551615]@Series[uint64] [0, 1,
18446744073709551615]@uint64
+float32:standard [0.0, 1.5, -1.5]@Series[float32] [0.0, 1.5,
-1.5]@float32
+float32:nullable [0.0, nan, 1.5]@Series[float32] [0.0, None, 1.5]@float32
+float32:empty []@Series[float32] []@float32
+float64:standard [0.0, 1.5, -1.5]@Series[float64] [0.0, 1.5,
-1.5]@float64
+float64:nullable [0.0, nan, 1.5]@Series[float64] [0.0, None, 1.5]@float64
+float64:empty []@Series[float64] []@float64
+bool:standard [True, False, True]@Series[bool] [True, False, True]@bool
+bool:empty []@Series[bool] []@bool
+object:string ['hello', 'world', '']@Series[object] [hello, world, ]@string
+object:string-nullable ['hello', None, 'world']@Series[object] [hello, None,
world]@string
+string:inferred ['hello', 'world']@Series[object] [hello,
world]@string
+object:bytes [b'hello', b'world']@Series[object] [b'hello',
b'world']@binary
+object:empty []@Series[object] []@null
+object:all-null [None, None]@Series[object] [None, None]@null
+object:decimal [Decimal('1.50'), Decimal('-2.25')]@Series[object] [1.50,
-2.25]@decimal128(3, 2)
+list<int64>:standard [[1, 2], [3]]@Series[object] [[1, 2],
[3]]@list<item: int64>
+list<int64>:nullable [[1, 2], None]@Series[object] [[1, 2],
None]@list<item: int64>
+list<int64>:null-element [[1, None], [3]]@Series[object] [[1, None],
[3]]@list<item: int64>
+list<string>:standard [['a', 'b'], ['c']]@Series[object] [['a', 'b'],
['c']]@list<item: string>
+list<list<int64>>:standard [[[1, 2], [3]], [[4]]]@Series[object] [[[1,
2], [3]], [[4]]]@list<item: list<item: int64>>
+list<struct>:standard [[{'a': 1}], [{'a': 2}]]@Series[object] [[{'a': 1}],
[{'a': 2}]]@list<item: struct<a: int64>>
+struct:standard [{'a': 1, 'b': 'x'}]@Series[object] [[('a', 1),
('b', 'x')]]@struct<a: int64, b: string>
+struct:nullable [{'a': 1, 'b': 'x'}, None]@Series[object] [[('a',
1), ('b', 'x')], None]@struct<a: int64, b: string>
+struct<struct>:standard [{'a': {'b': 1}}]@Series[object] [[('a',
{'b': 1})]]@struct<a: struct<b: int64>>
+struct<list<int64>>:standard [{'a': [1, 2]}]@Series[object] [[('a', [1,
2])]]@struct<a: list<item: int64>>
+datetime64[ns]:standard [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] [2024-06-15 18:30:00]@timestamp[ns]
+datetime64[ns]:nullable [Timestamp('2024-06-15 18:30:00'),
NaT]@Series[datetime64[ns]] [2024-06-15 18:30:00, None]@timestamp[ns]
+datetime64[ns]:empty []@Series[datetime64[ns]] []@timestamp[ns]
+datetime64[ns,tz]:standard [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] [2024-06-15
18:30:00+00:00]@timestamp[ns, tz=UTC]
+datetime64[ns,tz]:nullable [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC'), NaT]@Series[datetime64[ns, UTC]] [2024-06-15 18:30:00+00:00,
None]@timestamp[ns, tz=UTC]
+timedelta64[ns]:standard [Timedelta('1 days 00:00:00'), Timedelta('0
days 02:00:00')]@Series[timedelta64[ns]] [1 days 00:00:00, 0 days
02:00:00]@duration[ns]
+timedelta64[ns]:nullable [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]] [1 days 00:00:00, None]@duration[ns]
+datetime64[us]:standard [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[us]] [2024-06-15 18:30:00]@timestamp[us]
+datetime64[us]:nullable [Timestamp('2024-06-15 18:30:00'),
NaT]@Series[datetime64[us]] [2024-06-15 18:30:00, None]@timestamp[us]
+datetime64[us]:empty []@Series[datetime64[us]] []@timestamp[us]
+datetime64[us,tz]:standard [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[us, UTC]] [2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC]
+datetime64[us,tz]:nullable [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC'), NaT]@Series[datetime64[us, UTC]] [2024-06-15 18:30:00+00:00,
None]@timestamp[us, tz=UTC]
+timedelta64[us]:standard [Timedelta('1 days 00:00:00'), Timedelta('0
days 02:00:00')]@Series[timedelta64[us]] [1 day, 0:00:00,
2:00:00]@duration[us]
+timedelta64[us]:nullable [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[us]] [1 day, 0:00:00, None]@duration[us]
+datetime64:inferred [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] [2024-06-15 18:30:00]@timestamp[ns]
+timedelta64:inferred [Timedelta('1 days 00:00:00'), Timedelta('0 days
02:00:00')]@Series[timedelta64[ns]] [1 days 00:00:00, 0 days
02:00:00]@duration[ns]
+datetime64[us]:out-of-ns-range [Timestamp('1500-01-01
00:00:00')]@Series[datetime64[us]] [1500-01-01 00:00:00]@timestamp[us]
+date:standard [datetime.date(2024, 6, 15)]@Series[object]
[2024-06-15]@date32[day]
+time:standard [datetime.time(18, 30, 45)]@Series[object]
[18:30:45]@time64[us]
+object:datetime [datetime.datetime(2024, 6, 15, 18, 30)]@Series[object]
[2024-06-15 18:30:00]@timestamp[us]
+object:datetime-sub-us [Timestamp('2024-01-01
00:00:00.000000123')]@Series[object] [2024-01-01 00:00:00]@timestamp[us]
+object:timedelta [datetime.timedelta(days=1,
seconds=7200)]@Series[object] [1 day, 2:00:00]@duration[us]
+category:standard ['a', 'b', 'a']@Series[category] [a, b,
a]@dictionary<values=string, indices=int8, ordered=0>
+category:nullable ['a', nan, 'b']@Series[category] [a, None,
b]@dictionary<values=string, indices=int8, ordered=0>
+Int8:standard [0, 1, 127, -128]@Series[Int8] [0, 1, 127, -128]@int8
+Int8:nullable [0, 1, <NA>]@Series[Int8] [0, 1, None]@int8
+Int16:standard [0, 1, 32767, -32768]@Series[Int16] [0, 1, 32767,
-32768]@int16
+Int16:nullable [0, 1, <NA>]@Series[Int16] [0, 1, None]@int16
+Int32:standard [0, 1, 2147483647, -2147483648]@Series[Int32] [0, 1,
2147483647, -2147483648]@int32
+Int32:nullable [0, 1, <NA>]@Series[Int32] [0, 1, None]@int32
+Int64:standard [0, 1, 9223372036854775807, -9223372036854775808]@Series[Int64]
[0, 1, 9223372036854775807, -9223372036854775808]@int64
+Int64:nullable [0, 1, <NA>]@Series[Int64] [0, 1, None]@int64
+UInt64:standard [0, 1, 18446744073709551615]@Series[UInt64] [0, 1,
18446744073709551615]@uint64
+Int64:empty []@Series[Int64] []@int64
+Int64:all-null [<NA>, <NA>]@Series[Int64] [None, None]@int64
+Float64:standard [0.0, 1.5]@Series[Float64] [0.0, 1.5]@float64
+Float64:nullable [0.0, <NA>]@Series[Float64] [0.0, None]@float64
+boolean:standard [True, False]@Series[boolean] [True, False]@bool
+boolean:nullable [True, <NA>]@Series[boolean] [True, None]@bool
+string[python]:standard ['hello', 'world']@Series[string] [hello,
world]@string
+string[python]:nullable ['hello', <NA>]@Series[string] [hello,
None]@string
+string[python]:empty []@Series[string] []@string
+int64[pyarrow]:standard [0, 1, -1]@Series[int64[pyarrow]] [0, 1,
-1]@int64
+int64[pyarrow]:nullable [0, 1, <NA>]@Series[int64[pyarrow]] [0, 1,
None]@int64
+int64[pyarrow]:empty []@Series[int64[pyarrow]] []@int64
+double[pyarrow]:nullable [0.0, <NA>]@Series[double[pyarrow]] [0.0,
None]@float64
+bool[pyarrow]:nullable [True, <NA>]@Series[bool[pyarrow]] [True,
None]@bool
+string[pyarrow]:standard ['hello', 'world']@Series[string] [hello,
world]@large_string
+string[pyarrow]:nullable ['hello', <NA>]@Series[string] [hello,
None]@large_string
+string[pyarrow]:empty []@Series[string] []@large_string
+timestamp[us][pyarrow]:standard [Timestamp('2024-01-01
12:00:00')]@Series[timestamp[us][pyarrow]] [2024-01-01
12:00:00]@timestamp[us]
+int64[pyarrow]:single-chunk [1, 2]@Series[int64[pyarrow]] [1, 2]@int64
+int64[pyarrow]:multi-chunk [1, 2, 3]@Series[int64[pyarrow]] [1, 2,
3]@chunked<int64>
+int64:overflow [300, 1]@Series[int64] [300, 1]@int64
+float64:fractional [1.5, 2.5]@Series[float64] [1.5, 2.5]@float64
+float64:infinity [inf, 1.0]@Series[float64] [inf, 1.0]@float64
+float64:precision [1.1234567890123]@Series[float64]
[1.1234567890123]@float64
+datetime64[ns]:sub-us [Timestamp('2024-01-01
00:00:00.000000123')]@Series[datetime64[ns]] [2024-01-01
00:00:00.000000123]@timestamp[ns]
+object:date-then-datetime [datetime.date(2024, 1, 1),
datetime.datetime(2024, 1, 1, 5, 30)]@Series[object] [2024-01-01,
2024-01-01]@date32[day]
+object:datetime-then-date [datetime.datetime(2024, 1, 1, 5, 30),
datetime.date(2024, 1, 1)]@Series[object] ERR@ArrowTypeError
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.md
new file mode 100644
index 000000000000..502e494d468d
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.md
@@ -0,0 +1,100 @@
+| test case | pandas series
| arrow array
|
+|---------------------------------|--------------------------------------------------------------------------------------|-----------------------------------------------------------------|
+| int8:standard | [0, 1, -1, 127, -128]@Series[int8]
| [0, 1, -1, 127, -128]@int8
|
+| int8:empty | []@Series[int8]
| []@int8
|
+| int16:standard | [0, 1, -1, 32767, -32768]@Series[int16]
| [0, 1, -1, 32767, -32768]@int16
|
+| int16:empty | []@Series[int16]
| []@int16
|
+| int32:standard | [0, 1, -1, 2147483647,
-2147483648]@Series[int32] | [0, 1, -1,
2147483647, -2147483648]@int32 |
+| int32:empty | []@Series[int32]
| []@int32
|
+| int64:standard | [0, 1, -1, 9223372036854775807,
-9223372036854775808]@Series[int64] | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 |
+| int64:empty | []@Series[int64]
| []@int64
|
+| int64:nullable | [0.0, 1.0, nan]@Series[float64]
| [0.0, 1.0, None]@float64
|
+| uint8:standard | [0, 1, 255]@Series[uint8]
| [0, 1, 255]@uint8
|
+| uint16:standard | [0, 1, 65535]@Series[uint16]
| [0, 1, 65535]@uint16
|
+| uint32:standard | [0, 1, 4294967295]@Series[uint32]
| [0, 1, 4294967295]@uint32
|
+| uint64:standard | [0, 1,
18446744073709551615]@Series[uint64] |
[0, 1, 18446744073709551615]@uint64 |
+| float32:standard | [0.0, 1.5, -1.5]@Series[float32]
| [0.0, 1.5, -1.5]@float32
|
+| float32:nullable | [0.0, nan, 1.5]@Series[float32]
| [0.0, None, 1.5]@float32
|
+| float32:empty | []@Series[float32]
| []@float32
|
+| float64:standard | [0.0, 1.5, -1.5]@Series[float64]
| [0.0, 1.5, -1.5]@float64
|
+| float64:nullable | [0.0, nan, 1.5]@Series[float64]
| [0.0, None, 1.5]@float64
|
+| float64:empty | []@Series[float64]
| []@float64
|
+| bool:standard | [True, False, True]@Series[bool]
| [True, False, True]@bool
|
+| bool:empty | []@Series[bool]
| []@bool
|
+| object:string | ['hello', 'world', '']@Series[object]
| [hello, world, ]@string
|
+| object:string-nullable | ['hello', None, 'world']@Series[object]
| [hello, None, world]@string
|
+| string:inferred | ['hello', 'world']@Series[object]
| [hello, world]@string
|
+| object:bytes | [b'hello', b'world']@Series[object]
| [b'hello', b'world']@binary
|
+| object:empty | []@Series[object]
| []@null
|
+| object:all-null | [None, None]@Series[object]
| [None, None]@null
|
+| object:decimal | [Decimal('1.50'),
Decimal('-2.25')]@Series[object] | [1.50,
-2.25]@decimal128(3, 2) |
+| list<int64>:standard | [[1, 2], [3]]@Series[object]
| [[1, 2], [3]]@list<item: int64>
|
+| list<int64>:nullable | [[1, 2], None]@Series[object]
| [[1, 2], None]@list<item: int64>
|
+| list<int64>:null-element | [[1, None], [3]]@Series[object]
| [[1, None], [3]]@list<item: int64>
|
+| list<string>:standard | [['a', 'b'], ['c']]@Series[object]
| [['a', 'b'], ['c']]@list<item:
string> |
+| list<list<int64>>:standard | [[[1, 2], [3]], [[4]]]@Series[object]
| [[[1, 2], [3]], [[4]]]@list<item:
list<item: int64>> |
+| list<struct>:standard | [[{'a': 1}], [{'a': 2}]]@Series[object]
| [[{'a': 1}], [{'a': 2}]]@list<item:
struct<a: int64>> |
+| struct:standard | [{'a': 1, 'b': 'x'}]@Series[object]
| [[('a', 1), ('b', 'x')]]@struct<a:
int64, b: string> |
+| struct:nullable | [{'a': 1, 'b': 'x'}, None]@Series[object]
| [[('a', 1), ('b', 'x')],
None]@struct<a: int64, b: string> |
+| struct<struct>:standard | [{'a': {'b': 1}}]@Series[object]
| [[('a', {'b': 1})]]@struct<a:
struct<b: int64>> |
+| struct<list<int64>>:standard | [{'a': [1, 2]}]@Series[object]
| [[('a', [1, 2])]]@struct<a:
list<item: int64>> |
+| datetime64[ns]:standard | [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] | [2024-06-15
18:30:00]@timestamp[ns] |
+| datetime64[ns]:nullable | [Timestamp('2024-06-15 18:30:00'),
NaT]@Series[datetime64[ns]] | [2024-06-15 18:30:00,
None]@timestamp[ns] |
+| datetime64[ns]:empty | []@Series[datetime64[ns]]
| []@timestamp[ns]
|
+| datetime64[ns,tz]:standard | [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[ns, UTC]] | [2024-06-15
18:30:00+00:00]@timestamp[ns, tz=UTC] |
+| datetime64[ns,tz]:nullable | [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC'), NaT]@Series[datetime64[ns, UTC]] | [2024-06-15 18:30:00+00:00,
None]@timestamp[ns, tz=UTC] |
+| timedelta64[ns]:standard | [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:00:00')]@Series[timedelta64[ns]] | [1 days 00:00:00, 0
days 02:00:00]@duration[ns] |
+| timedelta64[ns]:nullable | [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[ns]] | [1 days 00:00:00,
None]@duration[ns] |
+| datetime64[us]:standard | [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[us]] | [2024-06-15
18:30:00]@timestamp[us] |
+| datetime64[us]:nullable | [Timestamp('2024-06-15 18:30:00'),
NaT]@Series[datetime64[us]] | [2024-06-15 18:30:00,
None]@timestamp[us] |
+| datetime64[us]:empty | []@Series[datetime64[us]]
| []@timestamp[us]
|
+| datetime64[us,tz]:standard | [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC')]@Series[datetime64[us, UTC]] | [2024-06-15
18:30:00+00:00]@timestamp[us, tz=UTC] |
+| datetime64[us,tz]:nullable | [Timestamp('2024-06-15 18:30:00+0000',
tz='UTC'), NaT]@Series[datetime64[us, UTC]] | [2024-06-15 18:30:00+00:00,
None]@timestamp[us, tz=UTC] |
+| timedelta64[us]:standard | [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:00:00')]@Series[timedelta64[us]] | [1 day, 0:00:00,
2:00:00]@duration[us] |
+| timedelta64[us]:nullable | [Timedelta('1 days 00:00:00'),
NaT]@Series[timedelta64[us]] | [1 day, 0:00:00,
None]@duration[us] |
+| datetime64:inferred | [Timestamp('2024-06-15
18:30:00')]@Series[datetime64[ns]] | [2024-06-15
18:30:00]@timestamp[ns] |
+| timedelta64:inferred | [Timedelta('1 days 00:00:00'),
Timedelta('0 days 02:00:00')]@Series[timedelta64[ns]] | [1 days 00:00:00, 0
days 02:00:00]@duration[ns] |
+| datetime64[us]:out-of-ns-range | [Timestamp('1500-01-01
00:00:00')]@Series[datetime64[us]] | [1500-01-01
00:00:00]@timestamp[us] |
+| date:standard | [datetime.date(2024, 6,
15)]@Series[object] |
[2024-06-15]@date32[day] |
+| time:standard | [datetime.time(18, 30, 45)]@Series[object]
| [18:30:45]@time64[us]
|
+| object:datetime | [datetime.datetime(2024, 6, 15, 18,
30)]@Series[object] | [2024-06-15
18:30:00]@timestamp[us] |
+| object:datetime-sub-us | [Timestamp('2024-01-01
00:00:00.000000123')]@Series[object] | [2024-01-01
00:00:00]@timestamp[us] |
+| object:timedelta | [datetime.timedelta(days=1,
seconds=7200)]@Series[object] | [1 day,
2:00:00]@duration[us] |
+| category:standard | ['a', 'b', 'a']@Series[category]
| [a, b, a]@dictionary<values=string,
indices=int8, ordered=0> |
+| category:nullable | ['a', nan, 'b']@Series[category]
| [a, None,
b]@dictionary<values=string, indices=int8, ordered=0> |
+| Int8:standard | [0, 1, 127, -128]@Series[Int8]
| [0, 1, 127, -128]@int8
|
+| Int8:nullable | [0, 1, <NA>]@Series[Int8]
| [0, 1, None]@int8
|
+| Int16:standard | [0, 1, 32767, -32768]@Series[Int16]
| [0, 1, 32767, -32768]@int16
|
+| Int16:nullable | [0, 1, <NA>]@Series[Int16]
| [0, 1, None]@int16
|
+| Int32:standard | [0, 1, 2147483647,
-2147483648]@Series[Int32] | [0, 1,
2147483647, -2147483648]@int32 |
+| Int32:nullable | [0, 1, <NA>]@Series[Int32]
| [0, 1, None]@int32
|
+| Int64:standard | [0, 1, 9223372036854775807,
-9223372036854775808]@Series[Int64] | [0, 1,
9223372036854775807, -9223372036854775808]@int64 |
+| Int64:nullable | [0, 1, <NA>]@Series[Int64]
| [0, 1, None]@int64
|
+| UInt64:standard | [0, 1,
18446744073709551615]@Series[UInt64] |
[0, 1, 18446744073709551615]@uint64 |
+| Int64:empty | []@Series[Int64]
| []@int64
|
+| Int64:all-null | [<NA>, <NA>]@Series[Int64]
| [None, None]@int64
|
+| Float64:standard | [0.0, 1.5]@Series[Float64]
| [0.0, 1.5]@float64
|
+| Float64:nullable | [0.0, <NA>]@Series[Float64]
| [0.0, None]@float64
|
+| boolean:standard | [True, False]@Series[boolean]
| [True, False]@bool
|
+| boolean:nullable | [True, <NA>]@Series[boolean]
| [True, None]@bool
|
+| string[python]:standard | ['hello', 'world']@Series[string]
| [hello, world]@string
|
+| string[python]:nullable | ['hello', <NA>]@Series[string]
| [hello, None]@string
|
+| string[python]:empty | []@Series[string]
| []@string
|
+| int64[pyarrow]:standard | [0, 1, -1]@Series[int64[pyarrow]]
| [0, 1, -1]@int64
|
+| int64[pyarrow]:nullable | [0, 1, <NA>]@Series[int64[pyarrow]]
| [0, 1, None]@int64
|
+| int64[pyarrow]:empty | []@Series[int64[pyarrow]]
| []@int64
|
+| double[pyarrow]:nullable | [0.0, <NA>]@Series[double[pyarrow]]
| [0.0, None]@float64
|
+| bool[pyarrow]:nullable | [True, <NA>]@Series[bool[pyarrow]]
| [True, None]@bool
|
+| string[pyarrow]:standard | ['hello', 'world']@Series[string]
| [hello, world]@large_string
|
+| string[pyarrow]:nullable | ['hello', <NA>]@Series[string]
| [hello, None]@large_string
|
+| string[pyarrow]:empty | []@Series[string]
| []@large_string
|
+| timestamp[us][pyarrow]:standard | [Timestamp('2024-01-01
12:00:00')]@Series[timestamp[us][pyarrow]] | [2024-01-01
12:00:00]@timestamp[us] |
+| int64[pyarrow]:single-chunk | [1, 2]@Series[int64[pyarrow]]
| [1, 2]@int64
|
+| int64[pyarrow]:multi-chunk | [1, 2, 3]@Series[int64[pyarrow]]
| [1, 2, 3]@chunked<int64>
|
+| int64:overflow | [300, 1]@Series[int64]
| [300, 1]@int64
|
+| float64:fractional | [1.5, 2.5]@Series[float64]
| [1.5, 2.5]@float64
|
+| float64:infinity | [inf, 1.0]@Series[float64]
| [inf, 1.0]@float64
|
+| float64:precision | [1.1234567890123]@Series[float64]
| [1.1234567890123]@float64
|
+| datetime64[ns]:sub-us | [Timestamp('2024-01-01
00:00:00.000000123')]@Series[datetime64[ns]] | [2024-01-01
00:00:00.000000123]@timestamp[ns] |
+| object:date-then-datetime | [datetime.date(2024, 1, 1),
datetime.datetime(2024, 1, 1, 5, 30)]@Series[object] | [2024-01-01,
2024-01-01]@date32[day] |
+| object:datetime-then-date | [datetime.datetime(2024, 1, 1, 5, 30),
datetime.date(2024, 1, 1)]@Series[object] | ERR@ArrowTypeError
|
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
new file mode 100644
index 000000000000..be0e518f416a
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
@@ -0,0 +1,413 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Tests for PyArrow Array.from_pandas() with default arguments using golden file
comparison.
+
+This test monitors the pandas -> Arrow direction, which PySpark relies on for
+``createDataFrame(pandas_df)`` and for every pandas UDF's return value.
PySpark calls
+``pa.Array.from_pandas(series, mask=mask, type=arrow_type, safe=safecheck)`` in
+``pyspark/sql/conversion.py`` and ``pyspark/sql/pandas/conversion.py``; the
non-default
+arguments are covered by ``test_pyarrow_array_from_pandas_non_default.py``,
which reuses
+the source Series built here.
+
+Rows are grouped by how pandas stores the Series, because that decides which
branch
+PySpark takes when computing ``mask``:
+
+- numpy-backed dtypes, where ``mask=series.isnull()`` is passed;
+- dtypes implementing the ``__arrow_array__`` protocol, where PySpark passes
``mask=None``
+ because supplying a mask raises. The protocol means "can export Arrow", not
"is stored
+ as Arrow": ``Int64`` is numpy values plus a byte mask and ``string[python]``
is an object
+ ndarray, yet both implement it alongside the genuinely Arrow-backed
``[pyarrow]`` dtypes;
+- values chosen to be lossy or ambiguous when coerced, which the non-default
type tests
+ need and whose unconverted baseline is recorded here.
+
+## Golden File Cell Format
+
+Each cell uses the value@type format:
+- pandas Series: "python_list_repr@Series[dtype]"
+- PyArrow Array: "python_list_repr@arrow_type"
+- PyArrow ChunkedArray: "python_list_repr@chunked<arrow_type>"
+- Error: "ERR@ExceptionClassName"
+
+``from_pandas`` returns a ChunkedArray rather than an Array when the input
Series is backed
+by a chunked Arrow array, and the two are otherwise indistinguishable because
both report
+the element type. The distinction is a real contract:
``create_arrow_table_from_pandas``
+builds a ``pa.Table`` (which accepts either) specifically because of it -- see
SPARK-46776.
+``chunked<...>`` follows Arrow's angle-bracket spelling for parameterized
types, keeping it
+distinct from the square-bracket ``Series[...]`` / ``ndarray[...]`` used for
pandas and
+numpy containers. The chunk count is deliberately not recorded: it varies
with batching,
+whereas chunked-versus-not is the behavior under test.
+
+## Regenerating Golden Files
+
+Set SPARK_GENERATE_GOLDEN_FILES=1 before running:
+
+ SPARK_GENERATE_GOLDEN_FILES=1 python -m pytest \\
+
python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
+
+## PyArrow and pandas Version Compatibility
+
+The golden files capture behavior for specific PyArrow and pandas versions.
+Regenerate when upgrading either dependency, as from_pandas() behavior may
change.
+"""
+
+import datetime
+import decimal
+import unittest
+
+from pyspark.loose_version import LooseVersion
+from pyspark.testing.utils import (
+ have_pyarrow,
+ have_pandas,
+ have_numpy,
+ pyarrow_requirement_message,
+ pandas_requirement_message,
+ numpy_requirement_message,
+)
+from pyspark.testing.goldenutils import GoldenFileTestMixin
+
+if have_pandas:
+ import pandas as pd
+if have_pyarrow:
+ import pyarrow as pa
+if have_numpy:
+ import numpy as np
+
+
[email protected](
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
+)
+class PyArrowArrayFromPandasDefaultTests(GoldenFileTestMixin,
unittest.TestCase):
+ """
+ Tests pa.Array.from_pandas() with default arguments via golden file
comparison.
+
+ Three group methods build the source Series so the non-default tests can
reuse the whole
+ inventory or one group. Groups are disjoint and their order fixes the row
order.
+ """
+
+ @staticmethod
+ def repr_from_pandas_result(value):
+ """
+ Format a ``from_pandas`` return value, marking a ChunkedArray as
``chunked<type>``.
+
+ ``repr_value`` reports the element type, which is the same for an
Array and a
+ ChunkedArray, so without this the two are indistinguishable in a cell.
+ """
+ rendered = GoldenFileTestMixin.repr_value(value, max_len=0)
+ if isinstance(value, pa.ChunkedArray):
+ values, _, arrow_type = rendered.rpartition("@")
+ return f"{values}@chunked<{arrow_type}>"
+ return rendered
+
+ def _numpy_backed_sources(self):
+ """
+ Series whose storage is numpy, so ``hasattr(series.array,
"__arrow_array__")`` is
+ False and PySpark passes ``mask=series.isnull()``.
+ """
+ sources = {}
+
+ # =====================================================================
+ # Integer types
+ # =====================================================================
+ for dtype in ["int8", "int16", "int32", "int64"]:
+ info = np.iinfo(dtype)
+ sources[f"{dtype}:standard"] = pd.Series([0, 1, -1, info.max,
info.min], dtype=dtype)
+ sources[f"{dtype}:empty"] = pd.Series([], dtype=dtype)
+ # A numpy integer Series cannot hold a null, so pandas widens it to
float64.
+ sources["int64:nullable"] = pd.Series([0, 1, None])
+
+ for dtype in ["uint8", "uint16", "uint32", "uint64"]:
+ sources[f"{dtype}:standard"] = pd.Series([0, 1,
np.iinfo(dtype).max], dtype=dtype)
+
+ # =====================================================================
+ # Float and boolean types
+ # =====================================================================
+ for dtype in ["float32", "float64"]:
+ sources[f"{dtype}:standard"] = pd.Series([0.0, 1.5, -1.5],
dtype=dtype)
+ sources[f"{dtype}:nullable"] = pd.Series([0.0, np.nan, 1.5],
dtype=dtype)
+ sources[f"{dtype}:empty"] = pd.Series([], dtype=dtype)
+
+ sources["bool:standard"] = pd.Series([True, False, True])
+ sources["bool:empty"] = pd.Series([], dtype="bool")
+
+ # =====================================================================
+ # Object types (string, binary, decimal)
+ # =====================================================================
+ # Only the string rows need an explicit dtype; pandas 3 would infer
its ``str``
+ # dtype and they would stop being object rows.
+ sources["object:string"] = pd.Series(["hello", "world", ""],
dtype=object)
+ sources["object:string-nullable"] = pd.Series(["hello", None,
"world"], dtype=object)
+ # Unpinned, so this records the dtype pandas infers from Python
strings: object on
+ # pandas 2, its dedicated str dtype on pandas 3, which Arrow reads as
large_string.
+ sources["string:inferred"] = pd.Series(["hello", "world"])
+ sources["object:bytes"] = pd.Series([b"hello", b"world"])
+ sources["object:empty"] = pd.Series([], dtype=object)
+ sources["object:all-null"] = pd.Series([None, None], dtype=object)
+ sources["object:decimal"] = pd.Series([decimal.Decimal("1.50"),
decimal.Decimal("-2.25")])
+
+ # =====================================================================
+ # Nested types
+ # =====================================================================
+ # Inference builds these from Python containers. It cannot reach
``map`` (that
+ # needs an explicit type), nor ``large_list`` / ``fixed_size_list``,
which
+ # ``to_arrow_type`` never requests.
+ sources["list<int64>:standard"] = pd.Series([[1, 2], [3]])
+ sources["list<int64>:nullable"] = pd.Series([[1, 2], None])
+ sources["list<int64>:null-element"] = pd.Series([[1, None], [3]])
+ sources["list<string>:standard"] = pd.Series([["a", "b"], ["c"]])
+ sources["list<list<int64>>:standard"] = pd.Series([[[1, 2], [3]],
[[4]]])
+ sources["list<struct>:standard"] = pd.Series([[{"a": 1}], [{"a": 2}]])
+ sources["struct:standard"] = pd.Series([{"a": 1, "b": "x"}])
+ sources["struct:nullable"] = pd.Series([{"a": 1, "b": "x"}, None])
+ sources["struct<struct>:standard"] = pd.Series([{"a": {"b": 1}}])
+ sources["struct<list<int64>>:standard"] = pd.Series([{"a": [1, 2]}])
+
+ # =====================================================================
+ # Temporal types
+ # =====================================================================
+ # Each row pins its resolution so the row name stays accurate on both
pandas
+ # majors: pandas 2 infers "ns" from a datetime, pandas 3 infers "us".
+ dt = datetime.datetime(2024, 6, 15, 18, 30, 0)
+ for unit in ["ns", "us"]:
+ sources[f"datetime64[{unit}]:standard"] = pd.Series([dt],
dtype=f"datetime64[{unit}]")
+ sources[f"datetime64[{unit}]:nullable"] = pd.Series(
+ [dt, None], dtype=f"datetime64[{unit}]"
+ )
+ sources[f"datetime64[{unit}]:empty"] = pd.Series([],
dtype=f"datetime64[{unit}]")
+ sources[f"datetime64[{unit},tz]:standard"] = pd.Series(
+ [dt], dtype=f"datetime64[{unit}, UTC]"
+ )
+ sources[f"datetime64[{unit},tz]:nullable"] = pd.Series(
+ [dt, None], dtype=f"datetime64[{unit}, UTC]"
+ )
+ sources[f"timedelta64[{unit}]:standard"] = pd.Series(
+ pd.to_timedelta(["1 days", "2 hours"]),
dtype=f"timedelta64[{unit}]"
+ )
+ sources[f"timedelta64[{unit}]:nullable"] = pd.Series(
+ pd.to_timedelta(["1 days", None]), dtype=f"timedelta64[{unit}]"
+ )
+
+ # Unpinned, so the cell always duplicates one of the pinned rows
above. Which one
+ # it matches is the behavior recorded: "ns" on pandas 2, "us" on
pandas 3.
+ sources["datetime64:inferred"] = pd.Series([dt])
+ sources["timedelta64:inferred"] = pd.Series(pd.to_timedelta(["1 days",
"2 hours"]))
+
+ # datetime64[ns] spans only 1677-2262, so a far-past value needs
microseconds.
+ sources["datetime64[us]:out-of-ns-range"] = pd.Series(
+ [datetime.datetime(1500, 1, 1)], dtype="datetime64[us]"
+ )
+
+ # pandas has no native date or time dtype, so these stay object on
their own.
+ sources["date:standard"] = pd.Series([datetime.date(2024, 6, 15)])
+ sources["time:standard"] = pd.Series([datetime.time(18, 30, 45)])
+
+ # In object dtype pyarrow takes the unit from datetime's own
microsecond
+ # resolution, not from a numpy dtype, so these give timestamp[us]
where the
+ # datetime64[ns] rows above give timestamp[ns] -- and sub-microsecond
digits are
+ # dropped silently, while an explicit type=timestamp("us") raises
ArrowInvalid.
+ sources["object:datetime"] = pd.Series([dt], dtype=object)
+ sources["object:datetime-sub-us"] = pd.Series(
+ [pd.Timestamp("2024-01-01 00:00:00.000000123")], dtype=object
+ )
+ sources["object:timedelta"] = pd.Series([datetime.timedelta(days=1,
hours=2)], dtype=object)
+
+ # =====================================================================
+ # Categorical type
+ # =====================================================================
+ # PySpark casts a categorical to its categories' dtype first, but the
raw
+ # categorical is what a user hands to createDataFrame.
+ sources["category:standard"] = pd.Series(pd.Categorical(["a", "b",
"a"]))
+ sources["category:nullable"] = pd.Series(pd.Categorical(["a", None,
"b"]))
+
+ return sources
+
+ def _protocol_sources(self):
+ """
+ Series implementing ``__arrow_array__``, so PySpark passes
``mask=None`` --
+ supplying one raises, because the protocol returns an array whose
validity bitmap
+ is already built.
+
+ The protocol means "can export Arrow", not "is stored as Arrow", and
the rows
+ below deliberately cover both: ``Int64`` is numpy values plus a byte
mask and
+ ``string[python]`` is an object ndarray, while the ``[pyarrow]``
dtypes hold Arrow
+ buffers. PySpark does not distinguish them -- ``conversion.py:435``
branches only
+ on the protocol -- so they share one group.
+ """
+ sources = {}
+
+ # =====================================================================
+ # Nullable extension dtypes (numpy values plus a byte mask)
+ # =====================================================================
+ for dtype in ["Int8", "Int16", "Int32", "Int64"]:
+ info = np.iinfo(dtype.lower())
+ sources[f"{dtype}:standard"] = pd.Series([0, 1, info.max,
info.min], dtype=dtype)
+ sources[f"{dtype}:nullable"] = pd.Series([0, 1, None], dtype=dtype)
+ sources["UInt64:standard"] = pd.Series([0, 1, np.iinfo("uint64").max],
dtype="UInt64")
+ sources["Int64:empty"] = pd.Series([], dtype="Int64")
+ sources["Int64:all-null"] = pd.Series([None, None], dtype="Int64")
+
+ sources["Float64:standard"] = pd.Series([0.0, 1.5], dtype="Float64")
+ sources["Float64:nullable"] = pd.Series([0.0, None], dtype="Float64")
+
+ sources["boolean:standard"] = pd.Series([True, False], dtype="boolean")
+ sources["boolean:nullable"] = pd.Series([True, None], dtype="boolean")
+
+ # StringArray: an object ndarray of Python str, yet it exports Arrow.
+ sources["string[python]:standard"] = pd.Series(["hello", "world"],
dtype="string[python]")
+ sources["string[python]:nullable"] = pd.Series(["hello", None],
dtype="string[python]")
+ sources["string[python]:empty"] = pd.Series([], dtype="string[python]")
+
+ # =====================================================================
+ # PyArrow-backed dtypes (Arrow buffers, so the handoff is zero-copy)
+ # =====================================================================
+ # Since pandas 2.2 a pyarrow-backed string stores large_string, and
the protocol
+ # hands that back rather than the 32-bit string the name suggests.
PySpark works
+ # around pyarrow < 19 ignoring a narrower request (SPARK-46776).
+ sources["int64[pyarrow]:standard"] = pd.Series([0, 1, -1],
dtype="int64[pyarrow]")
+ sources["int64[pyarrow]:nullable"] = pd.Series([0, 1, None],
dtype="int64[pyarrow]")
+ sources["int64[pyarrow]:empty"] = pd.Series([], dtype="int64[pyarrow]")
+ sources["double[pyarrow]:nullable"] = pd.Series([0.0, None],
dtype="double[pyarrow]")
+ sources["bool[pyarrow]:nullable"] = pd.Series([True, None],
dtype="bool[pyarrow]")
+ sources["string[pyarrow]:standard"] = pd.Series(["hello", "world"],
dtype="string[pyarrow]")
+ sources["string[pyarrow]:nullable"] = pd.Series(["hello", None],
dtype="string[pyarrow]")
+ sources["string[pyarrow]:empty"] = pd.Series([],
dtype="string[pyarrow]")
+ sources["timestamp[us][pyarrow]:standard"] = pd.Series(
+ [datetime.datetime(2024, 1, 1, 12, 0, 0)],
dtype="timestamp[us][pyarrow]"
+ )
+
+ # A chunked backing array makes from_pandas return a ChunkedArray,
which
+ # pa.RecordBatch.from_arrays rejects -- hence PySpark builds a
pa.Table.
+ sources["int64[pyarrow]:single-chunk"] = pd.Series(
+ pd.arrays.ArrowExtensionArray(pa.chunked_array([pa.array([1, 2],
pa.int64())]))
+ )
+ sources["int64[pyarrow]:multi-chunk"] = pd.Series(
+ pd.arrays.ArrowExtensionArray(
+ pa.chunked_array([pa.array([1, 2], pa.int64()), pa.array([3],
pa.int64())])
+ )
+ )
+
+ return sources
+
+ def _coercion_sources(self):
+ """
+ Series whose values are lossy or ambiguous once a target type is
requested.
+
+ These exist for the non-default type tests, where ``safe=`` decides
whether the
+ conversion raises or silently changes the value. Recording their
unconverted
+ results here gives those cells a baseline to be read against.
+ """
+ sources = {}
+
+ # =====================================================================
+ # Values that narrow lossily
+ # =====================================================================
+ sources["int64:overflow"] = pd.Series([300, 1])
+ sources["float64:fractional"] = pd.Series([1.5, 2.5])
+ sources["float64:infinity"] = pd.Series([np.inf, 1.0])
+ sources["float64:precision"] = pd.Series([1.1234567890123])
+
+ # The only temporal case where safe= flips.
+ sources["datetime64[ns]:sub-us"] = pd.Series(
+ pd.to_datetime(["2024-01-01 00:00:00.000000123"])
+ )
+
+ # =====================================================================
+ # Values whose inferred type depends on element order
+ # =====================================================================
+ # Inference commits to the first element's type, so these same values
either lose
+ # the time component or raise, depending on their order.
+ sources["object:date-then-datetime"] = pd.Series(
+ [datetime.date(2024, 1, 1), datetime.datetime(2024, 1, 1, 5, 30)],
dtype=object
+ )
+ sources["object:datetime-then-date"] = pd.Series(
+ [datetime.datetime(2024, 1, 1, 5, 30), datetime.date(2024, 1, 1)],
dtype=object
+ )
+
+ return sources
+
+ def _build_source_arrays(self):
+ """Build an ordered dict of named source pandas Series for testing."""
+ sources = {}
+ for group in [
+ self._numpy_backed_sources(),
+ self._protocol_sources(),
+ self._coercion_sources(),
+ ]:
+ sources.update(group)
+ return sources
+
+ def test_from_pandas_default(self):
+ """Test pa.Array.from_pandas() with default arguments against golden
file."""
+ sources = self._build_source_arrays()
+ row_names = list(sources.keys())
+ col_names = ["pandas series", "arrow array"]
+
+ overrides = {}
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ # Only the deliberately unpinned rows move: pandas 3 infers
microseconds where
+ # pandas 2 inferred nanoseconds, and its dedicated str dtype is
backed by
+ # large_string -- which a categorical's values inherit too.
+ overrides.update(
+ {
+ ("string:inferred", "pandas series"): "['hello',
'world']@Series[str]",
+ ("string:inferred", "arrow array"): "[hello,
world]@large_string",
+ ("datetime64:inferred", "pandas series"): (
+ "[Timestamp('2024-06-15
18:30:00')]@Series[datetime64[us]]"
+ ),
+ ("datetime64:inferred", "arrow array"): "[2024-06-15
18:30:00]@timestamp[us]",
+ ("timedelta64:inferred", "pandas series"): (
+ "[Timedelta('1 days 00:00:00'), "
+ "Timedelta('0 days 02:00:00')]@Series[timedelta64[us]]"
+ ),
+ ("timedelta64:inferred", "arrow array"): (
+ "[1 day, 0:00:00, 2:00:00]@duration[us]"
+ ),
+ ("category:standard", "arrow array"): (
+ "[a, b, a]@dictionary<values=large_string,
indices=int8, ordered=0>"
+ ),
+ ("category:nullable", "arrow array"): (
+ "[a, None, b]@dictionary<values=large_string,
indices=int8, ordered=0>"
+ ),
+ }
+ )
+
+ def compute_cell(row_name, col_name):
+ series = sources[row_name]
+ if col_name == "pandas series":
+ return self.repr_value(series, max_len=0)
+ else:
+ try:
+ return
self.repr_from_pandas_result(pa.Array.from_pandas(series))
+ except Exception as e:
+ return f"ERR@{type(e).__name__}"
+
+ self.compare_or_generate_golden_matrix(
+ row_names=row_names,
+ col_names=col_names,
+ compute_cell=compute_cell,
+ golden_file_prefix="golden_pyarrow_array_from_pandas_default",
+ index_name="test case",
+ overrides=overrides,
+ )
+
+
+if __name__ == "__main__":
+ from pyspark.testing import main
+
+ main(globals()["__file__"])
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
