This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 0ba352a2a4ca [SPARK-58765][PYTHON][TEST] Add tests for
pa.Array.from_pandas with the type and safe arguments
0ba352a2a4ca is described below
commit 0ba352a2a4caa561216e1ddd45752d00d487ce0a
Author: Spenser Sun <[email protected]>
AuthorDate: Wed Aug 19 20:11:52 2026 +0000
[SPARK-58765][PYTHON][TEST] Add tests for pa.Array.from_pandas with the
type and safe arguments
### What changes were proposed in this pull request?
This PR adds golden-file tests for `pa.Array.from_pandas(series, type=...,
safe=...)` — the `type=` and `safe=` arguments PySpark passes when converting a
pandas Series to an Arrow array. A new `PyArrowArrayFromPandasTypeScalarTests`
class is added to the existing `test_pyarrow_array_from_pandas_non_default.py`,
reusing its shared `_PyArrowFromPandasTestBase`. `safe=True` and `safe=False`
are two test methods with two golden files, each a `source \ target` matrix.
The target types covered here are **scalar** Arrow types; nested types
(list / map / struct) will follow in a separate PR. The source rows and targets
are curated to the cells that make `type=`/`safe=` observable (range and
precision flips, silent losses, protocol-dtype behavior) rather than a dense
product, since Spark only requests the Arrow type that matches a column's dtype.
The PR also adds one `large_binary[pyarrow]` source row to the shared base
— the binary counterpart of the existing `string[pyarrow]` row — which is why
the existing default and mask golden files each gain one row.
No production code change; tests and golden files only.
### Why are the changes needed?
PySpark uses `pa.Array.from_pandas(..., type=..., safe=...)` to coerce
pandas data into a Spark schema's Arrow type in `createDataFrame(pandas_df)`
and in pandas UDF return values. These tests run without a Spark session and
act as a drift canary: if a PyArrow or pandas upgrade changes how a value is
coerced, a golden cell changes and the test fails loudly, instead of PySpark
silently producing wrong data.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New golden-file tests that run without a Spark session. Verified across a
PyArrow 18-25 x pandas 2/3 matrix, all passing.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57996 from Spenserrrr/from-pandas-type-tests.
Authored-by: Spenser Sun <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
(cherry picked from commit 1459a7ca5024f75b79f7b4da2b676d25c39fda34)
Signed-off-by: Yicong-Huang <[email protected]>
---
.../golden_pyarrow_array_from_pandas_default.csv | 1 +
.../golden_pyarrow_array_from_pandas_default.md | 1 +
.../golden_pyarrow_array_from_pandas_mask.csv | 1 +
.../golden_pyarrow_array_from_pandas_mask.md | 1 +
..._pyarrow_array_from_pandas_type_scalar_safe.csv | 20 ++++
...n_pyarrow_array_from_pandas_type_scalar_safe.md | 21 ++++
...yarrow_array_from_pandas_type_scalar_unsafe.csv | 20 ++++
...pyarrow_array_from_pandas_type_scalar_unsafe.md | 21 ++++
.../test_pyarrow_array_from_pandas_default.py | 6 ++
.../test_pyarrow_array_from_pandas_non_default.py | 118 +++++++++++++++++++++
10 files changed, 210 insertions(+)
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.csv
index 725784fe911d..74a8cd3a3da2 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.csv
@@ -87,6 +87,7 @@ bool[pyarrow]:nullable [True,
<NA>]@Series[bool[pyarrow]] [True, None]@bool
string[pyarrow]:standard ['hello', 'world']@Series[string] [hello,
world]@large_string
string[pyarrow]:nullable ['hello', <NA>]@Series[string] [hello,
None]@large_string
string[pyarrow]:empty []@Series[string] []@large_string
+large_binary[pyarrow]:standard [b'hello',
b'world']@Series[large_binary[pyarrow]] [b'hello', b'world']@large_binary
timestamp[us][pyarrow]:standard [Timestamp('2024-01-01
12:00:00')]@Series[timestamp[us][pyarrow]] [2024-01-01
12:00:00]@timestamp[us]
int64[pyarrow]:single-chunk [1, 2]@Series[int64[pyarrow]] [1, 2]@int64
int64[pyarrow]:multi-chunk [1, 2, 3]@Series[int64[pyarrow]] [1, 2,
3]@chunked<int64>
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.md
index 502e494d468d..0ca6395cdb11 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_default.md
@@ -88,6 +88,7 @@
| string[pyarrow]:standard | ['hello', 'world']@Series[string]
| [hello, world]@large_string
|
| string[pyarrow]:nullable | ['hello', <NA>]@Series[string]
| [hello, None]@large_string
|
| string[pyarrow]:empty | []@Series[string]
| []@large_string
|
+| large_binary[pyarrow]:standard | [b'hello',
b'world']@Series[large_binary[pyarrow]] |
[b'hello', b'world']@large_binary |
| timestamp[us][pyarrow]:standard | [Timestamp('2024-01-01
12:00:00')]@Series[timestamp[us][pyarrow]] | [2024-01-01
12:00:00]@timestamp[us] |
| int64[pyarrow]:single-chunk | [1, 2]@Series[int64[pyarrow]]
| [1, 2]@int64
|
| int64[pyarrow]:multi-chunk | [1, 2, 3]@Series[int64[pyarrow]]
| [1, 2, 3]@chunked<int64>
|
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.csv
index 7af02dca017f..9546f28e5c72 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.csv
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.csv
@@ -87,6 +87,7 @@ bool[pyarrow]:nullable [True,
<NA>]@Series[bool[pyarrow]] [True, None]@bool ERR@
string[pyarrow]:standard ['hello', 'world']@Series[string] [hello,
world]@large_string ERR@ValueError
string[pyarrow]:nullable ['hello', <NA>]@Series[string] [hello,
None]@large_string ERR@ValueError
string[pyarrow]:empty []@Series[string] []@large_string ERR@ValueError
+large_binary[pyarrow]:standard [b'hello',
b'world']@Series[large_binary[pyarrow]] [b'hello', b'world']@large_binary
ERR@ValueError
timestamp[us][pyarrow]:standard [Timestamp('2024-01-01
12:00:00')]@Series[timestamp[us][pyarrow]] [2024-01-01
12:00:00]@timestamp[us] ERR@ValueError
int64[pyarrow]:single-chunk [1, 2]@Series[int64[pyarrow]] [1, 2]@int64
ERR@ValueError
int64[pyarrow]:multi-chunk [1, 2, 3]@Series[int64[pyarrow]] [1, 2,
3]@chunked<int64> ERR@ValueError
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.md
index 8b3c5ff0f85d..1b01725a1854 100644
---
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.md
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_mask.md
@@ -88,6 +88,7 @@
| string[pyarrow]:standard | ['hello', 'world']@Series[string]
| [hello, world]@large_string
| ERR@ValueError
|
| string[pyarrow]:nullable | ['hello', <NA>]@Series[string]
| [hello, None]@large_string
| ERR@ValueError
|
| string[pyarrow]:empty | []@Series[string]
| []@large_string
| ERR@ValueError
|
+| large_binary[pyarrow]:standard | [b'hello',
b'world']@Series[large_binary[pyarrow]] |
[b'hello', b'world']@large_binary |
ERR@ValueError |
| timestamp[us][pyarrow]:standard | [Timestamp('2024-01-01
12:00:00')]@Series[timestamp[us][pyarrow]] | [2024-01-01
12:00:00]@timestamp[us] | ERR@ValueError
|
| int64[pyarrow]:single-chunk | [1, 2]@Series[int64[pyarrow]]
| [1, 2]@int64
| ERR@ValueError
|
| int64[pyarrow]:multi-chunk | [1, 2, 3]@Series[int64[pyarrow]]
| [1, 2, 3]@chunked<int64>
| ERR@ValueError
|
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
new file mode 100644
index 000000000000..655dce556473
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
@@ -0,0 +1,20 @@
+source \ target int8 int64 float32 timestamp[us] date32[day]
duration[us] time64[ns] string binary
+int64:standard ERR@ArrowInvalid [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid ERR@OverflowError
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day,
23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError
[b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\xff',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+float64:standard ERR@ArrowInvalid ERR@ArrowInvalid [0.0,
1.5, -1.5]@float32 ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'', b'', b'']@binary
+bool:standard [1, 0, 1]@int8 [1, 0, 1]@int64 [1.0, 0.0, 1.0]@float32
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'\x01', b'', b'\x01']@binary
+object:string ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError [hello, world, ]@string
[b'hello', b'world', b'']@binary
+object:bytes ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError [hello, world]@string
[b'hello', b'world']@binary
+date:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError [2024-06-15]@date32[day]
ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError
+object:datetime ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid [2024-06-15 18:30:00]@timestamp[us]
[2024-06-15]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
+object:timedelta ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError [1 day,
2:00:00]@duration[us] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError
+time:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError [18:30:45]@time64[ns] ERR@ArrowTypeError
ERR@ArrowTypeError
+Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid ERR@OverflowError
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+string[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello,
world]@string [b'hello', b'world']@binary
+large_binary[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError [hello, world]@string [b'hello',
b'world']@binary
+int64:overflow ERR@ArrowInvalid [300, 1]@int64 [300.0, 1.0]@float32
[1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00.000300, 0:00:00.000001]@duration[us]
[00:00:00, 00:00:00]@time64[ns] ERR@ArrowTypeError [b',\x01',
b'\x01']@binary
+float64:fractional ERR@ArrowInvalid ERR@ArrowInvalid [1.5,
2.5]@float32 ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'', b'']@binary
+float64:infinity ERR@ArrowInvalid ERR@ArrowInvalid [inf,
1.0]@float32 ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'', b'']@binary
+float64:precision ERR@ArrowInvalid ERR@ArrowInvalid
[1.1234568357467651]@float32 ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowTypeError
[b'\x98nt\xd3\xad\xf9\xf1?']@binary
+datetime64[ns]:sub-us ERR@ArrowNotImplementedError
[1704067200000000123]@int64 ERR@ArrowNotImplementedError
ERR@ArrowInvalid [2024-01-01]@date32[day]
ERR@ArrowNotImplementedError [00:00:00]@time64[ns] ERR@ArrowTypeError
[b'{']@binary
+object:date-then-datetime ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError [2024-01-01,
2024-01-01]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
+object:datetime-then-date ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError [2024-01-01,
2024-01-01]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
new file mode 100644
index 000000000000..89f7ce26e4da
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.md
@@ -0,0 +1,21 @@
+| source \ target | int8 | int64
| float32 |
timestamp[us] |
date32[day] | duration[us]
| time64[ns] | string | binary
[...]
+|--------------------------------|------------------------------|-------------------------------------------------------------|------------------------------|------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|---------------------------------|-------------------------|------------------------------
[...]
+| int64:standard | ERR@ArrowInvalid | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| ERR@OverflowError |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, -1 day,
23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] | ERR@ValueError |
ERR@ArrowTypeError | [b'', b'\x01', b'\xff\xff\xff [...]
+| float64:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | [0.0, 1.5,
-1.5]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTypeError | [b'', b'',
b'']@binary [...]
+| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | ERR@ArrowTypeError |
[b'\x01', b'', b'\x01']@binar [...]
+| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError | [hello, world, ]@string |
[b'hello', b'world', b'']@bin [...]
+| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError
| ERR@ArrowTypeError | [hello, world]@string |
[b'hello', b'world']@binary [...]
+| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
+| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15 18:30:00]@timestamp[us]
| [2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
+| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | [1 day, 2:00:00]@duration[us]
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
+| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError
| [18:30:45]@time64[ns] | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
+| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| ERR@OverflowError |
ERR@ArrowNotImplementedError | [0:00:00, 0:00:00.000001, 106751991
days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
| ERR@ValueError | ERR@ArrowTypeError |
[b'', b'\x01', b'\xff\xff\xff [...]
+| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
| ERR@ArrowInvalid | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | [hello, world]@string |
[b'hello', b'world']@binary [...]
+| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | [hello, world]@string |
[b'hello', b'world']@binary [...]
+| int64:overflow | ERR@ArrowInvalid | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us] |
ERR@ArrowNotImplementedError | [0:00:00.000300,
0:00:00.000001]@duration[us]
| [00:00:00, 00:00:00]@time64[ns] |
ERR@ArrowTypeError | [b',\x01', b'\x01']@binary [...]
+| float64:fractional | ERR@ArrowInvalid |
ERR@ArrowInvalid | [1.5,
2.5]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTypeError | [b'', b'']@binary
[...]
+| float64:infinity | ERR@ArrowInvalid |
ERR@ArrowInvalid | [inf,
1.0]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTypeError | [b'', b'']@binary
[...]
+| float64:precision | ERR@ArrowInvalid |
ERR@ArrowInvalid |
[1.1234568357467651]@float32 | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTypeError |
[b'\x98nt\xd3\xad\xf9\xf1?']@ [...]
+| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError | ERR@ArrowInvalid
| [2024-01-01]@date32[day] |
ERR@ArrowNotImplementedError
| [00:00:00]@time64[ns]
| ERR@ArrowTypeError | [b'{']@binary [...]
+| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
+| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01, 2024-01-01]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTypeError |
ERR@ArrowTypeError [...]
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
new file mode 100644
index 000000000000..e1690c2003b5
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.csv
@@ -0,0 +1,20 @@
+source \ target int8 int64 float32 timestamp[us] date32[day]
duration[us] time64[ns] string binary
+int64:standard [0, 1, -1, -1, 0]@int8 [0, 1, -1, 9223372036854775807,
-9223372036854775808]@int64 [0.0, 1.0, -1.0, 9.223372036854776e+18,
-9.223372036854776e+18]@float32 ERR@OverflowError
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day,
23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError
[b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\xff',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+float64:standard [0, 1, -1]@int8 [0, 1, -1]@int64 [0.0, 1.5,
-1.5]@float32 ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'', b'', b'']@binary
+bool:standard [1, 0, 1]@int8 [1, 0, 1]@int64 [1.0, 0.0, 1.0]@float32
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'\x01', b'', b'\x01']@binary
+object:string ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError [hello, world, ]@string
[b'hello', b'world', b'']@binary
+object:bytes ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError [hello, world]@string
[b'hello', b'world']@binary
+date:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError [2024-06-15]@date32[day]
ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError
+object:datetime ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid [2024-06-15 18:30:00]@timestamp[us]
[2024-06-15]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
+object:timedelta ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError [1 day,
2:00:00]@duration[us] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError
+time:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError [18:30:45]@time64[ns] ERR@ArrowTypeError
ERR@ArrowTypeError
+Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807,
-9223372036854775808]@int64 ERR@ArrowInvalid ERR@OverflowError
ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us]
ERR@ValueError ERR@ArrowTypeError [b'', b'\x01',
b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
+string[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello,
world]@string [b'hello', b'world']@binary
+large_binary[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError [hello, world]@string [b'hello',
b'world']@binary
+int64:overflow [44, 1]@int8 [300, 1]@int64 [300.0, 1.0]@float32
[1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us]
ERR@ArrowNotImplementedError [0:00:00.000300, 0:00:00.000001]@duration[us]
[00:00:00, 00:00:00]@time64[ns] ERR@ArrowTypeError [b',\x01',
b'\x01']@binary
+float64:fractional [1, 2]@int8 [1, 2]@int64 [1.5, 2.5]@float32
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'', b'']@binary
+float64:infinity [0, 1]@int8 [-9223372036854775808, 1]@int64 [inf,
1.0]@float32 ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowTypeError [b'', b'']@binary
+float64:precision [1]@int8 [1]@int64
[1.1234568357467651]@float32 ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError
ERR@ArrowNotImplementedError ERR@ArrowTypeError
[b'\x98nt\xd3\xad\xf9\xf1?']@binary
+datetime64[ns]:sub-us ERR@ArrowNotImplementedError
[1704067200000000123]@int64 ERR@ArrowNotImplementedError [2024-01-01
00:00:00]@timestamp[us] [2024-01-01]@date32[day]
ERR@ArrowNotImplementedError [00:00:00]@time64[ns] ERR@ArrowTypeError
[b'{']@binary
+object:date-then-datetime ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError [2024-01-01,
2024-01-01]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
+object:datetime-then-date ERR@ArrowInvalid ERR@ArrowInvalid
ERR@ArrowInvalid ERR@ArrowTypeError [2024-01-01,
2024-01-01]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError
ERR@ArrowTypeError ERR@ArrowTypeError
diff --git
a/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
new file mode 100644
index 000000000000..ce6d7dcd2c61
--- /dev/null
+++
b/python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_unsafe.md
@@ -0,0 +1,21 @@
+| source \ target | int8 | int64
| float32
| timestamp[us]
| date32[day] |
duration[us]
| time64[ns]
| string [...]
+|--------------------------------|------------------------------|-------------------------------------------------------------|-------------------------------------------------------------------------|------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|---------------------------------|-------------
[...]
+| int64:standard | [0, 1, -1, -1, 0]@int8 | [0, 1, -1,
9223372036854775807, -9223372036854775808]@int64 | [0.0, 1.0, -1.0,
9.223372036854776e+18, -9.223372036854776e+18]@float32 | ERR@OverflowError
| ERR@ArrowNotImplementedError
| [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days,
4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] | ERR@ValueError
| ERR@ArrowTyp [...]
+| float64:standard | [0, 1, -1]@int8 | [0, 1,
-1]@int64 | [0.0, 1.5, -1.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
+| bool:standard | [1, 0, 1]@int8 | [1, 0,
1]@int64 | [1.0, 0.0, 1.0]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
+| object:string | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| [hello, worl [...]
+| object:bytes | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| [hello, worl [...]
+| date:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-06-15]@date32[day]
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTyp [...]
+| object:datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| [2024-06-15
18:30:00]@timestamp[us] |
[2024-06-15]@date32[day] | ERR@ArrowTypeError
| ERR@ArrowTypeError | ERR@ArrowTyp [...]
+| object:timedelta | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| [1 day, 2:00:00]@duration[us]
| ERR@ArrowTypeError
| ERR@ArrowTyp [...]
+| time:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| ERR@ArrowTypeError
| ERR@ArrowTypeError
|
[18:30:45]@time64[ns] | ERR@ArrowTyp [...]
+| Int64:standard | ERR@ArrowInvalid | [0, 1,
9223372036854775807, -9223372036854775808]@int64 | ERR@ArrowInvalid
| ERR@OverflowError
| ERR@ArrowNotImplementedError
| [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days,
19:59:05.224192]@duration[us] | ERR@ValueError
| ERR@ArrowTyp [...]
+| string[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowInvalid
| ERR@ArrowInvalid
| ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | [hello, worl [...]
+| large_binary[pyarrow]:standard | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
|
ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | [hello, worl [...]
+| int64:overflow | [44, 1]@int8 | [300,
1]@int64 | [300.0, 1.0]@float32
| [1970-01-01 00:00:00.000300,
1970-01-01 00:00:00.000001]@timestamp[us] | ERR@ArrowNotImplementedError
| [0:00:00.000300, 0:00:00.000001]@duration[us]
| [00:00:00,
00:00:00]@time64[ns] | ERR@ArrowTyp [...]
+| float64:fractional | [1, 2]@int8 | [1, 2]@int64
| [1.5, 2.5]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
+| float64:infinity | [0, 1]@int8 |
[-9223372036854775808, 1]@int64 | [inf,
1.0]@float32 |
ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError | ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
+| float64:precision | [1]@int8 | [1]@int64
| [1.1234568357467651]@float32
| ERR@ArrowNotImplementedError
| ERR@ArrowNotImplementedError |
ERR@ArrowNotImplementedError
|
ERR@ArrowNotImplementedError | ERR@ArrowTyp [...]
+| datetime64[ns]:sub-us | ERR@ArrowNotImplementedError |
[1704067200000000123]@int64 |
ERR@ArrowNotImplementedError |
[2024-01-01 00:00:00]@timestamp[us] |
[2024-01-01]@date32[day] | ERR@ArrowNotImplementedError
| [00:00:00]@time64[ns] | ERR@ArrowTyp [...]
+| object:date-then-datetime | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTyp [...]
+| object:datetime-then-date | ERR@ArrowInvalid |
ERR@ArrowInvalid | ERR@ArrowInvalid
| ERR@ArrowTypeError
| [2024-01-01,
2024-01-01]@date32[day] | ERR@ArrowTypeError
|
ERR@ArrowTypeError | ERR@ArrowTyp [...]
\ No newline at end of file
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
index 125ab62789e4..c3c2bb2b4f36 100644
---
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_default.py
@@ -296,6 +296,12 @@ class _PyArrowFromPandasTestBase(GoldenFileTestMixin,
unittest.TestCase):
sources["string[pyarrow]:standard"] = pd.Series(["hello", "world"],
dtype="string[pyarrow]")
sources["string[pyarrow]:nullable"] = pd.Series(["hello", None],
dtype="string[pyarrow]")
sources["string[pyarrow]:empty"] = pd.Series([],
dtype="string[pyarrow]")
+ # Binary is not auto-promoted to large_binary the way string is, so
large_binary
+ # has to be requested explicitly. This is the binary counterpart of
the string
+ # rows, and lets the type tests exercise the binary half of
SPARK-46776.
+ sources["large_binary[pyarrow]:standard"] = pd.Series(
+ [b"hello", b"world"], dtype="large_binary[pyarrow]"
+ )
sources["timestamp[us][pyarrow]:standard"] = pd.Series(
[datetime.datetime(2024, 1, 1, 12, 0, 0)],
dtype="timestamp[us][pyarrow]"
)
diff --git
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_non_default.py
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_non_default.py
index 6439e189ff43..67191696207c 100644
---
a/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_non_default.py
+++
b/python/pyspark/tests/upstream/pyarrow/test_pyarrow_array_from_pandas_non_default.py
@@ -83,6 +83,8 @@ from pyspark.tests.upstream.pyarrow import
test_pyarrow_array_from_pandas_defaul
if have_pandas:
import pandas as pd
+if have_pyarrow:
+ import pyarrow as pa
@unittest.skipIf(
@@ -161,6 +163,122 @@ class PyArrowArrayFromPandasMaskTests(
)
[email protected](
+ not have_pyarrow or not have_pandas or not have_numpy,
+ pyarrow_requirement_message or pandas_requirement_message or
numpy_requirement_message,
+)
+class PyArrowArrayFromPandasTypeScalarTests(
+ test_pyarrow_array_from_pandas_default._PyArrowFromPandasTestBase
+):
+ """
+ Tests pa.Array.from_pandas(series, type=..., safe=...) for SCALAR target
types via golden
+ file comparison. Nested targets (list / map / struct) are covered by a
separate class.
+
+ Spark uses ``type=`` diagonally (source dtype and requested Arrow type
share one
+ schema), so this pins a focused set of rows that make type=/safe=
observable rather
+ than a dense source x target product: the off-diagonal cells (e.g. int ->
large_binary)
+ are pyarrow-construction trivia Spark never hits, and general conversion is
+ pa.Array.cast's job. safe=True/False are two methods and two goldens, not
doubled
+ columns. ``mask`` stays None (fixed by storage backend, covered by the
mask tests),
+ isolating type=/safe=.
+ """
+
+ @staticmethod
+ def _get_target_types():
+ """Scalar to_arrow_type targets that discriminate type=/safe=, plus
duration/time64
+ for the timedelta and time-of-day diagonals."""
+ return [
+ pa.int8(),
+ pa.int64(),
+ pa.float32(),
+ pa.timestamp("us"),
+ pa.date32(),
+ pa.duration("us"),
+ pa.time64("ns"),
+ pa.string(),
+ pa.binary(),
+ ]
+
+ def _type_source_arrays(self):
+ """
+ Clean family representatives, the protocol rows that expose the safe=
drop and
+ SPARK-46776, and the shared coercion rows (reused so a changed cell is
attributable
+ across the default/mask/type goldens).
+ """
+ pool = {**self._numpy_backed_sources(), **self._protocol_sources()}
+ selected = [
+ # Clean family reps.
+ "int64:standard",
+ "float64:standard",
+ "bool:standard",
+ "object:string",
+ "object:bytes",
+ "date:standard",
+ "object:datetime",
+ "object:timedelta",
+ "time:standard",
+ # Protocol rows: Int64 = safe-drop contrast; last two =
SPARK-46776's pyarrow < 19
+ # followup (a narrower type is ignored on the __arrow_array__ path
-> stored type).
+ "Int64:standard",
+ "string[pyarrow]:standard",
+ "large_binary[pyarrow]:standard",
+ ]
+ sources = {name: pool[name] for name in selected}
+ # Coercion group reused in full; the numpy/protocol rows above are
cherry-picked.
+ sources.update(self._coercion_sources())
+ return sources
+
+ def _compare_type_matrix(self, safe, golden_file_prefix, overrides):
+ sources = self._type_source_arrays()
+ target_types = self._get_target_types()
+ target_names = [self.repr_type(t) for t in target_types]
+ target_lookup = dict(zip(target_names, target_types))
+
+ self.compare_or_generate_golden_matrix(
+ row_names=list(sources.keys()),
+ col_names=target_names,
+ compute_cell=lambda src, tgt: self._from_pandas_cell(
+ sources[src], type=target_lookup[tgt], safe=safe
+ ),
+ golden_file_prefix=golden_file_prefix,
+ overrides=overrides,
+ )
+
+ def test_from_pandas_type_scalar_safe(self):
+ """Test pa.Array.from_pandas(type=<scalar>, safe=True) against golden
file."""
+ # pyarrow < 19 ignores the requested type on the protocol path and
returns the stored
+ # type (the SPARK-46776 followup), for every target. Other sources
are version-stable.
+ overrides: dict[tuple[str, str], str] = {}
+ if LooseVersion(pa.__version__) < LooseVersion("19.0.0"):
+ for col in [self.repr_type(t) for t in self._get_target_types()]:
+ overrides[("string[pyarrow]:standard", col)] = "[hello,
world]@large_string"
+ overrides[("large_binary[pyarrow]:standard", col)] = (
+ "[b'hello', b'world']@large_binary"
+ )
+ self._compare_type_matrix(
+ safe=True,
+
golden_file_prefix="golden_pyarrow_array_from_pandas_type_scalar_safe",
+ overrides=overrides,
+ )
+
+ def test_from_pandas_type_scalar_unsafe(self):
+ """Test pa.Array.from_pandas(type=<scalar>, safe=False) against golden
file."""
+ # Same overrides as the safe method: the protocol drops the type
request before any
+ # cast, so safe=False changes none of these cells (SPARK-46776,
pyarrow < 19 followup).
+ overrides: dict[tuple[str, str], str] = {}
+ if LooseVersion(pa.__version__) < LooseVersion("19.0.0"):
+ for col in [self.repr_type(t) for t in self._get_target_types()]:
+ overrides[("string[pyarrow]:standard", col)] = "[hello,
world]@large_string"
+ overrides[("large_binary[pyarrow]:standard", col)] = (
+ "[b'hello', b'world']@large_binary"
+ )
+ self._compare_type_matrix(
+ safe=False,
+
golden_file_prefix="golden_pyarrow_array_from_pandas_type_scalar_unsafe",
+ overrides=overrides,
+ )
+
+
if __name__ == "__main__":
from pyspark.testing import main
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
