This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 73504bc8e003 [SPARK-57389][PYTHON][TEST] Use dedicated Pandas 3 golden
files for pandas UDF coercion tests
73504bc8e003 is described below
commit 73504bc8e0039628b9c8d3d643a7af61f56a7bf5
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Tue Jun 30 09:51:07 2026 +0800
[SPARK-57389][PYTHON][TEST] Use dedicated Pandas 3 golden files for pandas
UDF coercion tests
### What changes were proposed in this pull request?
The coercion golden file tests kept a single golden file generated under
Pandas 2. Under Pandas 3 — where several defaults differ — the pandas UDF
tests patched the loaded golden in memory, and the Python UDF legacy pandas
conversion variant simply failed:
- `datetime64` ndarrays default to `[us]` instead of `[ns]`
- `Categorical` categories use `str` instead of `object`
- the same casts return microseconds instead of nanoseconds
- string lists coerce to `Decimal` where Pandas 2 errored
- the legacy pandas conversion turns `None` into `nan` for str columns
This PR keeps a dedicated golden file per pandas major version for every
coercion path that routes data through pandas, and removes all in-memory
patching. The version-dependent goldens live in `pandas_2/` and `pandas_3/`
subdirectories, selected by a `pandas_dir` property next to the existing
`prefix`. File names are identical across versions, so the two can be
compared directly, e.g. `diff pandas_2/golden_X.csv pandas_3/golden_X.csv`:
```
python/pyspark/sql/tests/coercion/ # .md files omitted
golden_python_udf_input_type_coercion_vanilla.csv # pandas-free
paths:
golden_python_udf_input_type_coercion_with_arrow.csv # a single
golden
golden_python_udf_return_type_coercion_vanilla.csv
golden_python_udf_return_type_coercion_with_arrow.csv
pandas_2/ #
pandas-dependent paths:
golden_pandas_udf_input_type_coercion_base.csv # one golden per
golden_pandas_udf_return_type_coercion_base.csv # pandas major
version
golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv
pandas_3/
(same file names as pandas_2/)
```
The Python UDF `vanilla` and `with_arrow` paths do not involve pandas, so
they keep a single golden in the parent directory.
The input-type tests now pad failure rows to the full column count so rows
recording an exception roundtrip through the golden CSV; the `pandas_3/`
input-type golden records the `None` -> `nan` legacy conversion behavior for
str columns.
A temporary commit switched the default CI `PYSPARK_IMAGE_TO_TEST` to
`python-312-pandas-3` to validate the change under Pandas 3 and has since
been reverted, so this PR no longer touches the CI workflow.
### Why are the changes needed?
In-memory patching couples the golden data to hand-maintained assumptions
about how Pandas 3 differs from Pandas 2 (column renames, ns->us scaling,
cell flips). Generating real golden files under each pandas major version
captures the actual behavior directly, is easier to maintain and review, and
the directory layout itself documents which paths are
pandas-version-dependent.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
- Regenerated the `pandas_3/` golden files with
`SPARK_GENERATE_GOLDEN_FILES=1`
under pandas 3.0.2 (numpy 2.4.3, pyarrow 23.0.1, Python 3.13). The
`pandas_2/` goldens are moves of the existing goldens (100% git renames),
not regenerated.
- All 4 coercion test modules pass locally under both pandas 2.3.3 and
pandas 3.0.2.
- A CI run with the `python-312-pandas-3` image
(https://github.com/zhengruifeng/spark/actions/runs/27275940698, 70452
tests run) validated the pandas UDF goldens under Pandas 3; the remaining
coercion failure in that run (`test_python_udf_input_type`, legacy pandas
conversion variant) is fixed by this PR's Python UDF golden split and now
passes locally under Pandas 3.
- The default (Pandas 2) CI on this PR validates the `pandas_2/` goldens.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (model: claude-opus-4-8)
Closes #56402 from zhengruifeng/coercion-golden-pandas3-dev6.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 1ffd05bdcf3342926691403c70908603756dbfa5)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../golden_pandas_udf_input_type_coercion_base.csv | 0
.../golden_pandas_udf_input_type_coercion_base.md | 0
...golden_pandas_udf_return_type_coercion_base.csv | 0
.../golden_pandas_udf_return_type_coercion_base.md | 0
...f_input_type_coercion_with_arrow_and_pandas.csv | 0
...df_input_type_coercion_with_arrow_and_pandas.md | 0
..._return_type_coercion_with_arrow_and_pandas.csv | 0
...f_return_type_coercion_with_arrow_and_pandas.md | 0
.../golden_pandas_udf_input_type_coercion_base.csv | 4 +-
.../golden_pandas_udf_input_type_coercion_base.md | 4 +-
...golden_pandas_udf_return_type_coercion_base.csv | 6 +--
.../golden_pandas_udf_return_type_coercion_base.md | 17 +++++++
...f_input_type_coercion_with_arrow_and_pandas.csv | 2 +-
...df_input_type_coercion_with_arrow_and_pandas.md | 41 +++++++++++++++
..._return_type_coercion_with_arrow_and_pandas.csv | 0
...f_return_type_coercion_with_arrow_and_pandas.md | 0
.../tests/coercion/test_pandas_udf_input_type.py | 25 ++++++----
.../tests/coercion/test_pandas_udf_return_type.py | 58 ++++++----------------
.../tests/coercion/test_python_udf_input_type.py | 17 ++++++-
.../tests/coercion/test_python_udf_return_type.py | 13 ++++-
20 files changed, 124 insertions(+), 63 deletions(-)
diff --git
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.csv
b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_input_type_coercion_base.csv
similarity index 100%
copy from
python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.csv
copy to
python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_input_type_coercion_base.csv
diff --git
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.md
b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_input_type_coercion_base.md
similarity index 100%
copy from
python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.md
copy to
python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_input_type_coercion_base.md
diff --git
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_return_type_coercion_base.csv
b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv
similarity index 100%
copy from
python/pyspark/sql/tests/coercion/golden_pandas_udf_return_type_coercion_base.csv
copy to
python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.csv
diff --git
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_return_type_coercion_base.md
b/python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md
similarity index 100%
rename from
python/pyspark/sql/tests/coercion/golden_pandas_udf_return_type_coercion_base.md
rename to
python/pyspark/sql/tests/coercion/pandas_2/golden_pandas_udf_return_type_coercion_base.md
diff --git
a/python/pyspark/sql/tests/coercion/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
b/python/pyspark/sql/tests/coercion/pandas_2/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
similarity index 100%
copy from
python/pyspark/sql/tests/coercion/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
copy to
python/pyspark/sql/tests/coercion/pandas_2/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
diff --git
a/python/pyspark/sql/tests/coercion/golden_python_udf_input_type_coercion_with_arrow_and_pandas.md
b/python/pyspark/sql/tests/coercion/pandas_2/golden_python_udf_input_type_coercion_with_arrow_and_pandas.md
similarity index 100%
rename from
python/pyspark/sql/tests/coercion/golden_python_udf_input_type_coercion_with_arrow_and_pandas.md
rename to
python/pyspark/sql/tests/coercion/pandas_2/golden_python_udf_input_type_coercion_with_arrow_and_pandas.md
diff --git
a/python/pyspark/sql/tests/coercion/golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv
b/python/pyspark/sql/tests/coercion/pandas_2/golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv
similarity index 100%
copy from
python/pyspark/sql/tests/coercion/golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv
copy to
python/pyspark/sql/tests/coercion/pandas_2/golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv
diff --git
a/python/pyspark/sql/tests/coercion/golden_python_udf_return_type_coercion_with_arrow_and_pandas.md
b/python/pyspark/sql/tests/coercion/pandas_2/golden_python_udf_return_type_coercion_with_arrow_and_pandas.md
similarity index 100%
copy from
python/pyspark/sql/tests/coercion/golden_python_udf_return_type_coercion_with_arrow_and_pandas.md
copy to
python/pyspark/sql/tests/coercion/pandas_2/golden_python_udf_return_type_coercion_with_arrow_and_pandas.md
diff --git
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.csv
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_input_type_coercion_base.csv
similarity index 96%
rename from
python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.csv
rename to
python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_input_type_coercion_base.csv
index 4c2817525801..90596692a45d 100644
---
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.csv
+++
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_input_type_coercion_base.csv
@@ -13,8 +13,8 @@
11 double_null double [None, 2.71] ['float64', 'float64'] [None,
2.71]
12 decimal_values decimal(3,2) [Decimal('5.35'), Decimal('1.23')]
['object', 'object'] [Decimal('5.35'), Decimal('1.23')]
13 decimal_null decimal(3,2) [None, Decimal('9.99')] ['object',
'object'] [None, Decimal('9.99')]
-14 string_values string ['abc', '', 'hello'] ['object', 'object',
'object'] ['abc', '', 'hello']
-15 string_null string [None, 'test'] ['object', 'object'] [None,
'test']
+14 string_values string ['abc', '', 'hello'] ['str', 'str', 'str']
['abc', '', 'hello']
+15 string_null string [None, 'test'] ['str', 'str'] [None, 'test']
16 binary_values binary [b'abc', b'', b'ABC'] ['object', 'object',
'object'] [b'abc', b'', b'ABC']
17 binary_null binary [None, b'test'] ['object', 'object'] [None,
b'test']
18 boolean_values boolean [True, False] ['bool', 'bool'] [True,
False]
diff --git
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.md
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_input_type_coercion_base.md
similarity index 98%
rename from
python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.md
rename to
python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_input_type_coercion_base.md
index 6a028a978fe4..f2cbb8be58d4 100644
---
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_input_type_coercion_base.md
+++
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_input_type_coercion_base.md
@@ -14,8 +14,8 @@
| 11 | double_null | double |
[None, 2.71] |
['float64', 'float64'] | [None, 2.71]
|
| 12 | decimal_values | decimal(3,2) |
[Decimal('5.35'), Decimal('1.23')] |
['object', 'object'] | [Decimal('5.35'), Decimal('1.23')]
|
| 13 | decimal_null | decimal(3,2) |
[None, Decimal('9.99')] |
['object', 'object'] | [None, Decimal('9.99')]
|
-| 14 | string_values | string |
['abc', '', 'hello'] |
['object', 'object', 'object'] | ['abc', '', 'hello']
|
-| 15 | string_null | string |
[None, 'test'] |
['object', 'object'] | [None, 'test']
|
+| 14 | string_values | string |
['abc', '', 'hello'] |
['str', 'str', 'str'] | ['abc', '', 'hello']
|
+| 15 | string_null | string |
[None, 'test'] |
['str', 'str'] | [None, 'test']
|
| 16 | binary_values | binary |
[b'abc', b'', b'ABC'] |
['object', 'object', 'object'] | [b'abc', b'', b'ABC']
|
| 17 | binary_null | binary |
[None, b'test'] |
['object', 'object'] | [None, b'test']
|
| 18 | boolean_values | boolean |
[True, False] |
['bool', 'bool'] | [True, False]
|
diff --git
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_return_type_coercion_base.csv
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
similarity index 82%
rename from
python/pyspark/sql/tests/coercion/golden_pandas_udf_return_type_coercion_base.csv
rename to
python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
index 4f5d13daaf10..5e12cc57c84a 100644
---
a/python/pyspark/sql/tests/coercion/golden_pandas_udf_return_type_coercion_base.csv
+++
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.csv
@@ -1,9 +1,9 @@
-SQL Type \ Value@Type [None, None]@list [True, False]@list ['a',
'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list
[{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16]
[1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1
2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1.
2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1.
2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j
2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
+SQL Type \ Value@Type [None, None]@list [True, False]@list ['a',
'b']@list ['12', '34']@list [Decimal('1'), Decimal('2')]@list
[{'a': 1}, {'b': 2}]@list [1 2]@ndarray[int8] [1 2]@ndarray[int16]
[1 2]@ndarray[int32] [1 2]@ndarray[int64] [1 2]@ndarray[uint8] [1
2]@ndarray[uint16] [1 2]@ndarray[uint32] [1 2]@ndarray[uint64] [1.
2.]@ndarray[float16] [1. 2.]@ndarray[float32] [1.
2.]@ndarray[float64] [1.+0.j 2.+0.j]@ndarray[complex64] [1.+0.j
2.+0.j]@ndarray[complex128] [array([1, 2, 3], dtyp [...]
boolean [None, None] [True, False] X X X X
[True, True] [True, True] [True, True] [True, True] [True, True]
[True, True] [True, True] [True, True] X [True, True] [True,
True] X X X X X X X X
tinyint [None, None] [1, 0] X [12, 34] [1, 2] X
[1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2]
[1, 2] X X X X X X X X
smallint [None, None] [1, 0] X [12, 34] [1, 2] X
[1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2]
[1, 2] X X X X X X X X
int [None, None] [1, 0] X [12, 34] [1, 2] X [1, 2]
[1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2]
X X X X X X X X
-bigint [None, None] [1, 0] X [12, 34] [1, 2] X [1, 2]
[1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2]
X X X [0, 86400000000000] [18000000000000,
104400000000000] [86400000000000, 172800000000000] X X
+bigint [None, None] [1, 0] X [12, 34] [1, 2] X [1, 2]
[1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2] [1, 2]
X X X [0, 86400000000] [18000000000, 104400000000]
[86400000000, 172800000000] X X
string [None, None] X ['a', 'b'] ['12', '34'] X X
X X X X X X X X X X
X X X X X X X ['A', 'B'] X
date [None, None] X X X [datetime.date(1970, 1, 2),
datetime.dat X X X [datetime.date(1970, 1, 2),
datetime.dat X X X X X X X X
X X X [datetime.date(1970, 1, 1), datetime.dat
[datetime.date(1970, 1, 1), datetime.dat X X X
timestamp [None, None] X X X
[datetime.datetime(1969, 12, 31, 16, 0, X X X X
[datetime.datetime(1969, 12, 31, 16, 0, X X X X
X X X X X X [datetime.datetime(1970, 1, 1,
0, 0), da [datetime.datetime(1970, 1, 1, 5, 0), da X X
X
@@ -11,6 +11,6 @@ float [None, None] [1.0, 0.0] X [12.0, 34.0]
[1.0, 2.0] X [1.0, 2.0] [1.0, 2.0]
double [None, None] [1.0, 0.0] X [12.0, 34.0] [1.0, 2.0]
X [1.0, 2.0] [1.0, 2.0] [1.0, 2.0] [1.0, 2.0] [1.0,
2.0] [1.0, 2.0] [1.0, 2.0] [1.0, 2.0] [1.0, 2.0] [1.0,
2.0] [1.0, 2.0] X X X X X X X
X
array<int> [None, None] X X [[1, 2], [3, 4]] X
X X X X X X X X X X
X X X X [[1, 2, 3], [1, 2, 3]] X X X
X X
binary [None, None] [b'\x01', b''] [b'a', b'b'] [b'12', b'34'] X
X [b'\x01', b'\x02'] [b'\x01', b'\x02'] [b'\x01', b'\x02']
[b'\x01', b'\x02'] [b'\x01', b'\x02'] [b'\x01', b'\x02']
[b'\x01', b'\x02'] [b'\x01', b'\x02'] [b'', b''] [b'', b'']
[b'', b''] [b'', b''] [b'', b''] X [b'', b''] [b'',
b''] [b'', b''] [b'A', b'B'] X
-decimal(10,0) [None, None] X X X [Decimal('1'),
Decimal('2')] X [Decimal('1'), Decimal('2')] [Decimal('1'),
Decimal('2')] [Decimal('1'), Decimal('2')] X [Decimal('1'),
Decimal('2')] [Decimal('1'), Decimal('2')] [Decimal('1'), Decimal('2')]
X X [Decimal('1'), Decimal('2')] [Decimal('1'), Decimal('2')]
X X X X X X X X
+decimal(10,0) [None, None] X X [Decimal('12'), Decimal('34')]
[Decimal('1'), Decimal('2')] X [Decimal('1'), Decimal('2')]
[Decimal('1'), Decimal('2')] [Decimal('1'), Decimal('2')] X
[Decimal('1'), Decimal('2')] [Decimal('1'), Decimal('2')] [Decimal('1'),
Decimal('2')] X X [Decimal('1'), Decimal('2')] [Decimal('1'),
Decimal('2')] X X X X X X X X
map<string,int> [None, None] X X X X [{'a':
1}, {'b': 2}] X X X X X X X X
X X X X X X X X X X
X
struct<_1:int> X X X X X X X X
X X X X X X X X X X
X X X X X X [Row(_1=1), Row(_1=2)]
diff --git
a/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
new file mode 100644
index 000000000000..aa57ca6eec96
--- /dev/null
+++
b/python/pyspark/sql/tests/coercion/pandas_3/golden_pandas_udf_return_type_coercion_base.md
@@ -0,0 +1,17 @@
+| SQL Type \ Value@Type | [None, None]@list | [True, False]@list |
['a', 'b']@list | ['12', '34']@list | [Decimal('1'),
Decimal('2')]@list | [{'a': 1}, {'b': 2}]@list | [1 2]@ndarray[int8]
| [1 2]@ndarray[int16] | [1 2]@ndarray[int32]
| [1 2]@ndarray[int64] | [1 2]@ndarray[uint8] | [1
2]@ndarray[uint16] | [1 2]@ndarray[uint32] | [1
2]@ndarray[uint64] | [1. 2.]@ndarray[float16] [...]
+|--------------------------|---------------------|----------------------|-------------------|--------------------------------|------------------------------------------|-----------------------------|------------------------------|------------------------------|------------------------------------------|-----------------------------------------|------------------------------|------------------------------|------------------------------|-------------------------|---------------------------
[...]
+| boolean | [None, None] | [True, False] | X
| X | X
| X | [True, True] | [True,
True] | [True, True] | [True, True]
| [True, True] | [True, True]
| [True, True] | [True, True] | X
[...]
+| tinyint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
+| smallint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
+| int | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
+| bigint | [None, None] | [1, 0] | X
| [12, 34] | [1, 2]
| X | [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2]
| [1, 2] | [1, 2] | [1, 2]
[...]
+| string | [None, None] | X |
['a', 'b'] | ['12', '34'] | X
| X | X | X
| X | X
| X | X
| X | X | X
[...]
+| date | [None, None] | X | X
| X | [datetime.date(1970, 1, 2),
datetime.dat | X | X | X
| [datetime.date(1970, 1, 2), datetime.dat | X
| X | X
| X | X | X
[...]
+| timestamp | [None, None] | X | X
| X | [datetime.datetime(1969, 12,
31, 16, 0, | X | X | X
| X |
[datetime.datetime(1969, 12, 31, 16, 0, | X | X
| X | X
| X [...]
+| float | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0]
| X | [1.0, 2.0] | [1.0,
2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
[...]
+| double | [None, None] | [1.0, 0.0] | X
| [12.0, 34.0] | [1.0, 2.0]
| X | [1.0, 2.0] | [1.0,
2.0] | [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0]
| [1.0, 2.0] | [1.0, 2.0] | [1.0, 2.0]
[...]
+| array<int> | [None, None] | X | X
| [[1, 2], [3, 4]] | X
| X | X | X
| X | X
| X | X
| X | X | X
[...]
+| binary | [None, None] | [b'\x01', b''] |
[b'a', b'b'] | [b'12', b'34'] | X
| X | [b'\x01', b'\x02'] |
[b'\x01', b'\x02'] | [b'\x01', b'\x02'] |
[b'\x01', b'\x02'] | [b'\x01', b'\x02'] |
[b'\x01', b'\x02'] | [b'\x01', b'\x02'] | [b'\x01',
b'\x02'] | [b'', b''] [...]
+| decimal(10,0) | [None, None] | X | X
| [Decimal('12'), Decimal('34')] | [Decimal('1'), Decimal('2')]
| X | [Decimal('1'), Decimal('2')] |
[Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X
| [Decimal('1'), Decimal('2')] |
[Decimal('1'), Decimal('2')] | [Decimal('1'), Decimal('2')] | X
| X [...]
+| map<string,int> | [None, None] | X | X
| X | X
| [{'a': 1}, {'b': 2}] | X | X
| X | X
| X | X
| X | X | X
[...]
+| struct<_1:int> | X | X | X
| X | X
| X | X | X
| X | X
| X | X
| X | X | X
[...]
\ No newline at end of file
diff --git
a/python/pyspark/sql/tests/coercion/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
b/python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
similarity index 98%
rename from
python/pyspark/sql/tests/coercion/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
rename to
python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
index be52b9885f76..d8eda38c67c0 100644
---
a/python/pyspark/sql/tests/coercion/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
+++
b/python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_input_type_coercion_with_arrow_and_pandas.csv
@@ -14,7 +14,7 @@
12 decimal_values decimal(3,2) [Decimal('5.35'), Decimal('1.23')]
['Decimal', 'Decimal'] ['5.35', '1.23']
13 decimal_null decimal(3,2) [None, Decimal('9.99')] ['NoneType',
'Decimal'] ['None', '9.99']
14 string_values string ['abc', '', 'hello'] ['str', 'str', 'str']
['abc', '', 'hello']
-15 string_null string [None, 'test'] ['NoneType', 'str']
['None', 'test']
+15 string_null string [None, 'test'] ✗ Output ['nan', 'test'] !=
Input [None, 'test']
16 binary_values binary [b'abc', b'', b'ABC'] ['bytes', 'bytes',
'bytes'] "[""b'abc'"", ""b''"", ""b'ABC'""]"
17 binary_null binary [None, b'test'] ['NoneType', 'bytes']
"['None', ""b'test'""]"
18 boolean_values boolean [True, False] ['bool', 'bool']
['True', 'False']
diff --git
a/python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_input_type_coercion_with_arrow_and_pandas.md
b/python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_input_type_coercion_with_arrow_and_pandas.md
new file mode 100644
index 000000000000..fe6b9e897be4
--- /dev/null
+++
b/python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_input_type_coercion_with_arrow_and_pandas.md
@@ -0,0 +1,41 @@
+| | Test Case | Spark Type |
Spark Value |
Python Type | Python Value
|
+|----|---------------------------|------------------------------------------|---------------------------------------------------------------------------|--------------------------------------------------|-------------------------------------------------------------------------------------------------------|
+| 0 | byte_values | tinyint |
[-128, 127, 0] |
['int', 'int', 'int'] | ['-128', '127', '0']
|
+| 1 | byte_null | tinyint |
[None, 42] |
['float', 'float'] | ['nan', '42.0']
|
+| 2 | short_values | smallint |
[-32768, 32767, 0] |
['int', 'int', 'int'] | ['-32768', '32767', '0']
|
+| 3 | short_null | smallint |
[None, 123] |
['float', 'float'] | ['nan', '123.0']
|
+| 4 | int_values | int |
[-2147483648, 2147483647, 0] |
['int', 'int', 'int'] | ['-2147483648',
'2147483647', '0']
|
+| 5 | int_null | int |
[None, 456] |
['float', 'float'] | ['nan', '456.0']
|
+| 6 | long_values | bigint |
[-9223372036854775808, 9223372036854775807, 0] |
['int', 'int', 'int'] | ['-9223372036854775808',
'9223372036854775807', '0'] |
+| 7 | long_null | bigint |
[None, 789] |
['float', 'float'] | ['nan', '789.0']
|
+| 8 | float_values | float |
[0.0, 1.0, 3.140000104904175] |
['float', 'float', 'float'] | ['0.0', '1.0',
'3.140000104904175']
|
+| 9 | float_null | float |
[None, 3.140000104904175] |
['float', 'float'] | ['nan', '3.140000104904175']
|
+| 10 | double_values | double |
[0.0, 1.0, 0.3333333333333333] |
['float', 'float', 'float'] | ['0.0', '1.0',
'0.3333333333333333']
|
+| 11 | double_null | double |
[None, 2.71] |
['float', 'float'] | ['nan', '2.71']
|
+| 12 | decimal_values | decimal(3,2) |
[Decimal('5.35'), Decimal('1.23')] |
['Decimal', 'Decimal'] | ['5.35', '1.23']
|
+| 13 | decimal_null | decimal(3,2) |
[None, Decimal('9.99')] |
['NoneType', 'Decimal'] | ['None', '9.99']
|
+| 14 | string_values | string |
['abc', '', 'hello'] |
['str', 'str', 'str'] | ['abc', '', 'hello']
|
+| 15 | string_null | string |
[None, 'test'] | ✗
Output ['nan', 'test'] != Input [None, 'test'] |
|
+| 16 | binary_values | binary |
[b'abc', b'', b'ABC'] |
['bytes', 'bytes', 'bytes'] | ["b'abc'", "b''", "b'ABC'"]
|
+| 17 | binary_null | binary |
[None, b'test'] |
['NoneType', 'bytes'] | ['None', "b'test'"]
|
+| 18 | boolean_values | boolean |
[True, False] |
['bool', 'bool'] | ['True', 'False']
|
+| 19 | boolean_null | boolean |
[None, True] |
['NoneType', 'bool'] | ['None', 'True']
|
+| 20 | date_values | date |
[datetime.date(2020, 2, 2), datetime.date(1970, 1, 1)] |
['date', 'date'] | ['2020-02-02', '1970-01-01']
|
+| 21 | date_null | date |
[None, datetime.date(2023, 1, 1)] |
['NoneType', 'date'] | ['None', '2023-01-01']
|
+| 22 | timestamp_values | timestamp |
[datetime.datetime(2020, 2, 2, 12, 15, 16, 123000)] |
['Timestamp'] | ['2020-02-02
12:15:16.123000']
|
+| 23 | timestamp_null | timestamp |
[None, datetime.datetime(2023, 1, 1, 12, 0)] |
['NaTType', 'Timestamp'] | ['NaT', '2023-01-01
12:00:00']
|
+| 24 | array_int_values | array<int> |
[[1, 2, 3], [], [1, None, 3]] |
['list', 'list', 'list'] | ['[1, 2, 3]', '[]', '[1,
None, 3]'] |
+| 25 | array_int_null | array<int> |
[None, [4, 5, 6]] |
['NoneType', 'list'] | ['None', '[np.int32(4),
np.int32(5), np.int32(6)]'] |
+| 26 | map_str_int_values | map<string,int> |
[{'world': 2, 'hello': 1}, {}] |
['dict', 'dict'] | ["{'world': 2, 'hello': 1}",
'{}'] |
+| 27 | map_str_int_null | map<string,int> |
[None, {'test': 123}] |
['NoneType', 'dict'] | ['None', "{'test': 123}"]
|
+| 28 | struct_int_str_values | struct<a1:int,a2:string> |
[Row(a1=1, a2='hello'), Row(a1=2, a2='world')] |
['Row', 'Row'] | ["Row(a1=1, a2='hello')",
"Row(a1=2, a2='world')"] |
+| 29 | struct_int_str_null | struct<a1:int,a2:string> |
[None, Row(a1=99, a2='test')] |
['NoneType', 'Row'] | ['None', "Row(a1=99,
a2='test')"]
|
+| 30 | array_array_int | array<array<int>> |
[[[1, 2, 3]], [[1], [2, 3]]] |
['list', 'list'] | ['[[np.int32(1),
np.int32(2), np.int32(3)]]', '[[np.int32(1)], [np.int32(2), np.int32(3)]]']
|
+| 31 | array_map_str_int | array<map<string,int>> |
[[{'world': 2, 'hello': 1}], [{'a': 1}, {'b': 2}]] |
['list', 'list'] | ["[{'world': 2, 'hello':
1}]", "[{'a': 1}, {'b': 2}]"] |
+| 32 | array_struct_int_str | array<struct<a1:int,a2:string>> |
[[Row(a1=1, a2='hello')], [Row(a1=1, a2='hello'), Row(a1=2, a2='world')]] |
['list', 'list'] | ["[Row(a1=1, a2='hello')]",
"[Row(a1=1, a2='hello'), Row(a1=2, a2='world')]"] |
+| 33 | map_int_array_int | map<int,array<int>> |
[{1: [1, 2, 3]}, {1: [1], 2: [2, 3]}] |
['dict', 'dict'] | ['{1: [np.int32(1),
np.int32(2), np.int32(3)]}', '{1: [np.int32(1)], 2: [np.int32(2),
np.int32(3)]}'] |
+| 34 | map_int_map_str_int | map<int,map<string,int>> |
[{1: {'world': 2, 'hello': 1}}] |
['dict'] | ["{1: {'world': 2, 'hello':
1}}"] |
+| 35 | map_int_struct_int_str | map<int,struct<a1:int,a2:string>> |
[{1: Row(a1=1, a2='hello')}] |
['dict'] | ["{1: Row(a1=1,
a2='hello')}"]
|
+| 36 | struct_int_array_int | struct<a:int,b:array<int>> |
[Row(a=1, b=[1, 2, 3])] |
['Row'] | ['Row(a=1, b=[np.int32(1),
np.int32(2), np.int32(3)])'] |
+| 37 | struct_int_map_str_int | struct<a:int,b:map<string,int>> |
[Row(a=1, b={'world': 2, 'hello': 1})] |
['Row'] | ["Row(a=1, b={'world': 2,
'hello': 1})"] |
+| 38 | struct_int_struct_int_str | struct<a:int,b:struct<a1:int,a2:string>> |
[Row(a=1, b=Row(a1=1, a2='hello'))] |
['Row'] | ["Row(a=1, b=Row(a1=1,
a2='hello'))"] |
\ No newline at end of file
diff --git
a/python/pyspark/sql/tests/coercion/golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv
b/python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv
similarity index 100%
rename from
python/pyspark/sql/tests/coercion/golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv
rename to
python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_return_type_coercion_with_arrow_and_pandas.csv
diff --git
a/python/pyspark/sql/tests/coercion/golden_python_udf_return_type_coercion_with_arrow_and_pandas.md
b/python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_return_type_coercion_with_arrow_and_pandas.md
similarity index 100%
rename from
python/pyspark/sql/tests/coercion/golden_python_udf_return_type_coercion_with_arrow_and_pandas.md
rename to
python/pyspark/sql/tests/coercion/pandas_3/golden_python_udf_return_type_coercion_with_arrow_and_pandas.md
diff --git a/python/pyspark/sql/tests/coercion/test_pandas_udf_input_type.py
b/python/pyspark/sql/tests/coercion/test_pandas_udf_input_type.py
index a77a750e4684..0f36e142f359 100644
--- a/python/pyspark/sql/tests/coercion/test_pandas_udf_input_type.py
+++ b/python/pyspark/sql/tests/coercion/test_pandas_udf_input_type.py
@@ -78,6 +78,18 @@ class PandasUDFInputTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
def prefix(self):
return "golden_pandas_udf_input_type_coercion"
+ @property
+ def pandas_dir(self):
+ # Pandas >= 3.0 reports the dedicated 'str' dtype for string columns,
+ # whereas earlier versions report 'object', which changes the recorded
+ # Python types. Use a dedicated golden file per major pandas version,
+ # kept in a versioned subdirectory, instead of patching one golden
+ # in memory.
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ return "pandas_3"
+ else:
+ return "pandas_2"
+
@property
def test_cases(self):
def df(args):
@@ -235,7 +247,7 @@ class PandasUDFInputTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
def test_pandas_input_type_coercion_vanilla(self):
self._run_pandas_udf_input_type_coercion(
- golden_file=f"{self.prefix}_base",
+ golden_file=os.path.join(self.pandas_dir, f"{self.prefix}_base"),
test_name="Pandas UDF",
)
@@ -251,14 +263,6 @@ class PandasUDFInputTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
golden = None
if not generating:
golden = self.load_golden_csv(golden_csv)
- # Pandas >= 3.0 reports the dedicated 'str' dtype for string
columns,
- # whereas earlier versions report 'object'. Patch the in-memory
golden
- # so the same file works under both versions.
- if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
- str_rows = golden["Spark Type"] == "string"
- golden.loc[str_rows, "Python Type"] = golden.loc[
- str_rows, "Python Type"
- ].str.replace("'object'", "'str'")
results = []
for idx, (case_name, spark_type, data_func) in
enumerate(self.test_cases):
@@ -296,6 +300,9 @@ class PandasUDFInputTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
except Exception as e:
print("error_msg", e)
result.append(f"✗ {str(e)}")
+ # Pad to the full column count so the row roundtrips through
+ # the golden CSV (short rows come back with a trailing "").
+ result.append("")
# Clean up exception message to remove newlines and extra
whitespace
result = [self.clean_result(r) for r in result]
diff --git a/python/pyspark/sql/tests/coercion/test_pandas_udf_return_type.py
b/python/pyspark/sql/tests/coercion/test_pandas_udf_return_type.py
index f1ba3cd84723..e26efd2b999c 100644
--- a/python/pyspark/sql/tests/coercion/test_pandas_udf_return_type.py
+++ b/python/pyspark/sql/tests/coercion/test_pandas_udf_return_type.py
@@ -78,6 +78,19 @@ class PandasUDFReturnTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
def prefix(self):
return "golden_pandas_udf_return_type_coercion"
+ @property
+ def pandas_dir(self):
+ # Pandas 3 changed several defaults (datetime64 ndarrays use [us]
instead
+ # of [ns], Categorical categories use str instead of object, and the
same
+ # casts return microseconds instead of nanoseconds), which produces
+ # different coercion results. Use a dedicated golden file per major
+ # pandas version, kept in a versioned subdirectory, instead of patching
+ # one golden in memory.
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ return "pandas_3"
+ else:
+ return "pandas_2"
+
@property
def test_data(self):
return [
@@ -144,7 +157,7 @@ class PandasUDFReturnTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
def test_pandas_return_type_coercion_vanilla(self):
self._run_pandas_udf_return_type_coercion(
- golden_file=f"{self.prefix}_base",
+ golden_file=os.path.join(self.pandas_dir, f"{self.prefix}_base"),
test_name="Pandas UDF",
)
@@ -160,49 +173,6 @@ class PandasUDFReturnTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
golden = None
if not generating:
golden = self.load_golden_csv(golden_csv)
- # The golden file was generated under pandas 2; patch the loaded
- # copy in memory so the same file works under pandas >= 3.0, where
- # the defaults differ: datetime64 ndarrays use [us] instead of
[ns],
- # Categorical categories use str instead of object, and the same
- # casts return microseconds instead of nanoseconds.
- if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
- rename = {}
- scale_cols = []
- for value in self.test_data:
- new_key = self.repr_value(value)
- if isinstance(value, np.ndarray) and value.dtype.kind ==
"M":
- old_key =
self.repr_value(value.astype("datetime64[ns]"))
- if old_key != new_key:
- rename[old_key] = new_key
- scale_cols.append(new_key)
- elif isinstance(value, pd.Categorical) and
value.categories.dtype != object:
- old_key = self.repr_value(
- pd.Categorical(
- value.tolist(),
- categories=pd.Index(value.categories.tolist(),
dtype=object),
- )
- )
- if old_key != new_key:
- rename[old_key] = new_key
- elif isinstance(value, list) and value and
isinstance(value[0], pd.Timedelta):
- scale_cols.append(new_key)
-
- if rename:
- golden.rename(columns=rename, inplace=True)
-
- for col in scale_cols:
- golden[col] = golden[col].str.replace(
- r"\d{13,}",
- lambda m: str(int(m.group()) // 1000),
- regex=True,
- )
-
- # Pandas 3 succeeds at coercing string list -> Decimal where
- # pandas 2 errored, so the corresponding cell flips from "X".
- decimal_idx = self.repr_type(DecimalType(10, 0))
- decimal_col = self.repr_value(["12", "34"])
- if decimal_idx in golden.index and decimal_col in
golden.columns:
- golden.loc[decimal_idx, decimal_col] = "[Decimal('12'),
Decimal('34')]"
def work(arg):
spark_type, value = arg
diff --git a/python/pyspark/sql/tests/coercion/test_python_udf_input_type.py
b/python/pyspark/sql/tests/coercion/test_python_udf_input_type.py
index c772e97747f6..5f00d97b7d12 100644
--- a/python/pyspark/sql/tests/coercion/test_python_udf_input_type.py
+++ b/python/pyspark/sql/tests/coercion/test_python_udf_input_type.py
@@ -78,6 +78,18 @@ class UDFInputTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
def prefix(self):
return "golden_python_udf_input_type_coercion"
+ @property
+ def pandas_dir(self):
+ # Only the legacy pandas conversion path routes inputs through pandas,
+ # whose defaults changed in pandas 3 (e.g. None becomes nan in str
+ # columns). Use a dedicated golden file per major pandas version for
+ # that path, kept in a versioned subdirectory, instead of patching
+ # one golden in memory.
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ return "pandas_3"
+ else:
+ return "pandas_2"
+
@property
def test_cases(self):
def df(args):
@@ -253,7 +265,7 @@ class UDFInputTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
self._run_udf_input_type_coercion(
use_arrow=True,
legacy_pandas=True,
- golden_file=f"{self.prefix}_with_arrow_and_pandas",
+ golden_file=os.path.join(self.pandas_dir,
f"{self.prefix}_with_arrow_and_pandas"),
test_name="Arrow Optimized Python UDF with Legacy Pandas
Conversion",
)
@@ -319,6 +331,9 @@ class UDFInputTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
except Exception as e:
print("Exception", e)
result.append(f"✗ {str(e)}")
+ # Pad to the full column count so the row roundtrips through
+ # the golden CSV (short rows come back with a trailing "").
+ result.append("")
# Clean up exception message to remove newlines and extra
whitespace
result = [self.clean_result(r) for r in result]
diff --git a/python/pyspark/sql/tests/coercion/test_python_udf_return_type.py
b/python/pyspark/sql/tests/coercion/test_python_udf_return_type.py
index e3b9939fa51f..e5281e9ad3d6 100644
--- a/python/pyspark/sql/tests/coercion/test_python_udf_return_type.py
+++ b/python/pyspark/sql/tests/coercion/test_python_udf_return_type.py
@@ -82,6 +82,17 @@ class UDFReturnTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
def prefix(self):
return "golden_python_udf_return_type_coercion"
+ @property
+ def pandas_dir(self):
+ # Only the legacy pandas conversion path routes results through pandas,
+ # whose defaults changed in pandas 3. Use a dedicated golden file per
+ # major pandas version for that path, kept in a versioned subdirectory,
+ # instead of patching one golden in memory.
+ if LooseVersion(pd.__version__) >= LooseVersion("3.0.0"):
+ return "pandas_3"
+ else:
+ return "pandas_2"
+
@property
def test_data(self):
return [
@@ -154,7 +165,7 @@ class UDFReturnTypeTests(GoldenFileTestMixin,
ReusedSQLTestCase):
self._run_udf_return_type_coercion(
use_arrow=True,
legacy_pandas=True,
- golden_file=f"{self.prefix}_with_arrow_and_pandas",
+ golden_file=os.path.join(self.pandas_dir,
f"{self.prefix}_with_arrow_and_pandas"),
test_name="Arrow Optimized Python UDF with Legacy Pandas
Conversion",
)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
