This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 53e4901ef5c8 [SPARK-46776][PYTHON][FOLLOWUP] Cast
large_string/large_binary to the requested type on pyarrow < 19
53e4901ef5c8 is described below
commit 53e4901ef5c8c4ed6b01d42b33f5a8f492a1888e
Author: Yicong Huang <[email protected]>
AuthorDate: Fri Jun 12 18:59:55 2026 +0000
[SPARK-46776][PYTHON][FOLLOWUP] Cast large_string/large_binary to the
requested type on pyarrow < 19
### What changes were proposed in this pull request?
This is a follow-up to #56157 (commit `2f4ed64204e`, `[SPARK-46776][PYTHON]
Support pa.ChunkedArray columns in createDataFrame from pandas`), which
introduced a regression in the minimum-dependencies CI job (`pyarrow==18.0.0`).
In `create_arrow_array_from_pandas`
(`python/pyspark/sql/pandas/conversion.py`), after `pa.Array.from_pandas(...)`,
cast the result back to the requested `arrow_type` only when pyarrow is older
than 19.0.0 and the produced type is the `large_string`/`large_binary`
counterpart of the requested `string`/`binary` type. Both guards are required:
the version check avoids touching pyarrow >= 19.0.0 (which already honors the
requested type), and the exact type check ensures only this offset- [...]
### Why are the changes needed?
Since pandas 2.2.0, the `string[pyarrow]` extension dtype is backed by
`large_string` (64-bit offsets). When such a series is converted via the
`__arrow_array__` protocol, pyarrow < 19.0.0 ignores the requested `type`
argument, so `pa.Array.from_pandas(series, type=pa.string())` returns a
`large_string` array even though `string` was requested. The JVM then reads the
64-bit offset buffers against the 32-bit schema, silently corrupting the data.
pyarrow 19.0.0 fixed this in the protocol path (apache/arrow#44195) by
casting the result to the requested type. This change replicates that behavior
for older pyarrow so the minimum-dependencies build (`pyarrow==18.0.0`)
produces correct data instead of corrupt rows.
### Does this PR introduce _any_ user-facing change?
Yes. On pyarrow < 19.0.0, `createDataFrame` from a pandas object containing
`string[pyarrow]`/`large_string`-backed columns (and the `binary` equivalent)
previously produced silently corrupted data; it now produces correct data. On
pyarrow >= 19.0.0 there is no behavior change.
### How was this patch tested?
Existing `pyspark.sql.tests.test_conversion` suite passes locally. The
regression environment was validated by manually dispatching the "Build /
Python-only (Minimum dependencies of PySpark)" workflow on my fork against this
branch with `pyarrow==18.0.0`; the `pyspark-sql`, `pyspark-pandas`, and related
modules all passed:
https://github.com/Yicong-Huang/spark/actions/runs/27244934810
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #56380 from
Yicong-Huang/SPARK-46776-followup/fix/cast-large-string-pyarrow18.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
(cherry picked from commit 3b5cbeb1f05dc3b7d32ee1107e7e8147ad518cd1)
Signed-off-by: Yicong-Huang <[email protected]>
---
python/pyspark/sql/pandas/conversion.py | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/pandas/conversion.py
b/python/pyspark/sql/pandas/conversion.py
index b488a49bcb6f..d0b89354e100 100644
--- a/python/pyspark/sql/pandas/conversion.py
+++ b/python/pyspark/sql/pandas/conversion.py
@@ -88,6 +88,7 @@ def create_arrow_array_from_pandas(
"""
import pyarrow as pa
import pandas as pd
+ from pyspark.loose_version import LooseVersion
from pyspark.sql.pandas.types import to_arrow_type,
_create_converter_from_pandas
if isinstance(series.dtype, pd.CategoricalDtype):
@@ -113,7 +114,23 @@ def create_arrow_array_from_pandas(
else:
mask = series.isnull()
try:
- return pa.Array.from_pandas(series, mask=mask, type=arrow_type,
safe=safecheck)
+ result = pa.Array.from_pandas(series, mask=mask, type=arrow_type,
safe=safecheck)
+ # SPARK-46776: pyarrow < 19.0.0 ignores the requested ``type`` in the
+ # ``__arrow_array__`` protocol used by pyarrow-backed extension
dtypes, so a
+ # ``string[pyarrow]`` series (backed by ``large_string`` since pandas
2.2.0) can
+ # come back as ``large_string`` even when ``string`` was requested,
silently
+ # corrupting data on the JVM side. Cast back only for this exact
(large_)string /
+ # (large_)binary offset-width mismatch; pyarrow >= 19.0.0 already
honors the type.
+ if (
+ arrow_type is not None
+ and LooseVersion(pa.__version__) < LooseVersion("19.0.0")
+ and (
+ (pa.types.is_large_string(result.type) and
pa.types.is_string(arrow_type))
+ or (pa.types.is_large_binary(result.type) and
pa.types.is_binary(arrow_type))
+ )
+ ):
+ result = result.cast(arrow_type)
+ return result
except TypeError as e:
error_msg = (
"Exception thrown when converting pandas.Series (%s) "
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
