Fangchen Li created SPARK-60009:
-----------------------------------
Summary: Arrow-backed pandas columns ignore the requested Arrow
type in pandas-to-Arrow conversion
Key: SPARK-60009
URL: https://issues.apache.org/jira/browse/SPARK-60009
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.4.0
Reporter: Fangchen Li
pandas' ArrowExtensionArray, which backs pd.ArrowDtype and string[pyarrow]
columns, ignores the requested Arrow type when Spark converts a pandas Series
with pa.Array.from_pandas. On pyarrow 18, Spark's minimum supported version,
the result is not cast, so in both createDataFrame from pandas and pandas UDF
results, a column can reach the JVM with a type that doesn't match the schema,
for example, int32 data for a LongType field. In pyarrow 19 and later, pyarrow
casts the result itself, but always with safe=True, so createDataFrame ignores
spark.sql.execution.pandas.convertToArrowArraySafely for these columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
