Hyukjin Kwon created SPARK-61227:
------------------------------------
Summary: Convert a pandas frame with default column labels through
Arrow, as PySpark does
Key: SPARK-61227
URL: https://issues.apache.org/jira/browse/SPARK-61227
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
createDataFrame of a plain pandas frame with default (RangeIndex) column
labels, such as pd.DataFrame([[None], [3]]) or pd.DataFrame(ndarray), is not
accepted by the Arrow routes in the Python client, so it takes the native row
path. With an explicit schema that path fails where PySpark works:
spark.createDataFrame(pd.DataFrame([[None], [3]]), "x int") raises
PySparkRuntimeError "Type mismatch in row data", while PySpark 4.3.0 returns
[Row(x=None), Row(x=3)]. The same frame with named or Int64 labels already
converts with Arrow.
The fix accepts a default RangeIndex of labels under the same override checks
as an Index, so these frames take the Arrow routes. Subclassed and
instance-overridden labels keep the row path.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
