[
https://issues.apache.org/jira/browse/SPARK-61227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61227:
-----------------------------------
Labels: pull-request-available (was: )
> Convert a pandas frame with default column labels through Arrow, as PySpark
> does
> --------------------------------------------------------------------------------
>
> Key: SPARK-61227
> URL: https://issues.apache.org/jira/browse/SPARK-61227
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> createDataFrame of a plain pandas frame with default (RangeIndex) column
> labels, such as pd.DataFrame([[None], [3]]) or pd.DataFrame(ndarray), is not
> accepted by the Arrow routes in the Python client, so it takes the native row
> path. With an explicit schema that path fails where PySpark works:
> spark.createDataFrame(pd.DataFrame([[None], [3]]), "x int") raises
> PySparkRuntimeError "Type mismatch in row data", while PySpark 4.3.0 returns
> [Row(x=None), Row(x=3)]. The same frame with named or Int64 labels already
> converts with Arrow.
> The fix accepts a default RangeIndex of labels under the same override checks
> as an Index, so these frames take the Arrow routes. Subclassed and
> instance-overridden labels keep the row path.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
