[
https://issues.apache.org/jira/browse/SPARK-60902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60902:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Apply toDF in Spark Connect Rust createDataFrame where PySpark does, so
> isLocal() matches
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-60902
> URL: https://issues.apache.org/jira/browse/SPARK-60902
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's {{createDataFrame}} applies {{toDF}} to the local relation
> when it has column names to apply: for a list or tuple of column names, and
> for every pandas DataFrame (its column names, or the field names of a given
> schema). The server resolves {{toDF}} into a {{Project}}, so {{isLocal()}} is
> False for those DataFrames.
> The Rust-backed Python client puts the names in the local relation's schema
> instead, so {{isLocal()}} is True for them:
> * {{createDataFrame([(1,)], ["a"]).isLocal()}} is True; PySpark returns False.
> * {{createDataFrame(pandas_df).isLocal()}}, with no schema, a list of names,
> a StructType or a DDL string, is True; PySpark returns False.
> Rows with no schema or with a StructType, DDL, or atomic schema, and PyArrow
> tables, already match. Apply {{toDF}} in the same cases as PySpark. NumPy
> ndarray input, which the client does not accept yet, is tracked by
> SPARK-60772.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
