[
https://issues.apache.org/jira/browse/SPARK-60902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60902:
---------------------------------
Description:
The Rust-backed Python client's {{createDataFrame}} applies column-name lists
and the columns of a pandas DataFrame differently from PySpark Connect 4.3.0:
* PySpark applies {{toDF}} when it has column names to apply: for a list or
tuple of names, and for every pandas DataFrame (its column names, or the field
names of a given schema). The server resolves {{toDF}} into a {{Project}}, so
{{isLocal()}} is False. The Rust client puts the names in the local relation's
schema instead, so {{isLocal()}} is True, for example for
{{createDataFrame([(1,)], ["a"])}} and {{createDataFrame(pandas_df)}}.
* PySpark extends a list of names that is shorter than plain rows or a pandas
DataFrame with {{_2}}, {{_3}}, ... (an empty list gives {{_1}}, {{_2}}, ...);
the Rust client raises an error instead. A longer list raises
{{AXIS_LENGTH_MISMATCH}} in PySpark, and an atomic schema for a pandas
DataFrame {{UNSUPPORTED_DATA_TYPE_FOR_ARROW}}; the Rust client raises an
unclassified error for both.
Rows with no schema or with a StructType, DDL, or atomic schema, and PyArrow
tables, already match. NumPy ndarray input, which the client does not accept
yet, is tracked by SPARK-60772.
was:
PySpark Connect's {{createDataFrame}} applies {{toDF}} to the local relation
when it has column names to apply: for a list or tuple of column names, and for
every pandas DataFrame (its column names, or the field names of a given
schema). The server resolves {{toDF}} into a {{Project}}, so {{isLocal()}} is
False for those DataFrames.
The Rust-backed Python client puts the names in the local relation's schema
instead, so {{isLocal()}} is True for them:
* {{createDataFrame([(1,)], ["a"]).isLocal()}} is True; PySpark returns False.
* {{createDataFrame(pandas_df).isLocal()}}, with no schema, a list of names, a
StructType or a DDL string, is True; PySpark returns False.
Rows with no schema or with a StructType, DDL, or atomic schema, and PyArrow
tables, already match. Apply {{toDF}} in the same cases as PySpark. NumPy
ndarray input, which the client does not accept yet, is tracked by SPARK-60772.
Summary: Apply column-name lists and pandas columns in Spark Connect
Rust createDataFrame as PySpark does (was: Apply toDF in Spark Connect Rust
createDataFrame where PySpark does, so isLocal() matches)
> Apply column-name lists and pandas columns in Spark Connect Rust
> createDataFrame as PySpark does
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-60902
> URL: https://issues.apache.org/jira/browse/SPARK-60902
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> The Rust-backed Python client's {{createDataFrame}} applies column-name lists
> and the columns of a pandas DataFrame differently from PySpark Connect 4.3.0:
> * PySpark applies {{toDF}} when it has column names to apply: for a list or
> tuple of names, and for every pandas DataFrame (its column names, or the
> field names of a given schema). The server resolves {{toDF}} into a
> {{Project}}, so {{isLocal()}} is False. The Rust client puts the names in the
> local relation's schema instead, so {{isLocal()}} is True, for example for
> {{createDataFrame([(1,)], ["a"])}} and {{createDataFrame(pandas_df)}}.
> * PySpark extends a list of names that is shorter than plain rows or a pandas
> DataFrame with {{_2}}, {{_3}}, ... (an empty list gives {{_1}}, {{_2}}, ...);
> the Rust client raises an error instead. A longer list raises
> {{AXIS_LENGTH_MISMATCH}} in PySpark, and an atomic schema for a pandas
> DataFrame {{UNSUPPORTED_DATA_TYPE_FOR_ARROW}}; the Rust client raises an
> unclassified error for both.
> Rows with no schema or with a StructType, DDL, or atomic schema, and PyArrow
> tables, already match. NumPy ndarray input, which the client does not accept
> yet, is tracked by SPARK-60772.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
