[
https://issues.apache.org/jira/browse/SPARK-61961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61961:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Send PySpark's requests from the Rust API for model deletions, listeners,
> stars, reduce, frame bounds, bucketing, foreach sinks, empty Arrow data,
> cross-session subqueries, corr, unpersist and lateral joins, and raise
> PySpark's argument errors
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61961
> URL: https://issues.apache.org/jira/browse/SPARK-61961
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> A round-3 review of the Rust API (crates/spark-connect) against
> pyspark-client 4.3.0. Where PySpark and the Python binding do one thing, the
> Rust API did another:
> - The deletion of a dropped native ML model carried no tags or user-context
> extensions of the dropping thread. The listener-bus registration carried no
> tags.
> - DataFrame::col("*") was an attribute named "*", not PySpark's df["*"] star.
> - functions::reduce sent "aggregate".
> - Window frame offsets of 0, and those past PySpark's unbounded thresholds,
> were sent as literals. PySpark sends the current row and unbounded, so
> rangeBetween(0, 0) over a string order failed.
> - bucket_by with a count of 0 or less sent a bucketing that the server
> refuses.
> - foreach and foreachBatch sent an output_type that PySpark does not.
> - PySpark's client-side checks were missing:
> - a schema with changes();
> - await timeouts that are not positive;
> - a create_table_with schema that is not a struct;
> - DataStreamReader::name's regex.
> - create_dataframe_from_arrow of zero-row batches sent their data with the
> schema, which the server refused, where PySpark sends the schema alone.
> - Subqueries and table arguments of a DataFrame of another session sent a
> plan whose WithRelations references had other plan ids. So
> `s2.range(5).where(col("id").isin(spark.range(2).select("id")))` raised
> "Missing relation in WithRelations", where PySpark returns the rows. This
> applies to the binding too.
> - corr never sent its method ("pearson").
> - unpersist(blocking=None) sent blocking false, where PySpark leaves it unset.
> - lateral_join took any join type, where PySpark raises UNSUPPORTED_JOIN_TYPE
> for all but inner, left and cross.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]