[ 
https://issues.apache.org/jira/browse/SPARK-61961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61961:
------------------------------------

    Assignee: Hyukjin Kwon

> Send PySpark's requests from the Rust API for model deletions, listeners, 
> stars, reduce, frame bounds, bucketing, foreach sinks, empty Arrow data, 
> cross-session subqueries, corr, unpersist and lateral joins, and raise 
> PySpark's argument errors
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61961
>                 URL: https://issues.apache.org/jira/browse/SPARK-61961
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> A round-3 review of the Rust API (crates/spark-connect) against 
> pyspark-client 4.3.0. Where PySpark and the Python binding do one thing, the 
> Rust API did another:
> - The deletion of a dropped native ML model carried no tags or user-context 
> extensions of the dropping thread. The listener-bus registration carried no 
> tags.
> - DataFrame::col("*") was an attribute named "*", not PySpark's df["*"] star.
> - functions::reduce sent "aggregate".
> - Window frame offsets of 0, and those past PySpark's unbounded thresholds, 
> were sent as literals. PySpark sends the current row and unbounded, so 
> rangeBetween(0, 0) over a string order failed.
> - bucket_by with a count of 0 or less sent a bucketing that the server 
> refuses.
> - foreach and foreachBatch sent an output_type that PySpark does not.
> - PySpark's client-side checks were missing:
>   - a schema with changes();
>   - await timeouts that are not positive;
>   - a create_table_with schema that is not a struct;
>   - DataStreamReader::name's regex.
> - create_dataframe_from_arrow of zero-row batches sent their data with the 
> schema, which the server refused, where PySpark sends the schema alone.
> - Subqueries and table arguments of a DataFrame of another session sent a 
> plan whose WithRelations references had other plan ids. So 
> `s2.range(5).where(col("id").isin(spark.range(2).select("id")))` raised 
> "Missing relation in WithRelations", where PySpark returns the rows. This 
> applies to the binding too.
> - corr never sent its method ("pearson").
> - unpersist(blocking=None) sent blocking false, where PySpark leaves it unset.
> - lateral_join took any join type, where PySpark raises UNSUPPORTED_JOIN_TYPE 
> for all but inner, left and cross.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to