Hyukjin Kwon created SPARK-61961:
------------------------------------
Summary: Send PySpark's requests from the Rust API for model
deletions, listeners, stars, reduce, frame bounds, bucketing, foreach sinks,
empty Arrow data, cross-session subqueries, corr, unpersist and lateral joins,
and raise PySpark's argument errors
Key: SPARK-61961
URL: https://issues.apache.org/jira/browse/SPARK-61961
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
A round-3 review of the Rust API (crates/spark-connect) against pyspark-client
4.3.0. Where PySpark and the Python binding do one thing, the Rust API did
another:
- The deletion of a dropped native ML model carried no tags or user-context
extensions of the dropping thread. The listener-bus registration carried no
tags.
- DataFrame::col("*") was an attribute named "*", not PySpark's df["*"] star.
- functions::reduce sent "aggregate".
- Window frame offsets of 0, and those past PySpark's unbounded thresholds,
were sent as literals. PySpark sends the current row and unbounded, so
rangeBetween(0, 0) over a string order failed.
- bucket_by with a count of 0 or less sent a bucketing that the server refuses.
- foreach and foreachBatch sent an output_type that PySpark does not.
- PySpark's client-side checks were missing:
- a schema with changes();
- await timeouts that are not positive;
- a create_table_with schema that is not a struct;
- DataStreamReader::name's regex.
- create_dataframe_from_arrow of zero-row batches sent their data with the
schema, which the server refused, where PySpark sends the schema alone.
- Subqueries and table arguments of a DataFrame of another session sent a plan
whose WithRelations references had other plan ids. So
`s2.range(5).where(col("id").isin(spark.range(2).select("id")))` raised
"Missing relation in WithRelations", where PySpark returns the rows. This
applies to the binding too.
- corr never sent its method ("pearson").
- unpersist(blocking=None) sent blocking false, where PySpark leaves it unset.
- lateral_join took any join type, where PySpark raises UNSUPPORTED_JOIN_TYPE
for all but inner, left and cross.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]