[ 
https://issues.apache.org/jira/browse/SPARK-61896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61896:
-----------------------------------
    Labels: pull-request-available  (was: )

> Send PySpark's requests for star column names, the catalog's scalar 
> operations, the removal of checkpoints, columns of no rows and errorifexists
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61896
>                 URL: https://issues.apache.org/jira/browse/SPARK-61896
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> The core sends other requests than PySpark 4.3.0 in these cases, of the Rust 
> API and, where noted, of the drop-in:
> - spark_connect::col("*"), col("t.*") and every &str/String column name are 
> sent as an unresolved_attribute, not a star, so select(["*"]), 
> count(col("*")), struct and to_json of a struct fail on the server with 
> UNRESOLVED_COLUMN of `*`, where PySpark's col("*") is UnresolvedStar. (Rust 
> API only.)
> - The scalar catalog operations (currentDatabase, tableExists, cacheTable, 
> dropTempView, ...) send a relation with no plan id and no thread tags, where 
> PySpark runs them as a DataFrame. (Both.)
> - The removal of a dropped checkpoint's cached relation is reattachable and 
> followed by a ReleaseExecute, where PySpark's CachedRemoteRelation.__del__ 
> sends one non-reattachable execution. (Both.)
> - create_dataframe_from_columns of no rows reads the settings with a Config 
> request, where PySpark sends the schema alone. (Rust API only.)
> - mode("errorifexists") writes, where PySpark Connect raises 
> UNSUPPORTED_OPERATION at the save. (Both.)
> - golden_test.rs compares 23 of its 28 goldens, and plans_golden.rs builds 
> select_star by hand, which hid the first case.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to