[
https://issues.apache.org/jira/browse/SPARK-61896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61896:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Send PySpark's requests for star column names, the catalog's scalar
> operations, the removal of checkpoints, columns of no rows and errorifexists
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61896
> URL: https://issues.apache.org/jira/browse/SPARK-61896
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> The core sends other requests than PySpark 4.3.0 in these cases, of the Rust
> API and, where noted, of the drop-in:
> - spark_connect::col("*"), col("t.*") and every &str/String column name are
> sent as an unresolved_attribute, not a star, so select(["*"]),
> count(col("*")), struct and to_json of a struct fail on the server with
> UNRESOLVED_COLUMN of `*`, where PySpark's col("*") is UnresolvedStar. (Rust
> API only.)
> - The scalar catalog operations (currentDatabase, tableExists, cacheTable,
> dropTempView, ...) send a relation with no plan id and no thread tags, where
> PySpark runs them as a DataFrame. (Both.)
> - The removal of a dropped checkpoint's cached relation is reattachable and
> followed by a ReleaseExecute, where PySpark's CachedRemoteRelation.__del__
> sends one non-reattachable execution. (Both.)
> - create_dataframe_from_columns of no rows reads the settings with a Config
> request, where PySpark sends the schema alone. (Rust API only.)
> - mode("errorifexists") writes, where PySpark Connect raises
> UNSUPPORTED_OPERATION at the save. (Both.)
> - golden_test.rs compares 23 of its 28 goldens, and plans_golden.rs builds
> select_star by hand, which hid the first case.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]