Hyukjin Kwon created SPARK-61896:
------------------------------------
Summary: Send PySpark's requests for star column names, the
catalog's scalar operations, the removal of checkpoints, columns of no rows and
errorifexists
Key: SPARK-61896
URL: https://issues.apache.org/jira/browse/SPARK-61896
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
The core sends other requests than PySpark 4.3.0 in these cases, of the Rust
API and, where noted, of the drop-in:
- spark_connect::col("*"), col("t.*") and every &str/String column name are
sent as an unresolved_attribute, not a star, so select(["*"]), count(col("*")),
struct and to_json of a struct fail on the server with UNRESOLVED_COLUMN of
`*`, where PySpark's col("*") is UnresolvedStar. (Rust API only.)
- The scalar catalog operations (currentDatabase, tableExists, cacheTable,
dropTempView, ...) send a relation with no plan id and no thread tags, where
PySpark runs them as a DataFrame. (Both.)
- The removal of a dropped checkpoint's cached relation is reattachable and
followed by a ReleaseExecute, where PySpark's CachedRemoteRelation.__del__
sends one non-reattachable execution. (Both.)
- create_dataframe_from_columns of no rows reads the settings with a Config
request, where PySpark sends the schema alone. (Rust API only.)
- mode("errorifexists") writes, where PySpark Connect raises
UNSUPPORTED_OPERATION at the save. (Both.)
- golden_test.rs compares 23 of its 28 goldens, and plans_golden.rs builds
select_star by hand, which hid the first case.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]