Hyukjin Kwon created SPARK-61896:
------------------------------------

             Summary: Send PySpark's requests for star column names, the 
catalog's scalar operations, the removal of checkpoints, columns of no rows and 
errorifexists
                 Key: SPARK-61896
                 URL: https://issues.apache.org/jira/browse/SPARK-61896
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


The core sends other requests than PySpark 4.3.0 in these cases, of the Rust 
API and, where noted, of the drop-in:
- spark_connect::col("*"), col("t.*") and every &str/String column name are 
sent as an unresolved_attribute, not a star, so select(["*"]), count(col("*")), 
struct and to_json of a struct fail on the server with UNRESOLVED_COLUMN of 
`*`, where PySpark's col("*") is UnresolvedStar. (Rust API only.)
- The scalar catalog operations (currentDatabase, tableExists, cacheTable, 
dropTempView, ...) send a relation with no plan id and no thread tags, where 
PySpark runs them as a DataFrame. (Both.)
- The removal of a dropped checkpoint's cached relation is reattachable and 
followed by a ReleaseExecute, where PySpark's CachedRemoteRelation.__del__ 
sends one non-reattachable execution. (Both.)
- create_dataframe_from_columns of no rows reads the settings with a Config 
request, where PySpark sends the schema alone. (Rust API only.)
- mode("errorifexists") writes, where PySpark Connect raises 
UNSUPPORTED_OPERATION at the save. (Both.)
- golden_test.rs compares 23 of its 28 goldens, and plans_golden.rs builds 
select_star by hand, which hid the first case.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to