L. C. Hsieh created SPARK-61496:
-----------------------------------

             Summary: Type the join type and direction of the as-of join in the 
Rust client, and set an Observation whose relation has sent copies
                 Key: SPARK-61496
                 URL: https://issues.apache.org/jira/browse/SPARK-61496
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: L. C. Hsieh


Two code-quality findings in crates/spark-connect/src/plan.rs of 
spark-connect-rust.

1. The AsOfJoin plan node keeps its join type and its direction as strings. 
Give them enums in the core (AsOfJoinType: inner, left outer; 
AsOfJoinDirection: backward, forward, nearest). PySpark's DataFrame._joinAsOf 
checks neither `how` nor `direction` and sends them as they are, and the server 
reads more spellings than these names (any case, underscores) and refuses the 
rest in the analysis, so any other string is kept and sent as it is: no request 
changes, and the client raises no error that PySpark does not.

2. PlanIds::node_of returns whichever node of a plan ID the scan of its map 
meets first. A plan ID is of more than one node whenever a relation that the 
session sent has a DataFrame of its own (DataFrame::from_sent_relation), such 
as an unpickled copy of a DataFrame. When the scan meets the copy, the observed 
metrics of that plan ID are dropped, and Observation.get of an observed 
DataFrame that has such a copy returns nothing. Give every live node of the ID, 
and set the Observation from the one that observes.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to