[ 
https://issues.apache.org/jira/browse/SPARK-61496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61496:
-----------------------------------
    Labels: pull-request-available  (was: )

> Type the join type and direction of the as-of join in the Rust client, and 
> set an Observation whose relation has sent copies
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61496
>                 URL: https://issues.apache.org/jira/browse/SPARK-61496
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Two code-quality findings in crates/spark-connect/src/plan.rs of 
> spark-connect-rust.
> 1. The AsOfJoin plan node keeps its join type and its direction as strings. 
> Give them enums in the core (AsOfJoinType: inner, left outer; 
> AsOfJoinDirection: backward, forward, nearest). PySpark's DataFrame._joinAsOf 
> checks neither `how` nor `direction` and sends them as they are, and the 
> server reads more spellings than these names (any case, underscores) and 
> refuses the rest in the analysis, so any other string is kept and sent as it 
> is: no request changes, and the client raises no error that PySpark does not.
> 2. PlanIds::node_of returns whichever node of a plan ID the scan of its map 
> meets first. A plan ID is of more than one node whenever a relation that the 
> session sent has a DataFrame of its own (DataFrame::from_sent_relation), such 
> as an unpickled copy of a DataFrame. When the scan meets the copy, the 
> observed metrics of that plan ID are dropped, and Observation.get of an 
> observed DataFrame that has such a copy returns nothing. Give every live node 
> of the ID, and set the Observation from the one that observes.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to