[ 
https://issues.apache.org/jira/browse/SPARK-61496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

L. C. Hsieh updated SPARK-61496:
--------------------------------
    Description: 
Two code-quality findings in crates/spark-connect/src/plan.rs of 
spark-connect-rust.

1. The AsOfJoin plan node keeps its join type and its direction as strings. 
Give them enums in the core (AsOfJoinType: inner, left outer; 
AsOfJoinDirection: backward, forward, nearest). PySpark's DataFrame._joinAsOf 
checks neither `how` nor `direction` and sends them as they are, and the server 
reads more spellings than these names (any case, underscores) and refuses the 
rest in the analysis, so any other string is kept and sent as it is: no request 
changes, and the client raises no error that PySpark does not.

2. PlanIds::node_of returns whichever live node of a plan ID the scan of its 
map meets first. A plan ID is of more than one node whenever a relation that 
the session sent has a DataFrame of its own (DataFrame::from_sent_relation), 
such as an unpickled copy of a DataFrame. That used to lose the observed 
metrics of the plan ID when the node was a copy; SPARK-61426 fixed that by 
setting the Observations by name from the executed plan, as PySpark does, and 
left node_of one caller, from_sent_relation, which takes the Observations of 
the node for the new copy. Every node of an ID has the same ones, so any node 
is right there. Document that node_of gives any live node of the ID and why, so 
that a later caller does not assume one particular node, and test it. Its code 
does not change.

  was:
Two code-quality findings in crates/spark-connect/src/plan.rs of 
spark-connect-rust.

1. The AsOfJoin plan node keeps its join type and its direction as strings. 
Give them enums in the core (AsOfJoinType: inner, left outer; 
AsOfJoinDirection: backward, forward, nearest). PySpark's DataFrame._joinAsOf 
checks neither `how` nor `direction` and sends them as they are, and the server 
reads more spellings than these names (any case, underscores) and refuses the 
rest in the analysis, so any other string is kept and sent as it is: no request 
changes, and the client raises no error that PySpark does not.

2. PlanIds::node_of returns whichever node of a plan ID the scan of its map 
meets first. A plan ID is of more than one node whenever a relation that the 
session sent has a DataFrame of its own (DataFrame::from_sent_relation), such 
as an unpickled copy of a DataFrame. When the scan meets the copy, the observed 
metrics of that plan ID are dropped, and Observation.get of an observed 
DataFrame that has such a copy returns nothing. Give every live node of the ID, 
and set the Observation from the one that observes.


        Summary: Type the join type and direction of the as-of join in the Rust 
client, and document which node a shared plan ID gives  (was: Type the join 
type and direction of the as-of join in the Rust client, and set an Observation 
whose relation has sent copies)

> Type the join type and direction of the as-of join in the Rust client, and 
> document which node a shared plan ID gives
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61496
>                 URL: https://issues.apache.org/jira/browse/SPARK-61496
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Two code-quality findings in crates/spark-connect/src/plan.rs of 
> spark-connect-rust.
> 1. The AsOfJoin plan node keeps its join type and its direction as strings. 
> Give them enums in the core (AsOfJoinType: inner, left outer; 
> AsOfJoinDirection: backward, forward, nearest). PySpark's DataFrame._joinAsOf 
> checks neither `how` nor `direction` and sends them as they are, and the 
> server reads more spellings than these names (any case, underscores) and 
> refuses the rest in the analysis, so any other string is kept and sent as it 
> is: no request changes, and the client raises no error that PySpark does not.
> 2. PlanIds::node_of returns whichever live node of a plan ID the scan of its 
> map meets first. A plan ID is of more than one node whenever a relation that 
> the session sent has a DataFrame of its own (DataFrame::from_sent_relation), 
> such as an unpickled copy of a DataFrame. That used to lose the observed 
> metrics of the plan ID when the node was a copy; SPARK-61426 fixed that by 
> setting the Observations by name from the executed plan, as PySpark does, and 
> left node_of one caller, from_sent_relation, which takes the Observations of 
> the node for the new copy. Every node of an ID has the same ones, so any node 
> is right there. Document that node_of gives any live node of the ID and why, 
> so that a later caller does not assume one particular node, and test it. Its 
> code does not change.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to