L. C. Hsieh created SPARK-61496:
-----------------------------------
Summary: Type the join type and direction of the as-of join in the
Rust client, and set an Observation whose relation has sent copies
Key: SPARK-61496
URL: https://issues.apache.org/jira/browse/SPARK-61496
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
Two code-quality findings in crates/spark-connect/src/plan.rs of
spark-connect-rust.
1. The AsOfJoin plan node keeps its join type and its direction as strings.
Give them enums in the core (AsOfJoinType: inner, left outer;
AsOfJoinDirection: backward, forward, nearest). PySpark's DataFrame._joinAsOf
checks neither `how` nor `direction` and sends them as they are, and the server
reads more spellings than these names (any case, underscores) and refuses the
rest in the analysis, so any other string is kept and sent as it is: no request
changes, and the client raises no error that PySpark does not.
2. PlanIds::node_of returns whichever node of a plan ID the scan of its map
meets first. A plan ID is of more than one node whenever a relation that the
session sent has a DataFrame of its own (DataFrame::from_sent_relation), such
as an unpickled copy of a DataFrame. When the scan meets the copy, the observed
metrics of that plan ID are dropped, and Observation.get of an observed
DataFrame that has such a copy returns nothing. Give every live node of the ID,
and set the Observation from the one that observes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]