[ 
https://issues.apache.org/jira/browse/SPARK-61426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61426:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Check the observed metrics of only the Observations of the executed plan, 
> keep the metrics of earlier actions, and raise for nested unconvertible 
> metrics, in the Python client as PySpark does
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61426
>                 URL: https://issues.apache.org/jira/browse/SPARK-61426
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> SPARK-61363 made the Python client raise UNSUPPORTED_LITERAL from an action 
> whose observed metrics hold a kind that PySpark does not convert (a 
> year-month interval, a nanosecond timestamp), as PySpark's client does. Three 
> differences from PySpark remain:
> 1. The check matched the session's live Observations by name against the 
> metrics of the last execution, not the Observations of the executed plan. So 
> df.localCheckpoint() (and checkpoint) and spark.sql("SELECT * FROM {d}", 
> d=observed_df).count() raised UNSUPPORTED_LITERAL, where PySpark runs them: 
> its execute_command converts no observations, and a DataFrame action converts 
> only those of its plan (LogicalPlan.observations). After a failing action, 
> Observation.get was {} where PySpark keeps the metrics of the earlier actions.
> 2. For a string-named observe, df.executionInfo.flows and 
> PlanObservedMetrics.to_dict returned the client's internal 
> _UnsupportedLiteral marker nested in a list or Row (F.collect_list(ym), 
> F.first(F.struct(ym))), where PySpark raises UNSUPPORTED_LITERAL.
> 3. The registry of the Observations of a session was pruned only by actions 
> of that session, so a stopped session kept its entry.
> Fix: the core finds the Observations of the executed plan as PySpark's 
> observations properties collect them and checks their metrics per execution 
> before setting them, through a check that the Python client registers; 
> commands check none. A nested unconvertible literal is the marker of the 
> metric. The per-session registry is removed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to