[
https://issues.apache.org/jira/browse/SPARK-61426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61426:
-----------------------------------
Labels: pull-request-available (was: )
> Check the observed metrics of only the Observations of the executed plan,
> keep the metrics of earlier actions, and raise for nested unconvertible
> metrics, in the Python client as PySpark does
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61426
> URL: https://issues.apache.org/jira/browse/SPARK-61426
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> SPARK-61363 made the Python client raise UNSUPPORTED_LITERAL from an action
> whose observed metrics hold a kind that PySpark does not convert (a
> year-month interval, a nanosecond timestamp), as PySpark's client does. Three
> differences from PySpark remain:
> 1. The check matched the session's live Observations by name against the
> metrics of the last execution, not the Observations of the executed plan. So
> df.localCheckpoint() (and checkpoint) and spark.sql("SELECT * FROM {d}",
> d=observed_df).count() raised UNSUPPORTED_LITERAL, where PySpark runs them:
> its execute_command converts no observations, and a DataFrame action converts
> only those of its plan (LogicalPlan.observations). After a failing action,
> Observation.get was {} where PySpark keeps the metrics of the earlier actions.
> 2. For a string-named observe, df.executionInfo.flows and
> PlanObservedMetrics.to_dict returned the client's internal
> _UnsupportedLiteral marker nested in a list or Row (F.collect_list(ym),
> F.first(F.struct(ym))), where PySpark raises UNSUPPORTED_LITERAL.
> 3. The registry of the Observations of a session was pruned only by actions
> of that session, so a stopped session kept its entry.
> Fix: the core finds the Observations of the executed plan as PySpark's
> observations properties collect them and checks their metrics per execution
> before setting them, through a check that the Python client registers;
> commands check none. A nested unconvertible literal is the marker of the
> metric. The per-session registry is removed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]