Hyukjin Kwon created SPARK-61426:
------------------------------------
Summary: Check the observed metrics of only the Observations of
the executed plan, keep the metrics of earlier actions, and raise for nested
unconvertible metrics, in the Python client as PySpark does
Key: SPARK-61426
URL: https://issues.apache.org/jira/browse/SPARK-61426
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
SPARK-61363 made the Python client raise UNSUPPORTED_LITERAL from an action
whose observed metrics hold a kind that PySpark does not convert (a year-month
interval, a nanosecond timestamp), as PySpark's client does. Three differences
from PySpark remain:
1. The check matched the session's live Observations by name against the
metrics of the last execution, not the Observations of the executed plan. So
df.localCheckpoint() (and checkpoint) and spark.sql("SELECT * FROM {d}",
d=observed_df).count() raised UNSUPPORTED_LITERAL, where PySpark runs them: its
execute_command converts no observations, and a DataFrame action converts only
those of its plan (LogicalPlan.observations). After a failing action,
Observation.get was {} where PySpark keeps the metrics of the earlier actions.
2. For a string-named observe, df.executionInfo.flows and
PlanObservedMetrics.to_dict returned the client's internal _UnsupportedLiteral
marker nested in a list or Row (F.collect_list(ym), F.first(F.struct(ym))),
where PySpark raises UNSUPPORTED_LITERAL.
3. The registry of the Observations of a session was pruned only by actions of
that session, so a stopped session kept its entry.
Fix: the core finds the Observations of the executed plan as PySpark's
observations properties collect them and checks their metrics per execution
before setting them, through a check that the Python client registers; commands
check none. A nested unconvertible literal is the marker of the metric. The
per-session registry is removed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]