Hyukjin Kwon created SPARK-61426:
------------------------------------

             Summary: Check the observed metrics of only the Observations of 
the executed plan, keep the metrics of earlier actions, and raise for nested 
unconvertible metrics, in the Python client as PySpark does
                 Key: SPARK-61426
                 URL: https://issues.apache.org/jira/browse/SPARK-61426
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


SPARK-61363 made the Python client raise UNSUPPORTED_LITERAL from an action 
whose observed metrics hold a kind that PySpark does not convert (a year-month 
interval, a nanosecond timestamp), as PySpark's client does. Three differences 
from PySpark remain:

1. The check matched the session's live Observations by name against the 
metrics of the last execution, not the Observations of the executed plan. So 
df.localCheckpoint() (and checkpoint) and spark.sql("SELECT * FROM {d}", 
d=observed_df).count() raised UNSUPPORTED_LITERAL, where PySpark runs them: its 
execute_command converts no observations, and a DataFrame action converts only 
those of its plan (LogicalPlan.observations). After a failing action, 
Observation.get was {} where PySpark keeps the metrics of the earlier actions.

2. For a string-named observe, df.executionInfo.flows and 
PlanObservedMetrics.to_dict returned the client's internal _UnsupportedLiteral 
marker nested in a list or Row (F.collect_list(ym), F.first(F.struct(ym))), 
where PySpark raises UNSUPPORTED_LITERAL.

3. The registry of the Observations of a session was pruned only by actions of 
that session, so a stopped session kept its entry.

Fix: the core finds the Observations of the executed plan as PySpark's 
observations properties collect them and checks their metrics per execution 
before setting them, through a check that the Python client registers; commands 
check none. A nested unconvertible literal is the marker of the metric. The 
per-session registry is removed.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to