Hyukjin Kwon created SPARK-61013:
------------------------------------
Summary: Set the results of the Observations of observe in the
Python client as PySpark does
Key: SPARK-61013
URL: https://issues.apache.org/jira/browse/SPARK-61013
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{df.observe(observation, *exprs)}} makes the
{{Observation}} receive the metrics that the server observes when an action
runs the DataFrame: after {{df.observe(o,
count(lit(1)).alias("c")).collect()}}, {{o.get}} is {{{'c': 3}}}, also for
{{count()}}, {{toPandas()}} and {{show()}}. An {{Observation}} cannot be used
twice ({{REUSE_OBSERVATION}}), {{get}} raises {{NO_OBSERVE_BEFORE_GET}} before
it is used, and its name, which is a UUID by default, must be a non-empty
string.
The Rust-backed Python client's {{Observation.get}} is always {{{}}}: it never
receives the observed metrics. Set the results of Observations as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
