[
https://issues.apache.org/jira/browse/SPARK-61013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61013:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Set the results of the Observations of observe in the Python client as
> PySpark does
> -----------------------------------------------------------------------------------
>
> Key: SPARK-61013
> URL: https://issues.apache.org/jira/browse/SPARK-61013
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{df.observe(observation, *exprs)}} makes the
> {{Observation}} receive the metrics that the server observes when an action
> runs the DataFrame: after {{df.observe(o,
> count(lit(1)).alias("c")).collect()}}, {{o.get}} is {{{'c': 3}}}, also for
> {{count()}}, {{toPandas()}} and {{show()}}. An {{Observation}} cannot be used
> twice ({{REUSE_OBSERVATION}}), {{get}} raises {{NO_OBSERVE_BEFORE_GET}}
> before it is used, and its name, which is a UUID by default, must be a
> non-empty string.
> The Rust-backed Python client's {{Observation.get}} is always {{{}}}: it
> never receives the observed metrics. Set the results of Observations as
> PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
