[
https://issues.apache.org/jira/browse/SPARK-61940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61940:
------------------------------------
Assignee: Hyukjin Kwon
> Keep the metrics of PlanObservedMetrics as the Expression.Literal messages of
> the server, with PySpark's repr
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61940
> URL: https://issues.apache.org/jira/browse/SPARK-61940
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark's PlanObservedMetrics keeps the Expression.Literal messages that the
> server sends, and its
> pairs converts them with LiteralExpression._to_value when read. The drop-in's
> kept the decoded
> Python values, so its metrics, its repr and the UNKNOWN_RESPONSE message that
> names it differed:
> in DataFrame.executionInfo, in the ExecutionInfo of spark.client.to_table,
> to_pandas and
> execute_command, and in the attrs of to_pandas.
> Repro (Spark 4.3.0 server)
> from pyspark.sql import Observation, functions as F
> import pyspark.sql.connect.proto as pb2
> o = Observation("seen"); df = spark.range(4).observe(o,
> F.count(F.lit(1)).alias("rows"))
> p = pb2.Plan(); p.root.CopyFrom(df._plan.plan(spark.client))
> m = spark.client.to_pandas(p, {"seen": o})[0].attrs["observed_metrics"][0]
> m.metrics, repr(m)
> Actual (drop-in): [4], 'Plan observed(seen=[4])'
> Expected (pyspark-client 4.3.0): [long: 4\n] (Expression.Literal messages),
> 'Plan observed(seen=[long: 4\n])';
> pairs is {'rows': 4} on both, and a year-month interval metric raises
> UNSUPPORTED_LITERAL when
> flows is read.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]