Hyukjin Kwon created SPARK-61940:
------------------------------------
Summary: Keep the metrics of PlanObservedMetrics as the
Expression.Literal messages of the server, with PySpark's repr
Key: SPARK-61940
URL: https://issues.apache.org/jira/browse/SPARK-61940
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
PySpark's PlanObservedMetrics keeps the Expression.Literal messages that the
server sends, and its
pairs converts them with LiteralExpression._to_value when read. The drop-in's
kept the decoded
Python values, so its metrics, its repr and the UNKNOWN_RESPONSE message that
names it differed:
in DataFrame.executionInfo, in the ExecutionInfo of spark.client.to_table,
to_pandas and
execute_command, and in the attrs of to_pandas.
Repro (Spark 4.3.0 server)
from pyspark.sql import Observation, functions as F
import pyspark.sql.connect.proto as pb2
o = Observation("seen"); df = spark.range(4).observe(o,
F.count(F.lit(1)).alias("rows"))
p = pb2.Plan(); p.root.CopyFrom(df._plan.plan(spark.client))
m = spark.client.to_pandas(p, {"seen": o})[0].attrs["observed_metrics"][0]
m.metrics, repr(m)
Actual (drop-in): [4], 'Plan observed(seen=[4])'
Expected (pyspark-client 4.3.0): [long: 4\n] (Expression.Literal messages),
'Plan observed(seen=[long: 4\n])';
pairs is {'rows': 4} on both, and a year-month interval metric raises
UNSUPPORTED_LITERAL when
flows is read.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]