Hyukjin Kwon created SPARK-61940:
------------------------------------

             Summary: Keep the metrics of PlanObservedMetrics as the 
Expression.Literal messages of the server, with PySpark's repr
                 Key: SPARK-61940
                 URL: https://issues.apache.org/jira/browse/SPARK-61940
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
PySpark's PlanObservedMetrics keeps the Expression.Literal messages that the 
server sends, and its
pairs converts them with LiteralExpression._to_value when read. The drop-in's 
kept the decoded
Python values, so its metrics, its repr and the UNKNOWN_RESPONSE message that 
names it differed:
in DataFrame.executionInfo, in the ExecutionInfo of spark.client.to_table, 
to_pandas and
execute_command, and in the attrs of to_pandas.

Repro (Spark 4.3.0 server)
    from pyspark.sql import Observation, functions as F
    import pyspark.sql.connect.proto as pb2
    o = Observation("seen"); df = spark.range(4).observe(o, 
F.count(F.lit(1)).alias("rows"))
    p = pb2.Plan(); p.root.CopyFrom(df._plan.plan(spark.client))
    m = spark.client.to_pandas(p, {"seen": o})[0].attrs["observed_metrics"][0]
    m.metrics, repr(m)
Actual (drop-in): [4], 'Plan observed(seen=[4])'
Expected (pyspark-client 4.3.0): [long: 4\n] (Expression.Literal messages), 
'Plan observed(seen=[long: 4\n])';
pairs is {'rows': 4} on both, and a year-month interval metric raises 
UNSUPPORTED_LITERAL when
flows is read.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to