[ 
https://issues.apache.org/jira/browse/SPARK-61940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61940:
------------------------------------

    Assignee: Hyukjin Kwon

> Keep the metrics of PlanObservedMetrics as the Expression.Literal messages of 
> the server, with PySpark's repr
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61940
>                 URL: https://issues.apache.org/jira/browse/SPARK-61940
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark's PlanObservedMetrics keeps the Expression.Literal messages that the 
> server sends, and its
> pairs converts them with LiteralExpression._to_value when read. The drop-in's 
> kept the decoded
> Python values, so its metrics, its repr and the UNKNOWN_RESPONSE message that 
> names it differed:
> in DataFrame.executionInfo, in the ExecutionInfo of spark.client.to_table, 
> to_pandas and
> execute_command, and in the attrs of to_pandas.
> Repro (Spark 4.3.0 server)
>     from pyspark.sql import Observation, functions as F
>     import pyspark.sql.connect.proto as pb2
>     o = Observation("seen"); df = spark.range(4).observe(o, 
> F.count(F.lit(1)).alias("rows"))
>     p = pb2.Plan(); p.root.CopyFrom(df._plan.plan(spark.client))
>     m = spark.client.to_pandas(p, {"seen": o})[0].attrs["observed_metrics"][0]
>     m.metrics, repr(m)
> Actual (drop-in): [4], 'Plan observed(seen=[4])'
> Expected (pyspark-client 4.3.0): [long: 4\n] (Expression.Literal messages), 
> 'Plan observed(seen=[long: 4\n])';
> pairs is {'rows': 4} on both, and a year-month interval metric raises 
> UNSUPPORTED_LITERAL when
> flows is read.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to