[ 
https://issues.apache.org/jira/browse/SPARK-61963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61963:
------------------------------------

    Assignee: Hyukjin Kwon

> Give the plan of a DataFrame the Observations of the DataFrame by their 
> names, as PySpark's LogicalPlan.observations
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61963
>                 URL: https://issues.apache.org/jira/browse/SPARK-61963
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark's DataFrame._plan.observations is LogicalPlan.observations of the 
> DataFrame's plan: the
> Observations of observe on the DataFrame and on its inputs (both sides of a 
> join and of a set
> operation) by their names. PySpark's DataFrame gives them to the executions 
> of its client
> (client.to_table(query, self._plan.observations), connect/dataframe.py:1982, 
> 2002, 2228), and a plan
> of pyspark.sql.connect.plan over it, such as a SubqueryAlias, has them too 
> (plan.py:152-157,
> 1829-1837). The drop-in's DataFrame._plan, a _DataFramePlan, had none: its 
> observations was always
> {}. So code that runs the plan of a DataFrame through SparkSession.client, as 
> PySpark's DataFrame
> does, left the DataFrame's Observations unset.
> Repro (Spark 4.3.0 server)
>     import pyspark.sql.connect.proto as pb2
>     from pyspark.sql import Observation, functions as F
>     from pyspark.sql.connect.plan import SubqueryAlias
>     o = Observation("o1")
>     df = spark.range(3).observe(o, F.count(F.lit(1)).alias("n"))
>     df._plan.observations
>     SubqueryAlias(df._plan, "s").observations
>     plan = pb2.Plan(); plan.root.CopyFrom(df._plan.plan(spark.client))
>     spark.client.to_table(plan, df._plan.observations)
>     o.get
> Actual (drop-in): {}, {}, and o.get is {}.
> Expected (pyspark-client 4.3.0): {'o1': o} (the same Observation object), 
> {'o1': o}, and o.get is
> {'n': 3}. DataFrame(SubqueryAlias(df._plan, "s"), spark).collect() sets o on 
> both already.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to