[
https://issues.apache.org/jira/browse/SPARK-61963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61963.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 824
[https://github.com/apache/spark-connect-rust/pull/824]
> Give the plan of a DataFrame the Observations of the DataFrame by their
> names, as PySpark's LogicalPlan.observations
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61963
> URL: https://issues.apache.org/jira/browse/SPARK-61963
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark's DataFrame._plan.observations is LogicalPlan.observations of the
> DataFrame's plan: the
> Observations of observe on the DataFrame and on its inputs (both sides of a
> join and of a set
> operation) by their names. PySpark's DataFrame gives them to the executions
> of its client
> (client.to_table(query, self._plan.observations), connect/dataframe.py:1982,
> 2002, 2228), and a plan
> of pyspark.sql.connect.plan over it, such as a SubqueryAlias, has them too
> (plan.py:152-157,
> 1829-1837). The drop-in's DataFrame._plan, a _DataFramePlan, had none: its
> observations was always
> {}. So code that runs the plan of a DataFrame through SparkSession.client, as
> PySpark's DataFrame
> does, left the DataFrame's Observations unset.
> Repro (Spark 4.3.0 server)
> import pyspark.sql.connect.proto as pb2
> from pyspark.sql import Observation, functions as F
> from pyspark.sql.connect.plan import SubqueryAlias
> o = Observation("o1")
> df = spark.range(3).observe(o, F.count(F.lit(1)).alias("n"))
> df._plan.observations
> SubqueryAlias(df._plan, "s").observations
> plan = pb2.Plan(); plan.root.CopyFrom(df._plan.plan(spark.client))
> spark.client.to_table(plan, df._plan.observations)
> o.get
> Actual (drop-in): {}, {}, and o.get is {}.
> Expected (pyspark-client 4.3.0): {'o1': o} (the same Observation object),
> {'o1': o}, and o.get is
> {'n': 3}. DataFrame(SubqueryAlias(df._plan, "s"), spark).collect() sets o on
> both already.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]