Hyukjin Kwon created SPARK-61963:
------------------------------------
Summary: Give the plan of a DataFrame the Observations of the
DataFrame by their names, as PySpark's LogicalPlan.observations
Key: SPARK-61963
URL: https://issues.apache.org/jira/browse/SPARK-61963
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
PySpark's DataFrame._plan.observations is LogicalPlan.observations of the
DataFrame's plan: the
Observations of observe on the DataFrame and on its inputs (both sides of a
join and of a set
operation) by their names. PySpark's DataFrame gives them to the executions of
its client
(client.to_table(query, self._plan.observations), connect/dataframe.py:1982,
2002, 2228), and a plan
of pyspark.sql.connect.plan over it, such as a SubqueryAlias, has them too
(plan.py:152-157,
1829-1837). The drop-in's DataFrame._plan, a _DataFramePlan, had none: its
observations was always
{}. So code that runs the plan of a DataFrame through SparkSession.client, as
PySpark's DataFrame
does, left the DataFrame's Observations unset.
Repro (Spark 4.3.0 server)
import pyspark.sql.connect.proto as pb2
from pyspark.sql import Observation, functions as F
from pyspark.sql.connect.plan import SubqueryAlias
o = Observation("o1")
df = spark.range(3).observe(o, F.count(F.lit(1)).alias("n"))
df._plan.observations
SubqueryAlias(df._plan, "s").observations
plan = pb2.Plan(); plan.root.CopyFrom(df._plan.plan(spark.client))
spark.client.to_table(plan, df._plan.observations)
o.get
Actual (drop-in): {}, {}, and o.get is {}.
Expected (pyspark-client 4.3.0): {'o1': o} (the same Observation object),
{'o1': o}, and o.get is
{'n': 3}. DataFrame(SubqueryAlias(df._plan, "s"), spark).collect() sets o on
both already.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]