Hyukjin Kwon created SPARK-61963:
------------------------------------

             Summary: Give the plan of a DataFrame the Observations of the 
DataFrame by their names, as PySpark's LogicalPlan.observations
                 Key: SPARK-61963
                 URL: https://issues.apache.org/jira/browse/SPARK-61963
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
PySpark's DataFrame._plan.observations is LogicalPlan.observations of the 
DataFrame's plan: the
Observations of observe on the DataFrame and on its inputs (both sides of a 
join and of a set
operation) by their names. PySpark's DataFrame gives them to the executions of 
its client
(client.to_table(query, self._plan.observations), connect/dataframe.py:1982, 
2002, 2228), and a plan
of pyspark.sql.connect.plan over it, such as a SubqueryAlias, has them too 
(plan.py:152-157,
1829-1837). The drop-in's DataFrame._plan, a _DataFramePlan, had none: its 
observations was always
{}. So code that runs the plan of a DataFrame through SparkSession.client, as 
PySpark's DataFrame
does, left the DataFrame's Observations unset.

Repro (Spark 4.3.0 server)
    import pyspark.sql.connect.proto as pb2
    from pyspark.sql import Observation, functions as F
    from pyspark.sql.connect.plan import SubqueryAlias
    o = Observation("o1")
    df = spark.range(3).observe(o, F.count(F.lit(1)).alias("n"))
    df._plan.observations
    SubqueryAlias(df._plan, "s").observations
    plan = pb2.Plan(); plan.root.CopyFrom(df._plan.plan(spark.client))
    spark.client.to_table(plan, df._plan.observations)
    o.get
Actual (drop-in): {}, {}, and o.get is {}.
Expected (pyspark-client 4.3.0): {'o1': o} (the same Observation object), 
{'o1': o}, and o.get is
{'n': 3}. DataFrame(SubqueryAlias(df._plan, "s"), spark).collect() sets o on 
both already.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to