Hyukjin Kwon created SPARK-61939:
------------------------------------

             Summary: Replace the placeholders of DataFrames under every 
relation and in every command that holds one, as PySpark sends their relations
                 Key: SPARK-61939
                 URL: https://issues.apache.org/jira/browse/SPARK-61939
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Problem
Since SPARK-61856 (#779), SparkSession.client takes PySpark's plans and 
commands, which code builds
with pyspark.sql.connect.proto over df._plan.plan(client). The drop-in's 
placeholder of the
DataFrame's relation was replaced only as the root of a plan, in the plans of 
pyspark.ml, under a
WithRelations or SubqueryAlias, and in the messages of extensions. Under any 
other relation, or as
the relation of a command, the placeholder reached the server, which refuses it.

Repro (drop-in, Spark 4.3.0 server)
    import pyspark.sql.connect.proto as pb2
    c = spark.client; df = spark.range(5)
    p = pb2.Plan(); p.root.limit.input.CopyFrom(df._plan.plan(c)); 
p.root.limit.limit = 2
    c.to_table(p, {})
    cmd = pb2.Command(); 
cmd.create_dataframe_view.input.CopyFrom(df._plan.plan(c))
    cmd.create_dataframe_view.name = "v"; cmd.create_dataframe_view.replace = 
True
    c.execute_command(cmd)
Actual: InvalidPlanInput [CONNECT_INVALID_PLAN.NO_HANDLER_FOR_EXTENSION] ... 
DataFramePlaceholder,
also for a Join of two DataFrames' plans, a WriteOperation, a 
CheckpointCommand, and a
DataFrame(plan, session) of a LogicalPlan that builds such a relation.

Expected (pyspark-client 4.3.0): the server receives the relations: {'id': [0, 
1]} for the Limit,
a view of 5 rows, and the Observations of the DataFrames under a Join (both 
sides) receive their
metrics.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to