[ 
https://issues.apache.org/jira/browse/SPARK-61939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61939:
-----------------------------------
    Labels: pull-request-available  (was: )

> Replace the placeholders of DataFrames under every relation and in every 
> command that holds one, as PySpark sends their relations
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61939
>                 URL: https://issues.apache.org/jira/browse/SPARK-61939
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> Since SPARK-61856 (#779), SparkSession.client takes PySpark's plans and 
> commands, which code builds
> with pyspark.sql.connect.proto over df._plan.plan(client). The drop-in's 
> placeholder of the
> DataFrame's relation was replaced only as the root of a plan, in the plans of 
> pyspark.ml, under a
> WithRelations or SubqueryAlias, and in the messages of extensions. Under any 
> other relation, or as
> the relation of a command, the placeholder reached the server, which refuses 
> it.
> Repro (drop-in, Spark 4.3.0 server)
>     import pyspark.sql.connect.proto as pb2
>     c = spark.client; df = spark.range(5)
>     p = pb2.Plan(); p.root.limit.input.CopyFrom(df._plan.plan(c)); 
> p.root.limit.limit = 2
>     c.to_table(p, {})
>     cmd = pb2.Command(); 
> cmd.create_dataframe_view.input.CopyFrom(df._plan.plan(c))
>     cmd.create_dataframe_view.name = "v"; cmd.create_dataframe_view.replace = 
> True
>     c.execute_command(cmd)
> Actual: InvalidPlanInput [CONNECT_INVALID_PLAN.NO_HANDLER_FOR_EXTENSION] ... 
> DataFramePlaceholder,
> also for a Join of two DataFrames' plans, a WriteOperation, a 
> CheckpointCommand, and a
> DataFrame(plan, session) of a LogicalPlan that builds such a relation.
> Expected (pyspark-client 4.3.0): the server receives the relations: {'id': 
> [0, 1]} for the Limit,
> a view of 5 rows, and the Observations of the DataFrames under a Join (both 
> sides) receive their
> metrics.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to