[
https://issues.apache.org/jira/browse/SPARK-61939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61939:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Replace the placeholders of DataFrames under every relation and in every
> command that holds one, as PySpark sends their relations
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61939
> URL: https://issues.apache.org/jira/browse/SPARK-61939
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> Problem
> Since SPARK-61856 (#779), SparkSession.client takes PySpark's plans and
> commands, which code builds
> with pyspark.sql.connect.proto over df._plan.plan(client). The drop-in's
> placeholder of the
> DataFrame's relation was replaced only as the root of a plan, in the plans of
> pyspark.ml, under a
> WithRelations or SubqueryAlias, and in the messages of extensions. Under any
> other relation, or as
> the relation of a command, the placeholder reached the server, which refuses
> it.
> Repro (drop-in, Spark 4.3.0 server)
> import pyspark.sql.connect.proto as pb2
> c = spark.client; df = spark.range(5)
> p = pb2.Plan(); p.root.limit.input.CopyFrom(df._plan.plan(c));
> p.root.limit.limit = 2
> c.to_table(p, {})
> cmd = pb2.Command();
> cmd.create_dataframe_view.input.CopyFrom(df._plan.plan(c))
> cmd.create_dataframe_view.name = "v"; cmd.create_dataframe_view.replace =
> True
> c.execute_command(cmd)
> Actual: InvalidPlanInput [CONNECT_INVALID_PLAN.NO_HANDLER_FOR_EXTENSION] ...
> DataFramePlaceholder,
> also for a Join of two DataFrames' plans, a WriteOperation, a
> CheckpointCommand, and a
> DataFrame(plan, session) of a LogicalPlan that builds such a relation.
> Expected (pyspark-client 4.3.0): the server receives the relations: {'id':
> [0, 1]} for the Limit,
> a view of 5 rows, and the Observations of the DataFrames under a Join (both
> sides) receive their
> metrics.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]