Hyukjin Kwon created SPARK-61939:
------------------------------------
Summary: Replace the placeholders of DataFrames under every
relation and in every command that holds one, as PySpark sends their relations
Key: SPARK-61939
URL: https://issues.apache.org/jira/browse/SPARK-61939
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Problem
Since SPARK-61856 (#779), SparkSession.client takes PySpark's plans and
commands, which code builds
with pyspark.sql.connect.proto over df._plan.plan(client). The drop-in's
placeholder of the
DataFrame's relation was replaced only as the root of a plan, in the plans of
pyspark.ml, under a
WithRelations or SubqueryAlias, and in the messages of extensions. Under any
other relation, or as
the relation of a command, the placeholder reached the server, which refuses it.
Repro (drop-in, Spark 4.3.0 server)
import pyspark.sql.connect.proto as pb2
c = spark.client; df = spark.range(5)
p = pb2.Plan(); p.root.limit.input.CopyFrom(df._plan.plan(c));
p.root.limit.limit = 2
c.to_table(p, {})
cmd = pb2.Command();
cmd.create_dataframe_view.input.CopyFrom(df._plan.plan(c))
cmd.create_dataframe_view.name = "v"; cmd.create_dataframe_view.replace =
True
c.execute_command(cmd)
Actual: InvalidPlanInput [CONNECT_INVALID_PLAN.NO_HANDLER_FOR_EXTENSION] ...
DataFramePlaceholder,
also for a Join of two DataFrames' plans, a WriteOperation, a
CheckpointCommand, and a
DataFrame(plan, session) of a LogicalPlan that builds such a relation.
Expected (pyspark-client 4.3.0): the server receives the relations: {'id': [0,
1]} for the Limit,
a view of 5 rows, and the Observations of the DataFrames under a Join (both
sides) receive their
metrics.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]