Hyukjin Kwon created SPARK-62064:
------------------------------------
Summary: Send the relations of DataFrames that the extensions of
expressions hold, and with the changes a plan makes to them, and resolve the
messages of extensions in a time of their length, as PySpark sends its plans
Key: SPARK-62064
URL: https://issues.apache.org/jira/browse/SPARK-62064
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
df._plan.plan(session) gives a placeholder that holds the relation of the
DataFrame, which the core
replaces with that relation where a plan built in Python holds it
(plan::resolve_relation,
resolve_command). Three gaps were left:
1. The extensions of expressions. The core did not look into the expressions of
a plan built in
Python, so a placeholder in the message of a plugin's expression
(Expression.extension) was sent
as a placeholder, where PySpark's plan sends the relation.
2. A placeholder that a plan changes. PySpark's relation of plan(session) is a
message that a plan
or a plugin may change, such as relation.common.plan_id = ...:
- as a relation of a plan, the core replaced the placeholder with its
relation and dropped the
change, so a column of the changed plan ID failed with
CANNOT_RESOLVE_DATAFRAME_COLUMN;
- in the message of an extension, the core found a placeholder by its exact
encoding, which the
change alters, and sent it as a placeholder.
3. The time of the messages of extensions. The core read the message of an
extension of a relation
or a command level by level, and at each level searched all of its bytes for
the type URL of a
placeholder and copied them again. A crafted message that nests many
messages took a time of its
length times its depth (#785): in a release build, 0.29 s for 8,000 levels
and 20.9 s for
64,000; in a debug build, 41 to 69 s for 30,000 levels. PySpark sends the
bytes as they are.
Repro (master 80046f2f, against a Spark 4.3.0 server)
import pyspark.sql.connect.proto as pb2
client = spark.client
plan = pb2.Plan()
plan.root.project.input.CopyFrom(spark.range(3)._plan.plan(client))
plan.root.project.input.common.plan_id = 424242
column = plan.root.project.expressions.add().unresolved_attribute
column.unparsed_identifier = "id"; column.plan_id = 424242
client.to_table(plan, {})[0].to_pydict()
Actual (drop-in)
AnalysisException: [CANNOT_RESOLVE_DATAFRAME_COLUMN] Cannot resolve dataframe
column "id".
Expected (pyspark-client 4.3.0)
{'id': [0, 1, 2]}
For an expression extension, ExamplePluginRelation(input=df._plan.plan(client))
packed into a
project's expression is received with the input as the placeholder, where
PySpark's carries the
relation. (Checked with the recording mock server; a server has no handler of
such an expression.)
Fix
- Of each relation built in Python of a kind that holds expressions (a
projection, a filter, a
join, an aggregation, ...; an exhaustive match of the relation types), and of
each command built
in Python but an extension or an ML command, which are resolved as before,
the fields other than
the inputs are encoded, and read as messages for a placeholder only when that
encoding holds the
type URL of one (plan::resolve_own_fields). The relations of the
placeholders, which the core
builds, are not encoded, and nor are the kinds that hold no expressions:
reads, limits, set
operations, local data, ranges, catalog operations, ...
- A placeholder whose common is set, as a relation and in the message of an
extension, is
replaced by its relation with that common merged, as protobuf's MergeFrom
does.
- plan::resolve_encoded finds the offsets of the type URL once, reads a field
only when its value
holds one, and writes the resolved encoding once, of the lengths computed
first.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]