[
https://issues.apache.org/jira/browse/SPARK-62064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62064:
------------------------------------
Assignee: Hyukjin Kwon
> Send the relations of DataFrames that the extensions of expressions hold, and
> with the changes a plan makes to them, and resolve the messages of extensions
> in a time of their length, as PySpark sends its plans
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62064
> URL: https://issues.apache.org/jira/browse/SPARK-62064
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Problem
> df._plan.plan(session) gives a placeholder that holds the relation of the
> DataFrame, which the core
> replaces with that relation where a plan built in Python holds it
> (plan::resolve_relation,
> resolve_command). Three gaps were left:
> 1. The extensions of expressions. The core did not look into the expressions
> of a plan built in
> Python, so a placeholder in the message of a plugin's expression
> (Expression.extension) was sent
> as a placeholder, where PySpark's plan sends the relation.
> 2. A placeholder that a plan changes. PySpark's relation of plan(session) is
> a message that a plan
> or a plugin may change, such as relation.common.plan_id = ...:
> - as a relation of a plan, the core replaced the placeholder with its
> relation and dropped the
> change, so a column of the changed plan ID failed with
> CANNOT_RESOLVE_DATAFRAME_COLUMN;
> - in the message of an extension, the core found a placeholder by its
> exact encoding, which the
> change alters, and sent it as a placeholder.
> 3. The time of the messages of extensions. The core read the message of an
> extension of a relation
> or a command level by level, and at each level searched all of its bytes
> for the type URL of a
> placeholder and copied them again. A crafted message that nests many
> messages took a time of its
> length times its depth (#785): in a release build, 0.29 s for 8,000 levels
> and 20.9 s for
> 64,000; in a debug build, 41 to 69 s for 30,000 levels. PySpark sends the
> bytes as they are.
> Repro (master 80046f2f, against a Spark 4.3.0 server)
> import pyspark.sql.connect.proto as pb2
> client = spark.client
> plan = pb2.Plan()
> plan.root.project.input.CopyFrom(spark.range(3)._plan.plan(client))
> plan.root.project.input.common.plan_id = 424242
> column = plan.root.project.expressions.add().unresolved_attribute
> column.unparsed_identifier = "id"; column.plan_id = 424242
> client.to_table(plan, {})[0].to_pydict()
> Actual (drop-in)
> AnalysisException: [CANNOT_RESOLVE_DATAFRAME_COLUMN] Cannot resolve
> dataframe column "id".
> Expected (pyspark-client 4.3.0)
> {'id': [0, 1, 2]}
> For an expression extension,
> ExamplePluginRelation(input=df._plan.plan(client)) packed into a
> project's expression is received with the input as the placeholder, where
> PySpark's carries the
> relation. (Checked with the recording mock server; a server has no handler of
> such an expression.)
> Fix
> - Of each relation built in Python of a kind that holds expressions (a
> projection, a filter, a
> join, an aggregation, ...; an exhaustive match of the relation types), and
> of each command built
> in Python but an extension or an ML command, which are resolved as before,
> the fields other than
> the inputs are encoded, and read as messages for a placeholder only when
> that encoding holds the
> type URL of one (plan::resolve_own_fields). The relations of the
> placeholders, which the core
> builds, are not encoded, and nor are the kinds that hold no expressions:
> reads, limits, set
> operations, local data, ranges, catalog operations, ...
> - A placeholder whose common is set, as a relation and in the message of an
> extension, is
> replaced by its relation with that common merged, as protobuf's MergeFrom
> does.
> - plan::resolve_encoded finds the offsets of the type URL once, reads a field
> only when its value
> holds one, and writes the resolved encoding once, of the lengths computed
> first.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]