[ 
https://issues.apache.org/jira/browse/SPARK-62064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62064:
-----------------------------------
    Labels: pull-request-available  (was: )

> Send the relations of DataFrames that the extensions of expressions hold, and 
> with the changes a plan makes to them, and resolve the messages of extensions 
> in a time of their length, as PySpark sends its plans
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62064
>                 URL: https://issues.apache.org/jira/browse/SPARK-62064
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Problem
> df._plan.plan(session) gives a placeholder that holds the relation of the 
> DataFrame, which the core
> replaces with that relation where a plan built in Python holds it 
> (plan::resolve_relation,
> resolve_command). Three gaps were left:
> 1. The extensions of expressions. The core did not look into the expressions 
> of a plan built in
>    Python, so a placeholder in the message of a plugin's expression 
> (Expression.extension) was sent
>    as a placeholder, where PySpark's plan sends the relation.
> 2. A placeholder that a plan changes. PySpark's relation of plan(session) is 
> a message that a plan
>    or a plugin may change, such as relation.common.plan_id = ...:
>    - as a relation of a plan, the core replaced the placeholder with its 
> relation and dropped the
>      change, so a column of the changed plan ID failed with 
> CANNOT_RESOLVE_DATAFRAME_COLUMN;
>    - in the message of an extension, the core found a placeholder by its 
> exact encoding, which the
>      change alters, and sent it as a placeholder.
> 3. The time of the messages of extensions. The core read the message of an 
> extension of a relation
>    or a command level by level, and at each level searched all of its bytes 
> for the type URL of a
>    placeholder and copied them again. A crafted message that nests many 
> messages took a time of its
>    length times its depth (#785): in a release build, 0.29 s for 8,000 levels 
> and 20.9 s for
>    64,000; in a debug build, 41 to 69 s for 30,000 levels. PySpark sends the 
> bytes as they are.
> Repro (master 80046f2f, against a Spark 4.3.0 server)
>   import pyspark.sql.connect.proto as pb2
>   client = spark.client
>   plan = pb2.Plan()
>   plan.root.project.input.CopyFrom(spark.range(3)._plan.plan(client))
>   plan.root.project.input.common.plan_id = 424242
>   column = plan.root.project.expressions.add().unresolved_attribute
>   column.unparsed_identifier = "id"; column.plan_id = 424242
>   client.to_table(plan, {})[0].to_pydict()
> Actual (drop-in)
>   AnalysisException: [CANNOT_RESOLVE_DATAFRAME_COLUMN] Cannot resolve 
> dataframe column "id".
> Expected (pyspark-client 4.3.0)
>   {'id': [0, 1, 2]}
> For an expression extension, 
> ExamplePluginRelation(input=df._plan.plan(client)) packed into a
> project's expression is received with the input as the placeholder, where 
> PySpark's carries the
> relation. (Checked with the recording mock server; a server has no handler of 
> such an expression.)
> Fix
> - Of each relation built in Python of a kind that holds expressions (a 
> projection, a filter, a
>   join, an aggregation, ...; an exhaustive match of the relation types), and 
> of each command built
>   in Python but an extension or an ML command, which are resolved as before, 
> the fields other than
>   the inputs are encoded, and read as messages for a placeholder only when 
> that encoding holds the
>   type URL of one (plan::resolve_own_fields). The relations of the 
> placeholders, which the core
>   builds, are not encoded, and nor are the kinds that hold no expressions: 
> reads, limits, set
>   operations, local data, ranges, catalog operations, ...
> - A placeholder whose common is set, as a relation and in the message of an 
> extension, is
>   replaced by its relation with that common merged, as protobuf's MergeFrom 
> does.
> - plan::resolve_encoded finds the offsets of the type URL once, reads a field 
> only when its value
>   holds one, and writes the resolved encoding once, of the lengths computed 
> first.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to