Hyukjin Kwon created SPARK-62116:
------------------------------------
Summary: Skip the placeholder search of the plans built in Python
when their bytes do not hold the type URL of a placeholder
Key: SPARK-62116
URL: https://issues.apache.org/jira/browse/SPARK-62116
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
The Python client (pyspark-client-rust) searches the bytes of every plan built
in Python (outside the Rust core) for the placeholders of the relations of
DataFrames that the plan refers to, which it replaces with their relations
before it sends the plan. The search runs even when the bytes hold no
placeholder at all, which costs time proportional to the size of the plan on
every request.
Expected: skip the search when the bytes of the plan do not hold the type URL
of a placeholder, which no plan without one can hold, and send the plan as it
is; the plans that hold placeholders are resolved as before.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]