Hyukjin Kwon created SPARK-62116:
------------------------------------

             Summary: Skip the placeholder search of the plans built in Python 
when their bytes do not hold the type URL of a placeholder
                 Key: SPARK-62116
                 URL: https://issues.apache.org/jira/browse/SPARK-62116
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The Python client (pyspark-client-rust) searches the bytes of every plan built 
in Python (outside the Rust core) for the placeholders of the relations of 
DataFrames that the plan refers to, which it replaces with their relations 
before it sends the plan. The search runs even when the bytes hold no 
placeholder at all, which costs time proportional to the size of the plan on 
every request.

Expected: skip the search when the bytes of the plan do not hold the type URL 
of a placeholder, which no plan without one can hold, and send the plan as it 
is; the plans that hold placeholders are resolved as before.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to