[ 
https://issues.apache.org/jira/browse/SPARK-62111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62111:
-----------------------------------
    Labels: pull-request-available  (was: )

> Make the pickles and copies of DataFrames, Columns and table arguments carry 
> their UDF functions and Python plans, which each request builds and pickles 
> again, as PySpark does
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62111
>                 URL: https://issues.apache.org/jira/browse/SPARK-62111
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> The Python client (pyspark-client-rust) differs from pyspark-client 4.3.0 in 
> how it pickles and copies DataFrames, Columns and table arguments, as 
> dev/parity/known-differences.md lists:
> - A pickled Column or DataFrame carries the commands that its UDF functions 
> pickled to when it was pickled, where PySpark's carries the functions, which 
> its requests pickle again. So a value that a function reads, changed after 
> the pickle, is not seen; pickle.dumps of a Column of a UDF of a lambda 
> pickles where PySpark raises the PicklingError of the lambda; and a function 
> that does not pickle by value raises its PicklingError where PySpark pickles 
> it by reference.
> - A deep copy of a DataFrame into another session carries the commands of the 
> functions, where PySpark's holds copies of the functions.
> - DataFrame(plan, session) builds the relation of the plan when it is made, 
> so a plan that fails to build (a UDF that does not pickle, an option that 
> UTF-8 cannot encode) raises at construction, where PySpark keeps the plan and 
> raises at its first request. A pickle of such a DataFrame cannot carry the 
> plan.
> - An unpickled Column or table argument keeps the call sites of the 
> expressions only partly: with PySpark's origin capture on (an active 
> session), the unpickle of a sort order, a CASE WHEN or a window specification 
> records the call site of the unpickle, where PySpark's unpickled expressions 
> keep their origins.
> - Pickles of plans over a DataFrame's _plan, and of their Columns and table 
> arguments, fail without a session or on the server's Python worker.
> Expected: as pyspark-client, the pickles and copies carry the functions and 
> the Python plans, which each request builds and pickles again with the 
> request's session; a plan that fails to build raises its error at each 
> request, not at construction, of a fresh exception per request; unpickled 
> objects keep their origins; and the pickles load on the server's Python 
> worker as PySpark Connect's.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to