Hyukjin Kwon created SPARK-62111:
------------------------------------

             Summary: Make the pickles and copies of DataFrames, Columns and 
table arguments carry their UDF functions and Python plans, which each request 
builds and pickles again, as PySpark does
                 Key: SPARK-62111
                 URL: https://issues.apache.org/jira/browse/SPARK-62111
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


The Python client (pyspark-client-rust) differs from pyspark-client 4.3.0 in 
how it pickles and copies DataFrames, Columns and table arguments, as 
dev/parity/known-differences.md lists:

- A pickled Column or DataFrame carries the commands that its UDF functions 
pickled to when it was pickled, where PySpark's carries the functions, which 
its requests pickle again. So a value that a function reads, changed after the 
pickle, is not seen; pickle.dumps of a Column of a UDF of a lambda pickles 
where PySpark raises the PicklingError of the lambda; and a function that does 
not pickle by value raises its PicklingError where PySpark pickles it by 
reference.
- A deep copy of a DataFrame into another session carries the commands of the 
functions, where PySpark's holds copies of the functions.
- DataFrame(plan, session) builds the relation of the plan when it is made, so 
a plan that fails to build (a UDF that does not pickle, an option that UTF-8 
cannot encode) raises at construction, where PySpark keeps the plan and raises 
at its first request. A pickle of such a DataFrame cannot carry the plan.
- An unpickled Column or table argument keeps the call sites of the expressions 
only partly: with PySpark's origin capture on (an active session), the unpickle 
of a sort order, a CASE WHEN or a window specification records the call site of 
the unpickle, where PySpark's unpickled expressions keep their origins.
- Pickles of plans over a DataFrame's _plan, and of their Columns and table 
arguments, fail without a session or on the server's Python worker.

Expected: as pyspark-client, the pickles and copies carry the functions and the 
Python plans, which each request builds and pickles again with the request's 
session; a plan that fails to build raises its error at each request, not at 
construction, of a fresh exception per request; unpickled objects keep their 
origins; and the pickles load on the server's Python worker as PySpark 
Connect's.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to