[ 
https://issues.apache.org/jira/browse/SPARK-61881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61881:
---------------------------------
    Fix Version/s: connect-rust-4.4.0

> Pickle the functions of UDFs, UDTFs and the map and group-map functions when 
> the plan is built, as PySpark does
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61881
>                 URL: https://issues.apache.org/jira/browse/SPARK-61881
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.4.0
>
>
> The drop-in cloudpickles the functions of UDFs, pandas UDFs, UDTFs and the 
> map and group-map functions (mapInPandas, mapInArrow, applyInPandas, 
> applyInArrow, cogroup, applyInPandasWithState, transformWithState) when they 
> are called. PySpark 4.3.0 pickles them each time it builds a plan 
> (PythonUDF.to_plan, PythonUDTF.to_plan). So a global that the function reads, 
> changed after the UDF is applied, is seen by PySpark's next action but not by 
> the drop-in's, and an unpicklable function fails at the call in the drop-in, 
> where PySpark fails at the first request (df.schema or collect) with 
> PicklingError, or UDTF_SERIALIZATION_ERROR for UDTFs, and sends nothing.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to