[
https://issues.apache.org/jira/browse/SPARK-61881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61881:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Pickle the functions of UDFs, UDTFs and the map and group-map functions when
> the plan is built, as PySpark does
> ---------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61881
> URL: https://issues.apache.org/jira/browse/SPARK-61881
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> The drop-in cloudpickles the functions of UDFs, pandas UDFs, UDTFs and the
> map and group-map functions (mapInPandas, mapInArrow, applyInPandas,
> applyInArrow, cogroup, applyInPandasWithState, transformWithState) when they
> are called. PySpark 4.3.0 pickles them each time it builds a plan
> (PythonUDF.to_plan, PythonUDTF.to_plan). So a global that the function reads,
> changed after the UDF is applied, is seen by PySpark's next action but not by
> the drop-in's, and an unpicklable function fails at the call in the drop-in,
> where PySpark fails at the first request (df.schema or collect) with
> PicklingError, or UDTF_SERIALIZATION_ERROR for UDTFs, and sends nothing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]