[
https://issues.apache.org/jira/browse/SPARK-61125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61125:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Pickle the handler of a UDTF when it is used, and raise
> UDTF_SERIALIZATION_ERROR, in the Python client as PySpark does
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61125
> URL: https://issues.apache.org/jira/browse/SPARK-61125
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect pickles the handler class of a UDTF when the UDTF is used, as
> it builds the {{PythonUDTF}} of the plan, and raises a
> {{PySparkPicklingError}} of {{UDTF_SERIALIZATION_ERROR}} ("Please check the
> stack trace and make sure the function is serializable.") for a
> {{pickle.PicklingError}}. Apache Spark's {{test_udtf_pickle_error}} checks
> it, with a handler that refers to a file opened for writing.
> The Python client pickles the handler when {{udtf}} decorates the class, and
> raises the {{pickle.PicklingError}} as it is. So the error is raised where
> the UDTF is defined, rather than where it is used, and of another type.
> The fix is to pickle the handler when the UDTF is used, and to raise
> {{UDTF_SERIALIZATION_ERROR}}, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
