HyukjinKwon opened a new pull request, #323: URL: https://github.com/apache/spark-connect-rust/pull/323
### What changes were proposed in this pull request? [SPARK-61125](https://issues.apache.org/jira/browse/SPARK-61125): The handler class of a UDTF is pickled when the UDTF is used or registered, and a handler that does not pickle raises `UDTF_SERIALIZATION_ERROR`, as PySpark's. PySpark Connect pickles the handler of a UDTF as it builds the `PythonUDTF` of the plan. For a `pickle.PicklingError`, it raises a `PySparkPicklingError` of `UDTF_SERIALIZATION_ERROR`, of the name of the UDTF and "Please check the stack trace and make sure the function is serializable." `UserDefinedTableFunction.command` now pickles the handler when it is read, and raises that error. ### Why are the changes needed? The client pickled the handler when `udtf` decorated the class, and raised the `PicklingError` as it was. So a handler that does not pickle raised where the UDTF was defined, rather than where it was used, and with another type. Apache Spark's `test_udtf_pickle_error` failed on this client. ### Does this PR introduce _any_ user-facing change? Yes. A UDTF of a handler that does not pickle raises `UDTF_SERIALIZATION_ERROR` when it is used, as with PySpark 4.3.0. ### How was this patch tested? - A new offline test checks the error of a handler that refers to a file opened for writing, against a mock Connect server, when the UDTF is called and when it is registered. - Apache Spark's `test_udtf_pickle_error` of `test_parity_udtf.py` passes on this client against a local server, in its three variants. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2179 passed, 8 skipped), and the live drop-in and Arrow suites (174 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
