HyukjinKwon opened a new pull request, #212: URL: https://github.com/apache/spark-connect-rust/pull/212
### What changes were proposed in this pull request? [SPARK-60962](https://issues.apache.org/jira/browse/SPARK-60962): user-defined types now pickle as Python objects, and are made with the arguments of their `__init__`, as PySpark's do. - As PySpark's `UserDefinedType`, a Python object, a UDT pickles as its class and its state. cloudpickle sends a class of `__main__` by value, and the state keeps the parameters of the instance. It pickled as its JSON, which names the class by its module, and is rebuilt without arguments. - As PySpark's `DataType`, whose `__new__` is `object.__new__`, a subclass of `DataType`, such as a UDT, is made with the arguments of its `__init__`. ### Why are the changes needed? A UDT that `__main__` defines, as in a notebook or a script, failed as the return type of a UDF: the worker rebuilt it from its JSON, by importing its module, which `__main__` on the worker does not define. A UDT whose `__init__` keeps parameters was rebuilt without them. Even making one failed: `PointUDT(10.0)` raised `TypeError: DataType.__new__() takes 0 positional arguments but 1 was given`. ### Does this PR introduce _any_ user-facing change? Yes. A UDT of `__main__`, and of parameters, is a UDF return type, as in PySpark 4.3.0. ### How was this patch tested? - A new live test runs, in a new interpreter, a UDF whose return type is a UDT that `__main__` defines with a parameter, and collects its values. It also passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks that a UDT of parameters pickles and copies with them. It also loads, here, a UDT that another interpreter's `__main__` defined and pickled. It fails without this change. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1328 passed, 8 skipped), and the live drop-in and Arrow suites (87 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
