HyukjinKwon opened a new pull request, #321: URL: https://github.com/apache/spark-connect-rust/pull/321
### What changes were proposed in this pull request? [SPARK-61128](https://issues.apache.org/jira/browse/SPARK-61128): PySpark's `ArrayType`, `MapType`, `StructField` and `StructType` hold the type objects they are made of. So `ArrayType(udt).elementType is udt`, for a user-defined type `udt`, and a type pickles of those objects. The client's types now do the same: - **Giving back the user-defined types.** The client lowers its types into the Rust core. It now keeps each user-defined type it lowers, keyed by the JSON of its core type, and gives that object back wherever the type is turned back into a Python object: - the children of `ArrayType`, `MapType`, `StructField` and `StructType`, also nested - their `repr` and `simpleString`, and the conversions of their values A user-defined type that the client did not lower, such as one of the schema of a result, is still rebuilt from its JSON, as PySpark's `UserDefinedType.fromJson` does. - **Pickling.** A type with a user-defined type pickles as its class of `pyspark.sql.types` with its type objects, as PySpark's types do, and the server's Python workers rebuild it. A type without one still pickles by its JSON. - **`StructField` pickles**, of its name, type, nullability and metadata. Before, it could not be pickled at all. ### Why are the changes needed? The client rebuilt a user-defined type from its JSON, by importing its `pyClass`. That name need not be the module of the class: for Apache Spark's `ExamplePointUDT` it is `pyspark.sql.tests.ExamplePointUDT`, while the class is in `pyspark.testing.objects`. So on the client, these raised `PySparkValueError: [UNSUPPORTED_OPERATION] unpickling user defined types is not supported`: - `ArrayType(ExamplePointUDT()).elementType`, and the type of `StructField("p", ExamplePointUDT())` - their `repr` - `udf(returnType=ArrayType(ExamplePointUDT()))`, so Apache Spark's `test_udf_with_udt` failed on this client Pickled by its JSON, such a type would fail the same way on the server's Python worker. `pickle.dumps(StructField("a", IntegerType()))` raised `TypeError: cannot pickle 'pyspark.sql.types.StructField' object`. ### Does this PR introduce _any_ user-facing change? Yes: - The types give back the user-defined types they are made of, and pickle with them, as with PySpark 4.3.0. - A `StructField` can be pickled. ### How was this patch tested? - A new offline test checks: - the user-defined types that `ArrayType`, `MapType` (key and value), `StructField` and `StructType` give back, also nested, and their `repr` and `simpleString` - the round trips of these types with `pickle` and cloudpickle, also of a `StructField` with metadata and one without a user-defined type - what a type with a user-defined type pickles of The cases give the same results with official PySpark 4.3.0. - A new live test runs UDFs of an array of `ExamplePointUDT` on the server. It fails without this change. - Apache Spark's `test_parity_udf.py`, `test_parity_serde.py`, `test_parity_creation.py` and `test_parity_udf_combinations.py` run on this client against a local server. - `test_udf_with_udt` now passes, and so do the tests of user-defined types in `arrow/test_parity_arrow.py`, `arrow/test_parity_arrow_python_udf.py` and `test_parity_dataframe.py`. - The one other failure, `test_udf_with_string_return_type`, fails without this change too, because the client's `Row` is not a tuple. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2179 passed, 8 skipped), and the live drop-in and Arrow suites (175 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
