HyukjinKwon opened a new pull request, #248: URL: https://github.com/apache/spark-connect-rust/pull/248
### What changes were proposed in this pull request? [SPARK-61048](https://issues.apache.org/jira/browse/SPARK-61048): `spark.udf` now has the `sparkSession` of PySpark Connect's `UDFRegistration`, and `registerJavaFunction` sends its return type as PySpark does. - `spark.udf.sparkSession` is the session. - `spark.udf.registerJavaFunction(name, javaClassName, returnType)` sends its return type in the `JavaUDF` of the `RegisterFunction` command: - a DDL string as an unparsed data type, which the server parses - a DataType as itself - It raises `UNSUPPORTED_OPERATION` of "data type ..." for a return type that is neither. - It sends the function as not deterministic, as PySpark's `CommonInlineUserDefinedFunction` is by default. The core `SparkSession::register_java_function` now sends its DDL string unparsed. The new `register_java_function_with` takes a DataType. ### Why are the changes needed? The client named the session `spark_session`. It also parsed the `simpleString` of the return type in the client: - An invalid DDL string raised `INVALID_DATATYPE_FORMAT` rather than the `PARSE_SYNTAX_ERROR` of the server. - A return type that is not a DataType raised an `AttributeError`. ### Does this PR introduce _any_ user-facing change? Yes. `spark.udf` has the `sparkSession`, and `registerJavaFunction` the return types and errors, of PySpark 4.3.0. ### How was this patch tested? - A new live test checks `spark.udf.sparkSession`, the `PARSE_SYNTAX_ERROR` of an invalid DDL return type, and the `UNSUPPORTED_OPERATION` of a return type that is not a DataType. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the `JavaUDF`s that `registerJavaFunction` and `registerJavaUDAF` send to a mock Connect server: of a DDL string, a DataType and no return type, and not deterministic. - The lib tests of both crates (574 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1811 passed, 8 skipped), and the live drop-in and Arrow suites (110 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
