HyukjinKwon opened a new pull request, #205: URL: https://github.com/apache/spark-connect-rust/pull/205
### What changes were proposed in this pull request? [SPARK-61003](https://issues.apache.org/jira/browse/SPARK-61003): `spark.udtf.register(name, f)` now registers the UDTF `f` as `name` on the server, as PySpark's `UDTFRegistration.register` does, so that SQL calls it. It sends a `register_table_function` command of the `CommonInlineUserDefinedTableFunction` of the UDTF, with its handler, return type, eval type and determinism and without arguments, and returns `f` as it is. As PySpark's, it raises `CANNOT_REGISTER_UDTF` for an `f` that is not a UDTF, and `INVALID_UDTF_EVAL_TYPE` for one of another eval type. `spark.udtf` is the `UDTFRegistration` of the session. The core session has `register_table_function` for this. This is based on #204, whose commit is the first one here. The test of the construction of a UDTF no longer registers one, as it now needs a session. ### Why are the changes needed? `spark.udtf.register` only renamed the UDTF object, so the server did not know the name. For a registered UDTF, `spark.sql("SELECT * FROM name(2)")` raised `UNRESOLVABLE_TABLE_VALUED_FUNCTION` where PySpark 4.3.0 returns its rows. It also took a handler class rather than a UDTF. ### Does this PR introduce _any_ user-facing change? Yes. Registered UDTFs can be called from SQL, as in PySpark 4.3.0. ### How was this patch tested? - A new live test registers a UDTF and an `arrow_udtf`, and calls them from SQL. It also passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the `register_table_function` command that a mock Connect server receives, and the errors of `register`. It fails without this change. - A new lib test checks the `CommonInlineUserDefinedTableFunction` of a registration. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (665 passed, and 125 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1159 passed, 8 skipped), and the live drop-in and Arrow suites (85 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
