HyukjinKwon opened a new pull request, #209: URL: https://github.com/apache/spark-connect-rust/pull/209
### What changes were proposed in this pull request? [SPARK-61006](https://issues.apache.org/jira/browse/SPARK-61006): a call of a Python UDF or UDTF now passes its arguments as PySpark's `UserDefinedFunction` and `UserDefinedTableFunction` do. - Each keyword argument is sent as a `NamedArgumentExpression`, after the positional arguments. So `plus(a=lit(1), b=col("x"))` calls `plus` with `a` and `b`, and its column is `Column<'plus(a => 1, b => x)'>`. - A string argument of a UDTF call is a column name, as for a UDF. The core expressions have a `NamedArgument` for these. This is based on #206, whose commits are the first ones here. ### Why are the changes needed? The keyword arguments of a UDF call were dropped, so the call failed on the server with a `PythonException` of the missing arguments, and its column was `Column<'plus()'>`. A UDTF call with keyword arguments raised `TypeError`. A string argument of a UDTF call was taken as a literal, so `udtf_call("x")` returned a row of `'x'` where PySpark 4.3.0 raises `UNRESOLVED_COLUMN`. ### Does this PR introduce _any_ user-facing change? Yes. UDF and UDTF calls take the keyword arguments of PySpark 4.3.0, and a UDTF call takes column names. ### How was this patch tested? - A new live test calls a UDF, a pandas UDF and a UDTF with keyword arguments, alone and after positional ones, and a UDTF with a column name. It also passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks, against a mock Connect server, the named arguments and the column references that UDF and UDTF calls send, and the columns of UDF calls. It fails without this change. - A new lib test checks the rendering and the proto of the named argument. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (665 passed, and 125 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1161 passed, 8 skipped), and the live drop-in and Arrow suites (88 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
