HyukjinKwon opened a new pull request, #221: URL: https://github.com/apache/spark-connect-rust/pull/221
### What changes were proposed in this pull request? [SPARK-61018](https://issues.apache.org/jira/browse/SPARK-61018): the UDFs of the map and apply functions are now named after their functions, as PySpark's `UserDefinedFunction` names them. The name is `func.__name__`, or the name of the class of a callable object without one. This covers `mapInPandas` and `mapInArrow`, the `applyInPandas`, `applyInArrow` and `applyInPandasWithState` of `GroupedData`, and the `applyInPandas` and `applyInArrow` of `cogroup`. It also covers `transformWithState` and `transformWithStateInPandas`, whose UDF is the `transformWithStateUDF` of PySpark's utilities, as for PySpark. ### Why are the changes needed? The server shows the name in the plan. `df.mapInArrow(my_func, "id long").explain()` shows `MapInArrow my_func(id#0L)#3, [id#4L], false` for PySpark, but showed `MapInArrow mapInArrow(id#0L)#3, [id#4L], false`, because the UDF was named after the method. `transformWithState` named the UDF `transformWithState` rather than `transformWithStateUDF`. ### Does this PR introduce _any_ user-facing change? Yes. Plans and their explanations name these UDFs as PySpark 4.3.0 does. ### How was this patch tested? - A new live test checks the names in the plans that `explain` shows for `mapInArrow` and `applyInPandas`. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the names sent to a mock Connect server for each of the functions, including a lambda and a callable object. It fails without this change. - The plans of the five functions of a probe match those of official PySpark 4.3.0. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (671 passed, and 129 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1786 passed, 8 skipped), and the live drop-in and Arrow suites (95 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
