hzmsh commented on PR #59558: URL: https://github.com/apache/spark/pull/59558#issuecomment-6221394253
Hi @HyukjinKwon and @gaogaotiantian, could you take a look when you have a moment? This continues [#39974](https://github.com/apache/spark/pull/39974), an earlier version of which HyukjinKwon approved before it went stale, and it changes the `pandas_udf` type annotations. Thank you @wayneguow for the original PR and for letting me pick it up! Compared with it, the ten per-function-shape overloads are replaced by three `udf`-style ones that take any data type, and the `StructType` for `SCALAR_ITER` and `MapType` for `GROUPED_AGG` additions are taken back out because they type-checked UDFs that fail at runtime. The `**kwargs` change also affects `udf` results; I'm happy to split it into its own JIRA, and to add the same overloads for `arrow_udf` in a follow-up. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
