HyukjinKwon commented on pull request #31735: URL: https://github.com/apache/spark/pull/31735#issuecomment-791156444
I am sorry I wasn't clear. I meant to compare how it works in regular Python UDF and the pandas UDFs. I would be great if we can use a similar approach for both. Furthermore, we will probably have to do it for `toPandas` and `createDataFrame` with Arrow optimization on. It should be best to think about these cases as well. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
