HyukjinKwon commented on pull request #34505: URL: https://github.com/apache/spark/pull/34505#issuecomment-964656543
Sounds like that's close to the original proposal of #26783. I was skeptical about that idea and it might not be worthwhile because: - We will have to take care of other restrictions, and variants like aggregation, window, etc all together, and thought that might not be worthwhile - I still think that Arrow is rather an internal format instead of user-facing. So these made me propose one (developer) API that doesn’t require considering other restrictions (e.g., the length of input should be the same as output's in case of scalar UDF in `select`) or variants. In addition, I am pretty sure the current API can do what UDFs could do although it might be less optimizing. However, I am open to other suggestions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
