allisonwang-db commented on PR #43369: URL: https://github.com/apache/spark/pull/43369#issuecomment-1769005409
Hi @PhilDakin thanks for doing this! I personally think it's better to have the table here instead of a link to another page. Also, I think we should **explain why this conversion table matters**. For example, it is useful when users what to map a Python return type to a Spark return type in a Python UDF. Another thing we need to mention is type casting. What if I want to cast an int type in Python to a FloatType in Spark? Currently, for regular Python UDF, it will return NULL, I believe, but for arrow-optimized Python UDF, it can cast the value properly. It will be valuable to have a table like this: https://github.com/apache/spark/blob/b41ea9162f4c8fbc4d04d28d6ab5cc0342b88cb0/python/pyspark/sql/udf.py#L94-L119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
