[
https://issues.apache.org/jira/browse/SPARK-44856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng reassigned SPARK-44856:
-------------------------------------
Assignee: Michael Zhang
> Improve Python UDTF arrow serializer performance
> ------------------------------------------------
>
> Key: SPARK-44856
> URL: https://issues.apache.org/jira/browse/SPARK-44856
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 3.5.0, 4.0.0
> Reporter: Allison Wang
> Assignee: Michael Zhang
> Priority: Major
>
> Currently, there is a lot of overhead in the arrow serializer for Python
> UDTFs. The overhead is largely from converting arrow batches into pandas
> series and converting UDTF's results back to a pandas dataframe.
> We should try directly converting Python object into arrow and vice versa to
> avoid the expensive pandas conversion. Similar to this converter:
> [https://github.com/apache/spark/blob/be04ac1ace91f6da34b08a1510e41d3ab6f0377b/python/pyspark/sql/connect/conversion.py#L56]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
