[
https://issues.apache.org/jira/browse/SPARK-44559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takuya Ueshin reassigned SPARK-44559:
-------------------------------------
Assignee: Allison Wang
> Improve error messages for Python UDTF arrow type casts
> -------------------------------------------------------
>
> Key: SPARK-44559
> URL: https://issues.apache.org/jira/browse/SPARK-44559
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 3.5.0
> Reporter: Allison Wang
> Assignee: Allison Wang
> Priority: Major
> Fix For: 3.5.0
>
>
> Currently, if a Python UDTF outputs a type that is incompatible with the
> specified output schema, Spark will throw the following confusing error
> message:
> {code:java}
> File "pyarrow/array.pxi", line 1044, in pyarrow.lib.Array.from_pandas
> File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
> File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
> File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Could not convert [1, 2] with type list: tried to
> convert to int32{code}
> We should improve this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
