Allison Wang created SPARK-44559:
------------------------------------
Summary: Improve error messages for invalid Python UDTF arrow type
casts
Key: SPARK-44559
URL: https://issues.apache.org/jira/browse/SPARK-44559
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 3.5.0
Reporter: Allison Wang
Currently, if a Python UDTF outputs a type that is incompatible with the
specified output schema, Spark will throw the following confusing error message:
{code:java}
File "pyarrow/array.pxi", line 1044, in pyarrow.lib.Array.from_pandas
File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert [1, 2] with type list: tried to
convert to int32{code}
We should improve this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
