Hyukjin Kwon created SPARK-61478:
------------------------------------
Summary: Raise the done error of a progress handler over a decode
error of the results, as PySpark does
Key: SPARK-61478
URL: https://issues.apache.org/jira/browse/SPARK-61478
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Follow-up of SPARK-61398. When a result batch fails to decode, collect and
toLocalIterator return the decode error without finishing the progress, so the
done error of a progress handler goes to sys.unraisablehook later, named as a
LocalRowIterator also for collect. PySpark raises the done error with the
decode error as its context. A stream whose schema changes between batches can
also panic in the scan for invalid UTF-8.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]