[
https://issues.apache.org/jira/browse/SPARK-61342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61342:
------------------------------------
Assignee: Hyukjin Kwon
> Raise the exception of the done call of a progress handler from the
> execution, as PySpark does
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-61342
> URL: https://issues.apache.org/jira/browse/SPARK-61342
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> When a progress handler raised on its done call (done=True), the Python
> drop-in returned the result of the execution (collect, toPandas, toArrow,
> toLocalIterator, count, first, show, writes, spark.sql, catalog calls,
> foreachPartition) and sent the exception to sys.unraisablehook, and the Rust
> API's fallible handlers' done-call error was dropped. PySpark 4.3.0 makes the
> done call in Progress.__exit__, inside the with of the execution
> (client/core.py _execute_and_fetch and to_table_as_iterator), so the
> exception is raised from the execution, and replaces the execution's own
> error when there is one.
> Also, the executions that PySpark runs without a Progress (the registrations
> of functions, table functions, data sources and Java functions, through
> _execute, and pipeline runs, through execute_command_as_iterator) called the
> handlers in the drop-in; and the drop-in's removal of a server listener
> before a new add, which PySpark does not send, did too.
> The fix makes the done call when the stream of the execution ends, also of an
> error or before the stream starts, and raises its error from the execution;
> an execution dropped before its end (an iterator closed early) reports it as
> before, as PySpark's generator does. The executions that PySpark runs without
> a Progress no longer call the handlers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]