[ 
https://issues.apache.org/jira/browse/SPARK-61342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61342:
-----------------------------------
    Labels: pull-request-available  (was: )

> Raise the exception of the done call of a progress handler from the 
> execution, as PySpark does
> ----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61342
>                 URL: https://issues.apache.org/jira/browse/SPARK-61342
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> When a progress handler raised on its done call (done=True), the Python 
> drop-in returned the result of the execution (collect, toPandas, toArrow, 
> toLocalIterator, count, first, show, writes, spark.sql, catalog calls, 
> foreachPartition) and sent the exception to sys.unraisablehook, and the Rust 
> API's fallible handlers' done-call error was dropped. PySpark 4.3.0 makes the 
> done call in Progress.__exit__, inside the with of the execution 
> (client/core.py _execute_and_fetch and to_table_as_iterator), so the 
> exception is raised from the execution, and replaces the execution's own 
> error when there is one.
> Also, the executions that PySpark runs without a Progress (the registrations 
> of functions, table functions, data sources and Java functions, through 
> _execute, and pipeline runs, through execute_command_as_iterator) called the 
> handlers in the drop-in; and the drop-in's removal of a server listener 
> before a new add, which PySpark does not send, did too.
> The fix makes the done call when the stream of the execution ends, also of an 
> error or before the stream starts, and raises its error from the execution; 
> an execution dropped before its end (an iterator closed early) reports it as 
> before, as PySpark's generator does. The executions that PySpark runs without 
> a Progress no longer call the handlers.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to