Hyukjin Kwon created SPARK-61342:
------------------------------------

             Summary: Raise the exception of the done call of a progress 
handler from the execution, as PySpark does
                 Key: SPARK-61342
                 URL: https://issues.apache.org/jira/browse/SPARK-61342
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


When a progress handler raised on its done call (done=True), the Python drop-in 
returned the result of the execution (collect, toPandas, toArrow, 
toLocalIterator, count, first, show, writes, spark.sql, catalog calls, 
foreachPartition) and sent the exception to sys.unraisablehook, and the Rust 
API's fallible handlers' done-call error was dropped. PySpark 4.3.0 makes the 
done call in Progress.__exit__, inside the with of the execution 
(client/core.py _execute_and_fetch and to_table_as_iterator), so the exception 
is raised from the execution, and replaces the execution's own error when there 
is one.
Also, the executions that PySpark runs without a Progress (the registrations of 
functions, table functions, data sources and Java functions, through _execute, 
and pipeline runs, through execute_command_as_iterator) called the handlers in 
the drop-in; and the drop-in's removal of a server listener before a new add, 
which PySpark does not send, did too.
The fix makes the done call when the stream of the execution ends, also of an 
error or before the stream starts, and raises its error from the execution; an 
execution dropped before its end (an iterator closed early) reports it as 
before, as PySpark's generator does. The executions that PySpark runs without a 
Progress no longer call the handlers.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to