[ 
https://issues.apache.org/jira/browse/SPARK-61540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61540:
-----------------------------------
    Labels: pull-request-available  (was: )

> Let the removal of the last streaming query listener wait for the delivery in 
> flight before it removes the server's listener, as PySpark does
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61540
>                 URL: https://issues.apache.org/jira/browse/SPARK-61540
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> PySpark's StreamingQueryListenerBus.remove acquires the lock of the bus, 
> which post_to_all holds while it calls the listeners, before it sends the 
> REMOVE of the last listener; a signal interrupts that acquire. So, while a 
> listener callback runs, PySpark's removeListener (and streams.close()) of the 
> last listener waits for the callback, and a Ctrl-C raises KeyboardInterrupt 
> at once with no REMOVE sent and the listener still registered and receiving 
> events.
> The Python client sent the REMOVE at once while the callback ran, and then 
> waited for the worker in the join (abortable since SPARK-61481's follow-up). 
> So after a Ctrl-C, the server's listener was removed, and the listener 
> stopped getting events.
> The removal of the last listener from another thread than the worker now 
> waits for the delivery of the worker before it removes the server's listener, 
> as the removal of another listener already did. The bus is Stopping 
> meanwhile, as for a removal in flight. So a callback of that delivery that 
> removes a listener or closes the bus coalesces with the removal, which owns 
> the result, and one that adds a listener is refused, as before; one that 
> stops the session leaves nothing to remove. A signal ends the wait in the 
> Python client before anything is removed or sent.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to