[ 
https://issues.apache.org/jira/browse/SPARK-61532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61532:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Let a signal during a callback end the removal of the last listener before 
> its REMOVE, keep the listener of an add that a signal ends before the server 
> answers, and warn of a failed interrupt of the listener bus, as PySpark does
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61532
>                 URL: https://issues.apache.org/jira/browse/SPARK-61532
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> From a review of SPARK-61470 (Ctrl-C ends the waits of the Python client) and 
> its follow-ups. The listener bus of the Python client still differed from 
> PySpark's in three cases of a signal:
> 1. removeListener of the last listener (and streams.close()) while a listener 
> callback runs on the worker. PySpark's remove acquires the lock of the bus, 
> which post_to_all holds while it calls the listeners, and a signal interrupts 
> that acquire. So PySpark raises KeyboardInterrupt at once, sends no REMOVE, 
> and keeps the listener. Ours sent the REMOVE at once, and then waited for the 
> callback in the join of the worker. The last-listener removal now waits for 
> the delivery of the worker, as the removal of another listener already does, 
> through the bus's wait, which a signal ends.
> 2. A signal during addListener before the server's response headers came: an 
> unreachable or slow server, or the plan compression lookup. That signal 
> dropped the listener from the bus, while PySpark's append, which puts the 
> listener in the bus first, keeps it. The listener now stays, as for a signal 
> during the wait for the acknowledgement, and the server's listener, which the 
> ADD may have created, is removed by the next removal or subscription.
> 3. A failed Interrupt (the server fails the interrupt of the wait) during 
> addListener, removeListener or close was raised as the interrupt's error. 
> PySpark's append and remove catch it with `except Exception`, warn, and 
> return; append also removes the listener from its bus. These calls now warn 
> and return, and addListener drops the registration that the interrupted add 
> kept.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to