Hyukjin Kwon created SPARK-61481:
------------------------------------

             Summary: Let a second stop() release the session after an 
interrupted one, and let a signal end the waits of the streaming listener bus, 
in the Python client as PySpark does
                 Key: SPARK-61481
                 URL: https://issues.apache.org/jira/browse/SPARK-61481
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Two differences of the Python client's stop() and streaming listener bus from 
PySpark 4.3.0, which the review of SPARK-61353 and SPARK-61315 found:

1. Concurrent stop(): the core SparkSession::stop marked the session stopped 
before the release, so a second thread's stop() returned at once. If a signal 
then interrupted the first stop()'s release (which keeps the session open, as 
PySpark's does), the second caller had been told the session stopped while it 
stayed open. PySpark runs stop() under SparkSession._lock (an RLock), so a 
second stop() waits, and after the first one's KeyboardInterrupt it releases 
and closes the session itself; is_stopped is client.is_closed.

2. The removal of the last streaming listener (and close()) waited on a Condvar 
for another thread's started-event delivery, which no signal could interrupt. 
PySpark's remove waits on the bus's threading.Lock, whose acquire a signal 
interrupts.

Fix: serialize stop() per session with a re-entrant lock whose wait a signal 
ends in the Python client, and mark the session stopped after its release; wait 
for the listener bus through the same abortable wait, leaving the bus as it was 
when the wait is interrupted.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to