Hyukjin Kwon created SPARK-61481:
------------------------------------
Summary: Let a second stop() release the session after an
interrupted one, and let a signal end the waits of the streaming listener bus,
in the Python client as PySpark does
Key: SPARK-61481
URL: https://issues.apache.org/jira/browse/SPARK-61481
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Two differences of the Python client's stop() and streaming listener bus from
PySpark 4.3.0, which the review of SPARK-61353 and SPARK-61315 found:
1. Concurrent stop(): the core SparkSession::stop marked the session stopped
before the release, so a second thread's stop() returned at once. If a signal
then interrupted the first stop()'s release (which keeps the session open, as
PySpark's does), the second caller had been told the session stopped while it
stayed open. PySpark runs stop() under SparkSession._lock (an RLock), so a
second stop() waits, and after the first one's KeyboardInterrupt it releases
and closes the session itself; is_stopped is client.is_closed.
2. The removal of the last streaming listener (and close()) waited on a Condvar
for another thread's started-event delivery, which no signal could interrupt.
PySpark's remove waits on the bus's threading.Lock, whose acquire a signal
interrupts.
Fix: serialize stop() per session with a re-entrant lock whose wait a signal
ends in the Python client, and mark the session stopped after its release; wait
for the listener bus through the same abortable wait, leaving the bus as it was
when the wait is interrupted.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]