Hyukjin Kwon created SPARK-61208:
------------------------------------
Summary: Return from removeListener only after the event in
delivery reached its listener, so that a removed listener gets no more events,
in the Rust and Python clients as PySpark does
Key: SPARK-61208
URL: https://issues.apache.org/jira/browse/SPARK-61208
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's StreamingQueryListenerBus delivers each event to all its listeners
while it holds its lock, and removeListener takes the same lock. So once
removeListener returns, the removed listener gets no more events: an event that
was being delivered when removeListener was called reaches the listener before
removeListener returns.
The Rust client's listener bus takes a snapshot of its listeners for each event
and delivers it without the lock. A listener that is not the last is removed
locally and at once, so when it is removed while another listener's callback
runs, it still gets that event after removeListener has returned:
A busy in its progress callback; another thread removes B
official 4.3.0: A progress, B progress, removeListener(B) returned
this client: A progress, removeListener(B) returned, B progress
The fix makes a removal from another thread than the bus's wait until the event
in delivery has been delivered, as PySpark's lock does. A removal from a
callback of the bus does not wait, as it would wait for itself.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
