[
https://issues.apache.org/jira/browse/SPARK-61208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61208:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Return from removeListener only after the event in delivery reached its
> listener, so that a removed listener gets no more events, in the Rust and
> Python clients as PySpark does
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61208
> URL: https://issues.apache.org/jira/browse/SPARK-61208
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark's StreamingQueryListenerBus delivers each event to all its listeners
> while it holds its lock, and removeListener takes the same lock. So once
> removeListener returns, the removed listener gets no more events: an event
> that was being delivered when removeListener was called reaches the listener
> before removeListener returns.
> The Rust client's listener bus takes a snapshot of its listeners for each
> event and delivers it without the lock. A listener that is not the last is
> removed locally and at once, so when it is removed while another listener's
> callback runs, it still gets that event after removeListener has returned:
> A busy in its progress callback; another thread removes B
> official 4.3.0: A progress, B progress, removeListener(B) returned
> this client: A progress, removeListener(B) returned, B progress
> The fix makes a removal from another thread than the bus's wait until the
> event in delivery has been delivered, as PySpark's lock does. A removal from
> a callback of the bus does not wait, as it would wait for itself.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
