[
https://issues.apache.org/jira/browse/SPARK-60131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60131:
-----------------------------------
Labels: pull-request-available (was: )
> Removing an unregistered Scala Connect streaming listener stops the shared bus
> ------------------------------------------------------------------------------
>
> Key: SPARK-60131
> URL: https://issues.apache.org/jira/browse/SPARK-60131
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.0.0, 5.0.0
> Reporter: Vitthal
> Priority: Minor
> Labels: pull-request-available
>
> Problem
> In the Scala Spark Connect client, removing a streaming listener that was
> never registered can stop event delivery for another registered listener. The
> removal code checks whether one listener remains, but does not check whether
> it is removing that listener before stopping the shared bus.
> How to reproduce
> 1. Register listener A with spark.streams.addListener(A).
> 2. Call spark.streams.removeListener(B), where B is a different listener that
> was never registered.
> 3. The client sends a command to remove the shared server-side listener bus
> and interrupts its event worker.
> Expected behavior
> Removing B should do nothing. Listener A should remain registered and
> continue receiving events. This also matches the membership check already
> used by Python Connect.
> Why this fix is needed
> Removing an unrelated listener should not disable event handling for a
> registered listener. The server cannot prevent this mistake because the
> removal command addresses the shared bus, rather than an individual client
> listener.
> Proposed change and testing
> Check that the listener is registered before stopping the shared bus. Keep
> the existing behavior for removing a registered listener, including the last
> listener.
> A focused test against the production removal method fails before the fix
> because it records an unwanted bus removal command. After the fix, that test,
> the registered-listener removal test, and SparkSessionSuite pass: 12 tests in
> total. Module style checks also pass. The existing live-server streaming
> listener test also fails before the fix because the termination event is not
> delivered, and passes after the fix. It checks progress and termination
> events for both listener interface variants. No tests were ignored in either
> live run.
> Related issues
> SPARK-59847 and SPARK-59623 concern Python Connect deadlocks when removing
> the last registered listener. This issue concerns removing an unregistered
> listener in the Scala client, so the trigger and fix are different.
> SPARK-58000 concerns Scala listener-test flakiness caused by interrupted
> worker cleanup. It is related lifecycle context, but does not address this
> missing membership check.
> Affected versions
> The same removal path is present in the Spark 4.0.0 source. The regression
> was run on the current 5.0.0-SNAPSHOT master checkout; no released-version
> runtime test was run.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]