[
https://issues.apache.org/jira/browse/SPARK-55448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-55448:
-----------------------------------
Labels: pull-request-available (was: )
> Spark Connect query execution events dropped when session closes while query
> running
> --------------------------------------------------------------------------------------
>
> Key: SPARK-55448
> URL: https://issues.apache.org/jira/browse/SPARK-55448
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 3.4.4, 4.0.2, 3.5.8, 4.1.1
> Reporter: Biruk Tesfaye
> Priority: Major
> Labels: pull-request-available
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> There is a race condition in Spark Connect's execution lifecycle where
> SparkListenerBus query events (Cancelled/Closed) aren't sent if a session
> closes while a query is still running.
> The root cause is ExecuteEventsManager.assertStatus(), which validates that
> the session status is Started before allowing any event to be sent. When
> trying to close a session, the thread interrupts all execution threads for
> the session and proceeds to close the session, marking the session as closed.
> Each of the interrupted threads' cleanup path tries to send its terminal
> event which would fail if the main thread marked the session as closed
> before, causing an IllegalStateException that prevents the events from ever
> reaching the listener bus.
> The proposed fix allows ExecuteEventsManager.assertStatus() to accept
> SessionStatus.Closed alongside Started so terminal events can still be sent
> during execution cleanup. This is safe because the underlying SparkContext
> and its listener bus outlive the Spark Connect session, and all downstream
> operations after the assert are independent of session state. The assert
> continues to block SessionStatus.Pending as a guard against events on
> uninitialized sessions.
>
> A more robust solution might be to introduce a SessionStatus.Closing state
> that exists until all execution threads are cleaned up then transitions to
> SessionStatus.Closed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
