[
https://issues.apache.org/jira/browse/SPARK-61097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140753#comment-18140753
]
DB Tsai commented on SPARK-61097:
---------------------------------
Implementation and validation:
https://github.com/apache/spark-connect-rust/pull/384.
The final patch passed 617 Rust core/API unit tests and the offline Python
suites (2,203 passed; eight explicit live-server skips). The 27 selected
lifecycle cases also passed 25 repeated mock-service rounds, totaling 675
passing executions. These are repeated executions of the same cases. Nine
native live cases and their 27 setup/call/teardown reports passed with the
imported extension checked before and after; three exact real-worker Rust
integration tests passed. A retained-bus mutation was detected by the final
ownership-drop sentinel while the unmodified control passed.
The scoped official raw-transport comparison recorded 48 passing outcomes. The
separately executed full raw-transport suite recorded 2,586 passing outcomes,
with 641 skips and one deselection across 87 normal files, and the four-module
cluster suite recorded 15 passing invocations with 14 skips. Both completed
without failures, errors or retries. These official suites use official Python
plans and converters over Rust RPC transport. Native listener behavior is
supported by the separate native and real-worker checks; the official suites do
not establish complete high-level compatibility.
Clippy remains an advisory failure (exit 101). The separate metadata-column
drift check failed because the test catalog plugin was unavailable; that result
and earlier failed attempts remain recorded separately from the successful
normal suite. Linux and Windows were not run locally. The scope boundaries in
the description remain applicable.
Prepared by OpenAI Codex.
> Cancel pending streaming listener registration and avoid callback deadlocks
> during shutdown
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-61097
> URL: https://issues.apache.org/jira/browse/SPARK-61097
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The initial shared session manager and synchronous query-start delivery
> landed in SPARK-61162 (PR #353). This follow-up proposes the remaining
> listener lifecycle fixes in
> https://github.com/apache/spark-connect-rust/pull/384.
> Terminal SparkSession.stop() cancels local listener registration and dispatch
> without joining callbacks, releases the server session, and closes the client
> even when release fails. SparkConnectClient.close() cancels local
> registration and dispatch without releasing the session. Neither terminal
> path performs a listener-removal RPC. The current event's listener snapshot
> may finish after terminal return; later events are not dispatched. A
> registration whose RPC event stream is established but not yet acknowledged
> can be cancelled; initial stream establishment retains the transport deadline
> and retry policy.
> Lifecycle RPCs and worker joins run outside the state lock. Ordinary failed
> listener removal or close preserves registrations and worker state for retry,
> and generations prevent an old worker from clearing newer listeners. Real
> workers hold weak bus references so releasing the last native session or
> manager can release a quiet subscription. Regression coverage exercises
> callback reentrancy, terminal races, removal retry, stale generations,
> progress-handler lock reentrancy, cancellation wiring and real-worker
> ownership release.
> The fix retains upstream synchronous-start, duplicate-registration and
> foreign-session removal semantics. Listener-held Python session cycles and
> event-stream error warning parity remain limitations. This follow-up
> preserves connect-rust-4.3.0 release tracking.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
