[
https://issues.apache.org/jira/browse/SPARK-61098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai resolved SPARK-61098.
-----------------------------
Resolution: Duplicate
Synchronous query-start delivery landed in SPARK-61162 through
https://github.com/apache/spark-connect-rust/pull/353 (commit
e8b1d4284e11d119c4fda9563a913ece22f43e7c). This ticket tracks the same behavior.
Local Spark 4.3.0 validation passed the existing native listener regression:
both registered listeners receive the started event before start() returns, and
a callback-owned session can run an action. The separate shutdown and
pending-registration follow-up remains tracked in SPARK-61097.
Assessed by OpenAI Codex.
> Deliver streaming query start callbacks before returning the query
> ------------------------------------------------------------------
>
> Key: SPARK-61098
> URL: https://issues.apache.org/jira/browse/SPARK-61098
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> DataStreamWriter.start() reads the query ID and name from
> WriteStreamOperationStartResult but ignores query_started_event_json. The
> listener dispatcher accepts only progress, idle and termination events.
> Consequently a registered onQueryStarted callback is never called. Official
> PySpark dispatches this callback synchronously before start() returns.
> Deliver the start event exactly once to the session listeners and preserve
> callback error behavior, with regression coverage.
> Observed by source inspection of public apache/spark-connect-rust master
> 83d74462ec41012993176cf3af99017576688eb8. Runtime regression tests will
> accompany the fix.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
