Hyukjin Kwon created SPARK-61219:
------------------------------------
Summary: spark.stop() drops the terminated events of queries still
running at stop, because it cancels the listener bus before ReleaseSession
Key: SPARK-61219
URL: https://issues.apache.org/jira/browse/SPARK-61219
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
spark.stop() cancels the listener bus before it releases the session. The
server sends onQueryTerminated for the queries still running when it releases
the session, so listeners never receive it after stop(). Official PySpark 4.3.0
releases the session before it closes the client, and its listener gets the
terminated event (after stop, the probe prints ['start', 'term']; ours printed
['start'] in every run).
stop() now releases the server session first, then cancels the listener bus and
closes the client. It stays idempotent, and it keeps the listener teardown of
SPARK-61097 (no callback waits, no later event dispatched).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
