DB Tsai created SPARK-61097:
-------------------------------
Summary: Preserve a session streaming listener manager across
property access
Key: SPARK-61097
URL: https://issues.apache.org/jira/browse/SPARK-61097
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Repeated access to SparkSession.streams constructs independent
StreamingQueryManager instances and listener registries. For example,
spark.streams.addListener(listener) followed by spark.streams.close() closes a
different empty registry; removeListener also addresses a different registry.
Official PySpark caches the session manager. Preserve listener state for one
session while keeping new sessions isolated, and close the bus with the
session. Add regressions for repeated property access, listener removal and
cleanup.
Observed by source inspection of public apache/spark-connect-rust master
83d74462ec41012993176cf3af99017576688eb8. Runtime regression tests will
accompany the fix.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
