Hyukjin Kwon created SPARK-61162:
------------------------------------
Summary: Share the streaming query listener bus of a session, post
the started events of queries, and give the listeners the session, in the Rust
and Python clients, as PySpark does
Key: SPARK-61162
URL: https://issues.apache.org/jira/browse/SPARK-61162
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect's spark.streams is one StreamingQueryManager of the session,
whose StreamingQueryListenerBus holds the client-side listeners:
- addListener gives the listener the session (listener.spark), and, for the
first listener, returns once the server has added its listener
- start() of a query posts its started event, which the server sends with the
result of the start (query_started_event_json), to the listeners before it
returns
- removeListener of the last listener asks the server to remove its listener
and waits for the events left
In the Rust client, each spark.streams() / session.streams() is a new manager
with a new listener bus of its own, and in the Python client spark.streams is a
new object at each access:
- onQueryStarted is never called: the client ignores query_started_event_json
- listeners have no session (self.spark is None), so a listener that runs
queries fails
- a listener added through one spark.streams is removed through another only
because removing from an empty bus asks the server to remove the listener of
the session, which also cuts the other listeners off
- addListener returns before the server has added its listener, so a query
started right after misses its events
- spark.streams == spark.streams is False, and StreamingQueryManager has no
_sqlb (the bus of the listeners, which Apache Spark's tests read)
Six tests of Apache Spark's test_parity_listener.py and
test_query_manager_no_recreation of test_parity_streaming.py fail on this
client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
