Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/16186#discussion_r91356341
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingQueryListenerBus.scala
---
@@ -35,12 +43,24 @@ class StreamingQueryListenerBus(sparkListenerBus:
LiveListenerBus)
sparkListenerBus.addListener(this)
/**
- * Post a StreamingQueryListener event to the Spark listener bus
asynchronously. This event will
- * be dispatched to all StreamingQueryListener in the thread of the
Spark listener bus.
+ * RunIds of active queries whose events are supposed to be forwarded by
this ListenerBus
+ * to registered `StreamingQueryListeners`. Note that this list needs to
be maintained separately
+ * from the `StreamingQueryManager.activeQueries` because a terminated
query is cleared from
+ * `StreamingQueryManager.activeQueries` as soon as it is stopped, but
the this ListenerBus must
+ * clear a query only after the termination event of that query has been
posted.
+ */
+ private val activeQueryRunIds = new mutable.HashSet[UUID]
--- End diff --
+1
It's worth to document it. This is different from other Spark's listener
buses because of the SPARK-18764 `synchronous` event.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
