uncleGen commented on a change in pull request #28781:
URL: https://github.com/apache/spark/pull/28781#discussion_r437984275
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingQueryListenerBus.scala
##########
@@ -35,12 +35,12 @@ import org.apache.spark.util.ListenerBus
* and StreamingQueryManager. So this bus will dispatch events to registered
listeners for only
* those queries that were started in the associated SparkSession.
*/
-class StreamingQueryListenerBus(sparkListenerBus: LiveListenerBus)
+class StreamingQueryListenerBus(sparkListenerBus: Option[LiveListenerBus])
Review comment:
Make `sparkListenerBus `optional. When loaded by History Server, spark
use the `ReplayListenerBus` instead of `LiveListenerBus`. In live ui, streaming
query post the event into `StreamingQueryListenerBus`, and the
`StreamingQueryListenerBus` will post these events into `LiveListenerBus`.
`StreamingQueryListenerBus` also subscribes to `LiveListenerBus` events thus
getting back the posted event in a different thread. In history ui,
`StreamingQueryListenerBus` will subscribes to the `ReplayListenerBus`, and
process those events through `onOtherEvent()` directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
