[
https://issues.apache.org/jira/browse/SPARK-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15869368#comment-15869368
]
Eyal Zituny commented on SPARK-19594:
-------------------------------------
that will work but i will have to remove the "final" from the "postToAll"
method which is part of spark core
another option can be to change the method post(event:
StreamingQueryListener.Event):
def post(event: StreamingQueryListener.Event) {
event match {
case s: QueryStartedEvent =>
activeQueryRunIds.synchronized { activeQueryRunIds += s.runId }
sparkListenerBus.post(s)
// post to local listeners to trigger callbacks
postToAll(s)
case t: QueryTerminatedEvent =>
// run all the listeners synchronized before removing the id from the
list
postToAll(t)
activeQueryRunIds.synchronized { activeQueryRunIds -= t.runId }
case _ =>
sparkListenerBus.post(event)
}
}
> StreamingQueryListener fails to handle QueryTerminatedEvent if more then one
> listeners exists
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-19594
> URL: https://issues.apache.org/jira/browse/SPARK-19594
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.1.0
> Reporter: Eyal Zituny
> Priority: Minor
>
> reproduce:
> *create a spark session
> *add multiple streaming query listeners
> *create a simple query
> *stop the query
> result -> only the first listener handle the QueryTerminatedEvent
> this might happen because the query run id is being removed from
> activeQueryRunIds once the onQueryTerminated is called
> (StreamingQueryListenerBus:115)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
