Would the following change work for you ? diff --git a/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala b/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala index 61b5a4c..c330d25 100644 --- a/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala +++ b/core/src/main/scala/org/apache/spark/util/AsynchronousListenerBus.scala @@ -66,6 +66,7 @@ private[spark] abstract class AsynchronousListenerBus[L <: AnyRef, E](name: Stri self.synchronized { processingEvent = true } + if (stopped.get()) return try { val event = eventQueue.poll if (event == null) {
On Sat, Nov 7, 2015 at 12:17 PM, vonnagy <i...@vadio.com> wrote: > If I have a streaming job (Spark 1.5.1) and attempt to stop the stream > after > the first batch, the system locks up and never completes. The pseudo code > below shows that after the batch complete notification is called the stream > is stopped. I have traced the lockup to the call `listener.stop()`in > JobScheduler (line 114) which attempts to join the thread in > AsynchronousListenerBus. That thread never ends because it is still getting > messages `SparkListenerExecutorMetricsUpdate` from the DAGScheduler. The > thread never ends because the events continue to come in. > > Any thoughts/ideas on how I can effectively stop the stream after the first > batch would greatly appreciated. > > Psuedo Example: > > class SomeJob { > > val ssc = createStreamingContext() > val listener = new MyListener(ssc) > ssc.addStreamingListener(listener) > > val stream = getStream > > stream.foreachRDD { rdd => > // Do something with the data > } > } > > class MyListener(ctx: StreamingContext) extends StreamingListener { > override def onBatchCompleted(batchCompleted: > StreamingListenerBatchCompleted) = synchronized { > ctx.stop(false, false) > // NOTE: I get the same results with ctx.stop(), ctx.stop(true), > ctx.stop(true, true), or ctx.stop(false, false) > } > } > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Calling-stop-on-StreamingContext-locks-up-tp15063.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >