Issue i am having is similar to the one mentioned here : http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn
I am creating a rdd from sequence of 1 to 300 and creating streaming RDD out of it. val rdd = ssc.sparkContext.parallelize(1 to 300) val dstream = new ConstantInputDStream(ssc, rdd) dstream.foreachRDD{ rdd => rdd.foreach{ x => log(x) Thread.sleep(50) } } When i kill this job, i expect elements 1 to 300 to be logged before shutting down. It is indeed the case when i run it locally. It wait for the job to finish before shutting down. But when i launch the job in custer with "yarn-cluster" mode, it abruptly shuts down. Executor prints following log ERROR executor.CoarseGrainedExecutorBackend: Driver xx.xx.xx.xxx:yyyyy disassociated! Shutting down. and then it shuts down. It is not a graceful shutdown. Anybody knows how to do it in yarn ?