[
https://issues.apache.org/jira/browse/SPARK-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-15479.
-------------------------------
Resolution: Not A Problem
Yeah, you told YARN to kill all your containers, so it did that. You did not
shut down your app by telling your driver to start graceful shutdown.
> Spark job doesn't shut gracefully in yarn mode.
> -----------------------------------------------
>
> Key: SPARK-15479
> URL: https://issues.apache.org/jira/browse/SPARK-15479
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.5.1
> Reporter: Rakesh
> Attachments: driver.rtf, executor.rtf
>
>
> Issue i am having is similar to the one mentioned here :
> http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn
> I am creating a rdd from sequence of 1 to 300 and creating streaming RDD out
> of it.
> val rdd = ssc.sparkContext.parallelize(1 to 300)
> val dstream = new ConstantInputDStream(ssc, rdd)
> dstream.foreachRDD{ rdd =>
> rdd.foreach{ x =>
> log(x)
> Thread.sleep(50)
> }
> }
> When i kill this job, i expect elements 1 to 300 to be logged before shutting
> down. It is indeed the case when i run it locally. It waits for the job to
> finish before shutting down.
> But when i launch the job in cluster with "yarn-cluster" mode, it abruptly
> shuts down.
> Executor prints following log
> ERROR executor.CoarseGrainedExecutorBackend:
> Driver xx.xx.xx.xxx:yyyyy disassociated! Shutting down.
> and then it shuts down. It is not a graceful shutdown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
