YEs, since 1.4.0, it shuts down streamingContext without gracefully from
shutdown hook.
You can make it shutdown gracefully in that hook by setting the SparkConf
"spark.streaming.stopGracefullyOnShutdown" to "true"
Note to self, document this in the programming guide.
On Wed, Sep 23, 2015 at 3:33 AM, Bin Wang <wbi...@gmail.com> wrote:
> I'd like the spark application to be stopped gracefully while received
> kill signal, so I add these code:
>
> sys.ShutdownHookThread {
> println("Gracefully stopping Spark Streaming Application")
> ssc.stop(stopSparkContext = true, stopGracefully = true)
> println("Application stopped")
> }
>
> But the application is not stopped gracefully:
>
> 15/09/23 17:44:38 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
> ...
> 15/09/23 17:44:38 INFO streaming.StreamingContext: Invoking
> stop(stopGracefully=false) from shutdown hook
>
> Dose spark auto invoke StreamingContext.stop for me?
>
