Looks like currently there's no way for Spark Streaming to restart automatically in yarn-client mode, because in yarn-client mode, AM and driver are two processes, Yarn only control the restart of AM, not driver, so it is not supported in yarn-client mode.
You can write some scripts to monitor your driver process, if it is gone just restart it. Thanks Saisai On Fri, Oct 23, 2015 at 1:04 PM, y <[email protected]> wrote: > I'm managing Spark Streaming applications which run on Cloud Dataproc > (https://cloud.google.com/dataproc/). Spark Streaming applications running > on a Cloud Dataproc cluster seem to run in client mode on YARN. > > Some of my applications sometimes stop due to the application failure. > > I'd like YARN to restart the stopped Spark Streaming applications > automatically but I don't how to achieve it. I tried to set a value to > "yarn.resourcemanager.am.max-attempts" and restarted YARN, but it didn't > restart the failed application automatically. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-restart-a-failed-Spark-Streaming-Application-automatically-in-client-mode-on-YARN-tp25172.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
