Yes, what Sandy said. On top of that, I would suggest filing a bug for a new command line argument for spark-submit to make the launcher process exit cleanly as soon as a cluster job starts successfully. That can be helpful for code that launches Spark jobs but monitors the job through different means.
On Thu, Sep 18, 2014 at 7:37 AM, Sandy Ryza <sandy.r...@cloudera.com> wrote: > Hi Tobias, > > YARN cluster mode should have the behavior you're looking for. The client > process will stick around to report on things, but should be able to be > killed without affecting the application. If this isn't the behavior you're > observing, and your application isn't failing for a different reason, > there's a bug. > > -Sandy > > On Thu, Sep 18, 2014 at 10:20 AM, Nicholas Chammas > <nicholas.cham...@gmail.com> wrote: >> >> Dunno about having the application be independent of whether spark-submit >> is still alive, but you can have spark-submit run in a new session in Linux >> using setsid. >> >> That way even if you terminate your SSH session, spark-submit will keep >> running independently. Of course, if you terminate the host running >> spark-submit, you will still have problems. >> >> >> On Thu, Sep 18, 2014 at 4:19 AM, Tobias Pfeiffer <t...@preferred.jp> wrote: >>> >>> Hi, >>> >>> I am wondering: Is it possible to run spark-submit in a mode where it >>> will start an application on a YARN cluster (i.e., driver and executors run >>> on the cluster) and then forget about it in the sense that the Spark >>> application is completely independent from the host that ran the >>> spark-submit command and will not be affected if that controlling machine >>> shuts down etc.? I was using spark-submit with YARN in cluster mode, but >>> spark-submit stayed in the foreground and as far as I understood, it >>> terminated the application on the cluster when spark-submit was Ctrl+C'ed. >>> >>> Thanks >>> Tobias >> >> > -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
