got it! is it livy.spark.deploy-mode=yarn-cluster or livy.spark.deploy-mode = cluster ? Sorry to ask this question. I couldn't find it in docs or the comments in livy.conf and I am using livy 0.4.0
On Tue, Mar 20, 2018 at 5:01 PM, Meisam Fathi <meisam.fa...@gmail.com> wrote: > If you are running on cluster mode, the application should keep running on > YRAN. > > On Tue, Mar 20, 2018 at 3:34 PM kant kodali <kanth...@gmail.com> wrote: > >> @Meisam Fathi I am running with yarn and zookeeper as a state store. I >> spawned a job via livy that reads from kafka and writes to Kafka >> but the moment I kill the livy server the job also is getting killed. not >> sure why? I believe once the livy server crashes the spark context also >> get's killed so do I need to need to set the livy.spark.deploy.mode ? if >> so, what value should I set it to? >> >> >> On Mon, Mar 12, 2018 at 12:30 PM, Meisam Fathi <meisam.fa...@gmail.com> >> wrote: >> >>> On YARN, your application keeps running even if the launcher fails. So >>> after recovery, Livy reconnects to the application. On Spark standalone, I >>> am not sure what happens to the application of the launcher fails. >>> >>> Thanks, >>> Meisam >>> >>> On Mon, Mar 12, 2018 at 10:34 AM kant kodali <kanth...@gmail.com> wrote: >>> >>>> can someone please explain how YARN helps here? And why not spark >>>> master? >>>> >>>> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto < >>>> m.durighe...@miriade.it> wrote: >>>> >>>>> >>>>> >>>>> 2018-03-12 9:58 GMT+01:00 kant kodali <kanth...@gmail.com>: >>>>> >>>>>> Sorry I see there is a recovery mode and also I can set state store >>>>>> to zookeeper but looks like I need YARN? because I get the error message >>>>>> below >>>>>> >>>>>> "requirement failed: Session recovery requires YARN" >>>>>> >>>>>> >>>>>> I am using spark standalone and I don't use YARN anywhere in my >>>>>> cluster. is there any other option for recovery in this case? >>>>>> >>>>>> >>>>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <kanth...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> When my live server crashes it looks like all my spark jobs are >>>>>>> gone. I am trying to see how I can make it more resilient? other words, >>>>>>> I >>>>>>> would like spark jobs that were spawned by Livy to be running even if my >>>>>>> Livy server crashes because in theory Livy server can crash anytime and >>>>>>> Spark Jobs should run for weeks or months in my case. How can I achieve >>>>>>> this? >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>> Hello, >>>>> to enable recovery in Livy you need Spark on YARN >>>>> >>>>> ( https://spark.apache.org/docs/latest/running-on-yarn.html ) >>>>> >>>>> >>>>> >>>>> Kind Regards >>>>> >>>> >>>> >>
