Hi firemonk9, What you're doing looks interesting. Can you share some more details? Are you running the same spark context for each job, or are you running a seperate spark context for each job? Does your system need sharing of rdd's across multiple jobs? If yes, how do you implement that? Also what prompted you to run Yarn instead of standalone? Does this give some performance benefit? Have you evaluated yarn vs mesos? Also have you looked at spark jobserver by ooyala? It makes doing some if the stuff I mentioned easier. IIRC it also works with yarn. Definitely works with Mesos. Heres the link https://github.com/spark-jobserver/spark-jobserver
Thanks Anshul On 23 Apr 2015 20:32, "Dean Wampler" <deanwamp...@gmail.com> wrote: > I strongly recommend spawning a new process for the Spark jobs. Much > cleaner separation. Your driver program won't be clobbered if the Spark job > dies, etc. It can even watch for failures and restart. > > In the Scala standard library, the sys.process package has classes for > constructing and interoperating with external processes. Perhaps Java has > something similar these days? > > dean > > Dean Wampler, Ph.D. > Author: Programming Scala, 2nd Edition > <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) > Typesafe <http://typesafe.com> > @deanwampler <http://twitter.com/deanwampler> > http://polyglotprogramming.com > > On Tue, Apr 21, 2015 at 2:15 PM, Steve Loughran <ste...@hortonworks.com> > wrote: > >> >> On 21 Apr 2015, at 17:34, Richard Marscher <rmarsc...@localytics.com> >> wrote: >> >> - There are System.exit calls built into Spark as of now that could kill >> your running JVM. We have shadowed some of the most offensive bits within >> our own application to work around this. You'd likely want to do that or to >> do your own Spark fork. For example, if the SparkContext can't connect to >> your cluster master node when it is created, it will System.exit. >> >> >> people can block "errant" System.exit calls by running under a >> SecurityManager. Less than ideal (and there's a small performance hit) -but >> possible >> > >
