Hi, May I ask the reason for using the same spark application? Is it because of the time it takes in order to start a spark context?
On another note you may want to look at the number of contributors in a github repo before choosing a solution. Regards, Gourav On Tue, Feb 7, 2017 at 5:26 PM, vincent gromakowski < vincent.gromakow...@gmail.com> wrote: > Spark jobserver or Livy server are the best options for pure technical API. > If you want to publish business API you will probably have to build you > own app like the one I wrote a year ago https://github.com/elppc/akka- > spark-experiments > It combines Akka actors and a shared Spark context to serve concurrent > subsecond jobs > > > 2017-02-07 15:28 GMT+01:00 ayan guha <guha.a...@gmail.com>: > >> I think you are loking for livy or spark jobserver >> >> On Wed, 8 Feb 2017 at 12:37 am, Cosmin Posteuca < >> cosmin.poste...@gmail.com> wrote: >> >>> I want to run different jobs on demand with same spark context, but i >>> don't know how exactly i can do this. >>> >>> I try to get current context, but seems it create a new spark >>> context(with new executors). >>> >>> I call spark-submit to add new jobs. >>> >>> I run code on Amazon EMR(3 instances, 4 core & 16GB ram / instance), >>> with yarn as resource manager. >>> >>> My code: >>> >>> val sparkContext = SparkContext.getOrCreate() >>> val content = 1 to 40000 >>> val result = sparkContext.parallelize(content, 5) >>> result.map(value => value.toString).foreach(loop) >>> >>> def loop(x: String): Unit = { >>> for (a <- 1 to 30000000) { >>> >>> } >>> } >>> >>> spark-submit: >>> >>> spark-submit --executor-cores 1 \ >>> --executor-memory 1g \ >>> --driver-memory 1g \ >>> --master yarn \ >>> --deploy-mode cluster \ >>> --conf spark.dynamicAllocation.enabled=true \ >>> --conf spark.shuffle.service.enabled=true \ >>> --conf spark.dynamicAllocation.minExecutors=1 \ >>> --conf spark.dynamicAllocation.maxExecutors=3 \ >>> --conf spark.dynamicAllocation.initialExecutors=3 \ >>> --conf spark.executor.instances=3 \ >>> >>> If i run twice spark-submit it create 6 executors, but i want to run all >>> this jobs on same spark application. >>> >>> How can achieve adding jobs to an existing spark application? >>> >>> I don't understand why SparkContext.getOrCreate() don't get existing >>> spark context. >>> >>> >>> Thanks, >>> >>> Cosmin P. >>> >> -- >> Best Regards, >> Ayan Guha >> > >
