Response for vincent: Thanks for answer!
Yes, i need a business solution, that's the reason why i can't use Spark jobserver or Livy solutions. I will look on your github to see how to build such a system. But i don't understand, why spark doesn't have a solution for this kind of problem? and why can't i get the existing context and run some code on it? Thanks 2017-02-07 19:26 GMT+02:00 vincent gromakowski < vincent.gromakow...@gmail.com>: > Spark jobserver or Livy server are the best options for pure technical API. > If you want to publish business API you will probably have to build you > own app like the one I wrote a year ago https://github.com/elppc/akka- > spark-experiments > It combines Akka actors and a shared Spark context to serve concurrent > subsecond jobs > > > 2017-02-07 15:28 GMT+01:00 ayan guha <guha.a...@gmail.com>: > >> I think you are loking for livy or spark jobserver >> >> On Wed, 8 Feb 2017 at 12:37 am, Cosmin Posteuca < >> cosmin.poste...@gmail.com> wrote: >> >>> I want to run different jobs on demand with same spark context, but i >>> don't know how exactly i can do this. >>> >>> I try to get current context, but seems it create a new spark >>> context(with new executors). >>> >>> I call spark-submit to add new jobs. >>> >>> I run code on Amazon EMR(3 instances, 4 core & 16GB ram / instance), >>> with yarn as resource manager. >>> >>> My code: >>> >>> val sparkContext = SparkContext.getOrCreate() >>> val content = 1 to 40000 >>> val result = sparkContext.parallelize(content, 5) >>> result.map(value => value.toString).foreach(loop) >>> >>> def loop(x: String): Unit = { >>> for (a <- 1 to 30000000) { >>> >>> } >>> } >>> >>> spark-submit: >>> >>> spark-submit --executor-cores 1 \ >>> --executor-memory 1g \ >>> --driver-memory 1g \ >>> --master yarn \ >>> --deploy-mode cluster \ >>> --conf spark.dynamicAllocation.enabled=true \ >>> --conf spark.shuffle.service.enabled=true \ >>> --conf spark.dynamicAllocation.minExecutors=1 \ >>> --conf spark.dynamicAllocation.maxExecutors=3 \ >>> --conf spark.dynamicAllocation.initialExecutors=3 \ >>> --conf spark.executor.instances=3 \ >>> >>> If i run twice spark-submit it create 6 executors, but i want to run all >>> this jobs on same spark application. >>> >>> How can achieve adding jobs to an existing spark application? >>> >>> I don't understand why SparkContext.getOrCreate() don't get existing >>> spark context. >>> >>> >>> Thanks, >>> >>> Cosmin P. >>> >> -- >> Best Regards, >> Ayan Guha >> > >
