Spark jobserver or Livy server are the best options for pure technical API.
If you want to publish business API you will probably have to build you own
app like the one I wrote a year ago
https://github.com/elppc/akka-spark-experiments
It combines Akka actors and a shared Spark context to serve concurrent
subsecond jobs
2017-02-07 15:28 GMT+01:00 ayan guha <guha.a...@gmail.com>:
> I think you are loking for livy or spark jobserver
>
> On Wed, 8 Feb 2017 at 12:37 am, Cosmin Posteuca <cosmin.poste...@gmail.com>
> wrote:
>
>> I want to run different jobs on demand with same spark context, but i
>> don't know how exactly i can do this.
>>
>> I try to get current context, but seems it create a new spark
>> context(with new executors).
>>
>> I call spark-submit to add new jobs.
>>
>> I run code on Amazon EMR(3 instances, 4 core & 16GB ram / instance), with
>> yarn as resource manager.
>>
>> My code:
>>
>> val sparkContext = SparkContext.getOrCreate()
>> val content = 1 to 40000
>> val result = sparkContext.parallelize(content, 5)
>> result.map(value => value.toString).foreach(loop)
>>
>> def loop(x: String): Unit = {
>> for (a <- 1 to 30000000) {
>>
>> }
>> }
>>
>> spark-submit:
>>
>> spark-submit --executor-cores 1 \
>> --executor-memory 1g \
>> --driver-memory 1g \
>> --master yarn \
>> --deploy-mode cluster \
>> --conf spark.dynamicAllocation.enabled=true \
>> --conf spark.shuffle.service.enabled=true \
>> --conf spark.dynamicAllocation.minExecutors=1 \
>> --conf spark.dynamicAllocation.maxExecutors=3 \
>> --conf spark.dynamicAllocation.initialExecutors=3 \
>> --conf spark.executor.instances=3 \
>>
>> If i run twice spark-submit it create 6 executors, but i want to run all
>> this jobs on same spark application.
>>
>> How can achieve adding jobs to an existing spark application?
>>
>> I don't understand why SparkContext.getOrCreate() don't get existing
>> spark context.
>>
>>
>> Thanks,
>>
>> Cosmin P.
>>
> --
> Best Regards,
> Ayan Guha
>
