Our use case is trying to isolate the classes in shipped jars that are available to different users of the Spark context.
We have multiple users that can run against our Web Server JVM which will host a Spark Context. These users can compile their own jars and query the SparkContext to add them (addJar(…)). However, concurrent users could add jars that could possibly cause class namespace collisions and race conditions. In addition, one user may submit one job to the SparkContext with a jar, then finish that job and wish to update that jar on a subsequent job. That's why we want to create a SparkContext per compute-session to sandbox the jars in each user's job. Is this a use case that could be done by only using one SparkContext in the JVM? -Matt Cheah From: Dmitriy Lyubimov <dlie...@gmail.com<mailto:dlie...@gmail.com>> Reply-To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>" <user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>> Date: Wednesday, November 20, 2013 12:39 PM To: "user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>" <user@spark.incubator.apache.org<mailto:user@spark.incubator.apache.org>> Subject: Re: Multiple SparkContextx in one JVM As far as i can tell, the mesos back end would still not work correctly with multiple SparkContexts. However, if you are just after spark query concurrency, spark 0.8 seems to be supporting concurrent (reentrant) requests to the same session (SparkContext). One should also be able to use FAIR scheduler in this case it seems (at least that's what i request). So i just sempahore the same context while keeping pool of SparkContext's at 1. See the doc folder for details. On Wed, Nov 20, 2013 at 3:26 AM, Mingyu Kim <m...@palantir.com<mailto:m...@palantir.com>> wrote: Hi all, I’ve been searching to find out the current status of the multiple SparkContext support in one JVM. I found https://groups.google.com/forum/#!topic/spark-developers/GLx8yunSj0A and https://groups.google.com/forum/#!topic/spark-users/cOYP96I668I. According to the threads, I should be able to create multiple SparkContexts by setting the port to “0” every time. However, Matei mentioned that SparkEnv should be part of TaskContext rather than being thread local. Does this cause any problem with running multiple SparkContexts in one JVM right now or is this just a clean-up that needs to be done eventually? I’m wondering if Spark supports multiple SparkContexts as is right now or if there is anything I should be careful about when creating multiple SparkContexts. Thanks! Mingyu
