Hi Punya, This seems like a problem that the recently-announced job-server would likely have run into at one point. I haven't tested it yet, but I'd be interested to see what happens when two jobs in the job server have conflicting classes. Does the server correctly segregate each job's classes from other concurrently-running jobs?
>From my reading of the code I think it may not work the way I'd want it to, though there are a few classloader tricks going on. https://github.com/ooyala/spark-jobserver/blob/master/job-server/src/spark.jobserver/JobCache.scala In line 29 there the jar is added to the SparkContext, and in 30 the jar is added to the job-server's local classloader. Note all this PR related to classloaders - https://github.com/apache/spark/pull/119 Andrew On Tue, Mar 18, 2014 at 9:24 AM, Punya Biswal <[email protected]> wrote: > Hi Spark people, > > Sorry to bug everyone again about this, but do people have any thoughts on > whether sub-contexts would be a good way to solve this problem? I'm > thinking of something like > > class SparkContext { > // ... stuff ... > def inSubContext[T](fn: SparkContext => T): T > } > > > this way, I could do something like > > val sc = /* get myself a spark context somehow */; > val rdd = sc.textFile("/stuff.txt") > sc.inSubContext { sc1 => > sc1.addJar("extras-v1.jar") > print(sc1.filter(/* fn that depends on jar */).count) > } > sc.inSubContext { sc2 => > sc2.addJar("extras-v2.jar") > print(sc2.filter(/* fn that depends on jar */).count) > } > > > ... even if classes in extras-v1.jar and extras-v2.jar have name > collisions. > > Punya > > From: Punya Biswal <[email protected]> > Reply-To: <[email protected]> > Date: Sunday, March 16, 2014 at 11:09 AM > To: "[email protected]" <[email protected]> > Subject: Separating classloader management from SparkContexts > > Hi all, > > I'm trying to use Spark to support users who are interactively refining > the code that processes their data. As a concrete example, I might create > an RDD[String] and then write several versions of a function to map over > the RDD until I'm satisfied with the transformation. Right now, once I do > addJar() to add one version of the jar to the SparkContext, there's no way > to add a new version of the jar unless I rename the classes and functions > involved, or lose my current work by re-creating the SparkContext. Is there > a better way to do this? > > One idea that comes to mind is that we could add APIs to create > "sub-contexts" from within a SparkContext. Jars added to a sub-context > would get added to a child classloader on the executor, so that different > sub-contexts could use classes with the same name while still being able to > access on-heap objects for RDDs. If this makes sense conceptually, I'd like > to work on a PR to add such functionality to Spark. > > Punya > >
