Hey Mingyu, For this reason we are encouraging all users to run spark-submit. In Spark we capture closures and send them over the network from the driver to the executors. These are then deserialized on the executor. So if your driver program has different versions of certain classes than exist on the executor, it doesn't work well. We've even run into stranger issues, where the exact same version of Spark was used at the driver and the executor, but they were compiled at different times. Since Scala doesn't guarantee stable naming for certain types of anonymous classes, the class names didn't match up and it caused errors at runtime.
The most straightforward way to deal with this is to inject, at run-time, the exact version of Spark that the cluster expects if you are running the standalone mode. I think we'd be totally open to improving this to provide "API stability" for the case you are working with, i.e. the case where you have spark 1.0.X at the driver and 1.0.Y on the executors. But it will require looking at what exactly causes incompatibility and seeing if there is a solution. In this case I think we changed a publicly exposed class (the RDD class) in some way that caused compatibility issues... even though we didn't change any binary signatures. BTW - this is not relevant to YARN mode where you ship Spark with your job so there is no "cluster version of Spark". - Patrick On Fri, Aug 15, 2014 at 11:13 AM, Mingyu Kim <m...@palantir.com> wrote: > Thanks for your response. I think I misinterpreted the > stability/compatibility guarantee with 1.0 release. It seems like the > compatibility is only at the API level. > > This is interesting because it means any system/product that is built on > top of Spark and uses Spark with a long-running SparkContext connecting to > the cluster over network, will need to make sure it has the exact same > version of Spark jar as the cluster, even to the patch version. This would > be analogous to having to compile Spark against a very specific version of > Hadoop, as opposed to currently being able to use the Spark package with > CDH4 against most of the CDH4 Hadoop clusters. > > Is it correct that Spark is focusing and prioritizing around the > spark-submit use cases than the aforementioned use cases? I just wanted to > better understand the future direction/prioritization of spark. > > Thanks, > Mingyu > > From: Patrick Wendell <pwend...@gmail.com> > Date: Thursday, August 14, 2014 at 6:32 PM > To: Gary Malouf <malouf.g...@gmail.com> > Cc: Mingyu Kim <m...@palantir.com>, "dev@spark.apache.org" < > dev@spark.apache.org> > Subject: Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run > against a 1.0.1 cluster > > I commented on the bug. For driver mode, you'll need to get the > corresponding version of spark-submit for Spark 1.0.2. > > > On Thu, Aug 14, 2014 at 3:43 PM, Gary Malouf <malouf.g...@gmail.com> > wrote: > >> To be clear, is it 'compiled' against 1.0.2 or it packaged with it? >> >> >> On Thu, Aug 14, 2014 at 6:39 PM, Mingyu Kim <m...@palantir.com> wrote: >> >> > I ran a really simple code that runs with Spark 1.0.2 jar and connects >> to >> > a Spark 1.0.1 cluster, but it fails with java.io.InvalidClassException. >> I >> > filed the bug at https://issues.apache.org/jira/browse/SPARK-3050 >> <https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/browse/SPARK-3050&k=fDZpZZQMmYwf27OU23GmAQ%3D%3D%0A&r=UKDOcu6qL3KsoZhpOohNBR1ucPNmWnbd3eEJ9hVUdMk%3D%0A&m=qvQ59wZwD7EuezjTuLzmNTRUamDRDnI7%2F0%2BnULtXk4k%3D%0A&s=b7abf7638a3e6fac2ddac9d8f0ca52f1a92945465abfb2e2d996a96d2301fec5> >> . >> > >> > I assumed the minor and patch releases shouldn't break compatibility. Is >> > that correct? >> > >> > Thanks, >> > Mingyu >> > >> > >
