Thanks for the clarification. I don¹t have a deep knowledge of Scala, but I thought this was going to be reasonable to support since Java serialization framework provides relatively easy ways to support these kinds of backwards compatibility. I can see how this could be harder with closures.
Supporting at least the stability between different patch versions would help a lot. Mingyu From: Patrick Wendell <pwend...@gmail.com> Date: Friday, August 15, 2014 at 12:28 PM To: Mingyu Kim <m...@palantir.com> Cc: Gary Malouf <malouf.g...@gmail.com>, "dev@spark.apache.org" <dev@spark.apache.org> Subject: Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run against a 1.0.1 cluster Hey Mingyu, For this reason we are encouraging all users to run spark-submit. In Spark we capture closures and send them over the network from the driver to the executors. These are then deserialized on the executor. So if your driver program has different versions of certain classes than exist on the executor, it doesn't work well. We've even run into stranger issues, where the exact same version of Spark was used at the driver and the executor, but they were compiled at different times. Since Scala doesn't guarantee stable naming for certain types of anonymous classes, the class names didn't match up and it caused errors at runtime. The most straightforward way to deal with this is to inject, at run-time, the exact version of Spark that the cluster expects if you are running the standalone mode. I think we'd be totally open to improving this to provide "API stability" for the case you are working with, i.e. the case where you have spark 1.0.X at the driver and 1.0.Y on the executors. But it will require looking at what exactly causes incompatibility and seeing if there is a solution. In this case I think we changed a publicly exposed class (the RDD class) in some way that caused compatibility issues... even though we didn't change any binary signatures. BTW - this is not relevant to YARN mode where you ship Spark with your job so there is no "cluster version of Spark". - Patrick On Fri, Aug 15, 2014 at 11:13 AM, Mingyu Kim <m...@palantir.com> wrote: > Thanks for your response. I think I misinterpreted the stability/compatibility > guarantee with 1.0 release. It seems like the compatibility is only at the API > level. > > This is interesting because it means any system/product that is built on top > of Spark and uses Spark with a long-running SparkContext connecting to the > cluster over network, will need to make sure it has the exact same version of > Spark jar as the cluster, even to the patch version. This would be analogous > to having to compile Spark against a very specific version of Hadoop, as > opposed to currently being able to use the Spark package with CDH4 against > most of the CDH4 Hadoop clusters. > > Is it correct that Spark is focusing and prioritizing around the spark-submit > use cases than the aforementioned use cases? I just wanted to better > understand the future direction/prioritization of spark. > > Thanks, > Mingyu > > From: Patrick Wendell <pwend...@gmail.com> > Date: Thursday, August 14, 2014 at 6:32 PM > To: Gary Malouf <malouf.g...@gmail.com> > Cc: Mingyu Kim <m...@palantir.com>, "dev@spark.apache.org" > <dev@spark.apache.org> > Subject: Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run > against a 1.0.1 cluster > > I commented on the bug. For driver mode, you'll need to get the corresponding > version of spark-submit for Spark 1.0.2. > > > On Thu, Aug 14, 2014 at 3:43 PM, Gary Malouf <malouf.g...@gmail.com> wrote: >> To be clear, is it 'compiled' against 1.0.2 or it packaged with it? >> >> >> On Thu, Aug 14, 2014 at 6:39 PM, Mingyu Kim <m...@palantir.com> wrote: >> >>> > I ran a really simple code that runs with Spark 1.0.2 jar and connects to >>> > a Spark 1.0.1 cluster, but it fails with java.io.InvalidClassException. I >>> > filed the bug at https://issues.apache.org/jira/browse/SPARK-3050 >>> <https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/b >>> rowse/SPARK-3050&k=fDZpZZQMmYwf27OU23GmAQ%3D%3D%0A&r=UKDOcu6qL3KsoZhpOohNBR1 >>> ucPNmWnbd3eEJ9hVUdMk%3D%0A&m=qvQ59wZwD7EuezjTuLzmNTRUamDRDnI7%2F0%2BnULtXk4k >>> %3D%0A&s=b7abf7638a3e6fac2ddac9d8f0ca52f1a92945465abfb2e2d996a96d2301fec5> . >>> > >>> > I assumed the minor and patch releases shouldn¹t break compatibility. Is >>> > that correct? >>> > >>> > Thanks, >>> > Mingyu >>> > >
smime.p7s
Description: S/MIME cryptographic signature
