Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run against a 1.0.1 cluster

2014-08-15 Thread Mingyu Kim 
Thanks for your response. I think I misinterpreted the
stability/compatibility guarantee with 1.0 release. It seems like the
compatibility is only at the API level.

This is interesting because it means any system/product that is built on top
of Spark and uses Spark with a long-running SparkContext connecting to the
cluster over network, will need to make sure it has the exact same version
of Spark jar as the cluster, even to the patch version. This would be
analogous to having to compile Spark against a very specific version of
Hadoop, as opposed to currently being able to use the Spark package with
CDH4 against most of the CDH4 Hadoop clusters.

Is it correct that Spark is focusing and prioritizing around the
spark-submit use cases than the aforementioned use cases? I just wanted to
better understand the future direction/prioritization of spark.

Thanks,
Mingyu

From:  Patrick Wendell pwend...@gmail.com
Date:  Thursday, August 14, 2014 at 6:32 PM
To:  Gary Malouf malouf.g...@gmail.com
Cc:  Mingyu Kim m...@palantir.com, dev@spark.apache.org
dev@spark.apache.org
Subject:  Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run
against a 1.0.1 cluster

I commented on the bug. For driver mode, you'll need to get the
corresponding version of spark-submit for Spark 1.0.2.


On Thu, Aug 14, 2014 at 3:43 PM, Gary Malouf malouf.g...@gmail.com wrote:
 To be clear, is it 'compiled' against 1.0.2 or it packaged with it?
 
 
 On Thu, Aug 14, 2014 at 6:39 PM, Mingyu Kim m...@palantir.com wrote:
 
  I ran a really simple code that runs with Spark 1.0.2 jar and connects to
  a Spark 1.0.1 cluster, but it fails with java.io.InvalidClassException. I
  filed the bug at https://issues.apache.org/jira/browse/SPARK-3050
 https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/br
 owse/SPARK-3050k=fDZpZZQMmYwf27OU23GmAQ%3D%3D%0Ar=UKDOcu6qL3KsoZhpOohNBR1uc
 PNmWnbd3eEJ9hVUdMk%3D%0Am=qvQ59wZwD7EuezjTuLzmNTRUamDRDnI7%2F0%2BnULtXk4k%3D
 %0As=b7abf7638a3e6fac2ddac9d8f0ca52f1a92945465abfb2e2d996a96d2301fec5 .
 
  I assumed the minor and patch releases shouldn¹t break compatibility. Is
  that correct?
 
  Thanks,
  Mingyu
 





smime.p7s
Description: S/MIME cryptographic signature

[SPARK-3050] Spark program running with 1.0.2 jar cannot run against a 1.0.1 cluster

2014-08-14 Thread Mingyu Kim 
I ran a really simple code that runs with Spark 1.0.2 jar and connects to a
Spark 1.0.1 cluster, but it fails with java.io.InvalidClassException. I
filed the bug at https://issues.apache.org/jira/browse/SPARK-3050.

I assumed the minor and patch releases shouldn¹t break compatibility. Is
that correct?

Thanks,
Mingyu




smime.p7s
Description: S/MIME cryptographic signature

Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run against a 1.0.1 cluster

2014-08-14 Thread Gary Malouf 
To be clear, is it 'compiled' against 1.0.2 or it packaged with it?


On Thu, Aug 14, 2014 at 6:39 PM, Mingyu Kim m...@palantir.com wrote:

 I ran a really simple code that runs with Spark 1.0.2 jar and connects to
 a Spark 1.0.1 cluster, but it fails with java.io.InvalidClassException. I
 filed the bug at https://issues.apache.org/jira/browse/SPARK-3050.

 I assumed the minor and patch releases shouldn’t break compatibility. Is
 that correct?

 Thanks,
 Mingyu

Re: [SPARK-3050] Spark program running with 1.0.2 jar cannot run against a 1.0.1 cluster

2014-08-14 Thread Patrick Wendell 
I commented on the bug. For driver mode, you'll need to get the
corresponding version of spark-submit for Spark 1.0.2.


On Thu, Aug 14, 2014 at 3:43 PM, Gary Malouf malouf.g...@gmail.com wrote:

 To be clear, is it 'compiled' against 1.0.2 or it packaged with it?


 On Thu, Aug 14, 2014 at 6:39 PM, Mingyu Kim m...@palantir.com wrote:

  I ran a really simple code that runs with Spark 1.0.2 jar and connects to
  a Spark 1.0.1 cluster, but it fails with java.io.InvalidClassException. I
  filed the bug at https://issues.apache.org/jira/browse/SPARK-3050.
 
  I assumed the minor and patch releases shouldn't break compatibility. Is
  that correct?
 
  Thanks,
  Mingyu