Re: Using a different spark jars than the one on the cluster
+1 - I currently am doing what Marcelo is suggesting as I have a CDH 5.2 cluster (with Spark 1.1) and I'm also running Spark 1.3.0+ side-by-side in my cluster. On Wed, Mar 18, 2015 at 1:23 PM Marcelo Vanzin wrote: > Since you're using YARN, you should be able to download a Spark 1.3.0 > tarball from Spark's website and use spark-submit from that > installation to launch your app against the YARN cluster. > > So effectively you would have 1.2.0 and 1.3.0 side-by-side in your cluster. > > On Wed, Mar 18, 2015 at 11:09 AM, jaykatukuri wrote: > > Hi all, > > I am trying to run my job which needs spark-sql_2.11-1.3.0.jar. > > The cluster that I am running on is still on spark-1.2.0. > > > > I tried the following : > > > > spark-submit --class class-name --num-executors 100 --master yarn > > application_jar--jars hdfs:///path/spark-sql_2.11-1.3.0.jar > > hdfs:///input_data > > > > But, this did not work, I get an error that it is not able to find a > > class/method that is in spark-sql_2.11-1.3.0.jar . > > > > org.apache.spark.sql.SQLContext.implicits()Lorg/ > apache/spark/sql/SQLContext$implicits$ > > > > The question in general is how do we use a different version of spark > jars > > (spark-core, spark-sql, spark-ml etc) than the one's running on a > cluster ? > > > > Thanks, > > Jay > > > > > > > > > > > > -- > > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Using-a-different-spark-jars-than-the- > one-on-the-cluster-tp22125.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > - > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > > > > > -- > Marcelo > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Using a different spark jars than the one on the cluster
Since you're using YARN, you should be able to download a Spark 1.3.0 tarball from Spark's website and use spark-submit from that installation to launch your app against the YARN cluster. So effectively you would have 1.2.0 and 1.3.0 side-by-side in your cluster. On Wed, Mar 18, 2015 at 11:09 AM, jaykatukuri wrote: > Hi all, > I am trying to run my job which needs spark-sql_2.11-1.3.0.jar. > The cluster that I am running on is still on spark-1.2.0. > > I tried the following : > > spark-submit --class class-name --num-executors 100 --master yarn > application_jar--jars hdfs:///path/spark-sql_2.11-1.3.0.jar > hdfs:///input_data > > But, this did not work, I get an error that it is not able to find a > class/method that is in spark-sql_2.11-1.3.0.jar . > > org.apache.spark.sql.SQLContext.implicits()Lorg/apache/spark/sql/SQLContext$implicits$ > > The question in general is how do we use a different version of spark jars > (spark-core, spark-sql, spark-ml etc) than the one's running on a cluster ? > > Thanks, > Jay > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Using-a-different-spark-jars-than-the-one-on-the-cluster-tp22125.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > -- Marcelo - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Using a different spark jars than the one on the cluster
Hi all, I am trying to run my job which needs spark-sql_2.11-1.3.0.jar. The cluster that I am running on is still on spark-1.2.0. I tried the following : spark-submit --class class-name --num-executors 100 --master yarn application_jar--jars hdfs:///path/spark-sql_2.11-1.3.0.jar hdfs:///input_data But, this did not work, I get an error that it is not able to find a class/method that is in spark-sql_2.11-1.3.0.jar . org.apache.spark.sql.SQLContext.implicits()Lorg/apache/spark/sql/SQLContext$implicits$ The question in general is how do we use a different version of spark jars (spark-core, spark-sql, spark-ml etc) than the one's running on a cluster ? Thanks, Jay -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-a-different-spark-jars-than-the-one-on-the-cluster-tp22125.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org