> On 12 Sep 2015, at 09:14, Sean Owen <so...@cloudera.com> wrote: > > This is a question for the CDH list. CDH 5.4 has Spark 1.3, and 5.5 > has 1.5. The best thing is to update CDH as a whole if you can. > > However it's pretty simple to just run a newer Spark assembly as a > YARN app. Don't remove anything in the CDH installation. Try > downloading the assembly and prefixing your command with > SPARK_JAR=newassembly.jar ... >
...though you should try to make sure the the hadoop JARs in the spark JAR are equal to or less than the version of Hadoop you run on, i.e. don't try running a Hadoop-2.6 built artifact against a 2.4 based cluster. One way to guarantee that is to rebuild the spark binaries with the cloudera versions & repo in the build --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
