The thing is that the class it is complaining about is part of the spark assembly jar, not in my extra jar. The assembly jar was compiled with -Phive which is proven by the fact that it works with the same SPARK_HOME when run as shell.
On 23 July 2015 at 17:33, Akhil Das <ak...@sigmoidanalytics.com> wrote: > You can try adding that jar in SPARK_CLASSPATH (its deprecated though) in > spark-env.sh file. > > Thanks > Best Regards > > On Tue, Jul 21, 2015 at 7:34 PM, Michal Haris <michal.ha...@visualdna.com> > wrote: > >> I have a spark program that uses dataframes to query hive and I run it >> both as a spark-shell for exploration and I have a runner class that >> executes some tasks with spark-submit. I used to run against >> 1.4.0-SNAPSHOT. Since then 1.4.0 and 1.4.1 were released so I tried to >> switch to the official release. Now, when I run the program as a shell, >> everything works but when I try to run it with spark-submit it complains >> with this error: >> >> Exception in thread "main" java.lang.ClassNotFoundException: >> java.lang.NoClassDefFoundError: >> org/apache/hadoop/hive/ql/session/SessionState when creating Hive client >> using classpath: file:/home/mharis/dxp-spark.jar >> Please make sure that jars for your version of hive and hadoop are >> included in the paths passed to spark.sql.hive.metastore.jars. >> >> What is suspicious is firstly 'using classpath: ...' where the jar is my >> program, i.e. the paths that are passed along with --driver-class-path >> option are missing. When I switch to an older 1.4.0-SNAPSHOT on the driver, >> everything works. I observe the issue with 1.4.1. >> >> Are there any known obvious changes to how spark-submit handles >> configuration that I have missed ? >> >> -- >> Michal Haris >> Technical Architect >> direct line: +44 (0) 207 749 0229 >> www.visualdna.com | t: +44 (0) 207 734 7033 >> 31 Old Nichol Street >> London >> E2 7HR >> > > -- Michal Haris Technical Architect direct line: +44 (0) 207 749 0229 www.visualdna.com | t: +44 (0) 207 734 7033 31 Old Nichol Street London E2 7HR
