Re: Classpath problem trying to use DataFrames

2015-12-14 Thread Christopher Brady
Thanks for the response. I lost access to my cluster over the weekend, so I had to wait until today to check. All of the correct Hive jars are in classpath.txt. Also, this error seems to be happening in the driver rather than the executors. It's running in yarn-client mode, so it should use th

??????Classpath problem trying to use DataFrames

2015-12-12 Thread Ricky
encountor similar problems using hivecontext .When code print classload ,it was changed to multiclassloader from APPclassloader -- -- ??: Harsh J : 2015??12??12?? 12:09 ??: Christopher Brady , user : Re: Classpath problem trying

Re: Classpath problem trying to use DataFrames

2015-12-11 Thread Harsh J
Do you have all your hive jars listed in the classpath.txt / SPARK_DIST_CLASSPATH env., specifically the hive-exec jar? Is the location of that jar also the same on all the distributed hosts? Passing an explicit executor classpath string may also help overcome this (replace HIVE_BASE_DIR to the ro

Classpath problem trying to use DataFrames

2015-12-11 Thread Christopher Brady
I'm trying to run a basic "Hello world" type example using DataFrames with Hive in yarn-client mode. My code is: JavaSparkContext sc = new JavaSparkContext("yarn-client", "Test app")) HiveContext sqlContext = new HiveContext(sc.sc()); sqlContext.sql("SELECT * FROM my_table").count(); The except