Hm, I was thinking that the issue is that Spark has to use a forked hive-exec since hive-exec unfortunately includes a bunch of dependencies it shouldn't. It forked Hive 0.12.0: http://mvnrepository.com/artifact/org.spark-project.hive/hive-exec/0.12.0
... and then I was thinking maybe CDH wasn't able to harmonize it. But 5.1 did harmonize this dependency, it appears. https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-hive_2.10/1.0.0-cdh5.1.0/ Looking back at 5.0.3, which shipped 0.9.0 ... I don't think the Spark Hive module existed in 0.9.0? https://github.com/apache/spark/tree/branch-0.9/sql/hive So yeah using it with newer Spark 1.0 code probably means you'd have to build Spark to pull in the CDH version of hive-exec (or else keep it out the custom assembly and make sure only the CDH version is on the classpath at runtime) and hope you don't actually have to shade it like Spark does to get it to work. I think support will suggest this 1.0 Hive code is supported and works in the version that ships 1.0! (Sorry for extending a thread apparently about CDH but I think the issue is actually more broadly applicable.) On Sun, Aug 10, 2014 at 9:20 PM, Eric Friedman <eric.d.fried...@gmail.com> wrote: > Hi Sean, > > Thanks for the reply. I'm on CDH 5.0.3 and upgrading the whole cluster to > 5.1.0 will eventually happen but not immediately. > > I've tried running the CDH spark-1.0 release and also building it from > source. This, unfortunately goes into a whole other rathole of > dependencies. :-( > > Eric --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
