Then definitely its a jar conflict. Can you try removing this jar from the class path /opt/spark-poc/lib_managed/jars/org.spark-project.hive/hive-exec/ hive-exec-0.12.0.jar
Thanks Best Regards On Mon, Aug 18, 2014 at 12:45 PM, Cesar Arevalo <ce...@zephyrhealthinc.com> wrote: > Nope, it is NOT null. Check this out: > > scala> hiveContext == null > res2: Boolean = false > > > And thanks for sending that link, but I had already looked at it. Any > other ideas? > > I looked through some of the relevant Spark Hive code and I'm starting to > think this may be a bug. > > -Cesar > > > > On Mon, Aug 18, 2014 at 12:00 AM, Akhil Das <ak...@sigmoidanalytics.com> > wrote: > >> Looks like your hiveContext is null. Have a look at this documentation. >> <https://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables> >> >> Thanks >> Best Regards >> >> >> On Mon, Aug 18, 2014 at 12:09 PM, Cesar Arevalo < >> ce...@zephyrhealthinc.com> wrote: >> >>> Hello: >>> >>> I am trying to setup Spark to connect to a Hive table which is backed by >>> HBase, but I am running into the following NullPointerException: >>> >>> scala> val hiveCount = hiveContext.sql("select count(*) from >>> dataset_records").collect().head.getLong(0) >>> 14/08/18 06:34:29 INFO ParseDriver: Parsing command: select count(*) >>> from dataset_records >>> 14/08/18 06:34:29 INFO ParseDriver: Parse Completed >>> 14/08/18 06:34:29 INFO HiveMetaStore: 0: get_table : db=default >>> tbl=dataset_records >>> 14/08/18 06:34:29 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_table >>> : db=default tbl=dataset_records >>> 14/08/18 06:34:30 INFO MemoryStore: ensureFreeSpace(160296) called with >>> curMem=0, maxMem=280248975 >>> 14/08/18 06:34:30 INFO MemoryStore: Block broadcast_0 stored as values >>> in memory (estimated size 156.5 KB, free 267.1 MB) >>> 14/08/18 06:34:30 INFO SparkContext: Starting job: collect at >>> SparkPlan.scala:85 >>> 14/08/18 06:34:31 WARN DAGScheduler: Creating new stage failed due to >>> exception - job: 0 >>> java.lang.NullPointerException >>> at org.apache.hadoop.hbase.util.Bytes.toBytes(Bytes.java:502) >>> at >>> org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:418) >>> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:179) >>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) >>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) >>> at scala.Option.getOrElse(Option.scala:120) >>> at org.apache.spark.rdd.RDD.partitions(RDD.scala:202) >>> at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) >>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) >>> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202) >>> at scala.Option.getOrElse(Option.scala:120) >>> >>> >>> >>> >>> This is happening from the master on spark, I am running hbase version >>> hbase-0.98.4-hadoop1 and hive version 0.13.1. And here is how I am running >>> the spark shell: >>> >>> bin/spark-shell --driver-class-path >>> /opt/hive/latest/lib/hive-hbase-handler-0.13.1.jar:/opt/hive/latest/lib/zookeeper-3.4.5.jar:/opt/spark-poc/lib_managed/jars/com.google.guava/guava/guava-14.0.1.jar:/opt/hbase/latest/lib/hbase-common-0.98.4-hadoop1.jar:/opt/hbase/latest/lib/hbase-server-0.98.4-hadoop1.jar:/opt/hbase/latest/lib/hbase-client-0.98.4-hadoop1.jar:/opt/hbase/latest/lib/hbase-protocol-0.98.4-hadoop1.jar:/opt/hbase/latest/lib/htrace-core-2.04.jar:/opt/hbase/latest/lib/netty-3.6.6.Final.jar:/opt/hbase/latest/lib/hbase-hadoop-compat-0.98.4-hadoop1.jar:/opt/spark-poc/lib_managed/jars/org.apache.hbase/hbase-client/hbase-client-0.98.4-hadoop1.jar:/opt/spark-poc/lib_managed/jars/org.apache.hbase/hbase-common/hbase-common-0.98.4-hadoop1.jar:/opt/spark-poc/lib_managed/jars/org.apache.hbase/hbase-server/hbase-server-0.98.4-hadoop1.jar:/opt/spark-poc/lib_managed/jars/org.apache.hbase/hbase-prefix-tree/hbase-prefix-tree-0.98.4-hadoop1.jar:/opt/spark-poc/lib_managed/jars/org.apache.hbase/hbase-protocol/hbase-protocol-0.98.4-hadoop1.jar:/opt/spark-poc/lib_managed/bundles/com.google.protobuf/protobuf-java/protobuf-java-2.5.0.jar:/opt/spark-poc/lib_managed/jars/org.cloudera.htrace/htrace-core/htrace-core-2.04.jar:/opt/spark/sql/hive/target/spark-hive_2.10-1.1.0-SNAPSHOT.jar:/opt/spark-poc/lib_managed/jars/org.spark-project.hive/hive-common/hive-common-0.12.0.jar:/opt/spark-poc/lib_managed/jars/org.spark-project.hive/hive-exec/hive-exec-0.12.0.jar:/opt/spark-poc/lib_managed/jars/org.apache.thrift/libthrift/libthrift-0.9.0.jar:/opt/spark-poc/lib_managed/jars/org.spark-project.hive/hive-shims/hive-shims-0.12.0.jar:/opt/spark-poc/lib_managed/jars/org.spark-project.hive/hive-metastore/hive-metastore-0.12.0.jar:/opt/spark/sql/catalyst/target/spark-catalyst_2.10-1.1.0-SNAPSHOT.jar:/opt/spark-poc/lib_managed/jars/org.antlr/antlr-runtime/antlr-runtime-3.4.jar:/opt/spark-poc/lib_managed/jars/org.apache.thrift/libfb303/libfb303-0.9.0.jar:/opt/spark-poc/lib_managed/jars/javax.jdo/jdo-api/jdo-api-3.0.1.jar:/opt/spark-poc/lib_managed/jars/org.datanucleus/datanucleus-api-jdo/datanucleus-api-jdo-3.2.1.jar:/opt/spark-poc/lib_managed/jars/org.datanucleus/datanucleus-core/datanucleus-core-3.2.2.jar:/opt/spark-poc/lib_managed/jars/org.datanucleus/datanucleus-rdbms/datanucleus-rdbms-3.2.1.jar:/opt/spark-poc/lib_managed/jars/org.apache.derby/derby/derby-10.4.2.0.jar:/opt/spark-poc/sbt/ivy/cache/org.apache.hive/hive-hbase-handler/jars/hive-hbase-handler-0.13.1.jar:/opt/spark-poc/lib_managed/jars/com.typesafe/scalalogging-slf4j_2.10/scalalogging-slf4j_2.10-1.0.1.jar:/opt/spark-poc/lib_managed/bundles/com.jolbox/bonecp/bonecp-0.7.1.RELEASE.jar:/opt/spark-poc/sbt/ivy/cache/com.datastax.cassandra/cassandra-driver-core/bundles/cassandra-driver-core-2.0.4.jar:/opt/spark-poc/lib_managed/jars/org.json/json/json-20090211.jar >>> >>> >>> >>> Can anybody help me? >>> >>> Best, >>> -- >>> Cesar Arevalo >>> Software Engineer ❘ Zephyr Health >>> 450 Mission Street, Suite #201 ❘ San Francisco, CA 94105 >>> m: +1 415-571-7687 ❘ s: arevalocesar | t: @zephyrhealth >>> <https://twitter.com/zephyrhealth> >>> o: +1 415-529-7649 ❘ f: +1 415-520-9288 >>> http://www.zephyrhealth.com >>> >>> >>> >> > > > -- > Cesar Arevalo > Software Engineer ❘ Zephyr Health > 450 Mission Street, Suite #201 ❘ San Francisco, CA 94105 > m: +1 415-571-7687 ❘ s: arevalocesar | t: @zephyrhealth > <https://twitter.com/zephyrhealth> > o: +1 415-529-7649 ❘ f: +1 415-520-9288 > http://www.zephyrhealth.com > > >
