Looking at your classpath, it looks like you've compiled Spark yourself. Depending on which version of Hadoop you've compiled against (looks like it's Hadoop 2.2 in your case), Spark will have its own version of protobuf. You should try by making sure both your HBase and Spark are compiled against the same version of Hadoop.
On Thu, Apr 30, 2015 at 6:54 AM, Ted Yu <yuzhih...@gmail.com> wrote: > The error indicates incompatible protobuf versions. > > Please take a look at 4.1.1 under > http://hbase.apache.org/book.html#basic.prerequisites > > Cheers > > On Thu, Apr 30, 2015 at 3:49 AM, Saurabh Gupta <saurabh.gu...@semusi.com> > wrote: > >> Now able to solve the issue by setting >> >> SparkConf sconf = *new* SparkConf().setAppName(“App").setMaster("local") >> and >> >> conf.set(“zookeeper.znode.parent”, “/hbase-unsecure”) >> >> Standalone hbase has a table 'test' >> hbase(main):001:0> scan 'test' >> ROW COLUMN+CELL >> row1 column=cf:a, timestamp=1430234895637, >> value=value1 >> row2 column=cf:b, timestamp=1430234907537, >> value=value2 >> row3 column=cf:c, timestamp=1430234918284, >> value=value3 >> >> Now facing this issue: >> >> ERROR TableInputFormat: java.io.IOException: >> java.lang.reflect.InvocationTargetException >> at >> org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:416) >> at >> org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:393) >> at >> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:274) >> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194) >> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:156) >> at >> org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:101) >> at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:91) >> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >> at scala.Option.getOrElse(Option.scala:120) >> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1632) >> at org.apache.spark.rdd.RDD.count(RDD.scala:1012) >> at org.apache.spark.examples.HBaseTest$.main(HBaseTest.scala:58) >> at org.apache.spark.examples.HBaseTest.main(HBaseTest.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:607) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:167) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:190) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> Caused by: java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >> at >> org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:414) >> ... 23 more >> Caused by: java.lang.VerifyError: class >> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Result overrides >> final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:800) >> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) >> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> at >> org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:176) >> at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64) >> at >> org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:69) >> at >> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:857) >> at >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:662) >> ... 28 more >> >> Exception in thread "main" java.io.IOException: No table was provided. >> at >> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:154) >> at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95) >> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219) >> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217) >> at scala.Option.getOrElse(Option.scala:120) >> at org.apache.spark.rdd.RDD.partitions(RDD.scala:217) >> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1632) >> at org.apache.spark.rdd.RDD.count(RDD.scala:1012) >> at org.apache.spark.examples.HBaseTest$.main(HBaseTest.scala:58) >> at org.apache.spark.examples.HBaseTest.main(HBaseTest.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:607) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:167) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:190) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> On Thu, Apr 30, 2015 at 1:25 PM, Saurabh Gupta <saurabh.gu...@semusi.com> >> wrote: >> >>> I am using hbase -0.94.8. >>> >>> On Wed, Apr 29, 2015 at 11:56 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> Can you enable HBase DEBUG logging in log4j.properties so that we can >>>> have more clue ? >>>> >>>> What hbase release are you using ? >>>> >>>> Cheers >>>> >>>> On Wed, Apr 29, 2015 at 4:27 AM, Saurabh Gupta < >>>> saurabh.gu...@semusi.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am working with standalone HBase. And I want to execute >>>>> HBaseTest.scala (in scala examples) . >>>>> >>>>> I have created a test table with three rows and I just want to get the >>>>> count using HBaseTest.scala >>>>> >>>>> I am getting this issue: >>>>> >>>>> 15/04/29 11:17:10 INFO BlockManagerMaster: Registered BlockManager >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client >>>>> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client environment:host.name >>>>> =ip-10-144-185-113 >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client >>>>> environment:java.version=1.7.0_79 >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client >>>>> environment:java.vendor=Oracle Corporation >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client >>>>> environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client >>>>> environment:java.class.path=/home/ubuntu/sparkfolder/conf/:/home/ubuntu/sparkfolder/assembly/target/scala-2.10/spark-assembly-1.4.0-SNAPSHOT-hadoop2.2.0.jar:/home/ubuntu/sparkfolder/lib_managed/jars/datanucleus-core-3.2.10.jar:/home/ubuntu/sparkfolder/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar:/home/ubuntu/sparkfolder/lib_managed/jars/datanucleus-rdbms-3.2.9.jar >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client >>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client >>>>> environment:java.io.tmpdir=/tmp >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client environment:java.compiler=<NA> >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client environment:os.name=Linux >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client environment:os.arch=amd64 >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client >>>>> environment:os.version=3.13.0-49-generic >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client environment:user.name=root >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client environment:user.home=/root >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Client >>>>> environment:user.dir=/home/ubuntu/sparkfolder >>>>> 15/04/29 11:17:11 INFO ZooKeeper: Initiating client connection, >>>>> connectString=localhost:2181 sessionTimeout=90000 >>>>> watcher=hconnection-0x2711025f, quorum=localhost:2181, baseZNode=/hbase >>>>> 15/04/29 11:17:11 INFO RecoverableZooKeeper: Process >>>>> identifier=hconnection-0x2711025f connecting to ZooKeeper >>>>> ensemble=localhost:2181 >>>>> 15/04/29 11:17:11 INFO ClientCnxn: Opening socket connection to server >>>>> ip-10-144-185-113/10.144.185.113:2181. Will not attempt to >>>>> authenticate using SASL (unknown error) >>>>> 15/04/29 11:17:11 INFO ClientCnxn: Socket connection established to >>>>> ip-10-144-185-113/10.144.185.113:2181, initiating session >>>>> 15/04/29 11:17:11 INFO ClientCnxn: Session establishment complete on >>>>> server ip-10-144-185-113/10.144.185.113:2181, sessionid = >>>>> 0x14d04d506da0005, negotiated timeout = 40000 >>>>> 15/04/29 11:17:11 INFO ZooKeeperRegistry: ClusterId read in ZooKeeper >>>>> is null >>>>> >>>>> Its just stuck Not showing any error. There is no Hadoop on my >>>>> machine. What could be the issue? >>>>> >>>>> here is hbase-site.xml: >>>>> >>>>> <configuration> >>>>> <property> >>>>> <name>hbase.zookeeper.quorum</name> >>>>> <value>localhost</value> >>>>> </property> >>>>> >>>>> <property> >>>>> <name>hbase.zookeeper.property.clientPort</name> >>>>> <value>2181</value> >>>>> </property> >>>>> <property> >>>>> <name>zookeeper.znode.parent</name> >>>>> <value>/hbase</value> >>>>> </property> >>>>> </configuration> >>>>> >>>>> >>>> >>> >> >
