Why are you using zookeeper 3.3.0 with hbase 0.20.6? hbase uses zk 3.2.2 internally -- mixing and matching zk versions is not a great plan
Dave -----Original Message----- From: Gökhan Çapan [mailto:[email protected]] Sent: Wednesday, December 08, 2010 1:19 PM To: [email protected] Subject: Re: Importing to HBase from Java problem Sure. This is a prototype project for me to test some algorithms before deploying them to cloud for production. I wanted to persist results of a Mapreduced MinHash algorithm to a NoSQL storage. After that, I wanted to execute some scans, gets,... as a simple test for me. It is a maven project which has dependencies to hadoop 0.20.2, hbase 0.20.6 (i have added the downloaded jar to our internal archiva), zookeeper 3.3.0. Plus, an hbase-site.xml: <configuration> <property> <name>hbase.zookeeper.quorum</name> <value>localhost</value> <description> </description> </property> </configuration> is in its classpath. One module of the project is a series of MR jobs that finally persists results to an HBase table. When I tried to run it, I got some errors. Following your instructions, they've been solved. There is another module with exactly same dependencies. This is a "has nothing but a main" class that performs some scans on the HBase table produced in the previous step. after a mvn package with dependencies and adding the hbase-site.xml above as a resource, with SimpleClient as main class, I couldn't java -jar SimpleClient.jar, it gave me the exception I have posted. Thanks in advance. On Wed, Dec 8, 2010 at 8:11 PM, Stack <[email protected]> wrote: > Tell us more about your client project. What is it? Is it a > container of some type? A tomcat or something? You could try logging > the CLASSPATH your client sees. The ZooKeeper that we have in TRUNK > and 0.90 branch does it. It doesn't look like the one in 0.20.6 does > going by the log below. > St.Ack > > On Wed, Dec 8, 2010 at 6:44 AM, Gökhan Çapan <[email protected]> wrote: > > Hi, > > > > Now I am getting the same Exception while trying to run a simple HBase > > client. > > > > HBase 0.20.6 is both in classpath of Hadoop and the Client project. > > HBase and hadoop configuration is in the classpath of the Client Project > > HBase configuration is in the Hadoop classpath. > > HBase shell runs well. > > Hadoop runs well. > > My previous MapReduce app with same dependencies works well. > > > > Below is the log and Exception: > > > > 43:44 INFO zookeeper.ZooKeeper: Client > > > environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java > > 10/12/08 13:43:44 INFO zookeeper.ZooKeeper: Client > > > environment:java.io.tmpdir=/var/folders/9x/9x-OkSEyFDWn+13ntAMH6++++TI/-Tmp-/ > > 10/12/08 13:43:44 INFO zookeeper.ZooKeeper: Client > > environment:java.compiler=<NA> > > 10/12/08 13:43:44 INFO zookeeper.ZooKeeper: Client environment:os.name > =Mac > > OS X > > 10/12/08 13:43:44 INFO zookeeper.ZooKeeper: Client > > environment:os.arch=x86_64 > > 10/12/08 13:43:44 INFO zookeeper.ZooKeeper: Client > > environment:os.version=10.6.5 > > 10/12/08 13:43:44 INFO zookeeper.ZooKeeper: Client environment:user.name > > =gcapan > > 10/12/08 13:43:44 INFO zookeeper.ZooKeeper: Client > > environment:user.home=/Users/gcapan > > 10/12/08 13:43:44 INFO zookeeper.ZooKeeper: Client > > environment:user.dir=/Users/gcapan/workspace/*** > > 10/12/08 13:43:44 INFO zookeeper.ZooKeeper: Initiating client connection, > > connectString=localhost:2181 sessionTimeout=60000 > > > watcher=org.apache.hadoop.hbase.client.hconnectionmanager$tableserv...@2b2d96f2 > > 10/12/08 13:43:44 INFO zookeeper.ClientCnxn: Opening socket connection to > > server localhost/127.0.0.1:2181 > > 10/12/08 13:43:44 INFO zookeeper.ClientCnxn: Socket connection > established > > to localhost/127.0.0.1:2181, initiating session > > 10/12/08 13:43:44 INFO zookeeper.ClientCnxn: Session establishment > complete > > on server localhost/127.0.0.1:2181, sessionid = 0x12cc531bd4c0006, > > negotiated timeout = 60000 > > Exception in thread "main" java.lang.reflect.UndeclaredThrowableException > > at $Proxy0.getRegionInfo(Unknown Source) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:874) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:528) > > at > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491) > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:123) > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:101) > > at com.***.***.***.***HBaseClientDemo.main(HBaseClientDemo.java:26) > > Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: > > java.io.IOException: Could not find requested method, the usual cause is > a > > version mismatch between client and server. > > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) > > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > > > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:723) > > at > > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328) > > ... 13 more > > > > > > > > On Tue, Dec 7, 2010 at 11:30 AM, Gökhan Çapan <[email protected]> wrote: > > > >> I had forgotten add HBase-conf to MapReduce ClassPath. > >> > >> That solved the problem, > >> > >> Thanks. > >> > >> > >> > >> On Mon, Dec 6, 2010 at 8:40 PM, Gökhan Çapan <[email protected]> wrote: > >> > >>> > >>> > >>> On Mon, Dec 6, 2010 at 7:54 PM, Stack <[email protected]> wrote: > >>> > >>>> That looks like a mismatch between client and server hbase versions. > >>>> Ensure you have same running all over your cluster. Make sure you > >>>> don't have a mix of 0.20.x and 0.89... or 0.90 release candidates. > >>>> > >>>> You seem to be feeling your way. Have you seen > >>>> > >>>> > https://hudson.apache.org/hudson/view/G-L/view/HBase/job/hbase-0.90/ws/trunk/target/site/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description > >>>> ? > >>>> > >>>> You added zookeeper. Be sure to add the hbase conf directory to your > >>>> MapReduce CLASSPATH. > >>>> > >>>> St.Ack > >>>> > >>> > >>> I hadn't added the hbase conf directory to my MapReduce CLASSPATH. > >>> Unfortunately, now I am not able to try. I will try tomorrow, and let > you > >>> know. > >>> > >>> Thanks. > >>> > >>>> > >>>> > >>>> On Mon, Dec 6, 2010 at 8:12 AM, Gökhan Çapan <[email protected]> > wrote: > >>>> > Hi, > >>>> > > >>>> > I am facing with some issues when I try to write the results of my > >>>> Reducer > >>>> > to an HBase table, in pseudo-distributed mode. > >>>> > > >>>> > HBase version: 0.20.6 > >>>> > Hadoop version: 0.20.2 > >>>> > > >>>> > I let HBase manage Zookeeper. > >>>> > > >>>> > After starting HBase, I can successfully start HBase shell, and > create > >>>> a > >>>> > table. > >>>> > > >>>> > In my MapReduce app, I have a dependency to HBase 0.20.2. > >>>> > >>> > >>> Sorry, that was also HBase 0.20.6 > >>> > >>> > >>>> > > >>>> > However, it gave me the following error: > >>>> > Error: java.lang.ClassNotFoundException: > org.apache.zookeeper.Watcher > >>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > >>>> > at java.security.AccessController.doPrivileged(Native Method) > >>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > >>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > >>>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > >>>> > at java.lang.ClassLoader.defineClass1(Native Method) > >>>> > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) > >>>> > at java.lang.ClassLoader.defineClass(ClassLoader.java:616) > >>>> > at > >>>> > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > >>>> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > >>>> > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > >>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > >>>> > at java.security.AccessController.doPrivileged(Native Method) > >>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > >>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > >>>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:93) > >>>> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:119) > >>>> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:101) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.mapreduce.TableOutputFormat.getRecordWriter(TableOutputFormat.java:116) > >>>> > at > >>>> > > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:553) > >>>> > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408) > >>>> > at org.apache.hadoop.mapred.Child.main(Child.java:170) > >>>> > . > >>>> > . > >>>> > . > >>>> > > >>>> > > >>>> > > >>>> > So I have added ZooKeeper as a dependency. Now it throws the > following > >>>> > Exception for ZooKeeper 3.2.2, 3.3.0, and 3.3.1: > >>>> > > >>>> > 10/12/06 17:51:53 INFO mapred.JobClient: map 100% reduce 27% > >>>> > 10/12/06 17:51:58 INFO mapred.JobClient: Task Id : > >>>> > attempt_201012061721_0002_r_000000_2, Status : FAILED > >>>> > java.lang.reflect.UndeclaredThrowableException > >>>> > at $Proxy2.getRegionInfo(Unknown Source) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:874) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:528) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491) > >>>> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:123) > >>>> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:101) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.mapreduce.TableOutputFormat.getRecordWriter(TableOutputFormat.java:116) > >>>> > at > >>>> > > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:553) > >>>> > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408) > >>>> > at org.apache.hadoop.mapred.Child.main(Child.java:170) > >>>> > Caused by: org.apache.hadoop.ipc.RemoteException: > java.io.IOException: > >>>> > java.io.IOException: Could not find requested method, the usual > cause > >>>> is a > >>>> > version mismatch between client and server. > >>>> > at > >>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) > >>>> > at > >>>> > > >>>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > >>>> > > >>>> > at > >>>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:723) > >>>> > at > >>>> > > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328) > >>>> > ... 16 more > >>>> > > >>>> > > >>>> > I guess this is a compatibility issue between HBase and ZooKeeper, > but > >>>> I > >>>> > couldn't find the right version of ZooKeeper to add as a dependency. > >>>> > > >>>> > > >>>> > Thanks in advance, > >>>> > >
