Hi hbase users, I am running HBase (on top of HDFS) in distributed mode (on 8 VMs), and things like JPS look fine on all the machines in the cluster. I am also able to run hbase shell and interact with HBase though it. But when I want to benchmark my HBase cluster with YCSB (Yahoo! Cloud System Benchmark, https://github.com/brianfrankcooper/YCSB/) I see this weird problem of slow start of the HBase operations and then picking up later.
Basically when I start the YCSB workload from a client machine, I see these problems in chronological order : 1) ERROR zookeeper.ZooKeeperWatcher: ZK is null on connection event ########### ERROR zookeeper.ZooKeeperWatcher: ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1322) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:584) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:827) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:172) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:131) at com.yahoo.ycsb.db.HBaseClient.getHTable(HBaseClient.java:155) ########### 2) org.apache.zookeeper.ClientCnxn - Error while calling watcher java.lang.NullPointerException: ZK is null ############ ERROR org.apache.zookeeper.ClientCnxn - Error while calling watcher java.lang.NullPointerException: ZK is null at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:334) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497) ############ 3) And then finally it starts the operation on HBase (which means Zookeeper is running fine and can be connected to ) 4) The operations remains below 10 ops/sec for first 60-70 sec, and then grow gradually to reach aroun 1300 ops/sec (normally expected number) Here are the actual logs :: http://pastebin.com/NC1zKwRF I am running 1) Hadoop-1.0.1 2) HBase-0.94.1 3) Zookeeper-3.3.6 4) Java 1.6.0_24 (openJDK-6) 5) OS : Ubuntu-11.10 6) YCSB-0.14 What I have already tried : 1) Checked my DNS setting (just to be sure .. using synced /etc/hosts file) .. no luck 2) Increasing "hbase.zookeeper.watcher.sync.connected.wait" to 10000 (default:2000), this get rid of "ZK is null ****" errors, but slow start is still the issue with no improvement. I am clueless as to what may be the reason behind this 'slowly picking up' behavior of my set-up. Please advise. Thanks, Akshay
