As far as I can tell the HMaster process is running correctly. There are no
obvious problems in the logs.
As suggested, I defined zookeeper.znode.parent in the client to match the
deployment, which did eliminate
the warning about the null ClusterId. However, the client still hangs in the
same place.
After some debugging, the client appears to be looking for a zookeeper node
named /hbase_unsecure/meta-region-server.
From the zookeeper cli (slightly edited):
[zk: ] ls /hbase-unsecure
returns
[splitlog, online-snapshot, unassigned, root-region-server, rs, table92,
backup-masters, draining, table, master, shutdown, hbaseid]
so meta-region-server node does not exist. As I read the code, the HBase Master
would have set that in a postOpenDeployTasks method, which
apparently is not being called. At this point it's not clear to me what else to
look for.
Brian
On Jul 9, 2013, at 1:28 PM, Gary Helmling wrote:
> Is the HMaster process running correctly on the cluster? Between the
> missing cluster ID and meta region not being available, it looks like
> HMaster may not have fully initialized.
>
> Alternately, if HMaster is running correctly, did you override the default
> value for zookeeper.znode.parent in your cluster configuration, but not set
> this in your client code?
>
>
> On Tue, Jul 9, 2013 at 10:05 AM, Brian Jeltema <
> [email protected]> wrote:
>
>> I'm new to HBase, and need a little guidance. I've set up a 6-node
>> cluster, with 3 nodes
>> running the ZooKeeper server. The database seems to be working from the
>> hbase shell; I can create tables, insert,
>> scan, etc.
>>
>> But when I try to perform operations in a Java app, I hang at:
>>
>> 13/07/09 12:40:34 INFO zookeeper.ZooKeeper: Initiating client connection,
>> connectString=cas-2:2181,cas-1:2181,cas-3:2181 sessionTimeout=90000
>> watcher=hconnection-0x6833f0de
>> 13/07/09 12:40:34 INFO zookeeper.RecoverableZooKeeper: Process
>> identifier=hconnection-0x6833f0de connecting to ZooKeeper
>> ensemble=cas-2:2181,cas-1:2181,cas-3:2181
>> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Opening socket connection to
>> server cas-1/10.4.0.1:2181. Will not attempt to authenticate using SASL
>> (Unable to locate a login configuration)
>> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Socket connection established
>> to cas-1/10.4.0.1:2181, initiating session
>> 13/07/09 12:40:34 INFO zookeeper.ClientCnxn: Session establishment
>> complete on server cas-1/10.4.0.1:2181, sessionid = 0x13fa5b5096e001f,
>> negotiated timeout = 40000
>> 13/07/09 12:40:34 INFO client.ZooKeeperRegistry: ClusterId read in
>> ZooKeeper is null
>>
>> The Java code is nothing more than:
>>
>> Configuration hConf = HBaseConfiguration.create();
>> hConf.set("hbase.zookeeper.quorum", "cas-1,cas-2,cas-3");
>> hConf.set("hbase.zookeeper.property.clientPort", "2181");
>> HTable hTable = new HTable(hConf, "tablename");
>>
>> a thread dump shows the app blocked:
>>
>> "main" prio=10 tid=0x00007f1424009000 nid=0x2976 waiting on condition
>> [0x00007f142a0c3000]
>> java.lang.Thread.State: TIMED_WAITING (sleeping)
>> at java.lang.Thread.sleep(Native Method)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.blockUntilAvailable(ZKUtil.java:1772)
>> at
>> org.apache.hadoop.hbase.zookeeper.MetaRegionTracker.blockUntilAvailable(MetaRegionTracker.java:175)
>> at
>> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:58)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:806)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:896)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:809)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:778)
>> at
>> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:245)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:186)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:147)
>>
>> Any suggestions as to the cause?
>>
>> TIA
>>
>> Brian
