Would there be a problem if our cluster is 0.94 and we use a 0.95 client?
I am not familiar with the HBase code base, but I did a dump of the
thread that is actually running (below). It seems like it is related to
the issue you mentioned as the running thread is doing the prefetch
logic. Would pre-splitting tables help here? We are doing some
performance tests and essentially starting from an empty instance.
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309)
- locked <0x00000000e10cf830> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.checkIfBaseNodeAvailable(ZooKeeperNodeTracker.java:208)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:77)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:874)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:987)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:885)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:846)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:383)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:947)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1002)
- locked <0x00000000e882f778> (a java.lang.Object)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:889)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:846)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1533)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1418)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:702)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:671)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:367)
Thanks
Kireet
On 5/30/13 12:16 AM, Anoop John wrote:
Can you have a look at issue HBASE-8476? Seems related? A fix is
available in HBASE-8346's patch..
-Anoop-
On Thu, May 30, 2013 at 9:21 AM, Kireet
<[email protected]> wrote:
We are running hbase 0.94.6 in a concurrent environment and we are seeing
the majority of our code stuck in this method at the synchronized block
(line 997). Is there some configuration we can change to avoid it? It seems
like we could disable prefetching, but that sounds like it could cause
other problems?
- org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegionInMeta(byte[], byte[], byte[],
boolean, java.lang.Object, boolean) @bci=149, line=997 (Compiled frame)
- org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegion(byte[], byte[], boolean,
boolean) @bci=212, line=889 (Compiled frame)
- org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegion(byte[], byte[]) @bci=5, line=846
(Compiled frame)
- org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**processBatchCallback(java.**util.List,
byte[], java.util.concurrent.**ExecutorService, java.lang.Object[],
org.apache.hadoop.hbase.**client.coprocessor.Batch$**Callback) @bci=194,
line=1533 (Compiled frame)
- org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**processBatch(java.util.List, byte[],
java.util.concurrent.**ExecutorService, java.lang.Object[]) @bci=30,
line=1418 (Compiled frame)
- org.apache.hadoop.hbase.**client.HTable.batch(java.util.**List)
@bci=24, line=702 (Compiled frame)
- org.apache.hadoop.hbase.**client.HTable.get(java.util.**List) @bci=2,
line=671 (Compiled frame)
- org.apache.hadoop.hbase.**client.HTablePool$**
PooledHTable.get(java.util.**List) @bci=5, line=367 (Compiled frame)
