Thanks, will give it a shot. So I should download 0.94.7 (latest stable)
and run the patch tool on top with the backport? This is a little new to me.
Also, I was looking at the stack below. From my reading of the code, the
HTable.batch() call will always cause the prefetch call to occur, which
will cause a new HTable object to get created. The constructor used in
creating a new thread pool. So every call to batch will create a new
thread? Or the HTable's thread pool never gets used as the pool is only
used for writes? I think I am missing something but just want to confirm.
Thanks
Kireet
On 5/30/13 12:48 PM, Himanshu Vashishtha wrote:
bq. Anoop attached backported patch in HBASE-8655. It should go into
0.94.9, the next release - current is 0.94.8
In case you want it sooner, you can apply 8655 patch and test/verify it.
Thanks,
Himanshu
On Thu, May 30, 2013 at 7:26 AM, Ted Yu
<[email protected]> wrote:
Anoop attached backported patch in HBASE-8655
It should go into 0.94.9, the next release - current is 0.94.8
Cheers
On Thu, May 30, 2013 at 7:01 AM, Kireet
<[email protected]> wrote:
How long do backports typically take? We have to go live in a month ready
or not. Thanks for the quick replies Anoop and Ted.
--Kireet
On 5/30/13 9:20 AM, Ted Yu wrote:
0.95 client is not compatible with 0.94 cluster. So you cannot use 0.95
client.
Cheers
On May 30, 2013, at 6:12 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
[email protected]
<kireet-teh5dpvpl8nqt0dzr%[email protected]
wrote:
Would there be a problem if our cluster is 0.94 and we use a 0.95
client?
I am not familiar with the HBase code base, but I did a dump of the
thread that is actually running (below). It seems like it is related
to the
issue you mentioned as the running thread is doing the prefetch logic.
Would pre-splitting tables help here? We are doing some performance
tests
and essentially starting from an empty instance.
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.**java:503)
at org.apache.zookeeper.**ClientCnxn.submitRequest(**
ClientCnxn.java:1309)
- locked <0x00000000e10cf830> (a org.apache.zookeeper.**
ClientCnxn$Packet)
at org.apache.zookeeper.**ZooKeeper.exists(ZooKeeper.**java:1036)
at org.apache.hadoop.hbase.**zookeeper.**RecoverableZooKeeper.exists(**
RecoverableZooKeeper.java:172)
at org.apache.hadoop.hbase.**zookeeper.ZKUtil.checkExists(**
ZKUtil.java:450)
at org.apache.hadoop.hbase.**zookeeper.**ZooKeeperNodeTracker.**
checkIfBaseNodeAvailable(**ZooKeeperNodeTracker.java:208)
at org.apache.hadoop.hbase.**zookeeper.RootRegionTracker.**
waitRootRegionLocation(**RootRegionTracker.java:77)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegion(**HConnectionManager.java:874)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegionInMeta(**
HConnectionManager.java:987)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegion(**HConnectionManager.java:885)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegion(**HConnectionManager.java:846)
at org.apache.hadoop.hbase.**client.HTable.finishSetup(**
HTable.java:234)
at org.apache.hadoop.hbase.**client.HTable.<init>(HTable.**java:174)
at org.apache.hadoop.hbase.**client.MetaScanner.metaScan(**
MetaScanner.java:160)
at org.apache.hadoop.hbase.**client.MetaScanner.access$000(**
MetaScanner.java:54)
at org.apache.hadoop.hbase.**client.MetaScanner$1.connect(**
MetaScanner.java:133)
at org.apache.hadoop.hbase.**client.MetaScanner$1.connect(**
MetaScanner.java:130)
at org.apache.hadoop.hbase.**client.HConnectionManager.**
execute(HConnectionManager.**java:383)
at org.apache.hadoop.hbase.**client.MetaScanner.metaScan(**
MetaScanner.java:130)
at org.apache.hadoop.hbase.**client.MetaScanner.metaScan(**
MetaScanner.java:105)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**prefetchRegionCache(**
HConnectionManager.java:947)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegionInMeta(**
HConnectionManager.java:1002)
- locked <0x00000000e882f778> (a java.lang.Object)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegion(**HConnectionManager.java:889)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**locateRegion(**HConnectionManager.java:846)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**processBatchCallback(**
HConnectionManager.java:1533)
at org.apache.hadoop.hbase.**client.HConnectionManager$**
HConnectionImplementation.**processBatch(**HConnectionManager.java:1418)
at org.apache.hadoop.hbase.**client.HTable.batch(HTable.**java:702)
at org.apache.hadoop.hbase.**client.HTable.get(HTable.java:**671)
at org.apache.hadoop.hbase.**client.HTablePool$**
PooledHTable.get(HTablePool.**java:367)
Thanks
Kireet
On 5/30/13 12:16 AM, Anoop John wrote:
Can you have a look at issue HBASE-8476? Seems related? A fix is
available in HBASE-8346's patch..
-Anoop-
On Thu, May 30, 2013 at 9:21 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
[email protected]<
kireet-teh5dpvpl8nqt0dzr%[email protected]>>
wrote:
We are running hbase 0.94.6 in a concurrent environment and we are
seeing
the majority of our code stuck in this method at the synchronized
block
(line 997). Is there some configuration we can change to avoid it? It
seems
like we could disable prefetching, but that sounds like it could
cause
other problems?
- org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegionInMeta(byte[], byte[],
byte[],
boolean, java.lang.Object, boolean) @bci=149, line=997 (Compiled
frame)
- org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegion(byte[], byte[], boolean,
boolean) @bci=212, line=889 (Compiled frame)
- org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegion(byte[], byte[]) @bci=5,
line=846
(Compiled frame)
- org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****processBatchCallback(java.****util.List,
byte[], java.util.concurrent.****ExecutorService, java.lang.Object[],
org.apache.hadoop.hbase.****client.coprocessor.Batch$****Callback)
@bci=194,
line=1533 (Compiled frame)
- org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****processBatch(java.util.List, byte[],
java.util.concurrent.****ExecutorService, java.lang.Object[])
@bci=30,
line=1418 (Compiled frame)
- org.apache.hadoop.hbase.****client.HTable.batch(java.util.**
**List)
@bci=24, line=702 (Compiled frame)
- org.apache.hadoop.hbase.****client.HTable.get(java.util.****List)
@bci=2,
line=671 (Compiled frame)
- org.apache.hadoop.hbase.****client.HTablePool$**
PooledHTable.get(java.util.****List) @bci=5, line=367 (Compiled
frame)
