Even if I initiate the call via a pooled htable, the MetaScanner seems
to use a concrete HTable instance. The constructor invoked seems to
create a java ThreadPoolExecutor. I am not 100% sure but I think as long
as nothing is submitted to the ThreadPoolExecutor it won't create any
threads. I just wanted to confirm this was the case. I do see the
connection is shared.
--Kireet
On 5/30/13 7:38 PM, Ted Yu wrote:
HTablePool$**PooledHTable is a wrapper around HTable.
Here is how HTable obtains a connection:
public HTable(Configuration conf, final byte[] tableName, final
ExecutorService pool)
throws IOException {
this.connection = HConnectionManager.getConnection(conf);
Meaning the connection is a shared one based on certain key/value pairs
from conf.
bq. So every call to batch will create a new thread?
I don't think so.
On Thu, May 30, 2013 at 11:28 AM, Kireet
<[email protected]> wrote:
Thanks, will give it a shot. So I should download 0.94.7 (latest stable)
and run the patch tool on top with the backport? This is a little new to me.
Also, I was looking at the stack below. From my reading of the code, the
HTable.batch() call will always cause the prefetch call to occur, which
will cause a new HTable object to get created. The constructor used in
creating a new thread pool. So every call to batch will create a new
thread? Or the HTable's thread pool never gets used as the pool is only
used for writes? I think I am missing something but just want to confirm.
Thanks
Kireet
On 5/30/13 12:48 PM, Himanshu Vashishtha wrote:
bq. Anoop attached backported patch in HBASE-8655. It should go into
0.94.9, the next release - current is 0.94.8
In case you want it sooner, you can apply 8655 patch and test/verify it.
Thanks,
Himanshu
On Thu, May 30, 2013 at 7:26 AM, Ted Yu <yuzhihong-**
Re5JQEeQqe8AvxtiuMwx3w@public.**gmane.org<yuzhihong-re5jqeeqqe8avxtiumwx3w-xmd5yjdbdmrexy1tmh2...@public.gmane.org>>
wrote:
Anoop attached backported patch in HBASE-8655
It should go into 0.94.9, the next release - current is 0.94.8
Cheers
On Thu, May 30, 2013 at 7:01 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
[email protected]
<kireet-teh5dpvpl8nqt0dzr%[email protected]>>
wrote:
How long do backports typically take? We have to go live in a month
ready
or not. Thanks for the quick replies Anoop and Ted.
--Kireet
On 5/30/13 9:20 AM, Ted Yu wrote:
0.95 client is not compatible with 0.94 cluster. So you cannot use 0.95
client.
Cheers
On May 30, 2013, at 6:12 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
AlfA-XMD5yJDbdMReXY1tMh2IBg@**public.gmane.org<alfa-xmd5yjdbdmrexy1tmh2ibg-xmd5yjdbdmrexy1tmh2...@public.gmane.org><
kireet-Teh5dPVPL8nQT0dZR%**2BAlfA-XMD5yJDbdMReXY1tMh2IBg@**
public.gmane.org<kireet-teh5dpvpl8nqt0dzr%252balfa-xmd5yjdbdmrexy1tmh2ibg-xmd5yjdbdmrexy1tmh2...@public.gmane.org>
wrote:
Would there be a problem if our cluster is 0.94 and we use a 0.95
client?
I am not familiar with the HBase code base, but I did a dump of the
thread that is actually running (below). It seems like it is related
to the
issue you mentioned as the running thread is doing the prefetch logic.
Would pre-splitting tables help here? We are doing some performance
tests
and essentially starting from an empty instance.
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.****java:503)
at org.apache.zookeeper.****ClientCnxn.submitRequest(**
ClientCnxn.java:1309)
- locked <0x00000000e10cf830> (a org.apache.zookeeper.**
ClientCnxn$Packet)
at org.apache.zookeeper.****ZooKeeper.exists(ZooKeeper.****java:1036)
at org.apache.hadoop.hbase.****zookeeper.****
RecoverableZooKeeper.exists(**
RecoverableZooKeeper.java:172)
at org.apache.hadoop.hbase.****zookeeper.ZKUtil.checkExists(****
ZKUtil.java:450)
at org.apache.hadoop.hbase.****zookeeper.****ZooKeeperNodeTracker.**
checkIfBaseNodeAvailable(****ZooKeeperNodeTracker.java:208)
at org.apache.hadoop.hbase.****zookeeper.RootRegionTracker.**
waitRootRegionLocation(****RootRegionTracker.java:77)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegion(****
HConnectionManager.java:874)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegionInMeta(**
HConnectionManager.java:987)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegion(****
HConnectionManager.java:885)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegion(****
HConnectionManager.java:846)
at org.apache.hadoop.hbase.****client.HTable.finishSetup(**
HTable.java:234)
at org.apache.hadoop.hbase.****client.HTable.<init>(HTable.****
java:174)
at org.apache.hadoop.hbase.****client.MetaScanner.metaScan(**
MetaScanner.java:160)
at org.apache.hadoop.hbase.****client.MetaScanner.access$000(****
MetaScanner.java:54)
at org.apache.hadoop.hbase.****client.MetaScanner$1.connect(****
MetaScanner.java:133)
at org.apache.hadoop.hbase.****client.MetaScanner$1.connect(****
MetaScanner.java:130)
at org.apache.hadoop.hbase.****client.HConnectionManager.**
execute(HConnectionManager.****java:383)
at org.apache.hadoop.hbase.****client.MetaScanner.metaScan(**
MetaScanner.java:130)
at org.apache.hadoop.hbase.****client.MetaScanner.metaScan(**
MetaScanner.java:105)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****prefetchRegionCache(**
HConnectionManager.java:947)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegionInMeta(**
HConnectionManager.java:1002)
- locked <0x00000000e882f778> (a java.lang.Object)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegion(****
HConnectionManager.java:889)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****locateRegion(****
HConnectionManager.java:846)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****processBatchCallback(**
HConnectionManager.java:1533)
at org.apache.hadoop.hbase.****client.HConnectionManager$**
HConnectionImplementation.****processBatch(****
HConnectionManager.java:1418)
at org.apache.hadoop.hbase.****client.HTable.batch(HTable.****java:702)
at org.apache.hadoop.hbase.****client.HTable.get(HTable.java:****671)
at org.apache.hadoop.hbase.****client.HTablePool$**
PooledHTable.get(HTablePool.****java:367)
Thanks
Kireet
On 5/30/13 12:16 AM, Anoop John wrote:
Can you have a look at issue HBASE-8476? Seems related? A fix is
available in HBASE-8346's patch..
-Anoop-
On Thu, May 30, 2013 at 9:21 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+**
AlfA-XMD5yJDbdMReXY1tMh2IBg@**public.gmane.org<alfa-xmd5yjdbdmrexy1tmh2ibg-xmd5yjdbdmrexy1tmh2...@public.gmane.org>
<
kireet-Teh5dPVPL8nQT0dZR%**2BAlfA-XMD5yJDbdMReXY1tMh2IBg@**
public.gmane.org<kireet-teh5dpvpl8nqt0dzr%252balfa-xmd5yjdbdmrexy1tmh2ibg-xmd5yjdbdmrexy1tmh2...@public.gmane.org>
wrote:
We are running hbase 0.94.6 in a concurrent environment and we are
seeing
the majority of our code stuck in this method at the synchronized
block
(line 997). Is there some configuration we can change to avoid it? It
seems
like we could disable prefetching, but that sounds like it could
cause
other problems?
- org.apache.hadoop.hbase.******client.HConnectionManager$**
HConnectionImplementation.******locateRegionInMeta(byte[], byte[],
byte[],
boolean, java.lang.Object, boolean) @bci=149, line=997 (Compiled
frame)
- org.apache.hadoop.hbase.******client.HConnectionManager$**
HConnectionImplementation.******locateRegion(byte[], byte[],
boolean,
boolean) @bci=212, line=889 (Compiled frame)
- org.apache.hadoop.hbase.******client.HConnectionManager$**
HConnectionImplementation.******locateRegion(byte[], byte[])
@bci=5,
line=846
(Compiled frame)
- org.apache.hadoop.hbase.******client.HConnectionManager$**
HConnectionImplementation.******processBatchCallback(java.******
util.List,
byte[], java.util.concurrent.******ExecutorService, java.lang.Object[],
org.apache.hadoop.hbase.******client.coprocessor.Batch$******
Callback)
@bci=194,
line=1533 (Compiled frame)
- org.apache.hadoop.hbase.******client.HConnectionManager$**
HConnectionImplementation.******processBatch(java.util.List,
byte[],
java.util.concurrent.******ExecutorService, java.lang.Object[])
@bci=30,
line=1418 (Compiled frame)
- org.apache.hadoop.hbase.******client.HTable.batch(java.util.**
**
**List)
@bci=24, line=702 (Compiled frame)
- org.apache.hadoop.hbase.******client.HTable.get(java.util.****
**List)
@bci=2,
line=671 (Compiled frame)
- org.apache.hadoop.hbase.******client.HTablePool$**
PooledHTable.get(java.util.******List) @bci=5, line=367 (Compiled
frame)
