Thanks Vinod, I'll try as you suggest next time the problem occurs and report back.
On Fri, 2011-10-07 at 01:36 -0700, Vinod Gupta Tankala wrote: > i haven't seen this before but what i can suggest you is these 2 steps and > you will probably get more insights - > 1) > # bin/hbase shell > # list > > see if the list commands returns back with your list of tables. > > 2) take a look at the latest hbase logs and see if there are any FATAL, > ERROR or WARN logs lately. > > thanks > > On Fri, Oct 7, 2011 at 1:31 AM, Joel Halbert <[email protected]> wrote: > > > Hi, > > > > Intermittently I am seeing my client threads stuck waiting on a call to > > HBaseClient.call(HBaseClient.java:722) > > > > This is a stand-along instance of HBase (no Hadoop). The client > > processes that use it are multi-threaded, and I am using a HTablePool to > > get HTable instances. > > > > Has anyone seen this before? Any ideas what could be the cause? > > > > A thread dump of a client process shows: > > > > e.g. > > UPDATE_CLIENT_SEEDS" prio=10 tid=0x09fbcc00 nid=0x34e0 in Object.wait() > > [0x0ae5a000] > > java.lang.Thread.State: WAITING (on object monitor) > > at java.lang.Object.wait(Native Method) > > at java.lang.Object.wait(Object.java:485) > > at > > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) > > - locked <0x30bdd308> (a org.apache.hadoop.hbase.ipc.HBaseClient > > $Call) > > at org.apache.hadoop.hbase.ipc.HBaseRPC > > $Invoker.invoke(HBaseRPC.java:333) > > at $Proxy0.get(Unknown Source) > > at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450) > > at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448) > > at org.apache.hadoop.hbase.client.HConnectionManager > > $TableServers.getRegionServerWithRetries(HConnectionManager.java:1040) > > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447) > > > > > > "R-PPQ-6" daemon prio=10 tid=0x0738e000 nid=0x3510 in Object.wait() > > [0x04ac4000] > > java.lang.Thread.State: WAITING (on object monitor) > > at java.lang.Object.wait(Native Method) > > at java.lang.Object.wait(Object.java:485) > > at > > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) > > - locked <0x155328e8> (a org.apache.hadoop.hbase.ipc.HBaseClient > > $Call) > > at org.apache.hadoop.hbase.ipc.HBaseRPC > > $Invoker.invoke(HBaseRPC.java:333) > > at $Proxy0.get(Unknown Source) > > at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450) > > at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448) > > at org.apache.hadoop.hbase.client.HConnectionManager > > $TableServers.getRegionServerWithRetries(HConnectionManager.java:1040) > > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447) > > at > > com.su3analytics.sitedelta.pages.PageTable.get(PageTable.java:111) > > at > > com.su3analytics.sitedelta.pages.PageTable.get(PageTable.java:256) > > > > > > "R-PPQ-9" daemon prio=10 tid=0x09ded400 nid=0x354a in Object.wait() > > [0x019fe000] > > java.lang.Thread.State: WAITING (on object monitor) > > at java.lang.Object.wait(Native Method) > > at java.lang.Object.wait(Object.java:485) > > at > > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) > > - locked <0x1576d880> (a org.apache.hadoop.hbase.ipc.HBaseClient > > $Call) > > at org.apache.hadoop.hbase.ipc.HBaseRPC > > $Invoker.invoke(HBaseRPC.java:333) > > at $Proxy0.get(Unknown Source) > > at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:450) > > at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:448) > > at org.apache.hadoop.hbase.client.HConnectionManager > > $TableServers.getRegionServerWithRetries(HConnectionManager.java:1040) > > at org.apache.hadoop.hbase.client.HTable.get(HTable.java:447) > > > > > > > > > >
