Has anyone seen this kind of behavior? i am wondering what is triggering it
and how can i resolve it.
I have a multithreaded app which is reading stuff out of hbase. i recently
made a change to switch from using Get for each row I am interested to
using Scan with row key ranges.. What i see this the process will pause
forever. and starts taking up a lot of cpu (100% on a 4 core machine - so
effectively 25%). all stacks look like this -
"pool-1-thread-13" prio=10 tid=0x00007fa094013000 nid=0x56b7 waiting for
monitor entry [0x00007fa0e4c56000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.TreeMap.put(TreeMap.java:571)
at
org.apache.hadoop.hbase.client.Result.getNoVersionMap(Result.java:360)
"pool-1-thread-11" prio=10 tid=0x00007fa0ec284000 nid=0x1c7 in
Object.wait() [0x00007fa0e5969000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f026f3b0> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:757)
- locked <0x00000000f026f3b0> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy4.get(Unknown Source)
at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:549)
at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:547)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1000)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:546)
"pool-1-thread-10" prio=10 tid=0x00007fa0ec282000 nid=0x1c6 waiting for
monitor entry [0x00007fa0e5a6a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.KeyValue.split(KeyValue.java:1048)
at org.apache.hadoop.hbase.client.Result.getMap(Result.java:309)
at
org.apache.hadoop.hbase.client.Result.getNoVersionMap(Result.java:345)
"pool-1-thread-6" prio=10 tid=0x00007fa0ec21f000 nid=0x1c2 in Object.wait()
[0x0
0007fa0e5e6d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000f0766b10> (a
org.apache.hadoop.hbase.ipc.HBaseCl
ient$Call)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:757)
- locked <0x00000000f0766b10> (a
org.apache.hadoop.hbase.ipc.HBaseClient
$Call)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257
)
at $Proxy4.next(Unknown Source)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.j
ava:79)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.j
ava:38)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
tation.getRegionServerWithRetries(HConnectionManager.java:1000)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1100)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1210)
The only thing I observed in the RS logs is there were some compactions
going on. And the GC logs did show some pauses as high as 5sec. but it is
past that state but the app still doesn't make progress.. any insights?
thanks
