Hi all,
We use thousand of threads doing the scan operations. One thread got blocked at
the position of "java.util.TreeMap.fixAfterDeletion". It can't come out of the
loop of "TreeMap.fixAfterDeletion":
So I think it maybe a concurrency issue. Has someone encountered this?
Thank you.
Jieshan.
Here's the thread dump:
"Thread-923" prio=10 tid=0x00007f3d40553000 nid=0x3ed6 runnable
[0x00007f3d05c1b000]
java.lang.Thread.State: RUNNABLE
at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2176)
at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
at java.util.TreeMap.remove(TreeMap.java:585)
at java.util.TreeMap$NavigableSubMap.remove(TreeMap.java:1395)
at
org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:101)
- locked <0x00007f3d92937748> (a
org.apache.hadoop.hbase.util.SoftValueSortedMap)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:846)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:668)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
at
org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
at
org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1018)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
at
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
at
com.huawei.icbc.query.SingleTabQuery.querybatch(SingleTabQuery.java:197)
at
com.huawei.icbc.benchmark.SingleTabQueryAction.query(SingleTabQueryAction.java:181)
at framework.QueryThread.run(QueryThread.java:47)
"Thread-923" prio=10 tid=0x00007f3d40553000 nid=0x3ed6 runnable
[0x00007f3d05c1a000]
java.lang.Thread.State: RUNNABLE
at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
at java.util.TreeMap.remove(TreeMap.java:585)
at java.util.TreeMap$NavigableSubMap.remove(TreeMap.java:1395)
at
org.apache.hadoop.hbase.util.SoftValueSortedMap.get(SoftValueSortedMap.java:101)
- locked <0x00007f3d94f24f70> (a
org.apache.hadoop.hbase.util.SoftValueSortedMap)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:846)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:668)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:594)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:559)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:416)
at
org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
at
org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:63)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1018)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1104)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1027)
at
org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:535)
at
com.huawei.icbc.query.SingleTabQuery.querybatch(SingleTabQuery.java:197)
at
com.huawei.icbc.benchmark.SingleTabQueryAction.query(SingleTabQueryAction.java:181)
at framework.QueryThread.run(QueryThread.java:47)
